Production Deployment & Cost

The core trade-off: latency, cost, and quality#

Every production LLM deployment decision is ultimately balancing three things against each other: how fast a response needs to come back (latency), how much each response costs to generate (cost), and how good the response needs to be (quality). Pushing on any one of these generally costs you on the others — a bigger, more capable model tends to be slower and more expensive per call; a faster, cheaper model tends to be less capable. Production engineering here is mostly about finding, and continuously re-tuning, the right point on that trade-off for a specific application, rather than assuming there's one "best" model to always use everywhere.

Where latency and cost actually come from#

Both scale primarily with tokens (see this site's LLM tutorials on tokens and context windows) — the number of input tokens the model has to process, plus the number of output tokens it has to generate, since generation happens one token at a time. A longer conversation history, a larger retrieved context (see this site's RAG tutorials), or a request for a longer response all directly increase both latency and cost, not just one of the two.

Caching#

Many applications see the same or near-identical requests repeatedly — a common FAQ, a repeated classification of similar inputs, a cached translation. Caching stores a previous response keyed by its (semantically or exactly matching) input, and serves the cached result instead of calling the model again:

• Exact-match caching — cache keyed on the literal input string; simple, but only helps for genuinely repeated requests.
• Semantic caching — cache keyed on embedding similarity (see this site's RAG tutorials on embeddings), so a near-identical but not word-for-word request can still hit the cache — more effective, but riskier if two subtly different questions are wrongly treated as "close enough."

Caching is often the single highest-leverage cost optimization available, precisely because it avoids the call entirely rather than trying to make each call cheaper.

Batching#

Batching groups multiple requests together to be processed in a single pass on the underlying hardware, rather than one at a time — this matters most for self-hosted model deployments, where GPU utilization (and therefore cost per request) improves significantly when the hardware processes several requests' worth of work simultaneously instead of sitting partially idle between individual requests. The trade-off is latency for any individual request in a batch — it may need to wait briefly for other requests to arrive before the batch is processed, trading a small amount of per-request latency for meaningfully better throughput and lower cost per request overall.

Quantization, briefly#

Quantization reduces the numerical precision of a model's weights (for example, from 16-bit to 8-bit or 4-bit numbers) — a self-hosting technique that shrinks the model's memory footprint and speeds up inference, at some cost to output quality. Well-executed quantization (particularly down to 8-bit) often costs little noticeable quality for a large speed and memory win, but more aggressive quantization trades away more accuracy, so it's a decision worth validating against your specific application's quality bar rather than assuming it's free — this is mainly a concern for teams self-hosting open-weight models, not for those calling a hosted provider's API, where the provider handles this trade-off internally.

Practical levers, roughly in order of leverage#

1. Use the smallest/cheapest model that meets your quality bar for each specific sub-task — not every step in a pipeline needs the most capable model (see this site's LangChain tutorials on production patterns for the same idea applied to chains).
2. Cache aggressively wherever requests repeat, exactly or semantically.
3. Trim context ruthlessly — send only what the model actually needs (the relevant retrieved chunks, not the whole document; a summarized history, not the entire conversation) rather than padding requests with unnecessary tokens.
4. Set explicit output limits (max_tokens) so a single request can't run away with cost.
5. Batch where you control the serving infrastructure, to improve throughput per unit of hardware cost.

Common mistakes#

• Defaulting to the most capable (and most expensive) model for every request in a pipeline, including simple sub-tasks that a cheaper model would handle just as well.
• Ignoring caching because "every request is different" — even partial repetition (a shared retrieved context, a common opening instruction) is often cacheable at some level, and it's usually the highest-leverage cost lever available.
• Assuming quantization is free — aggressive quantization can measurably degrade output quality on specific tasks, and that trade-off needs to be validated against your own quality bar, not assumed away.