Production Patterns & Error Handling

Why "it works in a notebook" isn't the same as "it works in production"#

Every LLM call in a LangChain chain is, underneath, a network request to a third-party API. Like any external HTTP call, it can time out, get rate-limited, fail with a transient server error, or simply be slow — the exact same class of failure any other production system has to plan for when calling an external service. A chain that works perfectly in a notebook, where every call happens to succeed quickly, will surface a raw, unhandled exception straight to a real user the first time the underlying API has a bad moment — which, at any meaningful production traffic volume, is a "when," not an "if."

Retries with backoff#

Transient failures (a momentary rate limit, a dropped connection) are often resolved by simply trying again a moment later. LangChain's runnables support .with_retry() directly, wrapping a chain or model call with automatic retry-and-backoff behavior:

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o-mini").with_retry(
    stop_after_attempt=3,
    wait_exponential_jitter=True,  # backs off longer between each retry, with jitter
)

response = model.invoke("Summarize this support ticket in one sentence.")

Retrying blindly isn't free, though — a request that fails because the input itself is malformed (invalid arguments, a prompt that violates a content policy) will fail identically on every retry, just slower and at extra cost. Retries are the right tool specifically for transient, infrastructure-level failures, not for errors caused by the request's own content.

Timeouts#

Without an explicit timeout, a hung request can block a request thread indefinitely — a single slow LLM call can quietly take down an entire request pipeline's throughput. Setting an explicit timeout turns an indefinite hang into a bounded, handleable failure:

model = ChatOpenAI(model="gpt-4o-mini", timeout=15)  # seconds

try:
    response = model.invoke("Analyze this 50-page document...")
except TimeoutError:
    response = "Sorry, that took too long -- please try a shorter document."

Streaming responses#

For anything user-facing, waiting for the entire response to generate before showing anything feels slow, even when total generation time is reasonable — users perceive a response that starts appearing immediately, token by token, as much faster than one that appears all at once after the same total wait. LangChain's .stream() yields output incrementally as the model generates it:

for chunk in model.stream("Write a short paragraph about renewable energy."):
    print(chunk.content, end="", flush=True)  # prints as each token/chunk arrives

Streaming doesn't reduce total generation time — it changes when the user starts seeing output, which is usually what actually matters for perceived responsiveness in an interactive product like a chat interface.

Cost control#

LLM API costs scale with tokens in and out (see this site's LLM tutorials for how tokens are counted), which makes cost control a real, ongoing engineering concern in production, not a one-time setup step:

• Set max_tokens on the response to cap runaway generations — without a cap, a model that gets into a repetitive loop or misunderstands "be brief" can generate far more (and cost far more) than intended.
• Cache repeated or near-identical requests. If the same question (or a close variant) gets asked repeatedly — a common FAQ, a repeated batch job — cache the response instead of re-calling the model every time.
• Route sub-tasks to the cheapest model that handles them well. Not every step in a chain needs the most capable (and most expensive) model — a classification or extraction sub-step might do just as well on a smaller, cheaper model, reserving the most capable model for the step that actually needs it (see this site's LLM tutorials on production deployment and cost for more on this trade-off).
• Track token usage per request, not just in aggregate — LangChain callbacks expose token counts per call, which makes it possible to spot a specific chain or prompt that's unexpectedly expensive before it shows up as a surprise on a monthly bill.

from langchain_community.callbacks import get_openai_callback

with get_openai_callback() as cb:
    response = chain.invoke({"input": "Summarize this document..."})
    print(cb.total_tokens, cb.total_cost)  # per-call visibility into cost

Common mistakes#

• Retrying every failure identically, including ones caused by bad input — this wastes time and money re-running a request that will fail the same way every time, rather than failing fast and surfacing a clear error.
• Leaving no timeout at all on model calls — a single hung request can silently exhaust a request-handling thread pool or worker queue under real production load.
• Optimizing cost only by picking a cheaper model globally — often the bigger win is architectural: caching, shorter prompts, capping max_tokens, and routing only the sub-tasks that need it to a more expensive model.