Production deployment patterns

A working prototype is 10% of the work. The other 90% is making it survive real traffic, real outages, real users. Retries, fallbacks, rate limits, observability — the unglamorous stack.

What production deployment actually requires

A working AI prototype is roughly 10% of the work. The other 90% is the unglamorous stack that keeps it running: rate limits, retries, multi-provider fallback, validation, tracing. Every layer covers a specific failure mode that will happen — not might, will.

The cheapest insurance is multi-provider fallback: when one vendor has an outage (and they all do, regularly), you fail over to another. Then come the things that prevent self-inflicted wounds: per-user spend caps so a viral post doesn't generate a $40k bill, output validation so a weird LLM response doesn't crash your downstream code, and tracing every call so when something feels off, you can prove what changed.

That's the whole concept. Below: the architecture in detail, the patterns that matter, real cautionary tales, and a pre-launch checklist.

When you'd reach for it

The day you stop demoing and start having real users:

• Customer-facing features — they will hit edge cases you didn't test for.
• Anything billed by the call — costs grow with traffic; without caps, surprises happen.
• Multi-tenant SaaS — one rogue user shouldn't break the experience for everyone else.
• Anything with an SLA — your model provider's uptime becomes part of yours.
• Long-running agents — the longer the loop, the more places it can go wrong.

You can skip the heavy ops for an internal tool used by you and three coworkers. Anything past that, the checklist applies.

How it's actually built

The production picture, layer by layer:

Each layer covers one failure mode. Skip a layer, accept the failure mode it prevents.

The patterns that matter

Retries — LLM APIs return transient errors all the time (429, 503, network blips). Exponential backoff (1s/2s/4s/8s) with jitter to avoid retry stampedes. Don't retry 4xx-that-aren't-429; the input is broken.

Fallback chain — primary provider down? Try secondary. Both down? Self-hosted. Maintain [primary, secondary, tertiary] behind a common interface and switch on failure with structured logging so you know which fallback fired.

Rate limiting — two layers: respect provider limits (or get 429s), and your own per-user limits to prevent one user consuming the whole budget. Token-bucket algorithm; managed service from your cloud provider is fine.

Timeouts — per-call. Frontier on a complex query: 60+ seconds. Reasoning model: longer. Stream with separate TTFT timeout (give up if no first token in N seconds) and total timeout.

Idempotency — for tool calls that act on the world (send email, charge cards, modify records) — every request has a unique idempotency_key so duplicate retries return the original result without re-executing.

Validation before action — schema-validate structured outputs before use; sanity-check tool calls before executing; run a guard model over user-facing text. Cheap layers, big save when the LLM produces something weird.

Real-world examples

Multi-provider abstraction (optional code)

A common production pattern: wrap all providers behind one interface. The wrapper becomes the chokepoint where you add caching, retries, observability, routing, validation. One file becomes your AI infrastructure.

class LLMProvider(Protocol):
    def chat(self, system: str, messages: list, **opts) -> str: ...
 
# Concrete implementations: AnthropicProvider, OpenAIProvider, LocalLlamaProvider, …
 
class FailoverLLM:
    def __init__(self, providers: list[LLMProvider]):
        self.providers = providers
 
    def chat(self, system, messages, **opts):
        last_err = None
        for p in self.providers:
            try:
                return p.chat(system, messages, **opts)
            except (RateLimitError, ServiceUnavailableError, TimeoutError) as e:
                last_err = e
                logger.warning(f"{p} failed: {e}, trying next")
        raise last_err

Frameworks that do this for you: LangChain, LlamaIndex, Vercel AI SDK, Portkey, OpenRouter. All trade convenience for control. Roll your own if you want full visibility.

Pre-launch checklist

Before flipping an LLM-powered feature live:

1. ☐ Retries with backoff for transient errors
2. ☐ Per-user rate limits AND per-call max_tokens caps + spend alerts
3. ☐ Multi-provider fallback (or accepted single-provider risk)
4. ☐ Response and prompt caching where applicable
5. ☐ Schema validation for structured outputs + guard layer for user-facing text
6. ☐ Tracing every call with input/output/cost/latency
7. ☐ Eval suite in CI on every prompt change + online sampler scoring ~1% of prod
8. ☐ Runbook for: provider outage, prompt regression, cost spike, abuse

Every one you skip is a known way you'll be paged at 2 AM.

Further reading

• Anthropic — API best practices (errors, retries, rate limits) — official patterns.
• OpenAI — Production best practices — same on OpenAI's side.
• Eugene Yan — Building LLM applications for production — the most thorough production-pattern catalogue online.
• Hamel Husain — A Field Guide to Rapidly Improving AI Products — production-grade thinking on operational AI.
• Langfuse / LangSmith / Braintrust — pick a tracing platform; they all do similar things.