Caching & cost optimization

The cheapest LLM call is the one you don't make. The next cheapest is the one with a warm cache.

What caching actually does

LLM calls cost money and take time. Caching is the discipline of doing fewer of them — or doing the same one more cheaply. Three layers, three different wins:

• Don't call the model again when you already answered the same thing.
• Don't pay full price for the long, repeated chunk at the start of every prompt.
• Match similar questions to a single cached answer instead of regenerating.

That's it. There's no exotic trick — most of the savings come from a single config flag (prompt caching) and basic discipline about what doesn't need to be regenerated.

That's the whole concept. Below: the three caches in detail and the other cost levers worth pulling.

When you'd reach for it

The moment cost or latency starts mattering:

• Repeated workloads — the same support questions, the same code-review prompts, the same extraction templates.
• Long static prompts — multi-thousand-token system messages or few-shot blocks that get re-sent every turn.
• Chatty agents — a 30-step loop with the same tool descriptions on every turn is paying for the same tokens 30 times.
• High-volume features — anywhere your monthly bill has a comma in it.

You wouldn't bother on prototypes or one-shot internal scripts. Caching pays off when the same shape of call happens repeatedly.

How it's actually built

In rough order of how much they save in production:

1. Response cache (your code)

Same question, same context → same answer. Cache it.

def cache_key(model, system, messages, params):
    blob = json.dumps({"m": model, "s": system, "msgs": messages, "p": params}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()

Works perfectly for deterministic prompts (temperature=0) with stable inputs. For RAG, key by (query, retrieved_chunk_ids) so the cache invalidates when knowledge changes.

2. Prompt cache (provider-side)

Most long prompts have a long static prefix: system message, few-shot examples, retrieved doc set. Re-sending those tokens every turn is wasteful. Prompt caching lets the provider cache the prefix server-side; subsequent calls within the cache window read those tokens at ~10% of the normal price.

response = client.messages.create(
  model="claude-sonnet-4-6",
  system=[
    {"type": "text", "text": LONG_SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}},
  ],
  messages=[{"role": "user", "content": current_user_message}],
)

One line. Often a 70%+ cost cut.

3. Semantic cache (smart response cache)

"What are your hours?" and "when are you open?" should hit the same cache entry. A semantic cache embeds the query, looks for a near-neighbour in past Q→A pairs, returns the answer if similarity is high. Be careful with the threshold — too low and you serve wrong answers; too high and you miss obvious paraphrases.

Where it bites in real life

Other cost levers (in rough impact order)

1. Right-size the model. Cheap models for the easy 80%, frontier for the hard 20%.
2. Trim the prompt. Remove turns the user no longer needs. Concise system messages.
3. Cap max_tokens. Bounded output = bounded bill.
4. Batch APIs. Non-realtime jobs (overnight extraction, evals) at half-price.
5. Smaller embedding model. text-embedding-3-small ≈ 1/5 the cost of -large, often within a couple of percentage points on retrieval.
6. Truncate embedding dimensions. OpenAI's MRL embeddings let you keep e.g. 512 of 3072 dims for 6× storage savings.
7. Don't call the LLM at all for the cheap path. Some questions don't need an LLM — a regex routes faster and free.

Under the hood (optional)

A response-cache wrapper in ~15 lines: hash the request, check Redis, return the cached answer if there is one, otherwise call the LLM and store the result. Skip if you don't code — the takeaway is "deterministic same-input-same-output calls don't need to hit the LLM twice."

import json, hashlib, redis
r = redis.Redis()
 
def llm_call(model, system, messages, **params):
    if params.get("temperature", 0) > 0:
        return _raw(model, system, messages, **params)   # don't cache stochastic
    key = "llm:" + hashlib.sha256(
        json.dumps({"m": model, "s": system, "msgs": messages, **params},
                   sort_keys=True).encode()
    ).hexdigest()
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    out = _raw(model, system, messages, **params)
    r.setex(key, 86400, json.dumps(out))
    return out

Further reading

• Anthropic — Prompt caching — official guide with pricing.
• OpenAI — Prompt caching — same concept on OpenAI.
• Anthropic — Batch API — half-price for non-realtime workloads.
• GPTCache — open semantic cache — production-grade semantic-cache layer.