Cost & latency

Whether an AI feature ships or dies usually comes down to two numbers: what does each call cost, and how long does the user wait?

What's actually happening

The going rates (mid-2026, per million tokens):

A single chat turn is a fraction of a cent. So where do the big bills come from? Scale and bloat:

• Long conversation histories re-sent every turn.
• RAG dumping huge context for tiny answers.
• Verbose system prompts paid for on every call.
• A viral moment with no rate limits (see the story above).

The latency formula worth memorising: Total time = TTFT + (output tokens ÷ TPS). Streaming the output doesn't change total time, but it does mean the user sees the first word at TTFT instead of staring at a spinner — which is why streaming-by-default is now standard.

Where it shows up in real life

Where the wins come from (in order)

1. Right-size the model. Cheap models for the easy 80%, frontier for the hard 20%.
2. Prompt caching. Static prefixes billed at a fraction of normal rate.
3. Trim the prompt. Drop turns the user no longer needs.
4. Cap max_tokens. Bounded output = bounded bill.
5. Batch APIs. Non-realtime jobs at half-price.
6. Stream responses. Cost stays the same; perceived latency drops dramatically.

Under the hood (optional)

A simple cost-estimator: price table × token counts. ~12 lines of TypeScript. Skip if you don't code — the price table itself tells you what you need.

const PRICING = {
  "claude-haiku-4-5":  { in: 0.25 / 1e6, out: 1.25 / 1e6 },
  "claude-sonnet-4-6": { in: 3 / 1e6,    out: 15 / 1e6 },
  "claude-opus-4-7":   { in: 15 / 1e6,   out: 75 / 1e6 },
} as const;
 
function estimateCost(model: keyof typeof PRICING, inputTokens: number, outputTokens: number) {
  const p = PRICING[model];
  return inputTokens * p.in + outputTokens * p.out;
}
 
// Realistic: 4k input, 500 output, mid model
console.log(estimateCost("claude-sonnet-4-6", 4_000, 500));
// → $0.0195 per call. At 1M calls/month, that's $19,500.

Try it yourself (no coding, ~5 minutes)

1. Open console.anthropic.com → Workbench (or platform.openai.com/playground).
2. Type "Write a 200-word summary of the French Revolution."
3. Run with a small, mid, then big model. Note token counts, time-to-first-word, and total cost (most playgrounds show this).

You'll feel the trade-off: small = cheap + fast, sometimes lower quality. Big = slower + pricier, sharper writing. Pick the smallest model that meets your bar — in miniature.

Further reading

• Anthropic — Pricing and OpenAI — Pricing.
• Anthropic — Prompt caching — official guide.
• Artificial Analysis — independent cost/latency/quality across providers.