Context windows

The total amount of text the model can hold in its head at once. Measured in tokens.

What's actually happening

Two non-obvious points the diagram doesn't make:

• Both your input and the model's reply share the same budget. A 200k window with 199k of input leaves 1k for the reply.
• The model has no memory between requests. Anything you want it to "remember" must be re-sent every turn — that's why long chats get expensive (every turn re-sends the full history) and why "memory" features in chat products are scaffolding around the model, not part of it.

Where it bites in real life

Under the hood (optional)

Guard before sending: add up tokens, refuse if overflow. ~5 lines of TypeScript. Skip if you don't code — the idea is "check the total before hitting send."

import { encode } from "gpt-tokenizer";
 
const MAX_INPUT = 180_000;   // leave room for the reply
 
function fitsInContext(messages: { role: string; content: string }[]) {
  const total = messages.reduce((n, m) => n + encode(m.content).length, 0);
  return total <= MAX_INPUT;
}

Try it yourself (no coding, ~5 minutes)

1. Open claude.ai (200k context).
2. Find a public document of ~5,000+ words (Wikipedia article, long blog post). Copy it.
3. Test 1: paste, ask "What's the main argument?" — note the answer.
4. Test 2: paste again, ask "What does paragraph 7 say verbatim?" — note the answer.

Test 1 plays to the model's strength (holistic compression). Test 2 tests precise recall — you'll often see drift, paraphrase, or wrong details. "Lost in the middle" in action.

Further reading

• Lost in the Middle — Liu et al. 2023 — landmark paper.
• Anthropic — Long context prompting tips — production tactics.
• Needle in a Haystack benchmarks — open methodology for measuring long-context recall.