Long-context strategies

A 2-million-token window doesn't mean you should stuff 2 million tokens in. The strategies for actually leveraging long context are where production systems differentiate.

What long context actually means

Frontier models advertise context windows in the hundreds of thousands or millions of tokens. That's an upper bound, not a recipe. Two facts make "just stuff it all in" the wrong move most of the time:

• Cost grows quadratically. Doubling the input quadruples the attention compute. A 200k-token call isn't twice as expensive as 100k — it's roughly four times.
• Attention is U-shaped. The model attends best to what's at the very start and very end of its context. The middle is where things get forgotten — the famous "lost in the middle" problem.

So the real engineering question isn't "can it fit?" — it's "what's the smallest, smartest context I can hand it?" Smarter retrieval, summarisation, and ordering beat brute-force stuffing.

That's the whole concept. Below: the techniques that actually work — KV cache, prompt caching, hierarchical summarisation, reranking — and when stuffing it all in is the right move.

When you'd reach for it

Long-context tactics earn their keep when:

• You have more content than fits comfortably — multi-document analysis, long codebases, full transcripts.
• You want one-shot understanding — summarise this book, compare these contracts.
• You're building agents — long-running loops that accumulate tool results and conversation.
• Costs or latencies are getting ugly — prompt caching alone often cuts 70% of the bill.

You wouldn't reach for these when your inputs already fit comfortably and the model isn't getting confused. Don't over-engineer.

How it's actually built

The compute reality

Self-attention is O(n²). A 200k-token request both costs more and is slower than four 50k-token requests on the same content.

The KV cache

During generation, the model would otherwise recompute K and V for the whole prefix at every step. The KV cache stores them once. This is why TTFT scales with input length but TPS stays roughly constant — and why streaming generation is fast.

Prompt caching (provider-side)

For inputs with stable prefixes (system prompt, large doc set), the provider caches the prefix's KV state server-side. Subsequent calls skip ahead. On 200k contexts, this can be the difference between $3 and $0.30 per call.

Sliding-window attention

Some open models (Mistral, Llama variants) restrict each token to attend to the last N tokens. Stacked layers let distant info still flow through indirectly. Cheaper on smaller hardware; small quality cost.

Hierarchical summarisation

When you genuinely have more content than fits:

Lose detail, gain reach. For specific-fact retrieval, combine with RAG over the original chunks.

Where it bites in real life

Under the hood (optional)

A summarise-forward pass: chop the long text into chunks, walk through them while maintaining a running summary, fold each new chunk into the summary. ~12 lines. Skip if you don't code — the technique (rolling summary) is the takeaway.

def summarize_long(text: str, chunk_tokens=8000) -> str:
    chunks = split_by_tokens(text, chunk_tokens)
    running = ""
    for chunk in chunks:
        prompt = f"""So far we've established:
{running}
 
New segment:
{chunk}
 
Update the running summary, preserving important details and replacing things that have
been superseded. Stay under 800 tokens."""
        running = call_llm(prompt, max_tokens=900)
    return running

Further reading

• Lost in the Middle (Liu et al. 2023) — the long-context attention U-curve.
• Anthropic — Long context tips — production-tested guidance.
• FlashAttention (Dao et al. 2022) — the algorithm that made long context efficient.
• Greg Kamradt — Needle in a Haystack benchmarks — how to actually measure long-context retrieval.