Sampling: temperature, top-p, top-k
The dials that control how creative — or how predictable — the model is.
What's actually happening
| Knob | What it does | Typical values |
|---|---|---|
temperature | Sharpness of the distribution | 0 (deterministic) → 1 (natural) → >1 (chaotic) |
top_p | Keep the smallest set of words whose probabilities sum to ≥ p, sample from those | 0.9–0.95 is the default sweet spot |
top_k | Keep only the k highest-probability words | Less common; mostly replaced by top-p |
Two production rules of thumb:
- Most teams ship at
temperature=0or0.2and leave top-p alone. Easier to debug, fewer surprises. - Crank temperature only when variety is the product (creative writing, prompt brainstorming) — not because you "want better answers."
Prompt: "My favourite colour is". These are the (illustrative) probabilities the model assigns to the next token at different sampling settings.
Green = inside the top-p nucleus (eligible to be sampled). Grey = excluded by top-p. Lower temperature sharpens the distribution; higher temperature flattens it.
Defaults that just work
| Task | Temperature |
|---|---|
| Code generation, JSON extraction | 0 |
| Summarisation, classification | 0–0.3 |
| Default chatbot reply | 0.5–0.8 |
| Creative writing, brainstorming | 0.8–1.2 |
Where it shows up in real life
Under the hood (optional)
The sampling knobs are just request parameters. ~8 lines of Python. Skip if you don't code.
›Show the example code (Python, ~8 lines)click to expand
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=512,
temperature=0.7, # 0 for code, 1.0+ for creative
top_p=0.95, # nucleus sampling
stop_sequences=["\n\n##"], # stop at a heading
messages=[{"role": "user", "content": "Write a haiku about debugging."}],
)Try it yourself (no coding, ~5 minutes)
Most chat products hide temperature; playgrounds expose it. Free option:
- console.anthropic.com → Workbench (free signup). Temperature slider on the right.
- Test 1: temp = 0. "Write a one-sentence opening line for a fantasy novel." Run 3×.
- Test 2: temp = 1. Same prompt. Run 3×.
- Test 3: temp = 0, "What's 247 × 89?". Run twice.
Test 1: nearly identical. Test 2: wildly different. Test 3: same answer twice (right or wrong, but consistent). You just felt why deterministic tasks want low temperature.
Check your understanding
- 1. What does temperature=0 do?
- 2. Building a JSON extractor — what temperature first?
- 3. What does top-p = 0.9 mean?
Found this useful? Share it with someone learning AI.
Further reading
- The Curious Case of Neural Text Degeneration (Holtzman et al. 2019) — paper that introduced top-p.
- Anthropic — Messages API parameters — exact behaviour in Claude.
- Hugging Face — How to generate text — visual walkthrough.