Tokens & tokenization

The "chunks" an LLM is billed by. Common words are one chunk; rare words are several.

What's actually happening

A small program called the tokenizer runs before the model sees anything. It uses BPE, an algorithm trained on a huge text corpus to find which character sequences appear together often enough to deserve a single ID.

A few non-obvious consequences of how that lookup works:

• Spaces matter. " the" (with leading space) and "the" are two different tokens with two different IDs. This is why where punctuation goes in a prompt matters.
• Misspellings cost more. A typo turns a common word into a rare string the tokenizer has to spell out byte by byte.
• Non-English text is denser. Languages with less training data get split into more tokens per character.
• The vocabulary is finite — typically ~100,000 distinct tokens. Anything not in the table is assembled from smaller pieces.

Where it shows up in real life

Under the hood (optional)

Counting tokens before sending a request is how you avoid surprise bills. ~6 lines of Python with tiktoken. Skip if you don't code — the takeaway is "count before you send."

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
 
ids = enc.encode("Hello, strawberries!")
# → [9906, 11, 76050, 0]   (4 tokens, not 3 words)
 
print(len(ids))                                # 4
print([enc.decode([i]) for i in ids])
# → ['Hello', ',', ' strawberries', '!']

Try it yourself (no coding, ~2 minutes)

1. Use the interactive demo above. Type your full name, the longest English word you know, a sentence in another language, an emoji.
2. Watch how each splits differently — and how your bill would scale.
3. For three tokenizers side-by-side: tiktokenizer.vercel.app.

Further reading

• Tiktokenizer playground — paste text, see three tokenizers side by side.
• Anthropic — Counting tokens — canonical way for Claude.
• Karpathy — Let's build the GPT Tokenizer — 2-hour video building BPE from scratch.