Pre-training, post-training, RLHF, RLAIF

A frontier chat model isn't trained in one step — it's trained in three or four. Knowing the stages explains a lot of the model's behaviour.

What the training stages actually do

A frontier chat model isn't trained once — it's trained in three or four stages, each one shaping a different aspect of the final product.

• Pre-training stuffs the model with knowledge by reading the internet. Expensive, slow, and where most of the model's facts come from.
• Supervised fine-tuning (SFT) teaches it to behave like an assistant — to answer instead of continue.
• RLHF polishes the answers by training against human preferences: shown two replies, humans pick the better one, the model learns what "better" looks like.
• RLAIF (or Constitutional AI) does the same but with another AI as the rater, scaling the polishing further.

That layering is why the same architecture and the same parameter count can feel completely different across base/instruct/chat versions. Knowledge lives in stage 1; behaviour lives in stages 2 and beyond.

That's the whole concept. Below: each stage in detail and the practical implication of where your costs and behaviour come from.

When you'd reach for it

Knowing the stages matters when you have to make a choice:

• Picking a model — chat-tuned vs base, refusal-trained vs uncensored, RLHF vs DPO.
• Deciding fine-tune vs RAG — knowledge problem (RAG) or behaviour problem (fine-tune)?
• Reading model cards and papers — they all reference these stages.
• Debugging odd behaviour — "the model is too refusal-happy" is post-training; "the model doesn't know our internal product" is pre-training (and you can't fix that — use RAG).

If you're a pure consumer of frontier APIs and never tune anything, the conceptual map is enough.

How it's actually built

Stage 1: Pre-training

The model reads ~10–15 trillion tokens (web pages, books, code, papers). Objective: predict the next token, on everything. No human labels.

Output: a base model. It writes in the style of whatever it last read. Ask "what is the capital of France?" and it might continue your question, or write an article, or just keep predicting plausible internet text. Most expensive stage by far — months on thousands of GPUs, costing tens of millions at frontier scale.

Stage 2: Supervised fine-tuning (SFT)

Show the model thousands or millions of (prompt, ideal-response) pairs written by humans. Train it to imitate.

Output: an instruction-tuned model. Now it answers questions instead of continuing them.

Stage 3: RLHF — Reinforcement Learning from Human Feedback

The breakthrough that made ChatGPT work. Three sub-steps:

1. Collect preference data. Sample two responses to the same prompt; a human picks the better one. Repeat thousands of times.
2. Train a reward model to predict that human preference: given (prompt, response), output a quality score.
3. Fine-tune with RL (typically PPO) — gradients push the chat model toward higher reward-model scores. A KL-divergence penalty keeps the policy anchored to the SFT model; without it, the policy drifts into gibberish that exploits the reward model's blind spots ("reward hacking").

The shift: training on "what humans prefer" rather than "what humans demonstrate."

Stage 4: RLAIF / Constitutional AI

Same loop, but preferences come from another AI scoring against a written constitution, not from humans. Cheaper, scales further, less dependent on potentially biased raters. Anthropic's Constitutional AI paper showed it's competitive with RLHF for harmlessness training.

Where it bites in real life

Under the hood (optional)

The RLHF training loop in pseudocode: train a reward model on (chosen, rejected) preference pairs, then fine-tune the chat model with reinforcement learning to maximise that reward — with a KL penalty to keep it from going off the rails. Skip if you don't code; the conceptual story above is the lesson.

# 1. Reward model trained on preferences
def train_reward_model(prefs):       # prefs: list of (prompt, chosen, rejected)
    for prompt, chosen, rejected in prefs:
        score_chosen   = reward_model(prompt, chosen)
        score_rejected = reward_model(prompt, rejected)
        loss = -log(sigmoid(score_chosen - score_rejected))
        optimize(loss)
 
# 2. RL fine-tuning of the policy (chat model)
for batch_of_prompts in dataset:
    completions = policy.generate(batch_of_prompts)
    rewards = reward_model(batch_of_prompts, completions)
    # KL penalty keeps the policy close to the SFT model
    loss = -rewards + beta * kl_divergence(policy, sft_reference)
    optimize(loss)

Further reading

• InstructGPT paper (Ouyang et al. 2022) — production-grade RLHF.
• Constitutional AI (Bai et al. 2022, Anthropic) — RLAIF; principles-driven training.
• Hugging Face — RLHF blog — clear walkthrough with diagrams.
• Karpathy — Intro to LLMs (talk) — covers all three stages in plain English.