Beginner Basics

For Post-Training Stages and Preference Data, Judging Is Much Easier Than Generating

LLM post-training shapes behavior through supervised fine-tuning, preference data, and verifiable rewards, making judging easier than generating.

For Post-Training Stages and Preference Data, Judging Is Much Easier Than Generating

Key Takeaways

  • Post-training uses far less compute than pre-training but strongly shapes how well a model follows instructions, stays safe, and reasons.
  • SFT teaches format and tone by imitating ideal answers, while preference tuning learns from rankings and can push beyond the quality of the examples.
  • DPO is simpler and more stable because it trains offline on preference pairs, but online RL methods like PPO can explore more and often reach higher performance at greater cost.
  • RLVR uses objective programmatic checkers for verifiable tasks such as math and code, making judging easier and harder to game than using a learned reward model.
  • Modern pipelines often train domain specialists first, then distill them into one general model, with human rubrics and expert feedback still playing an important role.

Developers, researchers and product teams seeking a practical map of LLM post-training now have a stage-by-stage description from a 2026 summary by Lambert and Timbers. It says the recipe—the sequence of post-training steps a lab follows, with a canonical recipe being the standard version of its time—changed more in the last year than in the three years before. Developers Digest (n.d.) notes that post-training uses much less compute than pre-training but largely decides how well a model follows instructions, how safe it is and how it reasons.

For Post-Training Stages and Preference Data, Judging Is Much Easier Than Generating

From Knowledge to Behavior

Pre-training turns billions of random weight values into grammar, facts and reasoning patterns by making the model predict the next token. Post-training then makes comparatively small adjustments to shape behavior. The data flowing through the model is not the weights: a token is a text chunk with a fixed ID and no meaning of its own; a vector is a list of numbers representing meaning; the embedding table maps each token ID to a starting vector. Only the embedding table and the layers that transform vectors are weights.

The first post-training stage is SFT, or supervised fine-tuning. It trains on example prompts paired with ideal answers. Unlike pre-training, which continues raw text, SFT learns to answer. It teaches format and tone quickly, but the model can only imitate what it has been shown.

The source frames the difference as studying grandmaster games versus playing many games and learning only whether you won: SFT is capped near the quality of its examples, while RL can go beyond them.

Preference Tuning, RLVR and Checking

The second stage is preference tuning. It uses reinforcement learning, or RL, where the model learns from scores on its own attempts. RLHF is RL from human feedback, where scores come from a judge trained on human preferences. PPO, proximal policy optimization, is an RL algorithm that keeps each update small. DPO, direct preference optimization, trains directly on pairs of answers marked better and worse, with no separate reward model and no RL loop, according to Rafailov et al. (2023).

Preference tuning differs from SFT because it learns from rankings of several answers, not one ideal answer. The rankings train a reward model, a separate judge model, and the main model adjusts to earn higher scores. The weakness is reward hacking: the model learns to please the judge instead of being right. DPO skips the reward model and the RL loop.

DPO is simpler, more stable and faster to iterate on because it trains offline on a fixed dataset, but it is limited by that dataset’s coverage and has a slightly lower performance ceiling. Online RL such as PPO generates fresh completions and can explore new regions, often reaching higher peak performance, at the cost of more compute and more moving parts, according to Ivison et al. (2024), Tajwar et al. (2024) and Xu et al. (2024). DPO needs feedback data, not necessarily human feedback. UltraFeedback was the first prominent open synthetic preference dataset and helped start the DPO era, Cui et al. (2023) say, and Tülu 3 followed with larger synthetic-feedback recipes, Lambert et al. (2024) note. Rejection sampling, another approach mentioned in the source, keeps only the critic’s best of ten versions for practice.

The third stage is RLVR, reinforcement learning with verifiable rewards. AI2, the Allen Institute for AI, introduced the RLVR term, Lambert et al. (2024) say. A program checks whether the answer is correct, for example in math, code tests or numeric ranges. Unlike a learned judge, a checker is cheap, objective and hard to game, but it only works where an answer can be checked. GRPO, group relative policy optimization, scores a group of answers and reinforces the above-average ones. The source says this is also how models learn to think: careful attempts win more often, so they get reinforced.

Specialists, Distillation and Human Rubrics

The fourth stage separates specialists first, then distills them. One RLVR run on every domain became expensive and conflict-prone, because mixing math, code and agentic RL can trade capabilities off against each other. Specialists are cheap and parallelizable, and on-policy distillation matured. A common pattern trains one specialist per domain, then lets a general student write its own answers while the relevant teacher marks it token by token, according to Lambert and Timbers (2026) and Lu and Thinking Machines Lab (2025).

For Post-Training Stages and Preference Data, Judging Is Much Easier Than Generating

Distillation means a student learns from a teacher’s outputs, Hinton et al. (2015) describe. On-policy means the student learns from its own attempts, including its own mistakes, Agarwal et al. (2024) add. MOPD, multi-teacher on-policy distillation, lets several specialist teachers train one student on the student’s own answers. KL, Kullback-Leibler divergence, scores how different two models’ next-word probabilities are. Reverse KL penalizes the student for saying things the teacher would never say, Gu et al. (2024) explain. The source says teacher marking is not a golden answer or a final reward-model score: the student writes its own answer, and the relevant teacher gives an opinion on every token.

The source says conflicting skills in the student are much less than in one mixed RL run, because each question gets feedback from one teacher only, but the conflict does not vanish because the student has finite capacity and the mix of questions still has to be tuned.

Human experts still matter at different points. SFT can use expert-written answers. Preference tuning can use expert rankings. RLVR can use rubrics, for example in finance and law. A rubric is a checklist of yes-or-no items any judge can run on any answer. Rubric weights, from 1 to 10, set how many points each item is worth; they are not model weights, but the weighted score becomes the reward that adjusts the model’s weights. A good rubric accepts more than one valid method, and in a MOPD pipeline the finance teacher is only as good as the tasks and rubrics behind it. Gunjal et al. (2025) say rubrics are already delivering gains beyond verifiable domains.

The source describes Llama 2 as using two reward models, helpfulness and safety, which can pull against each other; for risky prompts, the safety score takes priority.

The DeepSeek-V4 report describes rubric-guided RL data scored by a generative reward model, which works with only a small set of diverse human annotations, DeepSeek-AI (2026) notes. The source also contrasts R1-Zero and R1: R1-Zero was an experiment with pure RL, strong reasoning but messy output, while R1 was the product. A cold start gives the model a small SFT warm-up so it already writes in a clean, consistent format, and RL then spends its effort on correctness instead of readability.

For trace distillation SFT in MAI-Thinking-1, the source says traces from three specialists—SWE and agentic; STEM; helpfulness and safety—are collected, and one model is fine-tuned on them. The final climb, a term Microsoft AI uses for a long RL run that keeps improving a score, is one more RL run on that consolidated model. The source says this is closer to R1 than to V4, though its attribution note breaks off at “Micros,” so no further detail is provided there.

Preference Data and the Unaudited Gap

Preferences are what people reach for when there is no single right answer. Lambert (n.d.) says this works because judging is far easier than generating: people can tell which answer is better even when they could not write it themselves.

Lambert et al. (2023) trace RLHF to three branches, retaining original attribution in their figure: optimal control, which supplied dynamic programming and mathematical foundations that led to reinforcement learning and then deep RL; deep learning, which scaled neural networks, data and problem complexity into LLMs; and economics and decision theory, which studied individual preferences and methods for comparing them.

For Post-Training Stages and Preference Data, Judging Is Much Easier Than Generating

The older idea behind utility appears in Port-Royal Logic, which supports the view that decision quality is an outcome weighted by its probability, Arnauld (1662) writes. Singh et al. (2009) describe rewards through value functions: a value function gives direction toward preferred behavior, and decisions transition to higher-valued states.

Preference data can be collected through several interfaces. A Likert score lets a rater give a response a score on a fixed scale, as in early Claude’s preference interface, Bai et al. (2022) note. An A/B test asks the rater to choose between two responses. Pairwise with ties, as used in the Chatbot Arena public evaluation, lets the rater choose between two responses or mark a tie, Chiang et al. (2024) say. A single bit uses thumbs up or thumbs down on one response. Pick from many asks the rater to choose the best from several responses, the default in image models, Lambert (n.d.) notes.

Ratings and rankings are different. A rating is a score for one completion in isolation. A ranking is a relative comparison between responses, often on a Likert scale, Bai et al. (2022) say.

The chain is Spec, then Data, then Behaviour. The Spec is the documented intended behavior, for example the Model Spec, OpenAI (2024) notes. Lambert (n.d.) identifies the unaudited gap: the step from Data to Behaviour is not checked, meaning nobody verifies whether the values written down by annotators are actually reflected in the final model.

For readers, the practical takeaway is that LLM post-training depends on matching the training signal to the task. SFT teaches shape, preference tuning learns what people like, RLVR learns what can be checked, and specialist distillation tries to keep skills from colliding. The next thing to watch is whether rubric-guided and generative-reward methods can extend verified learning into hard-to-verify domains while closing the gap between documented intent and observed behavior.

Meta description: LLM post-training explained with SFT, RLHF, DPO, RLVR, preference data, specialist distillation, human rubrics, reward models and model behavior trade-offs.

Tags: LLM post-training, preference data, RLHF, DPO, RLVR

Featured image: Abstract friendly illustration of four tuning stages as colored lanes merging into a model, with no real people, no logos and no text.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.