Beginner Basics

This Is an Overview of Reasoning Models.

Verifiable rewards made reinforcement learning central to reasoning models. Compute scales at training and inference, shaping modern frontier recipes.

This Is an Overview of Reasoning Models.

Key Takeaways

  • Reinforcement learning shifted from a final polish to a core training stage, enabling large language models to learn longer reasoning chains.
  • RLVR, reinforcement learning with verifiable rewards, replaced expensive reward models with automatic checkers for tasks like math and code, making reasoning training scalable.
  • Reasoning models improve performance through both training-time scaling and inference-time scaling, often increasing coverage first and reliability later.
  • Key RL algorithms include PPO, GRPO, DAPO, and CISPO, each differing in how they compare, clip, or weight model updates.
  • The 2026 reasoning-model landscape reflects major shifts: large-scale RL, open-weight progress, agentic tool use, and multi-teacher training designs.

For anyone trying to follow how large language models learned to "think" before answering, the most important shift of the past two years is this: reinforcement learning (RL) stopped being a final polish and became a core training stage. This article explains that change for readers who know basic machine learning but have not tracked the research literature—covering why RL with verifiable rewards (RLVR) unlocked reasoning models, how training-time and inference-time compute scaling differ, and which design choices now define frontier training recipes.

This Is an Overview of Reasoning Models.

From cherry on the cake to load-bearing wall

The old mental model came from Yann LeCun's 2016 "cake" analogy, as cited in Lambert (2026): unsupervised or predictive learning is the cake, supervised learning is the icing, and RL is the cherry. The logic was about signal density—each layer up gives the model far fewer bits of signal per sample than the layer below, so RL could only ever be a finishing touch.

RLVR broke that assumption. It keeps the RL loop but makes the reward model optional, replacing it with a verification function that scores completions (Lambert et al., 2024). With a reliable checker—think math answers or code tests—the reward signal becomes cheap and scalable, and RL can run long enough to change how a model behaves, not just how it phrases things. That is the door through which reasoning models arrived (Lambert, 2026; Lambert, n.d.).

A reasoning model, in this framing, is a model trained with RL (specifically RLVR) to think at length before answering: it produces a long chain of thought and spends more compute at test time.

Two ways to spend more compute

Performance can be scaled in two places, and the lecture material treats them as complementary (OpenAI, 2024, as cited in Lambert, n.d.). RL training scales compute during learning; inference-time scaling spends compute while the model is answering. The analogy in the slides, the cost rows, and the sequential-versus-parallel split are a summary rather than measured results—but the practical point is that both routes improve performance, and labs now budget for both.

This Is an Overview of Reasoning Models.

This is also where the pass@1 versus pass@K distinction matters. pass@1 is one shot and measures reliability: can the model get the right answer on the first try? pass@K allows K attempts with any one counting, and measures coverage—whether the model can solve the problem at all, even if it only sometimes gets there (Lambert, n.d.). Reasoning training tends to improve coverage first; reliability is the harder, later win.

The algorithm stack: PPO, GRPO, DAPO, CISPO

The base methods are worth keeping straight because the newer variants are modifications of them.

PPO, Proximal Policy Optimization (Schulman et al., 2017), works like a student with a personal coach. The coach—the critic—predicts how well each attempt should go, and the reward is compared against that prediction. GRPO, Group Relative Policy Optimization (Shao et al., 2024), works like a class solving the same question: there is no coach, and each answer is scored against the group's average. The group-with-no-signal case corresponds to the zero-gradient filtering described in Lambert (n.d.).

DAPO—Decoupled Clip and Dynamic sAmpling Policy Optimization—is a relaxed-clipping variant of GRPO (Yu et al., 2025, as cited in Lambert, n.d.). Note that the lecture slide expands the acronym as "Dynamic Advantage Policy Optimization"; this article uses the DAPO paper's own name. CISPO, Clipped Importance Sampling Policy Optimization, comes from MiniMax-M1 (MiniMax, 2025, as cited in Lambert, n.d.). The importance sampling (IS) weight in both families corrects for training on answers written by an older version of the model (Lambert, n.d.).

What the 2026 reasoning model wave looks like

Release dates in the source material come from benchr (n.d.) and Interconnects AI (2026b), with pros and cons summarized from "Every AI Model Family That Matters" (2026), Interconnects AI (2026a, 2026b) and AI Weekly (2026). The source presents 12 models with one pro and one con each—a selection, not a ranking—and vendor names stand in for logos.

This Is an Overview of Reasoning Models.

On accuracy, only three of the 12 have an Artificial Analysis Intelligence Index v5 score. Claude Opus 5.5 leads at 58, ahead of Gemini 4 Argon at 50 and GLM-5.3 at 32 (Artificial Analysis, 2026). The index is a composite, not a pure accuracy measure, so treat the ordering as a rough signal rather than a verdict.

How the landscape got here

Five transitions explain the current shape of the field.

  • Cold-start SFT to large-scale RL. DeepSeek R1's recipe began with cold-start supervised fine-tuning and then relied on large-scale RL (Guo et al., 2025).
  • Closed weights to an open-weight standard. Kimi 1.5 published a detailed report but no open-weight release (Kimi Team et al., 2025); Qwen 3 then set a new open-weight standard (Lambert, n.d.; Yang et al., 2025).
  • RLHF to RLVR. RLVR keeps the RL loop, makes the reward model optional, and swaps in a verification function—the change that opened the door to reasoning models (Lambert et al., 2024; Lambert, 2026; Lambert, n.d.).
  • Reasoning to agentic reasoning. The frontier moved from reasoning models to long-horizon, tool-using agents (Lambert, n.d.), and DeepSeek V3.2 builds thinking directly into tool use (DeepSeek-AI, 2025).
  • One teacher to many. MiMo-V2-Flash combines multi-teacher on-policy distillation (MOPD) with large-scale agentic RL, folding several teachers into one model (LLM-Core Xiaomi, 2025; Lambert, 2026).

The RLVR recipe choices that actually matter

If you are reading training reports, these are the knobs to look for.

  • Online filtering and curriculum. Skip prompts that have become too easy or too hard, and keep harder problems for later in training (e.g., Kimi Team et al., 2025; Mistral AI, 2025).
  • Relaxed upper clipping. Widen the upper clip bound so exploratory updates survive (Yu et al., 2025).
  • Per-token loss. Average the loss over total tokens rather than per sequence, which removes a length bias (Liu et al., 2025).
  • Language consistency rewards. Penalize language switching inside a trace (Guo et al., 2025; Mistral AI, 2025).
  • Length control. Extend the target length progressively so the model first learns what it can at a shorter length (Kimi Team et al., 2025).

What to watch next

The open questions follow directly from the transitions above: whether verification stays cheap as tasks move from math and code into agentic, tool-using settings; whether open-weight releases keep pace with closed frontier training; and whether multi-teacher distillation plus agentic RL becomes the default recipe. For readers, the practical takeaway is that "reasoning" is now a training-stage property, not a prompting trick—so when a new model claims better thinking, the first thing to check is what reward signal it was trained against.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.