Do Dense Rewards Justify Their Token Costs in Group Relative Policy Optimization?
A study shows dense rewards lose to group-relative policy optimization when compute is matched; prompt selection near failure boundary helps.
Key Takeaways
- Dense Monte-Carlo process rewards can look better per update but lose to GRPO when compared at equal generated tokens and GPU-hours.
- Sampling continuations for dense rewards consumes most training tokens, making comparisons unfair; single-batch gradient checks are only directional and may overestimate gains.
- Selecting prompts near the success boundary, especially those the model is predicted to fail often, yields small but replicated accuracy gains under matched compute.
- Fair compute evaluation is essential: some apparent improvements come from extra computation, and simple optimizer changes may capture similar benefits.
Meta description: Equal compute reverses process reward gains in 1,668 runs: Monte-Carlo rewards can lose to group-relative policy optimization, and prompt selection helps.
Tags: matched compute, language model reasoning, process rewards, group-relative policy optimization, prompt selection
Featured image: abstract friendly illustration of two training paths meeting at an equal budget scale, no real people, no logos, no text.
A study of language-model reasoning training shows that comparing reward methods at matched compute can reverse the apparent winners. The researchers ran 1,668 experiments across six models and found that a dense process reward based on Monte-Carlo sampling can look better than group-relative policy optimization when judged only per update, but falls behind when every method receives the same number of generated tokens and the same training GPU-hours. The result matters for readers who build, tune, or evaluate reasoning systems, because it suggests that some apparent gains may reflect extra computation rather than better learning.
Why process rewards looked strong
Training methods that reward correct final answers can teach language models to reason, but they give every token of a solution the same credit. To address that limitation, researchers have proposed dense process rewards that try to credit intermediate steps. One example is a Monte-Carlo reward that scores each step by sampling continuations from it.
The problem is that checking each step this way can be expensive. The study notes that comparing such a reward with group-relative policy optimization per training update, or testing it with a per-token gradient check on a single batch, leaves out what it costs. In the setting studied, the sampled continuations alone took 91% of all generated tokens. That means the method may appear more accurate per update because it uses far more generated output during training.
Matching tokens and GPU-hours changes the ranking
To make the comparison fairer, the researchers introduced a matched-compute protocol. The protocol gives every training intervention the same budget, measured both in generated tokens and in training GPU-hours, and tests it in preregistered studies with controls for side effects.
Across 1,668 runs on six models, matching compute reversed the ranking. On Qwen2.5-1.5B, the Monte-Carlo reward led standard group-relative policy optimization per update. Yet it ended 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time. None of four dense-reward variants outperformed group-relative policy optimization even when given as much training time.
The study also examined a cheaper diagnostic: a per-token gradient test. That test ordered these rewards in the same way as training did, but it overestimated each one by one to two points. That gap is as large as the gains the test is meant to detect, so a single-batch gradient check may not be enough to decide whether a new reward is worth adopting.
What still works: prompts near the success boundary
Not every intervention disappeared under equal budgets. The study found that training on prompts whose predicted success is nearest 1/2 improved accuracy at matched compute. The gain was 1.7 points on Qwen3-0.6B and 0.8 points on Qwen2.5-1.5B.
Those gains replicated on fresh seeds and on unseen problems. The researchers also tested a simple rule: the candidate prompts should be predicted to fail more often than not. That rule predicted where the matched-compute gains appeared in five of six preregistered tests.
For readers, the practical reading is that prompt selection can help, but the effect is specific and should be checked against a fair compute budget. The study points to prompts where the model is predicted to fail often, rather than to prompts the model is already likely to solve.
How much the gains are worth
The matched-compute protocol does more than rank methods. It also measures what a gain costs. The study reports that the accuracy gains from selecting prompts near the success boundary come early in training. It also finds that standard group-relative policy optimization with twice the initial learning rate comes within half a point of those gains.
That comparison is important. If a method improves accuracy by 0.8 or 1.7 points at matched compute, a simpler change to the existing optimizer settings may capture much of the same benefit. The paper does not say the prompt-selection result is useless; it says its practical value should be judged against alternatives that use the same compute.
What readers should take away
The study argues that training interventions should be validated at matched compute. That standard overturns some apparent gains and confirms others. It gives readers a clearer test: do not ask only whether a reward scores each step more finely or whether a gradient check looks promising. Ask whether the method wins after both methods are allowed the same number of generated tokens and the same GPU-hours.
For teams developing reasoning models, the result suggests three checks. First, count generated tokens, not just updates, when comparing rewards that sample continuations. Second, treat single-batch gradient tests as directional signals, not final proof. Third, when a prompt-selection trick helps, test whether a simple change such as doubling the initial learning rate gets close to the same result.
The broader message is that compute fairness can change the conclusion. A method that looks better because it spends more output tokens during training may lose when those costs are counted. A method that survives equal-token and equal-time tests has stronger evidence that it actually improves learning.

Comments (0)