Converting Database Grading into Reinforcement Learning Rewards to Post-Train Agents with ThinkingBox

ThinkingBox-Training is now on GitHub, and the headline number is this: after 50 steps of GRPO, Qwen3.8-27B raised its average success on a held-out agent benchmark from 51.70% to 60.78%, and the count of tasks it completed correctly on all twenty attempts jumped from 38 to 107. That second figure is the one worth pausing on. It is the difference between a model that sometimes gets there and a model you could plausibly leave alone with a real backend.

The work follows an earlier finding that headline accuracy hides a reliability gap. ThinkingBox-Bench, a 507-task benchmark, gives an agent a real backend, a set of MCP tools and a simulated user with private context, then checks the terminal database state rather than the tool calls the agent made. In the first post, eighteen models each ran the benchmark twenty times. Kimi-K3 solved 93.89% of tasks at least once but completed only 68 of 507 (13.41%) on all twenty attempts. Claude Opus 5 was the mirror image: narrower coverage, far more consistency. Claude Opus 5.5 scored 0.66 percentage points higher than Opus 5 on pass@1 yet passed exactly the same 241 tasks on all twenty attempts, so the extra headline accuracy bought no extra dependability.

That framing matters for anyone picking a model to touch production records. Pass@1 alone will not tell you which tasks survive every attempt.

From grader to reward

The insight behind the training release is simple. For most ThinkingBox tasks, an executable check returns 1 or 0 straight from the final database state. That is not only a grader; it is a reward function. The team built a training pool from ThinkingBox-format tasks outside the benchmark and used those executable outcomes as rewards.

The hard part was choosing which tasks to train on. Most tasks are either already solved or hopeless for the model in front of you, and neither teaches anything. The team calls its fix offline policy-relative difficulty: estimate each task's success probability once with the base policy before training, then keep the selected pool fixed for the whole run. Difficulty is treated as relative to the policy, since a task is hard for the current model only until that model learns it. No ThinkingBox-Bench task entered the pool; the benchmark stayed reserved for evaluation.

What moved, and what it cost

The fine-tuned checkpoint was evaluated on all 507 benchmark tasks, twenty trials each, on a set it never trained on. The benchmark is out of distribution relative to the training pool: different domains, databases, policies and most tools. Shared elements — the ThinkingBox framework, the simulated-user machinery, executable terminal-state grading, a few common tools and a similar task style — limit how far the transfer claim can be pushed.

Pass@1 and observed 20/20 rose in all five domains. Pass@20 was mixed, rising in neobank and consulting but falling in retail, travel and auto insurance. Auto insurance lost the most breadth at 7.0 points; retail gained the most observed-20/20 at 24.5 points. The pattern is consistent with sharpening: probability mass moved toward trajectories that work repeatedly while some low-probability paths disappeared. The model did not become broader. It became more repeatable.

The ordering effects are worth noting. GPT-5.6 Sol scored slightly higher on pass@1 than the fine-tuned Qwen3.8-27B, 61.91% versus 60.78%, but passed fewer tasks on all twenty attempts, 82 versus 107. Claude Sonnet 4.6 scored lower on pass@1 at 59.19% yet passed 102 tasks every time. For context, observed 20/20 stands at 68 for Kimi-K3, 128 for GPT-5.4, 231 for GPT-6 Astra and 241 for Claude Opus 5.5.

Using the same cost math as the first post — the estimated cost of the 507 × 20 campaign divided by tasks passing 20/20 — the fine-tuned checkpoint lands at $8.00 per dependable task, close to GPT-5.4 at $6.80, GPT-6 Astra at $7.45 and Claude Opus 5.5 at $7.80. Cost per successful attempt fell from $0.177 to $0.139 while pass@1 rose. Those dollar values are a comparative serving index, not invoices; serving the checkpoint would cost the same as the base model, and the figure includes inference but excludes the training run and the policy-probe cost of curating the dataset.

Inside the loop

Selection runs once, offline, against the base policy. Training then uses that fixed pool. Every rollout is a complete episode with a fresh backend, an isolated MCP session and GPT-5 Chat as the simulated user, not a text completion. Most rewards come from deterministic terminal-state checks; a minority of narrow rubric requirements can invoke the configured judge. Training and evaluation share interpretation code so that reasoning extraction, tool calls, argument typing and rendering are read identically — otherwise the reward would be measuring a different agent from the one being evaluated.

The reference run fine-tuned all parameters of Qwen3.8-27B on 187 tasks with token-mean GRPO. Each update generated 18 groups of 8 rollouts, or 144 episodes, across 24 H100s, running synchronously with parameter and optimizer offload. AdamW used a 1e-6 learning rate and zero weight decay, for 50 steps. Zero-variance groups were replaced DAPO-style, and backend timeouts were refilled rather than scored as agent failures, since a system error is not an incorrect outcome.

Why selection dominates comes down to GRPO arithmetic. If all eight rollouts fail the reward vector is all zeros; if all eight succeed it is all ones. Either way, centering within the group yields no advantage, because no trajectory beat another. Only mixed groups carry signal.

One task shows the change concretely. A traveler asks for a room downgrade, but company policy requires manager approval first. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After fine-tuning it preserved the booking and payment state in all 20, with all expected records matching in 19.

What to watch

The tradeoff is now explicit: repeatable success up, at-least-once coverage slightly down. If your work depends on database records being right every time, observed 20/20 is the column to read alongside pass@1, and the gap between them is where production surprises live. The training stack is public on GitHub, so the next useful signal will be whether this sharpening holds when the pool is rebuilt against a different base policy — the one path the team deliberately did not take during the run.