To What Extent Can On-Policy Distillation Tolerate Mismatch Between Training and Inference?
On-policy distillation can learn from quantized or asynchronous rollouts with much greater tolerance to training-inference mismatch than tested reinforcement learning methods. A seven-billion-parameter student remains high quality above 200 times the mismatch of a standard 16-bit engine and fails near twice that level. Because tolerance depends on model and layer, a one-GPU screen can reject unsafe rollout settings before long runs. Within the safe budget, 8-bit key-value caches improve capacity and asynchronous rollouts reach comparable training quality sooner.
Key Takeaways
- On-policy distillation tolerates far more training-inference mismatch than tested RL methods, staying viable above 200× the mismatch of a standard BF16 engine and failing only at roughly 2× that level.
- Rollout mismatch mainly comes from quantized inference or asynchronous rollouts using older weights, but its impact varies by training method.
- A lightweight one-GPU screening check can reject unsafe rollout configurations before full training, reducing wasted compute.
- Within the tolerance budget, FP8 key-value caches can hold at least 1.93× more tokens than BF16 caches, and FP8 with async rollouts can reach similar training quality 1.62× faster.
- Safe mismatch budgets are model-dependent, with some layers more precision-sensitive, so teams should evaluate per model rather than rely on a global rule.
For teams lowering rollout cost via quantized inference or async older-weight rollouts, how much rollout-engine/model mismatch can a method survive? On-policy distillation, where a student learns from current-policy rollouts, keeps quality at over 200× the mismatch of a standard BF16 engine and fails only at about twice that level.
Why rollout engines can stop matching the model
Rollouts are generations used for training examples. Quantized inference lowers precision; async rollouts use older weights. Both create training-inference mismatch. Damage varies by method; measured tolerance shows absorbable engine drift.
On-policy distillation tolerates more mismatch than the tested methods
On a 7B student, on-policy distillation keeps quality at over 200× the mismatch of a standard BF16, or 16-bit floating-point, engine and fails at about twice that level. Under the same engines, two RL methods break at about 12× and 60×. Acceptable OPD rollouts can be unsafe for tested RL.
The budget depends on model; some layers are precision-sensitive, so check per model rather than one global rule.
A one-GPU screen can reject bad rollout settings
Before full training, a lightweight check screens a rollout configuration against the tolerance budget in minutes on one GPU. It filters engines outside the budget instead of failing after a long job, reducing search; it is not full validation.
Inside the budget, FP8 and asynchronous rollouts pay off
Inside the budget: FP8 key-value caches, using 8-bit floating-point, hold at least 1.93× as many tokens as BF16 caches, a cache-limited capacity gain. FP8 with async rollouts reaches the same training quality 1.62× sooner than sync BF16.
Takeaway: OPD appears to have a large mismatch budget; tested RL methods do not. Teams may use quantized/async rollouts after screening exact model and engine; layer sensitivity can change safe limit. Code: https://github.com/benyucong/train-inference-mismatch-opd.
Meta description: On-policy distillation tolerates 200× mismatch; two RL methods fail earlier, enabling cheaper rollouts.
Tags: on-policy distillation, training-inference mismatch, rollout quantization, asynchronous rollout, model training
Featured image: Abstract illustration of training and inference panels with mismatch gauges and quantized tokens, no people, logos, or text.

Comments (0)