Beginner Basics

To What Extent Can On-Policy Distillation Tolerate Mismatch Between Training and Inference?

On-policy distillation can learn from quantized or asynchronous rollouts with much greater tolerance to training-inference mismatch than tested reinforcement learning methods. A seven-billion-parameter student remains high quality above 200 times the mismatch of a standard 16-bit engine and fails near twice that level. Because tolerance depends on model and layer, a one-GPU screen can reject unsafe rollout settings before long runs. Within the safe budget, 8-bit key-value caches improve capacity and asynchronous rollouts reach comparable training quality sooner.

To What Extent Can On-Policy Distillation Tolerate Mismatch Between Training and Inference?

Key Takeaways

  • On-policy distillation tolerates far more training-inference mismatch than tested RL methods, staying viable above 200× the mismatch of a standard BF16 engine and failing only at roughly 2× that level.
  • Rollout mismatch mainly comes from quantized inference or asynchronous rollouts using older weights, but its impact varies by training method.
  • A lightweight one-GPU screening check can reject unsafe rollout configurations before full training, reducing wasted compute.
  • Within the tolerance budget, FP8 key-value caches can hold at least 1.93× more tokens than BF16 caches, and FP8 with async rollouts can reach similar training quality 1.62× faster.
  • Safe mismatch budgets are model-dependent, with some layers more precision-sensitive, so teams should evaluate per model rather than rely on a global rule.

For teams lowering rollout cost via quantized inference or async older-weight rollouts, how much rollout-engine/model mismatch can a method survive? On-policy distillation, where a student learns from current-policy rollouts, keeps quality at over 200× the mismatch of a standard BF16 engine and fails only at about twice that level.

To What Extent Can On-Policy Distillation Tolerate Mismatch Between Training and Inference?

Why rollout engines can stop matching the model

Rollouts are generations used for training examples. Quantized inference lowers precision; async rollouts use older weights. Both create training-inference mismatch. Damage varies by method; measured tolerance shows absorbable engine drift.

On-policy distillation tolerates more mismatch than the tested methods

On a 7B student, on-policy distillation keeps quality at over 200× the mismatch of a standard BF16, or 16-bit floating-point, engine and fails at about twice that level. Under the same engines, two RL methods break at about 12× and 60×. Acceptable OPD rollouts can be unsafe for tested RL.

To What Extent Can On-Policy Distillation Tolerate Mismatch Between Training and Inference?

The budget depends on model; some layers are precision-sensitive, so check per model rather than one global rule.

A one-GPU screen can reject bad rollout settings

Before full training, a lightweight check screens a rollout configuration against the tolerance budget in minutes on one GPU. It filters engines outside the budget instead of failing after a long job, reducing search; it is not full validation.

Inside the budget, FP8 and asynchronous rollouts pay off

Inside the budget: FP8 key-value caches, using 8-bit floating-point, hold at least 1.93× as many tokens as BF16 caches, a cache-limited capacity gain. FP8 with async rollouts reaches the same training quality 1.62× sooner than sync BF16.

To What Extent Can On-Policy Distillation Tolerate Mismatch Between Training and Inference?

Takeaway: OPD appears to have a large mismatch budget; tested RL methods do not. Teams may use quantized/async rollouts after screening exact model and engine; layer sensitivity can change safe limit. Code: https://github.com/benyucong/train-inference-mismatch-opd.

Meta description: On-policy distillation tolerates 200× mismatch; two RL methods fail earlier, enabling cheaper rollouts.
Tags: on-policy distillation, training-inference mismatch, rollout quantization, asynchronous rollout, model training
Featured image: Abstract illustration of training and inference panels with mismatch gauges and quantized tokens, no people, logos, or text.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.