Every Token Needs Its Own Teacher Through Adaptive Multi-Teacher On-Policy Self-Distillation
Multi-teacher adaptive on-policy self-distillation lets each LLM token choose a teacher, improving Qwen3 math reasoning by up to 5.9 over base model.
Multi-teacher adaptive on-policy self-distillation lets each LLM token choose a teacher, improving Qwen3 math reasoning by up to 5.9 over base model.