On-policy distillation methods for LLMs

On-policy distillation departs from classic knowledge distillation by grading the student on trajectories the student itself generates, rather than on fixed text written by the teacher in advance. A stronger teacher model scores every token of the student's own rollout, typically using reverse KL divergence, which gives a dense, per-token reward signal similar to reinforcement learning but far cheaper to compute ^[1]. Because supervision happens on states the student actually visits, on-policy distillation avoids the exposure-bias problem that hurts purely off-policy approaches, where a model trained only on teacher-written text tends to drift and compound errors once it starts generating on its own ^[2]. In Thinking Machines Lab's original benchmark, this combination let a student reach 74.4 percent on AIME'24 using only 1,800 GPU-hours, versus 67.6 percent for pure reinforcement learning at 17,920 GPU-hours and 60 percent for off-policy distillation trained on 400,000 examples - a claimed 7 to 10x speed advantage over RL and a 50 to 100x reduction in cumulative compute ^[1].

A year after that original post, late September 2026 produced a dense cluster of academic papers that each target a specific crack in the base recipe rather than proposing a wholesale replacement. Interpolated Policy Distillation (IPD) turns the off-policy/on-policy split into a tunable continuum, blending student and teacher token distributions token-by-token with speculative decoding to keep the extra compute manageable ^[3]. Interactive-Policy Distillation addresses what researchers call teacher unanchoring - early in training a weak student wanders into prefixes far from anything the teacher has seen - by having student and teacher alternate propose-and-verify roles, reporting meaningful accuracy gains using roughly a quarter of the training examples of standard on-policy distillation ^[4]. SIPO folds reinforcement learning and self-distillation into one contrastive self-teacher, pairing correct reference answers with the student's own mistakes for dense token-level credit assignment without needing an external teacher at all ^[5]. SAKI, from a Meituan and KTH collaboration, routes supervision between reverse-KL and direct token correction depending on where the student's rollout diverges from the teacher, and reports a large rollout-throughput improvement via a speculative verifier ^[6]. B-OPSD tackles the anchoring problem differently, temporarily training the policy ahead of itself to create a 'future teacher' before resetting and supervising the original student against it ^[7]. Other papers in the same window target orthogonal weaknesses: one shows that smaller students distilled in thinking mode fail to inherit the teacher's stopping behavior and keep generating past a correct answer even though they still learn to solve the problem ^[8]; another finds that prompt usefulness for distillation is highly uneven and relational to the specific teacher rather than an intrinsic property of the prompt - a handful of prompts can match the performance of thousands, but deliberately curating which ones does not reliably beat picking them at random ^[9]; and a third introduces per-token weighting via bilevel optimization, letting a smaller student actually surpass its own larger teacher on math benchmarks ^[10].

None of this comes for free. On-policy distillation requires the teacher model to stay resident in memory and run inference in parallel with the student for the full duration of training, a structural cost that off-policy approaches simply do not carry. That is part of why one practitioner reflection frames on-policy distillation as a tool that only pays off when off-policy data genuinely fails to cover the states the student will actually visit, and works best when the student and teacher are reasonably well matched in capability and the task does not require very long generations ^[11]. Most published variants, including the original recipe, are also largely restricted to teacher-student pairs from the same model family so that token-level supervision lines up cleanly - a constraint that has pushed a separate strand of community work toward extending the technique across different tokenizers, since real deployments rarely get to pick a same-family teacher for free.

The efficiency numbers have not settled the question of what on-policy distillation is actually teaching the student. Some observers read the mechanism as straightforward knowledge transfer: the teacher's per-token judgments propagate genuine reasoning ability into the student's weights. Others, pointing to how the reverse-KL-style reward term behaves in practice, argue the visible gains lean more toward suppressing the student's bad tokens - a bounded, mostly negative correction - than toward injecting new positive knowledge from the teacher. A separate line of research lends some support to the suppression reading: rather than grading every token, one recent paper found that concentrating the teacher's supervision on as little as one percent of a trajectory's tokens, and sometimes as little as a tenth of that, can match the results of scoring every token, which suggests only a small fraction of tokens are actually carrying the corrective signal. That distinction matters for how far the technique can scale: if the real mechanism is closer to pruning bad behavior than transferring capability, the eye-catching efficiency numbers may not extrapolate cleanly to harder tasks or bigger capability gaps between teacher and student. The debate has not stayed confined to papers and forum threads, either - within a year of the original announcement, the topic had spread to standalone explainer videos and a podcast segment walking through the mechanism for a broader technical audience, a sign the idea has moved past a narrow research niche.