Socratic skill learning interleaves teaching and practice to develop transferable agent skills.
Socratic Skill Learning interleaves teaching and student practice to build reusable LLM Skills, improving transfer and token efficiency.
Key Takeaways
- Socratic Skill Learning helps LLM agents build reusable Skills by interleaving educator guidance with student-led testing, rather than merely saving successful trajectories.
- The method focuses on transferable Skills whose procedures and conditions of use are actively checked, making them more reliable in new task sessions.
- On SKILLTRANSFERBENCH, Fixed-Window SSL reaches 0.801 utility, outperforming no-Skill and adapted distillation baselines, while Issue-Guided SSL reaches 0.805 utility.
- Issue-Guided SSL reduces Educator token use by 61.7% and total acquisition tokens by 16.9% compared with Fixed-Window SSL by focusing review on unresolved questions.
Meta description: Socratic Skill Learning helps LLM agents reuse task knowledge, raising utility on SKILLTRANSFERBENCH while Issue-Guided SSL cuts Educator token use.
Tags: LLM agents, Socratic Skill Learning, SKILLTRANSFERBENCH, agent Skills, token efficiency
Featured image: abstract, friendly illustration of a student agent testing a procedure with a guide, no real people, no logos, no text
Socratic Skill Learning is a method for LLM agents that makes task knowledge reusable across sessions. The work introduces it to address a gap in Agent Skills: making knowledge reusable is not enough if a successful task attempt leaves evidence about a procedure and its conditions of use untested. On SKILLTRANSFERBENCH, Fixed-Window SSL reaches 0.801 utility on the main 22-family, 155-case cohort, and Issue-Guided SSL, a Socratic Skill Learning variant, reaches 0.805 utility while using fewer tokens.
The problem: reusable Skills need tested conditions
Agent Skills are meant to carry knowledge from one task attempt into later sessions. That sounds simple: record what worked, save it as a Skill, and reuse it. The harder part is transfer. A Skill needs evidence not only about the procedure, but also about when that procedure should apply. A successful task attempt may show that a procedure worked in one context, yet leave the boundary conditions untested. If an LLM agent stores only the successful trajectory, the Skill may not behave well when the task varies or when a fresh solver session uses it later.
Socratic Skill Learning starts from that problem. It is designed to create Skills that can be reused beyond the learning task, not merely replay the original success.
How Socratic Skill Learning works
The method interleaves expert teaching with Student practice. An Educator guides investigation, while the Student remains the sole environment operator and Skill author. The Student can question, test, and adapt advice rather than passively copying it into a Skill package.
This separation matters. The Educator provides guidance, but the Student controls the environment and the final Skill. That keeps learning tied to practice: advice is useful only if the Student can check it against the task and revise the Skill around what remains uncertain. The process is closer to a guided investigation than to a lecture. The Student writes and revises the package, and the Educator helps shape what is worth testing.
Benchmark and reported utility scores
To test reuse beyond the learning task, the study constructs SKILLTRANSFERBENCH, with 35 task families and 408 executable cases. The benchmark pairs public variation specifications with held-out evaluation in fresh solver sessions. That setup is important: it checks whether a learned Skill helps in new sessions, not only in the session where it was written.
On the main 22-family, 155-case cohort, Fixed-Window SSL reaches 0.801 utility. The comparison points are 0.723 without a Skill, 0.761 for adapted Trace2Skill, a trajectory distillation approach, and 0.751 for adapted SKILL-KD, a contrastive distillation approach. Across three model pairs on shared families, interleaved guidance outperforms expert authoring and post-hoc teaching.
Process analysis links guidance to Student writing and package revision. A case study shows the Student testing and refining a proposed procedure for later reuse. These details point to the central mechanism: the Skill improves through active checking, not through a one-time copy of advice.
Issue-Guided SSL reduces token costs
Fixed-Window SSL has a cost pattern that can become less efficient as work continues. As demand for new lessons falls, the cost of fixed-window review grows. In practical terms, the Educator may keep spending review effort even when the amount of new learning is decreasing.
Issue-Guided SSL addresses that by tracking unresolved questions across Skill revisions. Instead of reviewing on a fixed window alone, it selects and focuses review around issues that remain open. On the main cohort, Issue-Guided SSL reaches 0.805 utility with 61.7% fewer Educator tokens and 16.9% fewer total acquisition tokens than Fixed-Window SSL. Tokens are the units of text processed in the learning and review process, so the lower totals point to a more focused acquisition cost.
The result does not turn Skill learning into a guaranteed shortcut. It shows a measured trade-off: better guidance can raise utility, and issue-focused review can keep the gain while reducing token use. For agent builders, the practical takeaway is that reusable Skills may need an active learning loop: question the advice, test it in the environment, revise the Skill, and track what still needs attention.
Comments (0)