Beginner Basics

This Study Explains Actor, Judge, and Teacher Decision Models in Long-Horizon Agents

A study shows weak LLM actors can teach agents via step-level judge rewards, improving ALFWorld and WebShop performance in long-horizon tasks.

This Study Explains Actor, Judge, and Teacher Decision Models in Long-Horizon Agents

Key Takeaways

  • A single decision model can serve as an Actor, Judge, or Teacher by outputting typed decisions such as probabilities over candidate actions.
  • Models that perform poorly as Actors can still be strong Judges, because judging completed trajectories reveals which steps mattered.
  • Step-level judgment provides finer credit assignment than final success signals, making it useful for reinforcement learning.
  • Using JEV step-level signals to train a Qwen2.5-1.5B policy improved performance on ALFWorld and WebShop.
  • Decision models should not be evaluated only by raw action selection, since judging ability can become a valuable teaching signal.

Meta description: ZJU-REAL decision models show weak actor models can still teach a Qwen2.5-1.5B policy with JEV step-level signals, improving ALFWorld and WebShop performance.

Tags: decision models, reinforcement learning, Qwen2.5-1.5B, ALFWorld, WebShop

Featured image: action steps and reward signals, no people, logos, or text.

Lead: one model, three jobs

Decision models output typed decisions, such as a probability over candidate actions, so one model can be an Actor, Judge, or Teacher that gives a reinforcement-learning policy step-level rewards. The materials ask: if a model performs badly while choosing actions, can it still teach another model which actions mattered?

The answer compares 3 decision models and 3 general LLMs, plus policy training with JEV, Qwen2.5-1.5B, ALFWorld, and WebShop.

All six models failed the same way as actors

This Study Explains Actor, Judge, and Teacher Decision Models in Long-Horizon Agents

In ALFWorld, all six models used as actors failed the same way: each was pulled toward an easier subgoal and skipped a prerequisite step.

The pattern concerns subgoal choice, not missing task knowledge. If a model acts poorly but judges well, its knowledge may still be useful.

Judging revealed the missing prerequisite

As judges, given the full trajectory, the same models correctly credited steps that advanced the skipped prerequisite.

The materials separate predicting whole-task success from judging whether each step helped. The useful signal came from step-level judgment.

For reinforcement learning, a final outcome is too coarse: it says whether the task succeeded, but not which action made it possible. Step-level judging recovers that missing credit.

A weak actor can still be a strong teacher because an actor must anticipate consequences, while a judge can inspect a completed episode.

This Study Explains Actor, Judge, and Teacher Decision Models in Long-Horizon Agents

JEV turned step judgments into training gains

The teacher method is JEV. It supplied step-level signals to a Qwen2.5-1.5B policy trained with GRPO.

ALFWorld moved from 72.8% to 82.9%. WebShop moved from 56.8% to 71.4%.

The gains come from action-level guidance, not only a final success signal, making steps clearer even when actor choices were unreliable.

Decision models should not be valued only by raw action selection. A weak Actor may be a valuable Judge, turning judging ability into a reinforcement-learning teaching signal.

Where to look next

Project page: https://zjureal.com/decision-agents/; code: https://github.com/ZJU-REAL/decision-agents. The open question is whether this pattern holds beyond ALFWorld and WebShop, and whether step-level judging helps where final outcomes are sparse or hard to attribute to individual actions.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.