Learning to Act with World Models Involves Valuing Future Outcomes
WM-PPO trains a world model to value imagined successor states, improving sparse-reward credit assignment and success on ALFWorld without overhead.
Key Takeaways
- WM-PPO adds a learned world model to PPO to improve long-horizon reinforcement learning under sparse terminal rewards.
- It extracts training signal from unexecuted sampled actions by predicting successor states and using them for both policy advantages and critic targets.
- The imagined outcomes are weighted by a reliability gate tied to real environment transitions, so the model is not trusted blindly.
- At inference time, only the policy is deployed; the world model is used during training and does not add runtime dependency.
- In tested environments, WM-PPO improves success rates over PPO, including ALFWorld, ScienceWorld, and text Sokoban, and encourages more ordered prerequisite behavior.
Meta description: WM-PPO uses a world model and PPO critic to improve sparse-reward success from 63.5% to 87.5% on ALFWorld and 39.3% to 52.9% on ScienceWorld.
Tags: reinforcement learning, world model, PPO, sparse rewards, agent training
Featured image: An abstract, friendly illustration of branching paths around a delayed goal, with no real people, logos, or text.
The Core Change
WM-PPO is a reinforcement learning method that adds one learned component to PPO: a world model trained on real state transitions. The method is described in a paper listed at arXiv under 2611.12200, with code and checkpoints released under the name nuskonghou on GitHub and Hugging Face. Its purpose is to address a common failure mode in long-horizon tasks, where a sparse terminal reward says little about individual steps.
A reward that arrives only at the end may tell an agent whether an episode succeeded, but not which earlier actions mattered. The paper's premise is that each state transition can change the chances of success long before the reward arrives. WM-PPO uses that gap. The world model predicts what each sampled action would change. PPO's own critic then values the predicted successor by the current policy's probability of success.
Why Sparse Rewards Are Hard
In many text-based agent tasks, the environment gives no useful score after each action. The reward may appear only when the agent places an item, gives an answer, or finishes an episode. That leaves a credit assignment problem: the agent must infer which early moves changed the probability of a later success.
WM-PPO targets that problem by using a model of transitions, not just a record of final outcomes. The world model is trained on real transitions, so it learns how the environment changes after an action. When the policy samples an action, the model predicts the next state that would result. That predicted state is the successor.
Learning From Unexecuted Actions
The key detail is that the method does not only learn from actions that were executed. For every action the policy samples, the value of its predicted successor enters training twice.
In the policy channel, comparisons among the sampled candidates give the policy auxiliary advantages. In plain terms, the agent gets extra training signals about which actions look better than others, even if only one of those actions is actually taken.
In the value channel, the mean of those predicted values gives the critic a one-step target. This gives the critic a more immediate signal than waiting for the final reward. The method therefore turns imagined next states into training information, rather than treating unexecuted actions as wasted samples.
A Gate Tied to Real Outcomes
The imagined values are not trusted blindly. A shared gate weights both channels by how reliably comparisons with similar value margins predicted real branch outcomes earlier in training. If a particular style of comparison has historically pointed to the correct branch, its signal gets more weight. If it has not, the method can lean less on it.
This makes the update grounded in real outcomes. The world model proposes a successor, but the training weight is still tied to evidence from earlier transitions in the real environment. The gate does not choose a single channel; the same reliability signal weights both the policy channel and the value channel.
A Minimal Addition at Inference
The world model is the only learned component added. The method uses no teacher model and no privileged information. That matters for deployment: only the policy runs at inference. The extra machinery is used during training, not as a runtime dependency.
Because the world model is only used during training, the added complexity does not have to be shipped with the policy. For teams working with agents that must act in the same environment they learned from, this keeps the final system closer to ordinary PPO deployment.
Results Across Tested Tasks
With a comparable number of real environment steps, success rises from PPO's 63.5% to 87.5% on ALFWorld. On ScienceWorld, the reported success rises from 39.3% to 52.9%. Among the PPO refinements tested in the paper, WM-PPO matches the strongest on ALFWorld. On ScienceWorld, it exceeds the best of those refinements by 6.5 points. It also surpasses the teacher-based methods tested.
On text Sokoban, the method reaches 87.0% against 77.3% for ProAct's MC-PPO. The gains are reported under the paper's tested conditions, so the practical value depends on the environment, the reward structure, and the training budget.
What The Behavior Change Suggests
The behavior reported is not just a higher score. The trained policy more often completes prerequisite steps, such as heating, cooling or cleaning an object, before placing it or committing to an answer.
That is a meaningful sign in sparse-reward tasks. A higher terminal success rate could come from lucky rollouts, but a shift toward prerequisite steps suggests the policy is learning a more ordered process. It acts less like a system waiting for a final reward and more like one that understands which earlier moves make later moves possible.
What This Means For Readers
For readers working with agents that receive little feedback until the end, the contribution is simple: WM-PPO tries to extract more learning signal from each sampled action without changing PPO's basic deployment. It does not require a teacher model, and it does not require running a model at inference.
The results are limited to the tested environments and methods, so the practical value depends on the task. But the mechanism points to a useful direction: if real transitions are expensive and terminal rewards are rare, a learned world model plus a gated critic signal may help an agent learn more from the actions it only imagines taking.
Comments (0)