Agent Lightning version 1.0 is a lightweight 3,500-line agentic reinforcement learning framework that trains agents using real harnesses.

Meta description: Microsoft Research Asia open-sources Agent Lightning v1.0, a 3,500-line Harnessed Agentic RL framework that trains real agent harnesses with Kubernetes jobs.

Tags: Agent Lightning, Harnessed Agentic RL, Kubernetes, Qwen3.5-9B, reinforcement learning

Featured image: Abstract friendly illustration of a lightweight control plane connecting an agent harness, Kubernetes jobs, and a training loop; no real people, no logos, no text.

For developers and research teams who train large language model agents, Agent Lightning v1.0 offers a simpler way to use reinforcement learning with the same agent harness that will be deployed. Microsoft Research Asia has introduced a training paradigm called Harnessed Agentic RL and open-sourced a fully rebuilt Agent Lightning v1.0. Compared with the original Agent Lightning, v1.0 focuses more on staying lightweight, integrating with real harnesses, and building a complete, reproducible agent RL training pipeline. Instead of making developers reimplement the agent inside the training framework, the project lets the deployed harness take part directly in learning. The result is a Kubernetes-friendly control plane that can train real agents without changing their existing code.

Reinforcement learning, or RL, is an approach where AI systems learn through trial and error, guided by rewards and penalties for their actions. It can improve agent behavior, but agent RL has been harder to apply than single-model RL. AI agents have evolved from single models into complex full-stack systems built from models, tools, and execution environments. Their capabilities increasingly depend on the agent harness, the software layer that coordinates those pieces outside the model. If training uses a simplified copy of that harness while deployment uses a different one, the learned behavior may not match the real system.

Why deployed and trained agents usually diverge

Traditional agentic RL assumes the training framework owns the interaction loop with the environment. In a ReAct-style loop, the model generates an action, the environment returns an observation, the observation is appended to the context, and the model generates the next action. The whole rollout maps onto one continuous token trajectory. Early RL systems such as verl, AReaL, and slime were built around this model, so training an agent required rebuilding its loop inside the RL framework.

That assumption became strained as real harnesses grew more complex. Coding agents such as mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex each bring their own context management, tool protocols, execution logic, and dependencies. General-purpose agent systems face similar issues. Rebuilding one for training is expensive, and the rebuilt agent may no longer behave the same way as the deployed agent.

Agent Lightning takes a different route. It places a large language model (LLM) proxy between the agent and the model. The agent continues to run as before: the endpoint that previously called the model API is pointed at Agent Lightning, and the training framework can observe and record its model calls. In v1.0, Microsoft Research Asia formalizes this approach as Harnessed Agentic RL. The core rule is simple: whichever agent harness is used in deployment is the harness that takes part directly in reinforcement learning during training. Existing harness code remains unchanged.

A lightweight control plane for hard training problems

The rebuilt framework is designed to be lightweight by default. Agent Lightning v1.0 delivers a complete agent RL control plane in about 3,500 lines of code. The codebase is small enough to understand, modify, and extend, while still implementing the key parts of Harnessed Agentic RL. The system is presented through Foundry Labs, which bridges Microsoft breakthroughs and builders, scientists, and curious minds who can put them to work.

The architecture has three core components. The API Gateway stores rollouts, models, and events, and serves as an OpenAI-compatible LLM proxy, meaning it can accept requests in the common OpenAI API format. It links every model call from the harness to its rollout and records the prompts, responses, and log probabilities that training needs. The Rollout Controller starts and manages agent execution, either as local processes or as standard Kubernetes jobs, keeping agent execution separate from the trainer. The Customized Trainer is built on verl. It creates rollouts, waits for them to finish, collects samples, and assembles the final training samples through a sample adapter. For an existing agent harness, pointing the model endpoint at the Agent Lightning proxy is usually enough to connect quickly to RL training.

Training with a real harness is not as simple as swapping an endpoint, because the environment interaction loop is handled by the harness rather than the training framework. The training system can only observe a series of LLM request and response pairs, so a single rollout may be split into a variable number of training samples. That creates four challenges.

First is retokenization and sample merging. Harnesses keep context as text, but RL training needs the token IDs sampled during the rollout. Passing text through the chat template and tokenizer again can shift token boundaries, so adjacent calls cannot always be merged into one sample.

Second is advantage calculation. Retokenization, subagents, and context summarization can split one rollout into several samples. Computing baselines and advantages directly at the sample level causes rollouts that produce more samples to be counted repeatedly, which alters the original statistical relationships at the rollout level.

Third is loss normalization. Averaging loss by sample count gives more weight to rollouts that produce more samples. Since sample count is often just a product of harness behavior, loss normalization must avoid being distorted by it.

Fourth is training backend scheduling. Sample count and length are known only after the harness finishes, while GPU counts and data/tensor parallel configurations are usually fixed. The backend has to map a variable workload into fixed resources.

Kubernetes jobs and faster rollouts

Collecting enough rollouts means running many agents at once, which consumes substantial CPU, memory, and compute resources. Other Harnessed Agentic RL frameworks often host those agents on commercial sandbox services such as Modal Sandbox or E2B, where cost climbs quickly with scale. Agent Lightning v1.0 avoids that dependency by running agents directly as standard Kubernetes jobs, with no dependency on paid commercial sandbox services. Kubernetes, a system for running containerized jobs, lets the framework reuse existing self-managed clusters, cloud Kubernetes, or local infrastructure. That makes rollout collection cheaper at scale and keeps the pipeline open source and reproducible.

The framework also addresses GPU idle time. Rollout times vary widely across agents. Synchronous RL waits for the slowest agent in a batch and leaves GPUs idle. Fully asynchronous RL raises utilization but needs separate GPU pools for rollout and training. Agent Lightning v1.0 introduces Collocated Async RL, which lets rollout and model updates share the same set of GPUs.

Once the system has collected enough rollouts, the update begins. The API Gateway pauses accepting new requests and waits for requests already in progress to finish. Rollout resumes after the update completes. The entire state transition is transparent to the external agent harness. In experiments, this approach achieved about a 2x end-to-end speedup over synchronous RL while using fewer GPUs than conventional asynchronous RL.

What the Qwen3.5-9B coding result shows

The reported result is an end-to-end coding agent pipeline built on SWE-smith, mini-SWE-agent, and Qwen3.5-9B. The pipeline covers data cleaning, environment construction, reward-hacking safeguards, and RL training. The training set holds about 6,000 training samples based on an open-source dataset and needs no large-scale compute. RL training alone raised Qwen3.5-9B from 41.8% to 56.4% Pass@1, a first-attempt success metric, on the SWE-bench Verified coding benchmark, an absolute gain of 14.6 percentage points.

The experiments also validate the design choices around rollout-level statistics. Compared with sample-level handling, using rollout-level advantage calculation together with rollout-level normalization achieves a higher validation reward and keeps policy entropy more stable during training. Pass rate and policy entropy were tracked for Qwen3.5-9B on the SWE-smith validation set.

For readers, the practical takeaway is that Agent Lightning v1.0 tries to remove one of the biggest barriers in agent RL: the gap between the agent being trained and the agent being deployed. If its endpoint-based proxy, Kubernetes support, and compact control plane hold up in real projects, teams may be able to reuse existing harnesses, run rollouts on their own infrastructure, and explore RL without paying for external sandbox services. The next thing to watch is whether other agent systems can connect through the same pattern while preserving their unique context management, tool protocols, and execution logic.