Accelerating agentic workflows using WebSockets in the Responses API

If you build or operate AI agents, you know the pain: every turn of an agent conversation triggers fresh API calls, and those round trips add up fast. This article walks through how the Codex agent loop was reworked to reduce that overhead, focusing on two key changes—WebSockets for persistent connections and connection-scoped caching. By the end, you'll understand the core ideas well enough to consider applying similar patterns to your own agent systems.

What the Agent Loop Actually Does

An agent loop is the heartbeat of any autonomous coding assistant. The model receives a prompt, generates a response, calls a tool (like reading a file or running a command), gets the result back, and repeats until the task is complete. Each iteration means sending context to the model and waiting for output.

The problem is that naive implementations treat each turn as an isolated request. Context gets re-sent from scratch, connections are torn down and rebuilt, and the infrastructure re-does work it has already done. For short tasks this barely matters. But agents that run for minutes—refactoring code, debugging across files—pay this tax over and over.

The Shift to WebSockets

Traditionally, agent-to-model communication used standard HTTP requests. Every turn opened a new connection, negotiated TLS, sent headers, and waited for a response. That handshake overhead is small per request, but multiplied across dozens or hundreds of agent turns, it becomes measurable.

Switching to WebSockets changes the picture. A WebSocket is a persistent, bidirectional channel: once established, it stays open, and messages flow both ways without repeated handshakes. For an agent loop, this means:

• No repeated connection setup. The connection cost is paid once, not per turn.
• Lower per-message latency. Messages travel over an already-warm channel.
• Streaming-friendly design. Model output can stream back as tokens arrive, letting the agent begin processing tool calls sooner.

The practical effect is that the time between "model finished thinking" and "agent received the answer" shrinks, which compounds across every turn of a long-running task.

Connection-Scoped Caching

The second change tackles a subtler problem: unnecessary recomputation. Every time an agent turn is processed, parts of the request are identical to the previous turn—the system prompt, conversation history, tool definitions. Under a naive design, infrastructure may re-validate, re-parse, or re-process these unchanged pieces each time.

Connection-scoped caching ties cached state to a specific connection. When the same connection handles the next turn, the system recognizes it can reuse what it already computed for that session: parsed request structures, session metadata, and context that has not changed. When the connection closes, the cache goes with it, which avoids the complexity of managing stale global caches.

The benefit is twofold. First, it reduces CPU work on the serving side, since repeated parsing and validation are skipped. Second, it reduces API overhead overall—fewer redundant operations between the agent and the model backend means each turn costs less time and fewer resources.

Why These Two Changes Work Together

Individually, WebSockets and caching each help. Together, they reinforce each other. A persistent connection gives the system a stable "session" to hang its cache on. Without WebSockets, connection-scoped caching is harder to define—requests might arrive from different sources, making reuse unreliable. With a stable channel, the cache has a natural lifetime and scope.

The result is an agent loop that feels faster to users and cheaper to run. Latency per turn drops because messages skip handshake costs and reuse prepared context. Total resource consumption drops because the system stops redoing identical work.

Takeaways for Your Own Agent Systems

You don't need to copy this architecture exactly to benefit from it:

1. Keep connections warm. If your agent makes many sequential model calls, avoid reopening connections each time. Persistent protocols like WebSockets (or HTTP/2 connection reuse) can help.
2. Scope caches to sessions. Cache whatever is deterministic per session—parsed prompts, tool schemas, metadata—and tie it to a session identifier so it dies when the session ends.
3. Measure per-turn overhead. Log the time spent on everything that isn't model inference. That overhead is often larger than teams expect, and it's usually the easiest place to win back performance.

The broader lesson: agent performance isn't just about model speed. The plumbing around the model—connections, caching, request handling—shapes real-world latency just as much, and it's often easier to improve.