The investor's final exam is whether AI agents can make equity investments without seeing the future.
The versioned benchmark tests LLM agents with point-in-time financial data, finding strong evidence organization but weaker investment forecasting.
Key Takeaways
- The ILE benchmark tests whether LLM agents can make point-in-time financial forecasts, not merely summarize research.
- It enforces a time boundary using a large versioned corpus so agents cannot rely on information unavailable at the decision moment.
- The benchmark is built around buy-side analyst deliverables, using rubrics and financial dependency graphs to trace evidence, calculations, and forecasts.
- Results show current models are stronger at organizing evidence than at making forward-looking investment judgments.
- The timestamped evaluation setup may also serve as a reusable training environment for domain-specific reinforcement learning.
The ILE benchmark is for readers who want to know whether large language model (LLM) agents can do financial forecasting, not just summarize financial research. It asks a narrow but difficult question: can an agent, using only what was visible at decision time, independently complete the full chain from evidence to assumptions to forecast? That matters because most work in LLM4Finance, the research area applying LLMs to finance, has focused on the research stage rather than the investment stage, which is where the field's core objective lies.
Waiting for Outcomes Can Blur the Model's Skill
Traditional forecasting evaluation usually means waiting for outcomes to materialize, which can take months, sometimes years. During that time, market conditions, fundamentals, and the information environment can shift enough that scores can no longer be cleanly attributed to the model. A forecast that looks good later may have depended on information that was not visible when the decision was made, or on assumptions that changed as the environment changed.
The ILE approach tries to separate those effects by enforcing a time boundary. Specify a timestamp and you get exactly the versions visible at that moment. In practice, this is point-in-time evaluation: the agent can use only the evidence that existed, or could be seen, at the chosen decision time. That makes the result easier to interpret as model capability rather than as a function of later information. It also helps readers see whether a score reflects model capability or access to later information.
A Versioned Corpus Built Around Analyst Deliverables
To make that boundary enforceable, ILE uses a versioned corpus of 3.43 million document versions, 1.52TB, and 9 source classes. The corpus is designed so the cutoff applies across the entire toolchain, not only to raw documents. A timestamp selects the evidence available at that point in time for the whole workflow. That design keeps the task closer to the analyst workflow the benchmark is trying to test.
The benchmark is organized around buy-side analyst deliverables, with 86 assignments. Buy-side work is close to investment decisions, so the tasks test more than general summarization. Scoring uses rubrics anchored to sources and reproducible calculations through financial dependency graphs. A financial dependency graph can link source inputs, intermediate calculations, and final outputs, so a model's answer can be traced rather than judged only by how fluent it reads.
Evidence Organization Is Stronger Than Forecast Judgment
The results show a split between research-stage and investment-stage skill. Among current state-of-the-art systems, Astra scores 78.9 on Research & Evidence but only 52.5 on Forecast & Model. That gap suggests that today's models are very good at organizing evidence of what has already happened into coherent analysis. They can collect, compare, and explain sources in a way that may be useful for research workflows.
The harder test is turning that analysis into judgment about what has not happened yet. When the systems are ordered by realized outcomes, DeepSeek V4.1 Flash is ahead, 40.3 versus 37.7. This outcome-based ordering is different from the subcategory scores: it asks whether later results support the forecast, not whether the analysis reads well. The comparison suggests that strong evidence handling does not automatically become strong forecasting.
Why the Training Angle Matters
The longer-term significance may be on the training side. Once the time boundary is a mechanism, historical timestamps become a re-samplable environment for large-scale domain reinforcement learning. Instead of treating evaluation as a one-off scoring exercise, the same timestamped environment can support training: an agent can be asked to make decisions under a visible information set, then evaluated against the benchmark's rubrics. Evaluation and training can share one environment.
What It Means for Financial Research Tools
For readers, the practical message is simple. Current LLM agents in finance appear more capable at evidence organization than at forecast judgment. A model with a high Research & Evidence score may be useful for pulling sources together, but a lower Forecast & Model score warns against treating that fluency as proof of investment skill. The ILE benchmark adds value because it measures a point-in-time chain from evidence to assumptions to forecast, with rubrics tied to sources and calculations. The next question is not just whether agents can analyze finance, but whether they can make better judgments when the future has not yet arrived.
Meta description: ILE benchmark uses 3.43 million document versions and 86 buy-side assignments to test point-in-time financial forecasting and evidence handling by LLM agents.
Tags: financial forecasting, LLM agents, point-in-time data, ILE benchmark, buy-side research
Featured image: abstract, friendly illustration of timestamped financial documents and a forecasting arrow, no real people, no logos, no text.
Comments (0)