LLMs do not know what they know in temporal self-knowledge benchmarks for financial backtesting.
FinDendro benchmark tests if LLMs know their temporal knowledge; GPT-5 shows hindsight leakage, so backtests need validated knowledge boundaries.
Key Takeaways
- FinDendro is a benchmark for testing whether LLMs used in financial backtesting can accurately know what they know across historical time boundaries.
- The benchmark contains 2,226 dated U.S. financial and macroeconomic facts from 1999 to 2026, covering macro indicators, stock prices, and Federal Reserve decisions.
- GPT-5 showed a major gap between stated knowledge and actual knowledge: it answered 76% of controlled facts correctly but claimed knowledge for only 38%.
- The study finds that hindsight leakage can make backtests appear more accurate than real forecasting, because models may recall known outcomes instead of predicting them.
- Common safeguards, such as relying on knowledge cutoffs or instructing models to answer “as of” a historical date, do not reliably prevent models from producing correct answers.
Meta description: FinDendro tests whether large language models know what they know when predicting past U.S. financial facts, revealing hindsight risk in AI backtesting.

Tags: FinDendro, LLM financial backtesting, GPT-5, temporal knowledge, financial benchmarks
Featured image: An abstract, friendly illustration of a branching timeline of dated financial facts, with no real people, logos, or text.
A Test for Knowing What a Model Knows
Researchers have introduced FinDendro, a benchmark for LLM financial backtesting built to answer a question that matters before any AI-driven finance experiment begins: when a large language model is asked to predict a past event, is it forecasting or simply recalling an outcome it already knows? The paper is titled “LLMs Don’t Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting.”
The concern is that a model’s stated knowledge cutoff, its own claim about whether it knows an answer, or an instruction to behave as if it is operating at a historical date may not be enough to prevent hindsight leakage. Hindsight leakage happens when a system uses information that should not have been available at the time being tested. That can make a backtest look more accurate than it would be in a real, forward-looking setting.
What FinDendro Measures
FinDendro is built from 2,226 dated U.S. financial and macroeconomic facts spanning 1999-2026. The facts cover three types of information that are common in financial evaluation work: macroeconomic indicators, month-end stock prices, and Federal Reserve decisions.
The benchmark does not only ask whether a model can produce the right number or outcome. It examines a second layer: whether the model can correctly report what it knows across time. In other words, it tests temporal self-knowledge, the model’s awareness of the boundary between what it should know, what it might know, and what it should not know because it occurs after the information date.
This matters because backtesting often requires choosing a period in which the model can be treated as if it did not already know the answer. If that boundary is wrong, the test can mix prediction with recall.
What GPT-5’s Results Show
The findings show a gap between claimed knowledge and measured knowledge. GPT-5 correctly answers 76% of the controlled facts, while reporting knowledge of only 38%. That means a majority of its correct answers were not accompanied by a claim of knowledge.
The benchmark also finds that, for the facts GPT-5 says it does not know, it still answers 61% correctly. That result is important because it suggests that simply asking a model whether it knows an outcome may not reveal whether it can access the answer. A model can deny knowing something while still producing a correct response.
Self-reported knowledge can stretch as far as seven quarters past the measured boundary. This is another sign that what a model says it knows and what it demonstrates it knows may not line up neatly.
Why Common Safeguards May Not Be Enough
The study tested safeguards that users often rely on. Trusting the vendor’s knowledge cutoff can give a starting point, but it does not by itself show what the model can answer correctly. Asking the model whether it knows the outcome can produce a clean-sounding disclaimer, but the results indicate that the answer may not match the model’s actual performance.
Telling the model to answer as of a historical date changes what it claims to know. The findings say historical-date instructions eliminate most prior-knowledge claims, but the accuracy level does not move. That combination is the key warning: the model may stop saying it knows something, but it may not stop producing correct answers. For a backtest, accuracy without a credible explanation of knowledge boundaries is not enough.
What This Means for Readers
For readers who use large language models to evaluate financial ideas, the practical lesson is to treat model statements as evidence, not proof. The benchmark supplies a behavioral procedure: it identifies temporal knowledge boundaries and helps choose backtest periods based on demonstrated factual knowledge. That means the more reliable approach is to test what the model can answer, rather than trusting only what the model says it knows.
FinDendro is also useful for readers who want to understand why AI financial forecasts can be misleading. A model that appears to predict the past well may be doing something closer to remembering the past. A proper evaluation should separate those two behaviors, especially when the goal is to judge how the model might perform in a real, future forecasting task.
The result does not say that LLMs cannot be used in financial work. It says that their value in backtesting depends on a clearer measurement of knowledge. For anyone relying on GPT-5 or similar models for finance experiments, FinDendro points to a more careful starting point: identify the model’s factual boundaries first, then choose the period and questions that match those boundaries.
Comments (0)