The Inversion Curse: Why Language Models That Understand “A Is B” Fail to Infer “B Is A”
LLMs may know A is B but fail B is A because next-token training is one-way. Test, prompt, and index both directions to improve retrieval and QA.
Summary of Key Points
- The Reversal Curse is a failure mode where a language model can answer “A is B” but not “B is A” because learned associations are directional.
- This often comes from next-token training, which rewards predicting one side of a fact rather than storing facts as symmetric, reversible knowledge.
- Users and builders should test both directions of key facts to see whether failures are caused by missing knowledge or by phrasing and retrieval direction.
- Prompt design, training examples, and retrieval indexes can help by explicitly including inverse forms and restating relations in the requested direction.
Meta description: A Towards Data Science post explains the Reversal Curse: why language models trained to predict tokens may know A is B but fail to answer B is A.
Tags: Reversal Curse, language models, AI training, prompt design
Featured image: Abstract friendly illustration of two mirrored arrows forming a broken loop, no text, no logos, no people.
A post titled "The Reversal Curse: Why a Language Model That Knows “A Is B” Can’t Tell You “B Is A”" appeared first on Towards Data Science. It describes the Reversal Curse as a failure mode where a model can retrieve a fact along the direction it learned and miss it when asked the opposite way. It gives a toy version of why that happens. Readers building chatbots, assistants, or retrieval systems will get a simple mental model for why some question answering failures may not be caused by missing knowledge alone.
What the Reversal Curse tests
The core idea is directional. A system may answer from A to B but not from B to A. If training text or a document says "A is B", a model can learn to connect that context to the answer B. It may not learn the inverse link from B to A unless the reverse pattern also appears.
This matters because users phrase questions in many ways. A fact may exist in the model's context, but the requested direction may not be the direction the model was rewarded to predict. The post presents a toy version of the problem, not a claim that every model fails every reversed question.
Why next-token training makes directions uneven
Many large language models are trained as next-token predictors, also called autoregressive models. They learn to assign higher probability to likely continuations after a sequence of text. That objective is useful, but it is not a symmetric database. It does not automatically store every statement as an undirected fact with equal access from both sides.
Imagine a training pattern that repeatedly shows "A is B" after a context that mentions A. The model can succeed by learning that A predicts B. If the context mentions B and asks about A, the earlier pattern may not help, because the model was not trained to generate A after B. The association is learned through sequences, not through a table of reversible logical triples.
The asymmetry can become more likely when the inverse phrasing is rare, awkward, or absent. Natural text often states a relation once. It rarely restates every relation in both directions. A model may therefore answer one side of a fact because it saw that side often, while failing the other side because it did not.
What this means for builders and testers
For people using language models, the Reversal Curse suggests a practical test: ask the same relation from both sides. If a system can say "A is B" but cannot say "B is A", the issue may be training or retrieval direction, not simply knowledge size. That can help when debugging assistants, evaluating prompt templates, or checking whether a retrieval index contains the right inverse phrasing.
It also gives guidance for prompt design. A prompt may need to make the requested direction explicit. Asking "What does A map to?" and "What does B map to?" can behave differently even when the underlying fact is the same. Teams can add inverse examples to demonstrations, include both phrasings in search documents, or ask the model to restate a relation in the requested direction before answering.
For retrieval systems, the lesson is that matching is directional too. If documents only contain "A is B", a query about B may not retrieve the relevant passage unless embeddings, lexical expansion, or query rewriting connect the two sides. Adding inverse forms to indexed text can make retrieval more robust, but it can also increase data volume and maintenance cost.
What to watch next
The post's value lies in its simple explanation for a type of failure that can appear in next-token models: learned associations may point in one direction. As models and systems improve, the question is not only whether they contain facts, but whether those facts are reachable through the phrasings users actually produce.
Developers and evaluators should watch for three signs. First, failures that appear when a question is rephrased into the reverse direction. Second, cases where retrieval returns the forward fact but not the inverse relation. Third, prompt changes that fix a reversal without changing the model itself. Those signals suggest the issue is association direction, not missing information.
The broader takeaway is to treat model knowledge as a path through text, not as a perfect logical store. A fact can be available in one context and unreachable in another. For users, that means testing both sides of important facts. For builders, it means designing prompts, examples, and retrieval indexes that anticipate the directions people will ask about.
Comments (0)