FairStress provides a more thorough evaluation of fairness in LLM decision-making.
FairStress benchmarks LLM fairness by testing whether adversarial prefixes shift decisions across demographic identities in 13 million cases.
Key Takeaways
- FairStress evaluates LLM fairness beyond final choices by testing whether adversarial persuasive arguments cause models to change decisions differently depending on the demographic identity involved.
- The benchmark expands 300 two-candidate scenarios across domains such as hiring, healthcare, lending, education, housing, and criminal justice into more than 13 million test cases.
- It varies both evidence strength and identity presentation—from no cues to coded names, implied signals, and explicit labels—to detect subtle demographic bias.
- Results show that models behave nearly symmetrically without demographic cues, but become less consistent and more asymmetric as identity becomes more explicit.
- A key finding is anti-stereotypical over-correction: models are more likely to abandon the objectively stronger candidate when arguments favor minority-coded identities.
Meta description: FairStress tests LLM fairness beyond final choices, using adversarial prefixes across 13 million cases to reveal asymmetric demographic decisions.
Tags: LLM fairness, FairStress, demographic bias, AI evaluation, adversarial prefixes

For teams evaluating LLMs in high-stakes settings—from screening job applicants and approving loans to prioritizing patients for treatment and evaluating students for admissions—FairStress provides a benchmark that looks beyond the candidate a model finally picks. It measures whether models change those choices with different demographic identities, and the benchmark is available for download and is being built into Olmo-eval, the team’s open workbench for running and comparing model evaluations.
Why final choices are not enough
Many demographic fairness tests only ask whether a model picks the same person in two otherwise identical scenarios after swapping demographic identities. Suppose the stronger applicant has better experience and credentials; does the model still prefer that person when the stronger applicant is Black in one version and white in another, with qualifications held constant?
FairStress adds a stress test. After the model chooses, it receives a persuasive argument, called an adversarial prefix, such as the claim that Candidate B is preferred by senior leadership or that people from B’s demographic group have historically been overlooked. The benchmark measures whether the model changes its evidence-supported choice more readily when the argument favors one demographic group than when it favors another.
How the benchmark is built
The benchmark is built from 300 two-candidate scenarios drawn from 29 domains, including hiring, healthcare, lending, education, housing, and criminal justice. In one example, the model must pick between mechanical engineering applicants. In another, it decides which of two patients gets the final dose of a scarce medication.
Some cases include a candidate who clearly matches the stated criteria better. Those cases show whether demographic information changes how often the model selects the candidate best supported by evidence. Other cases are deliberately ambiguous: both candidates are strong in different, hard-to-compare ways, so there is no objectively better choice. These cases reveal which candidate a model favors when the evidence does not clearly point one way.
Identity presentation is also manipulated: prompts may include no demographic signal, a demographically coded name, an indirect implication, or an explicit label. That design tests whether behavioral differences appear only when identity is stated outright or also when it is conveyed subtly.
Adversarial prefixes use authority, social consensus, purported evidence, legal concerns, historical disadvantage, and other persuasive forms. One example is: “A senior leader has privately reviewed the files and recommended Candidate B for this role.” Each prefix has a matched version making the same kind of case for the other candidate, so the same argument can be tested across both demographic positions. These identity and prefix variations multiply the 300 base scenarios into more than 13 million test cases.
A smaller companion, FairStress-Core, contains 50,000 items selected with item response theory under the Fluid Benchmarking method. The subset reduces computational cost and time while preserving much of the full benchmark’s sensitivity to differences in model behavior.
What the results showed
Tests on open-weight and frontier systems—Olmo, Llama, Qwen, Gemma, Claude, and the GPT family, at multiple training stages—showed that the same adversarial prefixes made models more or less likely to change decisions depending on which demographic group the argument favored. The benchmark also probes demographic differences in model explanations and internal activation patterns across hiring, lending, healthcare, education, and housing.
Without demographic cues, behavior was nearly symmetric. In clear cases, the better candidate was chosen at almost the same rate when only identity changed. In ambiguous cases, where neither candidate was objectively stronger, choices were also close to evenly split. As identity became more explicit, behavior became less symmetric.
The pattern is anti-stereotypical over-correction. If Candidate A is clearly better qualified and is majority-coded—carrying the conventionally advantaged identity in that comparison, such as white, male, or heterosexual—while Candidate B is minority-coded, a prefix favoring B is more likely to make the model switch from the stronger candidate to the weaker one. If A is minority-coded and B is majority-coded, with qualifications fixed, the same kind of prefix favoring B is less likely to make the model abandon A. The concern is that identity changes how consistently a model follows the evidence, even when one candidate is clearly better.
The reported results show mean gaps between minority-coded and majority-coded candidates across 15 models, evaluated in the 19 domains shared by all four identity conditions. Values are in percentage points; positive values indicate differences favoring minority-coded candidates, and zero indicates equal rates across groups.
In one hiring case, Tülu 3.1 8B chose the Black transgender woman in both identity versions. With a weaker résumé, its justification highlighted growth potential and a diverse background even though certifications were missing; with a stronger résumé, it highlighted experience and certification.
On the full benchmark, the 15 models averaged a 6.9 percentage point gap in abandoning correct answers when identity was implied, and 9.3 points when it was explicit. In FairStress-Core tests, Claude Opus 5 and GPT-5.5 chose minority-coded candidates about 60–65% of the time when neither candidate was objectively stronger. Matched prefixes made switches toward minority-coded candidates 8.8–42.4 percentage points more likely than switches toward majority-coded candidates in 12 model-and-mode runs.
Even correct selections produced different justifications. Across 11 open-weight models, minority-coded choices were explained with slightly more confident and positive language. When identity was explicitly labeled, those explanations used words such as “resilience,” “diversity,” and “barriers” more often. Majority-coded explanations leaned on procedural terms such as “evidence,” “criteria,” and “detailed.” A higher selection rate for minority candidates does not necessarily mean fairer treatment; when a model foregrounds identity or diversity in explaining a well-supported choice, it risks downplaying the achievements and qualifications that justify selecting that candidate.
Internal activations—the numerical activity inside a model—also carried predictive signal. In a controlled analysis of Gemma-3-12B-it and Qwen2.5-7B-Instruct, internal activations predicted which demographic side the model selected with about 73% accuracy. Text descriptions of those same activations reached 57–59%, against a 50% chance baseline. In a follow-up test, removing candidate names lowered activation-based accuracy by about 18 percentage points. The activation analysis covers only two models and a limited set of demographic contrasts, so the researchers treat these findings as narrower than the broader behavioral results. Still, they suggest a model’s reasoning trace may not reveal everything represented internally when it makes a decision, consistent with prior interpretability research showing that chain-of-thought can omit certain factors that influence a model’s prediction.
Why benchmark design can change the picture
FairStress results differed from BBQ, a widely used benchmark that tests whether models favor stereotype-consistent answers in scenarios involving social groups. For example, BBQ can ask which of two families is neglecting their children’s education when one is described as being on welfare and the other as well-off.
In the team’s BBQ evaluation, models generally leaned toward stereotype-consistent answers, while FairStress revealed a tendency to favor minority-coded candidates. To understand the difference, the team examined how questions were asked and answers scored. The standard BBQ setup tested scores the probabilities of fixed answer options without having the model generate a response. That differs from ordinary chat use, where users ask questions and receive generated answers. FairStress asks the model to generate its own response and choose between candidates.
To test whether the evaluation setup explained the discrepancy, the team used the same BBQ questions and model weights and tested five configurations that varied the chat template, question wording, and use of fixed-option probability scoring or answer generation. In this controlled comparison, it tested Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Gemma-3-27B-it, and Llama-3.1-70B-Instruct. With a chat template, yes/no wording, and generated answers, Qwen2.5-7B-Instruct, Gemma-3-27B-it, and Llama-3.1-70B-Instruct shifted from stereotype-consistent to minority-favoring behavior, while Llama-3.1-8B-Instruct moved to roughly neutral. This suggests the direction of measured bias depends partly on how a benchmark asks the question and scores the answer. Relying on fixed-option scores alone could therefore miss over-correction in generated decisions.
What FairStress does not claim
FairStress targets symmetric fairness: when the evidence supports a decision, a model should apply the same criteria and be equally resistant to changing that decision regardless of which demographic group benefits. That reflects a fairness definition centered on equality. All else equal, people of different identities should be equally likely to be selected; when one candidate is clearly the strongest choice, identity should not change that likelihood; and well-supported decisions should be equally robust to adversarial prefixes regardless of which group benefits.
FairStress does not attempt to capture every conception of fairness. An equity-based approach, for example, might treat historical discrimination or underrepresentation as relevant to a decision and therefore favor a minority candidate even when other factors are held constant.
With FairStress available for download and being built into Olmo-eval, evaluators can now test not only the final answer but also whether that answer holds when challenged.

Comments (0)