A Netflix scientist recommends resampling your LLM judge, saying five runs reduce the error bar by 13 percent.
AlpacaEval shows LLM judge noise. Five judge runs cut the 95% error bar by 13%, but 260 added questions can match it with fewer total calls.
Key Takeaways
- A single sampled LLM-judge verdict adds measurable noise: on AlpacaEval, grading GPT-4o answers five times instead of once reduces the 95% error bar from ±3.45 to ±3.00, a 13% improvement, but requires 4,025 judge calls instead of 805.
- Judge resampling only reduces judge noise, not task-sampling noise; the same ±3.00 error bar can be achieved with one verdict per answer on 1,065 questions, meaning 260 extra test cases instead of 3,220 extra judge calls.
- The error bar has a floor because task noise remains: infinite re-grading for GPT-4o stops at ±2.88, while halving the one-run error bar would require roughly four times as many tasks.
- If the judge exposes token log-probabilities, use the win probability as the score rather than a sampled yes/no verdict; this captures the judge-noise reduction at the cost of one call, whereas temperature 0 removes noise but can shift the score.
- Judge resampling is most useful when the judge contributes a large share of total variance, so measure that share with repeated grading before deciding whether to add judge runs or more evaluation tasks.
If your LLM evaluation pipeline grades each answer once with a sampled verdict, you are paying for noise you could measure. A script built on public AlpacaEval data shows that for GPT-4o, grading every answer five times instead of once shrinks the 95% error bar from ±3.45 points to ±3.00 — a 13% reduction that costs 4,025 judge calls instead of 805. The same ±3.00 is reachable with one verdict per answer on 1,065 questions, which means 260 extra test cases rather than 3,220 extra judge calls.
The analysis comes from Cameron Wolfe, a staff research scientist at Netflix who also writes the Deep (Learning) Focus newsletter. He gave a talk titled "How Netflix Evaluates AI When There's No Right Answer," posted by the AI Engineer channel on November 5. The talk covers tasks like creative writing where pass or fail is a judgment call, and it lays out a three-part framework: human evaluation as the North Star, LLM judges and rubrics to scale offline, and A/B tests plus production monitoring to confirm with real users. In one Netflix project, an LLM judge's scores were causally linked to viewers abandoning shows.
Wolfe's central concern is noise. Both the model and its LLM judge are random. His suggested fixes are power analysis to size the eval, error bars on every number, and resampling — multiple rollouts from either the model or the judge. The question this article answers is what judge resampling actually buys, using a judge whose odds are public.
Where the judge's noise becomes visible
AlpacaEval 2.0 publishes what you need. Its GPT-4 Turbo judge compares each model's answer with GPT-4 Turbo's own answer across 805 questions, and the repository stores the judge's win probability for every question, read from its token odds.
That probability is what makes the test possible. The judge's config asks for one token at temperature 1, so if it reports 0.7 on a question, a single sampled verdict is effectively a coin that lands "win" 70% of the time. The spread of that coin, 0.7 × 0.3, is the judge's own noise on that question. The spread of probabilities across questions is the noise that comes from which questions you happened to pick.
Two sources of noise, two different cures
An eval score is an average over questions, so its error bar has two sources. The first is task noise: which questions ended up in your set. The cure is more questions. The second is judge noise: a coin flip per verdict. The cure is more judge runs.
Grading each answer K times divides only the judge part by K. The task part does not move. This distinction is the whole point of the exercise, because the two cures have very different price tags.
Running the numbers on GPT-4o
The script, judgenoise.py, is standard-library Python 3.8+. It reads either an AlpacaEval annotations.json or a CSV of your own grades, splits the variance into the two parts, and prints the error bar for each number of judge runs. For a CSV it estimates the judge portion from the spread between repeat runs of a task. Bad input stops it with a message and exit code 1; no arguments prints the usage line.
On GPT-4o's AlpacaEval results — 805 tasks, a 51.3% score — the split comes out to a variance of 0.1740 from tasks and 0.0761 from the judge, meaning the judge accounts for 30% of the noise. GPT-4o wins 51.3% of comparisons against GPT-4 Turbo's answers.
The resulting table is the core finding:
| Judge runs | 95% error bar | One run matches it with |
|---|---|---|
| 1 | ±3.45 pts | 805 tasks (+0%) |
| 2 | ±3.18 pts | 950 tasks (+18%) |
| 3 | ±3.08 pts | 1,010 tasks (+25%) |
| 5 | ±3.00 pts | 1,065 tasks (+32%) |
| 10 | ±2.94 pts | 1,109 tasks (+38%) |
| unlimited | ±2.88 pts | the floor: task noise stays |
Five verdicts per answer means 4,025 judge calls for a 13% smaller bar. The right-hand column is the comparison that matters: one verdict per answer on 1,065 questions gives the same ±3.00. That is 260 more test cases to write, run and grade, against 3,220 extra judge calls.
The unlimited row is the floor. Even infinite re-grading stops at ±2.88, because task noise remains. Halving the one-run bar takes four times the questions — 3,220 in this case — and no number of judge runs gets there.
The free version of unlimited
The floor row matches a number AlpacaEval already publishes. Its leaderboard lists GPT-4o's win rate as 51.33 with a standard error of 1.47, and 1.47 × 1.96 is the ±2.88 in the floor row.
The reason is that AlpacaEval's "weighted" judge never uses a sampled verdict. It reads the judge's probability that the model's answer is better and uses that probability as the score. That captures the entire judge-noise reduction for the price of one call. A temperature-0 judge also removes the noise but changes the score: it rounds each question to its likelier verdict, giving 53.7% for GPT-4o rather than 51.3%.
If your judge API returns token log-probabilities, that is the one setting to change: ask for logprobs on the verdict token and average the probability, not a sampled "yes" or "no."
When re-grading does pay
The judge share was 30% for GPT-4o and 28% for Claude 3.5 Sonnet. A different judge, prompt or task set can be noisier, so the split should be measured rather than assumed.
To check your own judge, grade each answer at least twice and save one line per run as a CSV next to the script, with columns for task and grade between 0 and 1. A small four-task example — refund-email, plot-summary, sql-fix and tone-rewrite, each graded three times — produces a score of 58.3% with a variance of 0.1204 from tasks and 0.1667 from the judge, a 58% judge share. In that toy case, five judge runs match doubling the task count. The error bars are huge because four tasks is far too few, but the split is the point: the higher the judge's share of the noise, the more re-running is worth.
What the analysis does not cover
The judge noise measured here is the GPT-4 Turbo judge's token odds on one verdict, with each question's answers shown in one order. A judge that reasons first, or a re-run that reshuffles the order, adds noise this data cannot show. Seven GPT-4o questions had identical answers and were never judged; counting them as coin flips barely moves the numbers. The talk also means resampling the model's answers; with one answer per question, that noise sits inside the task part. The leaderboard's headline score is length-controlled at 57.5 for GPT-4o; this analysis uses the raw win rate.
What this means for your eval pipeline
With a judge that samples one verdict, re-grading every answer five times would cut GPT-4o's error bar from ±3.45 to ±3.00 points, which 32% more questions also achieves. Using the judge's token probability gets the full judge-noise cut in a single call.
The practical takeaway is to measure your own split before paying for anything. If the judge dominates your noise, more runs help. If task noise dominates, more questions are the only lever that moves the floor. And if your judge exposes log-probabilities, switching to probability averaging may make the entire question moot.
How many times does your eval pipeline grade each answer, and have you checked whether it needs to?
Comments (0)