A Netflix scientist recommends resampling your LLM judge, saying five runs reduce the error bar by 13 percent.
AlpacaEval shows LLM judge noise. Five judge runs cut the 95% error bar by 13%, but 260 added questions can match it with fewer total calls.