LangChain’s Jev benchmark repeats fixed agent judgments 100 times

LangChain replayed five fixed weather-agent outputs through Jev, GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6, repeating each case 100 times and comparing pass/fail decisions with a human reviewer. Jev matched all 500 binary labels and showed 92–913× lower mean per-case variance on continuous quality scores, while averaging 0.44 seconds and $0.00035 per call. Why it matters: Judge selection is part of the evaluation harness, not an interchangeable model call. Measuring agreement and repeatability separately can reveal whether a cheap, stable judge is also aligned with the decisions a team actually wants to automate.

Try this: Replay one bounded rubric across five fixed production traces 100 times with Jev and your current judge. Save the human reference labels, agreement rate, score variance, latency, cost, and every disagreement in one eval report before routing more traffic to either judge.