Decision models for evals: tune the threshold before you compare

Hamel Husain and Antaripa Saha tested six decision models (Jev, Perplexity Decider, Liquid d1 and d1-ultra, Cloudflare Clef Flash, OpenAI Decisions) as cheap pass/fail judges on 98 human-labeled traces from an apartment-leasing agent. The default 0.50 threshold often leaves performance on the table: on the full sample, tuning lifted Clef Flash's F1 from 0.35 to 0.80. Only Perplexity, Liquid d1-ultra and OpenAI could reach at least 80% precision and recall. After tuning, OpenAI Decisions scored the best F1 (0.87) at 3.72 cents per 100 runs, about 4.5x Perplexity's 0.82 cents for an F1 of 0.79. Why it matters: Cheap judges make it affordable to run evals on every trace, but only if you treat them as classifiers. Set minimum precision and recall from what a false alarm or a missed failure costs the business, tune the threshold on labeled data with cross-validation, and only then compare models on cost. Don't read raw scores as probabilities without checking calibration: one Clef Flash group averaged about an 8% predicted failure rate while 60% actually failed. One caveat is that this is one broad task on a small dataset, and the authors say results vary a lot across tasks.

Try this: Take one LLM-judge eval you already have human labels for, write down the minimum precision and recall it needs, and have a coding agent tune its threshold with five-fold cross-validation before you compare it with a cheaper model.