Cold-start evals use floor-to-ceiling cases as the product spec

Aakash Gupta presents Daniel McKinnon’s cold-start eval method for an AI feature with no production traces: define one clearly solvable floor case, one deliberately difficult ceiling case, then generate and review cases between them. Why it matters: Examples can make a broad AI feature testable before launch. They help a team find tasks the model cannot yet handle, narrow the initial product boundary, and give product and engineering a shared artifact for judging changes.

Try this: Run a cold-start eval for one planned AI feature. Define one floor case and one ceiling case with defensible answers, then save 20 intermediate eval cases with the input, expected answer, and pass/fail rule for every row.