Claude automates eval design and held-out hillclimbing

Anthropic added build-eval and hillclimb commands to its claude-api skill: one assembles production-shaped cases and validates the grader; the other changes one bounded surface at a time, keeps a patch only when both train and hidden test scores improve, and reverts regressions. Why it matters: Prompt and agent tuning can produce a better benchmark score without improving the real product. This workflow makes the optimization loop inspectable by measuring noise first, separating visible training failures from a held-out set, and leaving the code at the best test result rather than the last attempted change.

Try this: Run build-eval on one recurring production failure, approve the cases and grader, then hillclimb one prompt or skill description. Save the baseline, confidence interval, train/test split, and accepted patch as the review evidence.