Evals for Your Harness

PROJECT CONTEXT:
Every harness change ships on vibes: the prompt feels better, the agent seems smarter. Agent-platform research treats evals as the discipline separating engineering from folklore - fixed task sets, honest grading, and change-gating.

GOAL:
Stand up a real eval system: representative task sets, automatic and rubric grading, statistical honesty, and eval-gated harness changes.

REQUIREMENTS:
- An eval task set representative of real usage, tiered by difficulty
- Automatic grading where possible (tests pass, output validates) and rubric grading elsewhere
- Multiple runs per task with variance reported - single runs lie
- A results store tracking scores per harness version over time
- Change-gating: harness PRs show their eval delta before merge

CONSTRAINTS:
- Eval tasks must not leak into training-by-iteration: hold out a subset from tuning
- Grading criteria are versioned artifacts like everything else

DEFINITION OF DONE:
- The suite runs end to end with one command and produces a scored report
- Scores include variance across runs, not single-sample numbers
- A deliberately degraded harness (weaker prompt) shows a clear score drop
- The last harness change merged with its eval delta attached

COMMON FAILURES TO AVOID:
- Evals that test toy tasks nothing like real usage
- Single-run scores treated as truth despite run-to-run variance
- Grading by the same model that did the work, uncalibrated
- A suite that exists but never gates anything