Design the task set

Representative, tiered, and partially held out.

First time here? Paste the Context Pack first so the AI understands your project - open it from the header above.

Prompt Capsule

Build the eval task set from real usage. Mine transcripts/issues for 15-25 tasks across the classes that matter: multi-step builds, bug fixes with verification, research/navigation questions, permission-sensitive operations, summarizations, and refusal-appropriate requests. Tier by difficulty (smoke: must always pass; standard: the working band; stretch: currently unreliable). For each task write: the exact starting state (repo fixture, files), the request, and success criteria (machine-checkable where possible: tests to pass, artifacts to exist, strings to appear). Hold out 20% of tasks - never used when tuning prompts, only for validation. Package fixtures reproducibly.

Paste into EmergentFull Build: complete implementation prompt with explicit requirements

Quick is short. Full Build is recommended for most steps. Strict forces real logic when the AI keeps faking output.

Actual change check

Expected after this step

A tiered, fixtured task set with held-out validation tasks.

Should NOT happen

An existing feature broke
A button only logs to console
Data disappears after refresh
Errors fail silently with no visible state

This is what should exist before you continue. If reality does not match, do not move on.

Your notes for this step

Track what changed, failed, or needs follow-up. Notes export with the flow.

Pass the Verify Gate to complete this step

Verify before continuing

Do not move on until every check is true. The complete button stays locked until then.

Tasks mined from real usageSuccess criteria machine-checkable where possibleHoldout set marked and respected

Do not continue if…

! Evals that test toy tasks nothing like real usage
! Single-run scores treated as truth despite run-to-run variance
! Grading by the same model that did the work, uncalibrated
! A suite that exists but never gates anything

If the AI messes this up

Use this when:

- it only made visual forms or placeholder UI
- buttons do nothing or only log to console
- data is not persisted - state disappears on refresh
- something that worked before is now broken

Use this when the AI fakes progress or breaks the feature. It forces a real fix.

Tasks are toy-sized. Take three real sessions from history and convert them into fixtures verbatim - real mess (ambiguity, partial context) is exactly what the suite must contain.