Jailbreak Regression Suite

PROJECT CONTEXT:
A model update or prompt tweak can silently reopen a jailbreak I already fixed. Without a regression suite, safety boundaries erode invisibly. Red-teaming practice says: every jailbreak becomes a standing test that must keep failing.

GOAL:
Build a jailbreak regression suite: a corpus of known jailbreaks with expected refusals, automated evaluation, and CI gating on every model/prompt change.

REQUIREMENTS:
- A corpus of jailbreak attempts with the expected safe behavior for each
- Automated evaluation classifying agent responses as refused/complied
- Runs pinned to model version and prompt version for attribution
- CI gating: any regression (a previously-refused jailbreak now complies) blocks
- A contribution path: new jailbreaks are added the moment they are discovered

CONSTRAINTS:
- Expected behavior is defined per case - refusal, safe-completion, or redirect
- The suite must run on model upgrades, not only on prompt edits

DEFINITION OF DONE:
- The corpus covers the jailbreak categories relevant to my agent
- Evaluation reliably distinguishes genuine refusal from subtle compliance
- A model or prompt change that reopens a jailbreak fails CI
- A newly discovered jailbreak is added and immediately guards against regression

COMMON FAILURES TO AVOID:
- Fixing a jailbreak once with no test, so the next model update reopens it
- Naive keyword grading that counts a hedged compliance as a refusal
- Suite tied to one model, breaking silently on provider upgrades
- No process, so found jailbreaks live in a chat log, not the suite