Incident Response Runbook for Agent Mishaps

PROJECT CONTEXT:
When my agent eventually does something harmful - deletes the wrong data, leaks something, runs a destructive command - I have no plan, and improvising during an incident is how small mishaps become disasters. Agent operations needs an incident runbook like any production system.

GOAL:
Write and drill an incident-response runbook for agent mishaps: detection, immediate containment (kill switch), rollback, investigation via audit/traces, and a prevention feedback loop.

REQUIREMENTS:
- Detection signals and reporting paths for agent mishaps
- A kill switch: stop the agent and revoke its authority immediately
- Containment and rollback procedures using checkpoints and audit trails
- An investigation procedure using traces, audit records, and session replay
- A blameless postmortem template feeding fixes back into the harness

CONSTRAINTS:
- Containment speed beats diagnosis - stop the bleeding first, investigate second
- Every incident must produce a concrete prevention (rule, gate, or limit), not just a writeup

DEFINITION OF DONE:
- The kill switch halts agent action and revokes tool authority in seconds
- Rollback restores pre-incident state using existing checkpoints
- An investigation reconstructs the incident from audit trail and traces
- A drill runs the full runbook end to end and produces a prevention item

COMMON FAILURES TO AVOID:
- No kill switch, so a misbehaving agent keeps acting while you scramble
- Improvised response that makes things worse (deleting evidence, partial rollback)
- Investigations that cannot reconstruct what happened for lack of records
- Postmortems that assign blame and change nothing structural