Observability for Agent Runs

PROJECT CONTEXT:
When my agent misbehaves I scroll raw logs and guess. The agent-observability ecosystem has converged on the answer: structured traces of every step, session replay for debugging, and cost/latency metrics in the same view.

GOAL:
Instrument the agent with structured tracing (OpenTelemetry-compatible), build session replay, track cost per session, and run failure analysis on real data.

REQUIREMENTS:
- Every run traced: model calls, tool executions, decisions as spans with attributes
- Session replay: reconstruct any run step-by-step from stored events
- Cost and token metrics per call, aggregated per session and per task class
- Error and anomaly capture: failures linked to their trace context
- A failure-analysis workflow using the traces, demonstrated on real cases

CONSTRAINTS:
- Use open standards (OpenTelemetry semantics) over bespoke formats where possible
- Sensitive content (secrets, user data) must be scrubbed before storage

DEFINITION OF DONE:
- A session renders as a trace tree: turns, model calls, tool spans with timing
- Any past session replays step-by-step with full payloads (scrubbed)
- Cost per session appears in the session summary and aggregates over time
- One real failure is root-caused via traces, demonstrably faster than log-diving

COMMON FAILURES TO AVOID:
- Unstructured print-logging that cannot reconstruct a run
- Traces without token/cost attributes, blinding the economics
- Secrets in span payloads, turning observability into a breach
- Dashboards built, never used - observability without a debugging workflow