The AI observability starter: the exact seams to instrument

The short version

An overnight automated check found that 31 percent of AI-powered actions were failing. The AI call itself succeeded every time; the failure was in the code that unpacked and acted on the response, and it threw errors nobody was catching or reporting.

The failure was invisible because success was being measured at the wrong point. Measure the outcome the user wanted, not whether the request came back.

Copy these prompts

Paste them into Claude Code (or any AI that can see your project), in this order. Each one answers a different question, and the first is usually the one that matters.

1. Find where you record success

For every AI feature in my app, show me the exact line where I record it as having succeeded. I want to see the moment I decide it worked.

2. Move it to when the user got what they wanted

I suspect I am recording success when the AI response arrives rather than when the user actually got the outcome they asked for. Show me each place that gap exists and what the correct point would be.

3. Report the failures in between

Add error reporting to the code that parses and acts on AI responses, so a failure between a successful AI call and a successful outcome stops being silent. Tell me how I would see those from now on.

An overnight tracing run found 5 of 16 agent calls 502ing (a 31% failure rate) while every dashboard said everything was fine. The twist: the model call had SUCCEEDED every single time. My post-processing (response parsing and tool dispatch) was the killer, throwing untyped errors with no stack. The same run also caught my retry logic replaying yesterday's cached failure and calling it a fresh response.

The lesson

Instrument the seams around the model, not just the model. Provider dashboards show their side green; your users experience your whole pipeline.

The seams to instrument (in order of how often they burned me)

1. Response parsing. JSON extraction, schema validation, enum coercion: where "model succeeded" becomes "user got a 502." Wrap it in its own span and its own typed error:

try {
  intent = IntentSchema.parse(extractJson(raw));
} catch (e) {
  throw new ParseError('intent_parse_failed', { raw: raw.slice(0, 500), cause: e });
}

2. Tool dispatch. The model asked for a tool; did the tool run, and did its result make it back into the loop? Span per tool call, tagged with tool name + success.
3. Retry & cache layers. My retry was serving a cached failure as a fresh response. Instrument cache hits with the cached entry's age and status, and never cache non-2xx results without an explicit decision.
4. Pre-model assembly. Context building, memory recall, prompt construction. A recall query that returns garbage poisons everything downstream; give it a span so you can see it.
5. The provider call itself: last, not first. Tag spans with model, provider, variant, token counts, and cost so you can slice failures by configuration.

The starter setup

6. Typed errors at every seam. "Untyped error, no stack" cost me the diagnosis for weeks. Every seam throws its own error class tagged with phase + provider + model + experiment variant. Then your error tracker groups by seam, not by "Error: undefined."
7. One trace across the whole turn. Request → assembly → model → parse → dispatch → response as one distributed trace. The 31% was invisible in aggregate metrics; it was obvious in a single trace waterfall.
8. Run an overnight soak against production. Point an agent (or a script) at your real API for a night with tracing at 100%. Low traffic + full sampling = the failure patterns your daily glance misses. That single run found the 31%, the stale-retry bug, AND my docs lying about which model was in prod.

Steal this for your app

• Compute your end-to-end success rate (user got a valid result) and put it next to the provider's success rate. The gap between the two numbers is your roadmap.
• "The model is fine" and "the feature is broken" are usually both true. The bug lives in the plumbing you wrote in an afternoon and never instrumented.
• Cache + retry interactions deserve their own test: force a failure, then verify the retry does fresh work.

Run this on your codebase

Paste this into Claude Code in your repo:

Audit this repo's AI pipeline for uninstrumented seams.
Map the full turn: context assembly, provider call, response parsing, tool dispatch, retry and cache layers.
Flag any seam that throws untyped errors or lacks its own span; check parsing and dispatch first, the provider call last.
Verify JSON extraction and schema validation are wrapped in typed error classes tagged with phase, provider, model, and variant.
Check the cache layer: can a non-2xx or failed result get cached, and do retries check entry age and status before serving it?
Confirm one distributed trace spans the whole turn so a single waterfall exposes failures.
Find where end-to-end success (user got a valid result) is measured, versus only provider success.
Report findings as a checklist before changing anything.

R080: the alert set people will trust

An alert earns the right to interrupt someone only when it names a user outcome, has one owner, and points to an action they can take now. Start with three signals:

1. End-to-end outcome failure rate. Alert when users fail to receive a valid result, not when the provider returns an error. Include the affected workflow and sample run IDs.
2. Oldest stuck or retrying run. A growing queue can hurt users while the average success chart still looks normal. Page on age and count together.
3. Parse or dispatch failures after a successful model call. This catches the exact gap that hid the 31 percent failure rate.

Every alert definition should include:

signal | threshold and window | owner | first action | runbook link
deduplication key | recovery condition | test event | mute or delete rule

Send a test alert before enabling the real threshold. Confirm that only one notification appears for repeated copies of the same failure, the owner can open a representative trace, and the recovery notification fires. If nobody acts on an alert twice, tighten it or delete it.

Comment ALERT for this module.

R088: structured logs that survive production

console.log("failed") helps only while the exact request is still in your head. A production event needs enough safe structure to join one failure across services:

{
  "event": "agent.output_parse_failed",
  "run_id": "fictional-run-42",
  "stage": "parse",
  "severity": "error",
  "outcome": "user_result_missing",
  "duration_ms": 812,
  "error_type": "schema_validation",
  "safe_context": {"model": "configured-model", "attempt": 1}
}

Keep the event name stable. Carry the same run_id through assembly, provider call, parsing, tool dispatch, and response. Log duration and typed error at each seam. Redact prompt bodies, tokens, credentials, email addresses, and customer content before the logging call, not after ingestion.

Test the contract with one success, one parse failure, one tool failure, and a duplicate retry. The four runs should be searchable by ID without exposing the payload.

Comment LOGS for this module.

Download the runnable observability pack

The starter validates fictional event and alert definitions, returns a reviewable instrumentation draft, and performs no production changes.

• Download the complete pack
• Import the n8n workflow
• Open the sample input
• See the expected dry-run result
• Run with Docker Compose
• GitHub validation workflow
• GitHub manual run workflow