Unblocking a sixty-day AI stall in a five-day sprint
A three-agent clinical document pipeline worked in the demo and failed in production. Six failure modes had held it back for two months.
Agents looped without a recursion limit until the token budget was gone, a shared memory store leaked one patient document context into another session — a compliance exposure, not just a bug — no evaluation pipeline meant every prompt change was a blind gamble, and the average time to detect a production failure was 47 minutes.
We diagnosed structurally instead of patching symptoms. The original architecture had no path to production reliability, so we rebuilt the orchestration on correct foundations: typed tool results, enforced recursion limits, per-session memory isolation and a model fallback chain, with a regression benchmark and observability shipped alongside.
- —Orchestration rebuild with typed tool results and fallback chain
- —Recursion limit enforced at graph level
- —Per-session memory isolation for patient data
- —150-input regression benchmark gated in CI
- —Structured per-step logging, cost tracking and alerting
Deployed to 500 users in five days with zero production incidents across the following thirty days. p95 latency fell from 1,800ms to 195ms, time-to-detect from 47 minutes to under two, and three prompt regressions were caught before they reached users.
Have a problem that looks like this one?
Start with a 45-minute discovery call. No pitch, no deck — we align on the problem and tell you honestly whether we are the right team for it.