Case 02 · AI agent · AWS Bedrock · Insurance restoration
ClaimsAI — code picks the next question; the model only extracts.
A contractor describes a water loss in their own words and gets a priced estimate — from code that decides what to ask next and a model that only extracts.
01 — The problem
One prompt doing four jobs
Wave helps restoration contractors price water and property damage. A contractor describes the loss in chat, by text or voice, often as one long brain dump. The system has to pull out the customer, address, cause of loss, water category, rooms, dimensions, flooring, demolition and drying days, then ask only for what is missing.
The first version ran on a 99,000-character system prompt with rules that contradicted each other. One model call was asked to extract the facts, route the conversation, pick the next question and re-emit the entire discovery model every turn. State leaked between turns, and the agent needed more than 900 lines of merge, preserve and guard code just to put back what the model had dropped.
02 — What I built
A pipeline where each layer has one job
I wrote the V2 discovery agent alone, about 11,000 lines. The idea fits in two sentences. The server picks the next question, not the model. And the model emits a diff of what changed, not the whole state. I estimated those two changes would remove about 70% of the bug family we were fighting.
Each message now flows through narrow layers. Claude on Bedrock extracts a partial model through a tool-use schema. Pure Python merges it, and a deterministic state machine chooses the next question by field priority and dependencies. A phraser writes the reply and is skipped when there is nothing to acknowledge. On completion, a deterministic code assigner turns the model into priced line items.
V2 sits behind a flag with no conditional code at the call sites, and the Bedrock client sits behind its own pair of flags, so rollback is a single environment change. Around it I also built the Stripe tiers, quota consumed atomically when an estimate completes, and a free-tier cap of five AI refinement turns per estimate.
- Extractor: Claude Sonnet 4.5, temperature 0, structured tool-use output
- Merge and state machine: pure Python, under 30 ms per turn
- Phraser: an LLM call, bypassed on quiet turns
- Code assigner: CSV-driven pricing codes with regional pricing
03 — The hard part
Making the model's output boring
Narrow extraction only works if the output is reliable. I moved the extractor to native Bedrock tool-use, with tool specs generated from the Pydantic models. I inlined every schema reference, because Sonnet's tool-use was more reliable on flat schemas. A response adapter converts Bedrock's output to the shape the rest of the code already expected, so callers did not change.
Then I guarded the edges. A defensive unwrap catches the model wrapping its answer in an invented parameter key. When the extractor fails on a one-letter reply to a multiple-choice question, a deterministic salvage step reads the letter instead of looping.
I also wrote down which layer owns which kind of bug, with a decision tree for triage and five anti-patterns, each taken from a real shipped bug. One example: merge must never scan the user's text for keywords. A message like "scratch that… wait, demo IS needed, 2 days" had flipped both rooms to no demolition. I shipped more than 35 fixes as one bisectable commit per bug.
04 — How I verified it
Two gates and a replay
Prompt changes had to pass two gates. The first is deterministic tests of merge and the agent. The second runs the real model on Bedrock: five runs per scenario, and a scenario passes only if at least four succeed. The V2 suite has 851 test functions.
Tests only cover cases someone thought of, so I built a replay harness. It re-runs real production conversations against new code, with an LLM playing the contractor, grounded in the original thread so it cannot invent answers. The top-ten replay covered 10 estimates and 441 messages. It found a dimensions loop that wasted more than 100 turns, multi-letter answers to single-choice questions and missing square-footage parsing. Each one was fixed.
On the web app I built an AI browser suite: about 45 natural-language tests across 9 product areas, including 18 estimate-creation runs where an agent chats with Wave's AI until a dollar total appears. It runs on Claude Haiku by default, reports token cost per test, splits the estimate-creation tests across 6 CI workflows and runs from a pre-commit hook that picks only the tests a change touches.
05 — Where the seconds went
256 seconds to 12.7, and a failure nobody saw
I measured before changing anything. A baseline turn took 2.57 s at the median and 7.88 s at p95, and extraction was about 95% of that. Merge and the state machine took under 30 ms. The worst turns were much longer than p95, so I fixed the client first: TCP keep-alive, a 30-second read timeout, adaptive retries and cached clients. The maximum turn time fell from 256 s to 12.7 s.
Smaller wins came after. Skipping the phraser saves about 2–3 s on a happy-path turn. Warming the Bedrock clients when the server starts saves about 1–2 s on the first request after a deploy. Neither changes what the model sees; both stop the turn paying for work it does not need.
While profiling I found that language detection had been failing on every turn since the move to Bedrock. Bedrock did not enforce the JSON mode the code asked for, and the error handler returned a plausible default — English, 0.5 confidence — so nothing looked wrong. I switched it to a strict JSON schema, forced through tool-use on Bedrock, plus a parser that accepts fenced or prose-prefixed JSON.
06 — What I'd change
Next
The next steps were already planned. Route simple replies to Claude Haiku instead of Sonnet, and collapse the roughly 16 database round-trips in a turn into a single database call. Both cut time from every turn, not just the worst ones.
The client fixes capped the tail, but extraction is still about 95% of a normal turn. So the remaining gains come from fewer or smaller model calls, not from faster Python. The replay harness is what makes that safe to try: any cheaper model has to hold up on real conversations first.
Notes
- Maximum turn time before and after TCP keep-alive, a 30 s read timeout, adaptive retries and cached Bedrock clients.
- V1 asked one model call to extract, route, pick the next question and re-emit the whole model every turn.
- Test functions; the wider backend suite went from 309 to 784 passing as the work landed.
- Latency figures are my own measurements of the V2 pipeline: maximum turn time before and after the client fixes, and p50/p95 from the baseline run.
- Replay fixtures came from real production conversations; none of their content appears here.
- Figures are simplified drawings of the pipeline, not screenshots of the product.