Skip to content
Case 02 · 8 min read

Case 02 · AI agent · AWS Bedrock · Insurance restoration

ClaimsAI — code picks the next question; the model only extracts.

Role
Wrote V2 solo · #1 committer on the backend (451 of 1,494)
Stack
Python 3.13 · FastAPI · Pydantic v2 · Supabase · AWS Bedrock (Claude Sonnet 4.5)
Ships
Behind a V1/V2 flag · Bedrock rollback is one env flip
Result
12.7 s / worst turn, was 256 s

A contractor describes a water loss in their own words and gets a priced estimate — from code that decides what to ask next and a model that only extracts.

256 s to 12.7 s1
worst-case turn latency
99,000 chars to state machine2
system prompt replaced
8513
tests in the V2 suite
~11k
lines written solo for V2

01 — The problem

One prompt doing four jobs

Wave helps restoration contractors price water and property damage. A contractor describes the loss in chat, by text or voice, often as one long brain dump. The system has to pull out the customer, address, cause of loss, water category, rooms, dimensions, flooring, demolition and drying days, then ask only for what is missing.

The first version ran on a 99,000-character system prompt with rules that contradicted each other. One model call was asked to extract the facts, route the conversation, pick the next question and re-emit the entire discovery model every turn. State leaked between turns, and the agent needed more than 900 lines of merge, preserve and guard code just to put back what the model had dropped.

02 — What I built

A pipeline where each layer has one job

I wrote the V2 discovery agent alone, about 11,000 lines. The idea fits in two sentences. The server picks the next question, not the model. And the model emits a diff of what changed, not the whole state. I estimated those two changes would remove about 70% of the bug family we were fighting.

Each message now flows through narrow layers. Claude on Bedrock extracts a partial model through a tool-use schema. Pure Python merges it, and a deterministic state machine chooses the next question by field priority and dependencies. A phraser writes the reply and is skipped when there is nothing to acknowledge. On completion, a deterministic code assigner turns the model into priced line items.

V2 sits behind a flag with no conditional code at the call sites, and the Bedrock client sits behind its own pair of flags, so rollback is a single environment change. Around it I also built the Stripe tiers, quota consumed atomically when an estimate completes, and a free-tier cap of five AI refinement turns per estimate.

  1. Extractor: Claude Sonnet 4.5, temperature 0, structured tool-use output
  2. Merge and state machine: pure Python, under 30 ms per turn
  3. Phraser: an LLM call, bypassed on quiet turns
  4. Code assigner: CSV-driven pricing codes with regional pricing

03 — The hard part

Making the model's output boring

Narrow extraction only works if the output is reliable. I moved the extractor to native Bedrock tool-use, with tool specs generated from the Pydantic models. I inlined every schema reference, because Sonnet's tool-use was more reliable on flat schemas. A response adapter converts Bedrock's output to the shape the rest of the code already expected, so callers did not change.

Then I guarded the edges. A defensive unwrap catches the model wrapping its answer in an invented parameter key. When the extractor fails on a one-letter reply to a multiple-choice question, a deterministic salvage step reads the letter instead of looping.

I also wrote down which layer owns which kind of bug, with a decision tree for triage and five anti-patterns, each taken from a real shipped bug. One example: merge must never scan the user's text for keywords. A message like "scratch that… wait, demo IS needed, 2 days" had flipped both rooms to no demolition. I shipped more than 35 fixes as one bisectable commit per bug.

04 — How I verified it

Two gates and a replay

Prompt changes had to pass two gates. The first is deterministic tests of merge and the agent. The second runs the real model on Bedrock: five runs per scenario, and a scenario passes only if at least four succeed. The V2 suite has 851 test functions.

Tests only cover cases someone thought of, so I built a replay harness. It re-runs real production conversations against new code, with an LLM playing the contractor, grounded in the original thread so it cannot invent answers. The top-ten replay covered 10 estimates and 441 messages. It found a dimensions loop that wasted more than 100 turns, multi-letter answers to single-choice questions and missing square-footage parsing. Each one was fixed.

On the web app I built an AI browser suite: about 45 natural-language tests across 9 product areas, including 18 estimate-creation runs where an agent chats with Wave's AI until a dollar total appears. It runs on Claude Haiku by default, reports token cost per test, splits the estimate-creation tests across 6 CI workflows and runs from a pre-commit hook that picks only the tests a change touches.

05 — Where the seconds went

256 seconds to 12.7, and a failure nobody saw

I measured before changing anything. A baseline turn took 2.57 s at the median and 7.88 s at p95, and extraction was about 95% of that. Merge and the state machine took under 30 ms. The worst turns were much longer than p95, so I fixed the client first: TCP keep-alive, a 30-second read timeout, adaptive retries and cached clients. The maximum turn time fell from 256 s to 12.7 s.

Smaller wins came after. Skipping the phraser saves about 2–3 s on a happy-path turn. Warming the Bedrock clients when the server starts saves about 1–2 s on the first request after a deploy. Neither changes what the model sees; both stop the turn paying for work it does not need.

While profiling I found that language detection had been failing on every turn since the move to Bedrock. Bedrock did not enforce the JSON mode the code asked for, and the error handler returned a plausible default — English, 0.5 confidence — so nothing looked wrong. I switched it to a strict JSON schema, forced through tool-use on Bedrock, plus a parser that accepts fenced or prose-prefixed JSON.

06 — What I'd change

Next

The next steps were already planned. Route simple replies to Claude Haiku instead of Sonnet, and collapse the roughly 16 database round-trips in a turn into a single database call. Both cut time from every turn, not just the worst ones.

The client fixes capped the tail, but extraction is still about 95% of a normal turn. So the remaining gains come from fewer or smaller model calls, not from faster Python. The replay harness is what makes that safe to try: any cheaper model has to hold up on real conversations first.

Notes

  1. Maximum turn time before and after TCP keep-alive, a 30 s read timeout, adaptive retries and cached Bedrock clients.
  2. V1 asked one model call to extract, route, pick the next question and re-emit the whole model every turn.
  3. Test functions; the wider backend suite went from 309 to 784 passing as the work landed.

Ask my portfolio