Skip to content
Case 03 · 8 min read

Case 03 · AI testing · Playwright · Cost control

Qove — reason once, replay many, and never call a broken product green.

Role
Core engineer at BetterFutureLabs · ~1,100 of ~3,860 commits
Stack
Python · FastAPI · Postgres · Playwright · Claude on AWS Bedrock · Docker
Ships
In production with B2B SaaS pilot customers
Result
$0.50 / run, was $5.40

Real user sessions become browser tests that an AI writes once and a runner replays cheaply — without calling a broken product green.

$5.40 to $0.501
cost of one test run
$3.53 to $1.082
one run with prompt caching
953k to 1633
error rows collapsed to distinct errors
1,100+4
commits

01 — The problem

Tests nobody has to write

End-to-end tests are slow to write and quick to go stale. Qove takes a different route. A small JavaScript snippet records how real people use a customer's web app, with personal data masked in the browser. The backend groups sessions into journeys, and Claude turns those journeys into parameterised browser tests.

The tests ship in a per-customer Docker runner that the customer runs in CI or a pre-commit hook. The container never holds an LLM key; it calls Qove's relay, which forwards to Anthropic or AWS Bedrock and meters every token. The hard part is the run itself. An AI agent driving a browser on every run is slow and expensive — and it can be wrong about whether it passed.

02 — What I built

The runner, the triage and the cost controls

I am one of Qove's core engineers at BetterFutureLabs, with about 1,100 commits between April and October 2026. The idea started earlier as my own experiment: natural-language browser tests running in a pre-commit hook, about two months before Qove.

My work runs through the whole pipeline. On the way in: journey intelligence, test variables with encrypted secrets, and TestRail import. At run time: the record-and-replay engine and the first-run reasoning executor. After the run: autonomous failure triage, the gates that decide a verdict, and the relay metering and admin page that make LLM spend visible.

I also measure before I fix, and several production problems only showed up that way. Each fix below started from a number taken from production data and recorded in my write-up for the change, and each one shipped behind flags, with tests.

  1. 953,237 error rows described 163 distinct errors; repeats now fold into one row per session
  2. 114 of 114 scheduled report jobs had failed silently: a timeout cannot cancel a thread, so PDF rendering moved into a killable child process
  3. Two "green" SDK releases never reached a browser because of a one-year cache header; releases now purge the CDN and poll until the edge serves the new file
  4. A 212K-token prompt was being rejected by Claude's 200K limit; intent extraction now runs in token-budgeted batches

03 — The hard part

Reason once, replay many

The expensive model call should happen when a test is first run, not every time. On the first run, one Claude Opus agent walks the test over Playwright accessibility snapshots and acts on elements by reference. It repairs steps that have gone stale and records a recipe: for each step, the element's role and name, plus CSS, XPath and text fallbacks.

Later runs replay the recipe with close to zero LLM calls. Replay resolves elements by role and accessible name first, and only a step that no longer resolves goes back to the model. Recipes are keyed per customer, test and environment, with a quality guard so a worse recording can never overwrite a better one.

The verdict comes from an independent acceptance judge that sees the whole trajectory of observed page states but never the agent's own narration, so the agent cannot talk its way to a pass. Every change shipped behind per-organisation flags, with flag-off paths left byte-identical.

04 — Following the tokens

From $5.40 a run to $0.50

Cost problems hide in what gets re-sent. One dropdown with 2,737 items was about 70,000 tokens, and it was going back to the model 11 times in a single run. Pruning old snapshots from the agent's history and capping their size took a 4-step run from $5.40 to $0.50. Prompt caching with a rolling breakpoint took another run from $3.53 to $1.08.

Then the meter itself. A missing model price made the dashboard over-count spend about 2.5×, so I added date-stamped rates and a backfill. Skipping the cache-write premium on one-shot calls saved about 19% of a clean replay. The relay now attributes every call to a run and a test case, split four ways by token type, on an admin page I built.

For spend attribution on Bedrock, I checked the approach AWS guidance suggested against all 14 Bedrock regions in our account. It could not work there, and shipping it would have broken every Claude call. I built a working alternative that tags spend by project through a different Bedrock endpoint.

05 — Honest verdicts

A server error is not a pass

A test that should prove an action is blocked can pass for the wrong reason. In one customer test, 3 of 9 runs showed green while the server was returning HTTP 500. The prevention check now refuses to count a 5xx as "prevented". Criteria are probed at every step boundary and the strongest sighting is kept. A "this run created it" gate rejects records that existed before the run.

Clicks can lie too. The control that motivated this was disabled only visually, for about 25 seconds. Every click is now fingerprinted before and after, and the runner waits for the page to settle before calling a click inert.

When tests fail, triage groups them into root causes. Deterministic rules decide only what they can prove. One bounded, structured-output Claude investigation per run handles the rest, and the database owns identity, tenant scope and idempotency, so the model cannot invent candidates. It ran in shadow mode against blind human labels before it could publish anything.

  1. Validation replay: 13 runs, 112 failed tests, 20 accepted root causes, 18 cascades, 111 abstentions
  2. 28 metered model calls for the whole replay
  3. Paid calls are claimed exactly once and recovered after a crash without double billing

06 — What I'd change

Next

Most of my cost and correctness fixes started with a number nobody had looked at: a price missing from a table, a timeout that never fired, a CDN serving an old file. I would make those numbers visible from the first day — spend per run, error rows per session, job heartbeats — rather than finding them during an incident.

Shadow mode with blind human labels is the main reason I trust the triage. I would put every new AI judgement in Qove through the same process before it reaches a customer.

Notes

  1. A 4-step run, before and after pruning page snapshots from the agent's history and capping their size.
  2. Anthropic prompt caching with a rolling breakpoint, about 3.3× cheaper.
  3. Measured in the production database before folding repeats into one row per error per session.
  4. April to October 2026; about 30% of the lines added on main.

Ask my portfolio