How we test an agent before it touches a customer
Replays run a new agent on real cases from the last 90 days and put its answers next to what your team did. Here is how they work, and what they catch.
Most agent demos are tested on the happy path. Real work isn't. The ticket that says charged twice is sometimes a duplicate, sometimes a pre-authorisation that will drop off in three days, and sometimes a customer who really did buy two bikes.
You can't write test cases for that by hand. You don't know which cases are hard until you've seen them. So before an agent runs on anything new, Capstan runs it on what already happened.
What a replay does
A replay takes the last 30, 60 or 90 days of cases the agent would have handled, and runs the agent on each one with every write turned off. Nothing is sent, refunded or changed. The agent reads exactly what it would have read, decides what it would have decided, and drafts what it would have sent.
Then we put its answer next to what your team actually did, case by case.
That last part is the whole idea. A replay isn't graded against a rubric we wrote. It's graded against your team, on your cases, under your policy. If your team refunds duplicate charges without asking, an agent that asks first is slower than you. If your team always checks the order date, an agent that skips it is wrong, even when the refund happens to be right.
Reads are real, writes are recorded
The hard engineering problem is that tools don't know they're in a replay. A Stripe lookup should return what Stripe returns today. A Stripe refund must not happen.
Every tool in Capstan declares which of its actions read and which write. During a replay, reads go through as normal, and writes are caught at the edge and recorded with their full arguments:
// What the agent tried to do on case 4471, recorded instead of sent
{
tool: "stripe",
action: "refunds.create",
args: { charge: "ch_3Q…8f", amount: 4200, reason: "duplicate" },
replay: { sent: false, recordedAt: "2026-10-12T09:14:03Z" }
}
The recorded write is what we compare. Did the agent refund the right charge, for the right amount, for the right reason? A reply that reads well but refunds the wrong charge counts as worse, every time.
There's one subtlety. Reads made today about a case from last month can disagree with what the person saw then. An order that was "in transit" is now "delivered". We snapshot what we can (ticket text, order status at the time, the policy document as it was) and we flag any case where a read came back different from the record. Those cases are listed separately, so nobody mistakes stale data for a bad decision.
Same, better, worse
Every case lands in one of three columns.
- Same means the agent did what a person did, with the same outcome.
- Better means a reviewer preferred the agent's answer: usually a duplicate charge a person missed, or a faster reply that cited the right policy.
- Worse is the column that matters.
On a typical support replay of 186 cases, we see around 171 the same, 11 better and 4 worse. Teams expect to read the 171. They shouldn't. The four worse cases are where the work is, and they're almost never random. They cluster.
What the worse cases tell you
When we look across replays, the worse cases fall into a few groups.
- Policy the agent was never told. "We don't refund shipping on international orders" lives in a person's head, not in the doc. The agent refunds the shipping.
- Context in another tool. The customer already emailed sales about the same charge. The agent only read Zendesk.
- Cases that need a judgement call. A long-time customer, a third duplicate this year, a refund just over the limit. A person bent the rule; the agent didn't.
The first two are fixed by writing something down or connecting a tool. The third isn't a bug at all. It's a step that should stay with a person, and the replay just showed you where it is.
Approve topic by topic
That's why a replay groups cases by topic, and why you don't approve the agent. You approve topics.
Duplicate charges: yes. Order not arrived: not yet. Partial refunds over $50: never, keep those with people. The agent takes the approved topics and hands the rest to your team until you change your mind. When you do, you run the replay again on the new topic first.
We stopped asking “is the agent good enough?” and started asking “which of these could it do on Monday?” That's a question a support lead can actually answer.
Running one yourself
From the app, a replay is a button on any agent: pick the window, pick the topics, start. From the CLI it's one line, which is how most teams run it in CI before they publish a change to an agent:
npx capstan replay refund-desk --days 30 --fail-on worse>2
The command exits non-zero if more than two cases come back worse, and prints a link to the full comparison.
What it costs
Replays cost the same per case as a run, and never count against your plan's monthly runs. The 186-case replay above cost 92 cents. We price it that way on purpose: if testing costs more than shipping, people stop testing.
Sample post. Replace the files in content/blog with your own.