What 186 refund tickets taught us about policy

We expected the replay to grade the agent. It graded us.

I lead a support team of four at Northgate. We sell bikes and parts online in the US and Canada, and every Monday the queue opened at around 300 tickets. Almost a third were the same two questions: I was charged twice and where is my order? Easy, urgent to the customer, and always ahead of the hard ones.

So we built a refund agent in Capstan, and before turning it on we ran it against the last 30 days: 186 refund tickets.

The first read

The numbers looked good. The agent did the same as us on 171 cases. On 11 it did better, mostly by catching duplicate charges one of us had missed in a busy week. Four came back worse.

I opened the four worse cases expecting four agent mistakes. I found four places where our refund policy said one thing and we did another.

Four cases, four unwritten rules

Case one: international shipping. A customer in Canada was charged twice for a pair of tyres. The agent refunded the duplicate charge in full, including shipping. We had refunded the product but not the shipping, because "we don't refund cross-border shipping". That sentence was nowhere in the policy doc. It was in my head, and in the heads of two people on my team.

Case two: pre-authorisations. A customer saw two charges, one of which was a hold that drops off by itself after three days. The agent refunded it, which would have meant refunding money we never took. Whoever answered that ticket on the day knew to wait. The doc didn't say so.

Case three: the loyal customer. A refund just over our $50 limit, for someone who'd bought four bikes from us. We'd approved it without asking anyone. The agent sent it to a person, which was correct under the written policy and slower than what we actually do.

Case four: the third duplicate. The same customer had reported a duplicate charge twice before that year. A person on my team had flagged it to finance instead of refunding. The agent refunded it.

What we changed

None of the four was a model problem. Every one was a rule we followed but had never written down. So we wrote them down:

• Cross-border shipping isn't refunded on duplicate charges; the product is.
• A second charge marked pending is a hold. Reply, explain, and don't refund.
• Refunds up to $75 are fine for customers with three or more orders.
• A third duplicate report in 12 months goes to finance, not to a refund.

Then we ran the replay again on the same 186 tickets. Zero worse. The agent didn't get smarter. Our policy got honest.

The agent does the refunds under $50 and asks me about the rest. I read a summary at nine and I'm done with billing by ten past.

The part I didn't expect

The new policy doc helped the team more than the agent. Two of us had been following the shipping rule and two hadn't. We'd been giving different answers to the same question for over a year, and nobody had noticed because nobody reads 186 tickets side by side.

New hires now read the same doc the agent does. Onboarding a support person used to take two weeks of shadowing. It takes about one now, because most of what they used to learn by watching is written down.

Six weeks in

The queue opens with the billing tickets already answered. Every refund is one click away in the trace if anyone wants to check it, and none has been reversed. We ran a replay on where is my order? the same way, found two more unwritten rules about split shipments, and turned that agent on a week later.

If you're about to do the same, my advice is to read the worse cases before you read anything else. They won't tell you much about the agent. They'll tell you a lot about your team.

Sample post: Northgate Bikes is an invented company. Replace the files in content/blog with your own.