Inside an agentic customer-experience deployment

An anonymized walkthrough of an agentic customer-experience deployment that took support contact rates from 18% to 7% over four quarters. Here is what worked, what didn't, and what we would do differently.

Background

The client: a mid-market SaaS company, ~250k customers, ~45k support contacts per month. Goal: reduce contact volume without harming CSAT. Constraint: regulated data, on-prem requirements for some workloads, EU-resident for others.

Quarter 1: Foundations

We did not deploy an agent. We deployed evals. 200 historical tickets, expert-labeled, with a rubric. We instrumented the existing support stack. We built the data pipelines. The team was anxious that we were not shipping. We were shipping the thing that would let us measure whether anything we shipped later worked.

Quarter 2: Deflection on knowable questions

A retrieval system over the help center. Tightly scoped: handle the 30 known top intents, escalate everything else. Citations on every answer; CSAT survey on every interaction. By end of quarter: 22% of contacts deflected. CSAT held at 4.4/5.

Quarter 3: Agent for known workflows

Five workflows where the resolution was knowable but multi-step (password reset, billing question, plan change, downgrade, account merge). Tools for each, deterministic verification, HITL for any irreversible action. By end of quarter: another 18% of contacts handled. CSAT 4.5/5.

Quarter 4: Long-tail handling

For the long tail, we did not deploy a more capable agent. We deployed a better triage: classify the contact, summarize it, attach relevant precedents, route to the right human. Resolution time for long-tail tickets dropped 35%. Total contact rate now 7%, down from 18%.

What didn't work

• An open-ended chat interface. Pulled too many off-topic queries. We replaced it with intent-first navigation.
• Auto-resolution of refunds. Edge cases too varied. We moved to AI-prepared refund recommendations, human-approved.
• Vendor-led prompts. The vendor's defaults underperformed our tuned prompts by 11 points on our eval. We took back the prompts.

What we'd do differently

Spend less time in Q1 building infrastructure that we could have rented. We over-built the eval platform; a commercial one would have done 80% of the job in a tenth of the time. The cost of the infrastructure we built was higher than the cost of the infrastructure we should have bought.