How to pilot AI role-play training in four weeks, with a go/no-go decision

Most AI role-play pilots end with an impression instead of a decision. Agents liked it, managers aren’t sure, and the renewal conversation starts from opinions. A pilot that ends in a decision has four things in place before the first practice session: a narrow scope, a baseline measured the same way as the result, a final test on scenarios agents haven’t seen, and a written bar for buying.

This is the design Intalk uses for its four-week pilots with US agencies, IMOs and FMOs,¹ and it works just as well for evaluating any other vendor. It is written for agency owners, training directors and the managers who will run the pilot.

What should a four-week pilot include?

One workflow, 5 to 10 agents, one training manager who owns the pilot, a named budget sponsor, and decision criteria agreed in writing before the start.¹ Everything else follows from keeping the scope that narrow.

“Improve new-agent training” is not a workflow. “The first needs-discovery call for newly licensed Medicare agents” or “explaining surrender charges on a fixed indexed annuity” is. A single workflow lets you build equivalent test scenarios, keep the rubric fixed and have one manager review every answer. A group of 5 to 10 agents is small enough for that manager to keep up and large enough to show whether results vary from agent to agent.

The budget sponsor matters more than it seems. A pilot that clears its bar still needs someone with authority to approve the purchase, on a date set before the pilot begins.

How do you set a baseline?

Before agents touch the new tool, time the current process and score current practice on a fixed rubric that people review.¹ Without that, the end-of-pilot scores have nothing to be compared with.

A good rubric for sales conversations is short and observable. The four items in Intalk’s agent demo are a reasonable start:

• Needs confirmed: the agent asked what matters to the customer before recommending anything.
• Reasons explained: the recommendation is tied to what the customer said, with one trade-off named.
• Uncertainty disclosed: the agent said what still needs checking, such as underwriting or a provider network, before the customer had to ask.
• Agreed next step: the call ends with a concrete next step and a time the customer accepts.

Calibrate before the baseline counts. Have two reviewers score the same 10 practice answers separately, compare, and rewrite any item they read differently, with an example answer for each level. Then record how long a manager spends reviewing one agent’s practice today. That number becomes the baseline for manager time.

Which metrics decide the pilot?

Three numbers decide it, each with a bar agreed before the start; a fourth tells you whether the result can be trusted.

These measure learning and an early change in behavior, the second and third of the four levels in the Kirkpatrick model of training evaluation.² Agents’ reactions, the first level, are worth asking about but shouldn’t decide anything. Business results, the fourth level, such as retention, placement rates or complaint rates, take longer than four weeks to show up; plan to measure them in the first months after a purchase.

What happens in each week?

The four weeks are measured use; the setup work comes first.¹

Why test on scenarios agents haven’t practiced?

If the final test reuses the practice scenarios, scores rise because agents remember the answers, not because they can handle a new customer. The final test has to use equivalent scenarios that agents haven’t seen.¹ In practice that means the same rubric targets and difficulty, with different customer details and a different order of objections.

The simplest way to get there is to write two versions of each scenario before the pilot starts and decide at random which version agents practice and which one is held back for the final test. Keep the held-back versions away from managers’ coaching notes too.

What data does a pilot need?

Less than most teams expect. Scenarios can be built from synthetic or de-identified examples, so a pilot doesn’t need access to live customer records or protected health information (PHI).¹ Data permissions are agreed in writing before delivery starts.¹

If you want scenarios based on real conversations, de-identify them first. Under the HIPAA Privacy Rule, health information can be de-identified either by an expert determination or by removing the identifiers listed in the safe harbor method.³ The Department of Health and Human Services explains both methods in its guidance.⁴ For Medicare call centers, the practical rule is simple: don’t paste real recordings or transcripts into any vendor’s tool until your compliance team has approved how they are handled.

How do pilots go wrong?

Most failed pilots fail on design, not on the tool.

• The bar moves after the results. Write the criteria into the pilot plan and have the sponsor sign it.
• Only the best agents take part. Choose the cohort the way you’d choose it for a rollout, not from volunteers.
• The rubric changes mid-pilot. Fix scenario problems, but leave the rubric alone, or the baseline stops meaning anything.
• Nobody checks the scores. Whatever scores the practice, a person should review a sample, and any critical rule error should be confirmed by a person.
• Reaction stands in for results. Agents enjoying the tool is useful to know; it isn’t the decision.
• There is no decision date. Set it before the pilot starts, with the sponsor in the room.

Compliance-sensitive workflows add one more: a practice tool that teaches the wrong line is worse than none. Build scenarios from your approved material, and have the rule checks reviewed by your compliance lead before agents rely on them. The TPMO disclaimer changes for 2027 are a good test case for a Medicare team.

What happens after the pilot?

The decision is made on the agreed date.¹ If the pilot clears the bar, it converts to an annual license and expands team by team; if it misses, the scope is narrowed or the pilot stops.⁵ Write the outcome down in a one-page memo: the criteria, the baseline and final numbers, what the reviewers saw, and the decision.

To estimate what onboarding costs your team today, before a pilot changes it, try the onboarding cost calculator on the Intalk home page, and see the pilot terms for how Intalk scopes them.

Frequently asked questions