A scorecard for human and AI calls

Build one scorecard around call outcomes, require evidence for every score, keep shared criteria identical for people and AI agents, and add a small set of AI-specific checks that only apply when software took the call. A clear definition of quality keeps human and AI calls aligned, and turns subjective reviews into consistent feedback teams can act on.

A separate AI scorecard sounds tidy, but scores stop being comparable, and clients who ask "how are my calls going?" get two answers built on different standards. Start by listing what must be true at the end of a good call:

• The caller's request was resolved, or correctly handed off.
• The right thing changed in the right system, and nothing else changed.
• The caller knows what happens next.
• Required statements and checks for this program were completed.

Weight outcome criteria highest, and keep auto-fail items rare: reserve them for things that are unacceptable regardless of everything else on the call. Most criteria apply the same way to anyone on the phone. A handful need separate treatment when an AI agent takes the call:

For outbound AI calls, add a consent check. The FCC has confirmed that AI-generated voices are "artificial" voices under the Telephone Consumer Protection Act, so prior express consent is required where the TCPA applies (FCC Declaratory Ruling FCC 24-17). Your scorecard can confirm the consent source was checked; whether that consent is sufficient is a question for your counsel.

Ground each score in what was said and done, not a summary. For every criterion, the reviewer or the automated scorer should point to the transcript lines or system record that justify the result. Disputes are then settled by looking at the cited moment instead of re-listening to the whole call, and reviewers who score without evidence become visible in calibration.

Write each criterion so evidence is possible. "Showed empathy" is hard to evidence; "acknowledged the problem before troubleshooting" can be pointed to. In the excerpt below, resolution is supported by the completed action and the customer's confirmation together.

Before scores affect coaching, compensation or client reporting:

1. Have several reviewers score the same set of calls independently.
2. Compare results criterion by criterion, not just total scores.
3. Where reviewers disagree, rewrite the criterion or add an example until they agree.
4. If you use automated scoring, include it as one more reviewer and compare it the same way.
5. Repeat calibration on a regular schedule, and whenever the scorecard changes.

Version the scorecard, so older calls keep the version that applied at the time. Roll it out on one program first: score human calls to set a baseline, add AI-specific criteria when AI agents start handling that program, and track how often disputed scores are overturned. A rising overturn rate usually means a criterion needs rewriting.

A scorecard like this is a good starting point for a TrackAgent call quality discussion. If you are also choosing an AI agent, the guide on how to evaluate an AI voice agent shows how to use the same criteria during vendor testing.