QA Calibration Sessions: Why Evaluators Disagree and How to Align Them
If the same call scores 72 from one evaluator and 88 from another, the problem isn't the agent. A practical guide to running calibration sessions that make quality scores fair and trusted.
Picture a contact center agent who scores 72 out of 100 on this week's quality evaluation, then learns that a colleague scored 88 on a very similar call. The only difference is who evaluated it. At that moment the agent stops trusting the whole quality program, and every conversation about "development" loses its meaning.
That is exactly what calibration sessions fix: everyone scoring with the same ruler.
What is a calibration session?
A regular meeting where evaluators and team leaders review the same interactions, score them independently, then compare results and discuss the differences until they agree on one interpretation of every item on the scorecard.
The goal isn't for anyone to "win" the argument. It's to reach a shared understanding that gets documented and applied to every evaluation that follows.
Why do evaluators disagree in the first place?
- Vague scorecard items. "Agent showed empathy" has ten possible readings. Is "I understand" enough, or does the agent need to connect it to the customer's situation?
- Different backgrounds. A team leader from sales listens for selling opportunities; one from technical support listens for accuracy.
- The halo effect. When a call opens well, evaluators tend to go easy on the rest, and vice versa.
- Knowing the agent. Scoring someone you know and respect is subconsciously different from scoring a new hire.
- Changing procedures. A new process went out last week and not every evaluator has read it.
What's an acceptable target?
SQM Group, one of the best-known customer experience research firms for contact centers, says the primary goal of calibration is to keep evaluator scores within a 5% variance. It recommends sessions at least monthly, and preferably weekly, plus a session whenever the scorecard or its metrics change.
In practice: if the average score on a call is 80, evaluator scores should fall roughly between 76 and 84. Anyone outside that range needs to review how they score.
Steps for a successful calibration session
Before the session
- Pick 3 to 5 varied interactions. Don't choose only excellent or terrible calls. The "grey" calls reveal the real disagreements.
- Have each participant score independently and submit before the session. No discussion, no hints.
- Put all scores in one table showing who gave what on every item.
During the session
- Appoint a neutral facilitator to manage time and discussion, ideally from the quality team.
- Start with the items with the widest spread, not the total score. Totals can match while the details differ.
- Ask "why?", not "who's wrong?" What did each evaluator hear that led to their score?
- Agree on a written interpretation for every disputed item, with an example from the call.
After the session
- Update the scoring guide with the new interpretations within 48 hours and share it with every evaluator.
- Log the results of every session in one place and track each evaluator's variance over the months.
- Share the takeaways with agents. Knowing evaluators are calibrated regularly raises their trust in the program.
Should agents join the session?
SQM suggests involving agents, supervisors and evaluators together, and adding the customer's own view from the post-call survey as a fourth perspective. Including agents has two benefits: they understand the standard from the inside, and they sometimes add context the recording doesn't show, such as a system limitation or an unclear procedure.
Start with experienced, high-performing agents, then widen the circle gradually.
Common mistakes
- Turning the session into a trial of the agent on the call instead of a review of the standard.
- Letting the most senior person settle every disagreement. Others will stop sharing their real views.
- Calibrating only the total score, which hides the real disagreements in the items.
- Not documenting decisions, so the same argument returns next session.
What this means for your team
A quality score agents don't trust develops no one. Start this month with one calibration session on three calls, and calculate the gap between the highest and lowest score on each. That number alone will tell you how much work you have ahead.