Two reviewers, one customer, opposite decisions. Fix the gap before it costs you.

One analyst approves the customer. Another, given the same case, asks for more evidence. A third says the account should have been declined.

You may have a training issue. You may also have an ambiguous policy, incomplete case evidence or a screen that hides the detail the decision depends on.

Adding another reviewer will not resolve those problems by itself. First find out why reasonable people are reaching different conclusions.

Freeze the question before comparing the answers

Choose the decision you want to examine. Is the analyst resolving a name match, accepting address evidence or recommending an action on a transaction? Avoid combining unrelated review tasks into one exercise.

Give participants the same evidence snapshot, the same policy version and the same question. If one analyst can see a later document and another cannot, the exercise is comparing access to information rather than judgement.

Use permitted historical or synthetic case material in an authorised setting. A training review should not alter a live customer's status, send a message or trigger a payment restriction.

Select cases that reveal the problem

Include routine approvals, routine adverse decisions, unresolved cases and examples near your policy's boundaries. Do not select only famous fraud cases your team already recognises.

Label how you selected the sample. A deliberately difficult set is useful for finding disagreement, but it cannot establish your whole operation's error rate.

Make sure the evidence reflects what analysts actually receive. An immaculate training file can hide the problems caused by missing links, confusing timestamps or poor case organisation in production.

Ask for a decision and its reason independently

Have each analyst record the decision, evidence relied on, applicable policy and anything still missing before discussing the case with colleagues.

Independent first answers reduce the chance that everyone follows the most senior voice. The explanation is as valuable as the selected outcome. Two people can choose review for entirely different reasons, only one of which matches the policy.

An illustrative exercise might include a company-name match that one analyst clears using a registration-number mismatch, while another overlooks that field and escalates. The outcome difference suggests a specific evidence-use problem. “Be more careful” is not a useful fix.

Classify the cause before assigning training

Sort disagreements into practical categories:

• Evidence missing or visible to only some reviewers.

• Policy unclear, contradictory or silent on the case.

• A clear policy applied incorrectly.

• A tool or interface making the relevant detail difficult to use.

• A judgement call that needs a defined escalation owner.

Do not rewrite policy to match the most common answer automatically. The majority can be wrong. Have the responsible owner resolve the intended approach and explain the evidence supporting it.

For genuinely uncertain cases, an agreed escalation can be the correct outcome. Forcing a binary answer may produce consistency on paper while hiding the uncertainty.

Measure agreement without calling it accuracy

Agreement tells you whether reviewers reached the same result on the cases tested. It does not establish whether they were right.

A team can consistently clear a pattern it does not recognise. It can also disagree because one person spotted relevant evidence. Compare the exercise with confirmed downstream outcomes where available, keeping inconclusive cases distinct.

Our guide to decision quality covers outcome evidence and its completeness. Calibration serves a different purpose: finding differences in how people interpret the evidence and policy they already have.

Track disagreement by task and cause, not only as one blended score. Record the sample size, composition and unresolved cases so the result remains interpretable.

Make one improvement you can assess again

Choose an action tied to the cause. That might be a clearer policy example, a required evidence field, an interface change or focused training on a repeated error.

Use a fresh set of comparable cases to see whether the improvement helped. Do not simply repeat cases whose answers everyone has memorised.

Keep a record of the agreed interpretation, effective date and policy owner. Where the exercise reveals a material live-control weakness, assess its potential impact through your normal governance process. Do not silently rewrite earlier decisions.

Where Toden can help

Toden's case tools support investigation records, assignment and recorded outcomes. Those records can provide a foundation for reviewing how a decision was reached.

This article describes an operational calibration exercise. It does not claim a built-in blind-review module, an automatic analyst score or an independent quality-assurance certification.

Already using Toden? Pick one recurring review task and check whether two authorised analysts can reconstruct the same evidence and policy from its case record. If they cannot, talk to us about the missing context before adding more review steps.