When AI Screening Disagrees With Your Recruiter

Begin with two independent observations

Reviewers should make a first-pass assessment against predefined requirements before seeing a model recommendation, whenever the workflow makes this feasible. The AI evaluation should likewise use the same job criteria without incorporating the reviewers initial rating. Only after both are recorded should the system display the disagreement. This reduces the risk that the first visible opinion anchors the second.

Keep the outputs separated: the recruiter judgment, the AI finding, the source material cited by each, and the underlying requirement. Avoid a single blended score such as "82 percent match" unless the percentage has a validated and well-understood meaning. Ashby documentation already distinguishes its AI criteria met percentage from human application review ratings. That distinction is a helpful design precedent.

Disagreement is not only a difference in ranking. Two reviewers can assign the same label for different reasons. For example, the recruiter approves a resume because of a strong employer name, while the model approves it because of a keyword that appears in a hobby section. Agreement in score can conceal disagreement in evidence.

Classify the reason before discussing the candidate

A disciplined triage starts with six categories:

1. Extraction fault. The skill or date exists in the source CV but did not appear in parsed fields.
2. Requirement mismatch. The model used a broad synonym where a narrow production requirement was intended, or the reviewer applied an unofficial requirement.
3. Negation or scope error. A statement such as "no production Kubernetes ownership" was treated as positive evidence.
4. Seniority or context dispute. Evidence of participation was read as evidence of independent ownership.
5. Outdated information. The CV and application answers describe different points in time.
6. Unresolved judgment. Both sides point to credible but incomplete evidence and the question requires a structured follow-up.

The classification matters because the fix depends on the cause. Extraction faults call for parsing QA. Requirement mismatches call for job calibration. Scope disputes call for better evidence extraction and more precise questions. Unresolved judgment calls for a human review step, not a forced numerical compromise.

An example with an actual disagreement

A role needs someone who has operated distributed systems in production and participated in incident response. The CV says: "Supported service dashboards; coordinated with an on-call team during two launches." An AI summary claims "production incident ownership" because it recognizes related phrases. The recruiter marks the requirement unproven because support and coordination are not equivalent to being accountable for a production incident.

The source text supports exposure to operational work, but not the stronger claim. A fair resolution is not to reject the candidate or endorse the AI. Mark the criterion as "needs clarification" and ask a standardized question about the exact incident, responsibility, decisions made, and outcomes. If the response confirms ownership, the evidence can be updated. If not, the original claim remains unsupported.

Now reverse the scenario. A recruiter dismisses a candidate because the CV never contains the exact word "Kubernetes." The source describes container orchestration with kubectl, Helm charts and managed cluster upgrades. A model flags an adjacent skill worth review. This time, the useful next step is to inspect the quoted operational tasks and determine if the experience meets the actual job requirement. The AI has surfaced a question, not independently validated proficiency.

The most important status is often unresolved

A useful review queue must represent uncertainty explicitly. A binary fit or not-fit label pushes missing evidence into a negative conclusion. A queue with "source confirmed," "contradicted," "requires clarification" and "out of scope" keeps the next action visible. It also makes quality measurement possible because unanswered cases are not silently counted as correct rejects.

Teams need an escalation rule. For high-impact requirements, a second trained reviewer should check the source and the approved job criterion. For routine formatting errors, a designated operations reviewer can correct extraction fields. For suspected fairness problems, sample comparable cases to see whether the same standard is applied consistently. The escalation owner should be named, and reviewer notes should be brief enough to audit without exposing unnecessary private information.

Interactive: disagreement triage

Use the workbench below to record a single contested criterion. It does not score candidates or claim to verify credentials. Instead it helps the reviewer decide whether the available evidence supports the stated claim and what must happen next. Input stays in this browser session; do not paste confidential data unless your local policy permits it.

A compact disagreement review protocol

Step one: capture the job criterion as it stood when the candidate applied. Do not rewrite the requirement after reading the profile. Step two: preserve the exact candidate source excerpt and, separately, the model statement. Step three: identify whether the dispute is about text extraction, meaning, relevance, seniority or a possible contradiction. Step four: ask what additional evidence could resolve the case. Step five: have an authorized human record the next action and its rationale.

Use the same process whether the AI favors the candidate or the recruiter does. A system that reviews only candidates promoted by AI cannot tell you how many strong applicants it missed. Sample the lower-ranked queue and review model disagreements in both directions.

How to measure disagreement quality

An override rate alone says almost nothing. A recruiter can override an excellent system because the requirement was ambiguous. Or reviewers may rarely override a flawed system because recommendations look authoritative. Measure the resolution of reviewed disagreements using a carefully adjudicated reference standard: extraction corrected, requirement changed, claim confirmed, claim contradicted or still unresolved.

Track time to resolution and downstream outcomes for each category. Record when a corrected parser changes a decision. Review whether a new skill synonym increases false positives. Importantly, separate a recruiters opinion from an independently reviewed finding. Otherwise your evaluation may only measure agreement with whoever was on shift.

An evaluation set should include both cases where the AI ranked someone lower than a reviewer and cases where it ranked someone higher. Include CVs with ambiguous dates, unusual role titles, non-linear careers, strong adjacent skills and explicit negations. The purpose is to test judgment boundaries, not reward the system for repeating the dominant reviewers preferences.

FAQ

Who should win when the model and recruiter disagree?

Neither by default. Compare source evidence against the same approved criterion. Assign a documented next action when the information is incomplete, with an authorized human accountable for the hiring decision.

Does a confident model explanation make the finding reliable?

No. An explanation can be persuasive while being wrong. Check that the supporting excerpt exists, belongs to the candidate, refers to the right period and actually entails the claim being made.

Should every disagreement trigger a new interview?

No. Many disagreements can be resolved by correcting the parsed text, clarifying the job requirement or reading the source. Candidate outreach should be focused on material unresolved questions and handled consistently.

Turn disagreement into a controlled learning loop

The aim of human-in-the-loop screening is not to get reviewers to accept more AI suggestions. It is to improve both sides of the process: clearer requirements, fewer extraction mistakes, better evidence, traceable corrections and fairer candidate review. The best disagreement is one that produces a verifiable improvement without silently changing the standard for the next applicant.