Your Interview Scorecard Needs a “Not Observed” Option.

The number that was never earned

An engineering candidate completes two interviews. One interviewer evaluates debugging and communication; another spends the entire slot on architecture. The combined scorecard nevertheless gives the candidate a two out of five for incident response. No incident-response question was asked. The rating looks precise, but it describes the absence of an observation, not the candidate’s capability. ^{↗ ↗}

This is a measurement failure with operational consequences. An unobserved competency becomes a low score, the low score depresses an average, and the average becomes a rejection rationale. The panel has effectively transformed missing information into negative evidence. The fix is not to tell interviewers to be more generous. It is to make the data model admit uncertainty. ^↗

Use four distinct states instead of one numeric box

Every criterion should have a state: observed and scored; observed but insufficient evidence; not observed because it was outside this interviewer’s assignment; or not assessable because the assessment was interrupted or inaccessible. The last two states must not be converted into zeros. A weak response can legitimately receive a low anchored score; an unasked question cannot. ^↗

Attach the rating to a concrete observation. “Explained two production incidents, named the diagnostic sequence, and distinguished symptoms from root causes” is evidence. “Seemed senior” is an impression. The OPM structured-interview guidance emphasizes predefined job-related questions and rating benchmarks. The EEOC’s selection guidance reinforces the importance of job-related procedures. Neither supports manufacturing scores for skills that were never tested. ^{↗ ↗}

A coverage map before the first interview

Create a grid with competencies as rows and interview stages as columns. Assign a primary assessor to each required competency, a backup method, and a minimum acceptable evidence type. For example, the architecture interview might own trade-off reasoning while a live exercise owns debugging. If both assess communication, state what each will observe so duplicate scores are not mistaken for independent measurements. ^{↗ ↗}

Before the candidate enters the pipeline, ask a practical question: if one interviewer is absent, which competencies become unobserved? If the answer includes a must-have skill, the schedule needs a backup. A scorecard is only as complete as the observation plan that feeds it. ^{↗ ↗}

How to resolve missing evidence fairly

At the debrief, show coverage first, then scores. If a must-have criterion is unobserved, schedule one targeted, proportionate follow-up or use an already-approved equivalent piece of evidence. Do not ask the candidate to repeat an entire assessment day because one interviewer missed a question. Apply the same missing-evidence rule to every applicant in the same process. ^↗

Distinguish a genuine failure to demonstrate a skill after a fair opportunity from a process failure to provide that opportunity. A hiring manager can decline a candidate after documented low performance; they should not cite low performance when the evidence field is empty. Keep the follow-up question and scoring rubric stable across candidates. ^{↗ ↗}

The audit that finds false negatives

Pull 25 recent rejection scorecards and identify mandatory criteria with numeric scores but no supporting note. Then examine whether interviewers were actually assigned to assess those criteria. Count cases where the panel rejected on a criterion with no observation. This is not a statistical estimate of bias; it is a process-quality diagnostic. ^↗

For a monthly operating metric, report the percentage of decisions with complete coverage of mandatory competencies, the percentage requiring a targeted follow-up, and the number of rejections citing unsupported scores. A higher follow-up rate may initially be good news: it means the system stopped hiding missing evidence. ^↗

An implementation rule for tomorrow

Make “Not observed” a first-class field in the ATS or scorecard template. Require an evidence note for every numeric rating, and require a reason for each unobserved must-have criterion. Lock final rejection reasons until a reviewer has checked the coverage map. A well-designed process should make honest uncertainty easier to record than false certainty. ^↗

Keep the tool below as a planning aid, not a candidate-ranking algorithm. It estimates assessment coverage only; it does not infer anyone’s ability or decide whether an applicant should advance. ^↗

Run the operational check