Your Work-Sample Test Changed Halfway Through the Hiring Round.

The “small edit” that changes the bar

In week one, product-design candidates receive a two-hour exercise with a short data set. In week three, the team adds a more complex dataset and a stakeholder memo. The rubric still says “problem solving: 1–5.” Candidates from both weeks are ranked together as though they completed the same test. A higher score may now reflect a different assignment rather than stronger ability. ^↗

Work samples can provide useful job-related evidence, but only if administration and scoring are controlled. OPM describes work samples as simulations of tasks performed on the job; its assessment strategy guidance emphasizes reliability, validity and consistent administration. A prompt revision can alter all three. ^{↗ ↗}

What counts as a material version change?

Classify edits before publication. Cosmetic edits fix typos or accessibility without changing the task. Clarifying edits resolve ambiguity but may still affect candidate behavior. Material edits change required deliverables, time allowance, permitted tools, data difficulty, evaluator instructions or rating anchors. Record a new version for any material change. ^{↗ ↗}

Keep the version identifier on the invitation, task packet, scoring form and ATS record. If the assessment is delivered through a third-party platform, verify that the platform preserves the exact prompt and settings candidates received. A URL to a living document is not reliable evidence of what was administered. ^↗

Use equivalence evidence, not confidence

Before combining results from different versions, ask whether both assess the same competencies, provide similar opportunity to demonstrate them and produce comparable scores. Pilot both versions with trained evaluators and compare distributions, time-to-complete, common failure modes and rubric interpretations. A handful of examples is a warning signal, not formal psychometric validation. ^{↗ ↗}

If equivalence cannot be supported, avoid a single numeric ranking across versions. You may score within each version and conduct a targeted common assessment for finalists, or document a transition boundary and evaluate the old cohort under its original standards. Any approach should be consistent and proportionate. ^↗

Protect accessibility while preserving the construct

An adjustment such as an accessible file format or appropriate assistive technology should not automatically be classified as a different competence test. The question is whether the adjustment changes the job-related skill being measured. If the task accidentally measures typing speed when the role requires analysis, removing that barrier may improve validity. ^↗

Build an adjustment pathway into the test plan, communicate it before the assessment and protect unnecessary medical or personal details. EEOC, Acas and government accessibility guidance provide jurisdiction-specific context; local legal review is necessary when formal accommodation obligations apply. ^{↗ ↗}

A version-change decision record

Log six fields: old version, new version, changed elements, competencies affected, comparability decision and treatment of candidates already assessed. Add the approver and effective date. If a flaw is discovered after candidates have taken the test, document whether all candidates need a fair retest or whether the flawed item should be excluded uniformly. ^↗

Never silently rewrite a scoring rubric after reviewing individual candidates’ results. That creates an obvious incentive to move the bar toward a preferred applicant. Freeze the rubric before evaluating responses, and document any exceptional correction with an independent reviewer. ^{↗ ↗}

A useful metric: version mixing

Report the number of candidates per assessment version and the percentage of final comparison sets that contain mixed versions without an approved equivalence decision. The aim is not zero revisions. It is zero undocumented comparisons. ^↗

The interactive worksheet estimates how many comparisons need a version review. It is a triage aid, not proof that two assessment versions are equivalent. ^↗

Run the operational check