Case study · Competency-based assessment
From zero learners certified to a defensible, staged path to competence.
Training for a high-difficulty, fine-motor clinical/technical procedure in a regulated environment was producing completions, but almost no certified, independent performers. I rebuilt the training and assessment model around Direct Observation of Procedural Skills (DOPS), calibrated trainers to a shared standard, and built a staged authorization pathway with a defensibility rule built into the design.
Regulated clinical/technical training environment
Zero learners were certified in the roughly 18 months before the redesign. The first learner was certified within three months of it.
Completion was happening. Certification was not.
from zero learners certified in ~18 months to the first certified learner within 3 months of the redesign
the approach was positively noted during an external regulatory/accreditation assessment
the approach was later requested and adapted by another institution
All results are from this training and assessment redesign.
The problem
Learners were completing training. Almost none were being certified.
In the roughly 18 months before the redesign, training sessions were running and learners were completing them, but zero learners were certified as independently competent in that period. The training told learners what to do. Assessment did not give trainers a consistent, defensible way to confirm they could actually do it, so sign-off almost never happened.
Without a shared standard, one trainer’s sense of “ready” could be different from another trainer’s, and there was no staged path that let a learner move from close supervision to independent performance in a way anyone could point to and defend later.
No shared standard
Different trainers could reach different conclusions about the same performance, with no calibrated standard to align them.
No staged pathway
There was no defined route from needing supervision to working independently, so progress was hard to observe or document.
Assessment that did not lead to certification
Learners were completing sessions, but the assessment process was not producing certified, independently competent performers.
The constraint
The skill was high-stakes and fine-motor. The assessment had to be defensible, not just kinder.
This was not a case where the fix was simply more encouragement or a softer rubric. The skill involved fine-motor precision with real consequences for getting it wrong, so the redesign had to produce assessment decisions that a trainer could stand behind and that would hold up if questioned later. Feedback needed to happen with the learner present, not after the fact, and needed to be signed by both the trainer and the learner so there was a shared, documented understanding of where the learner stood.
The hardest constraint was the edge case: a learner could score well across most criteria and still not be safe to work independently if they missed one of the core expectations. The design had to treat that as an automatic “not yet,” not a judgment call.
If any core expectation is missed, the learner is automatically Supervision Required, regardless of how well everything else scored. That rule had to be built into the form itself, not left to individual trainer discretion.
What I built
A DOPS-based redesign with a staged, defensible path to independence.
I rebuilt the assessment around Direct Observation of Procedural Skills (DOPS), so every assessment is a structured, in-person observation against the same core expectations and rubric criteria, rather than an informal judgment call at the end of a session. Trainers were calibrated to that same standard before they assessed learners, so what counted as “ready” stopped depending on which trainer was in the room.
On top of that standard, I built a staged authorization pathway: Supervision Required, Partial Supervision, Independent, and Competent. A learner’s stage is set by the rubric, and one rule overrides everything else: if any core expectation is missed, the learner is automatically Supervision Required, no matter how the rest of the assessment scored. Every assessment is completed with the learner present, and the feedback is signed by both the trainer and the learner.
- StandardAssessment rebuilt around DOPS: structured, direct observation against the same core expectations and rubric every time.
- CalibrationTrainers were calibrated to the same standard before assessing learners, closing the gap between what different trainers would sign off on.
- PathwayA staged authorization pathway (Supervision Required, Partial Supervision, Independent, Competent) gives learners a clear, documented route to independence.
- Sign-offFeedback is completed with the learner present and signed by both trainer and learner, with any missed core expectation auto-triggering Supervision Required.
Artifacts
The staged pathway, and the assessment form that drives it.
The staged authorization pathway. While a learner is still being assessed, a missed core expectation overrides everything else and returns them to Supervision Required, no matter how the rest of the assessment scored. Once a learner is certified as Competent, they are no longer being assessed against this form.
Aligning trainers to a shared standard solved who was doing the assessing. The next problem was making sure the tool they were using actually measured the right thing.
The Assessment Tool, in Full
The rebuilt three-tier competency form, shown at the step level rather than the procedure level, so the structure carries over to any hands-on technical skill. Instructions: complete the entire form at the end of the session. The trainee feedback section must be completed with the learner present.
Core Expectations
- Wears appropriate PPE
- Keeps sanitary environment
- Organized
- Checks appropriate documentation
- Demonstrates respectful behaviour
Rubric
| Supervision Required | Partial Supervision | Independent |
|---|---|---|
| Target site identification or preparation is inconsistent and requires correction | Target site identification and preparation are adequate, with occasional guidance needed | Target site is consistently identified and prepared without guidance |
| Approach and technique are inconsistent, with little to no improvement seen during the session | Approach and technique are adequate, and improvement is seen during the session | No improvements required for approach and technique |
| Does not reliably confirm correct positioning before proceeding | Confirms correct positioning but requires occasional prompting | Consistently confirms correct positioning before proceeding, without prompting |
| Execution is uncontrolled or rushed, increasing risk | Execution is controlled with occasional lapses | Execution is consistently controlled and appropriately paced |
| Stabilization is incomplete or requires intervention | Stabilization is adequate with minor prompting | Stabilization is consistently thorough without prompting |
| Misses signs of distress or complication | Recognizes most signs of distress or complication, with occasional missed cues | Consistently monitors for, and responds appropriately to, signs of distress or complication |
Definitions
- Supervision Required3 or more selections in the Supervision Required column. The learner must always be with a certified trainer when practicing.
- Partial Supervision4 or more selections in the Partial Supervision column. The learner can practice with experienced colleagues and may practice alone under certain conditions, listed in the feedback section.
- Independent4 or more selections in the Independent column, none in Supervision Required. The learner can perform the task independently.
- Any other combinationA mix that meets none of the thresholds above defaults to Partial Supervision.
- Defensibility ruleIf any core expectation is not completed, the learner is automatically considered Supervision Required, regardless of other selections.
Trainee feedback (completed with the learner present)
For trainer only
Signature ______________________
Signature ______________________
Most training programs assess a hands-on technical skill with one of two tools: a supervisor’s general sign-off, or a written knowledge check. Neither answers the actual question, which is whether the person can perform the skill under real conditions. Someone can describe a procedure correctly and still be unable to execute it when the situation isn’t cooperating.
I designed this assessment around three things a task needs to actually measure competence: it has to resemble the real task, not a simplified version of it; it has to require judgment under pressure, not recall; and the learner needs to know the standard before they’re evaluated against it, not find out after. That shows up directly in the form. The rows are the real observable steps of the procedure, things a trainer watches happen rather than facts a trainee could recite. The three-tier scale states the criteria for each level up front instead of revealing them at the end, so feedback is against a known standard rather than a surprise verdict.
The main risk with this kind of assessment is consistency across trainers, since it depends more on the evaluator’s judgment than a standardized test does. I addressed that by setting numeric thresholds for each overall rating (three or more selections in one tier, four or more in another, and a set default for any mix in between) instead of leaving the final call to a trainer’s general impression. It doesn’t remove judgment from any single row, but it keeps the aggregation consistent, which is usually where inconsistency actually creeps in.
The underlying goal is distinguishing between a learner completing training and a learner demonstrating competence. Those are not the same claim, and a form that treats them as interchangeable will keep producing completions without producing anyone you’d actually trust to work alone.
A completion record tells you someone showed up. This form is built to tell you something closer to the truth.
Results
From zero certified to a standard other people wanted to borrow.
Zero learners were certified in the roughly 18 months before the redesign. The first learner was certified within three months of it, using the same staged pathway and defensibility rule.
The approach was positively noted during an external regulatory/accreditation assessment.
The approach was later requested and adapted by another institution.
Disclosure: This project reformed the training and assessment program for mouse intravenous injection procedures in a regulated research facility. The design challenge was broader: how to teach, assess, and document a high-stakes technical skill consistently across trainers and learners.
Role fit
Where this work maps.
This project sits at the intersection of competency-based assessment design and trainer standardization: building a defensible path from supervised practice to certified independence.
- Instructional Designer, Assessment & Competency
- Learning & Development Specialist
- Clinical/Technical Training Coordinator
- Compliance & Training Records Specialist
- Competency Framework / Credentialing roles