Video-Based Assessment of Operative Proficiency: A Pilot Study of Neurosurgical Skill Assessment

Introduction: Tools for assessing trainee progress and performance are an important part of competency-based education. Neurosurgical residencies do not yet have standardized assessment instruments for this purpose. In this prospective pilot study, we sought to develop and test a video-based assessment of operative techniques that can accurately predict neurosurgical operative experience in a cadaver-based operative setting.

Methods: Neurosurgical residents at our institution from postgraduate year (PGY) 1-5 were recruited. Participants performed a lumbar decompression with facetectomy and a unilateral cervicothoracic decompression and fusion on a cadaver specimen, which was recorded using a head-mounted camera. This video was then evaluated by blinded reviewers using our investigational assessment tool, which is comprised of qualitative and quantitative components. The qualitative component includes both a general surgical skill evaluation (mOSAT) as well as a checklist for the minimum necessary steps for that specific procedure (rubric). The quantitative component examines how participants utilize their time in order to better characterize areas that may be refined. The correlation between perative time (OT), instrument efficiency (IE), and overall operative efficiency (OE) from the quantitative assessment and the mOSAT and rubric scores from the qualitative assessment with experience level (measured as PGY level) using linear regression models and Mann-Whitney U tests.

Results: There were 12 residents enrolled. Increasing PGY correlated with decreased OT (ß= -18.50, p < 0.01), improved IE (ß = -23.26, p < 0.01), and OE (ß = -41.76, p < 0.01). PGY3 residents performed the procedure with more OT (p < 0.05), less IE (p = 0.01), and less OE (p = 0.01) compared to more senior residents. Increasing PGY correlated with increasing mOSAT (ß= 4.45, p < 0.01), rubric scores (ß = 1.57 p < 0.01), and composite scores (ß = 6.02, p < 0.01). PGY3 residents obtained lower mOSAT (p < 0.01), and composite scores (p < 0.01) compared to more senior residents.

Conclusion : Results of our pilot study suggest that our novel assessment instrument may accurately discriminate between early and more advanced learners. Further evidence of internal and external validity is needed, as well as correlations with actual operative performance.