Validity, reliability, and feasibility of video observation of procedural skills assessment of surgery residents - Rabe-Baquiran, Rannie R.; Nemuel S.Fajutagana, MD, MHPEd (Uni Maas) (Manila National Teacher Training Center for the Health Professions, University of the Philippines Manila, 2022)
Introduction. Surgery is one of the branches of medicine that is complex and high-risk for errors and ensuring safety in the realm of surgical training is a challenge. It is during training that residents are highly prone to mistakes and errors. Training technical skills is therefore often seen as the most important strategy to reduce adverse events in surgery. Despite the essential role those technical skills play in surgery, few objective, evidence-based methods exist to help surgeons assess their performance during an operation. The need for a scalable approach to objectively evaluate the technical skill remains unmet. Video observation of procedural skills is a novel way of assessment of technical skills of residents as videos have been utilized in sports and aviation as part of learning, evaluation and -self-improvement. This study evaluated the use of actual surgical videos of general surgery residents (open surgery) using the concept of video observation of procedural skills (VOPS) formative assessment of technical skills in the operating room. Methodology. Video recording of open operations (2 MRM, 1 Mesh hernioplasty and 1 Open Cholecystectomy) performed independently by surgery residents were edited to remove personal and institutional identifiers and were subjected to evaluation by the self-rater, senior resident and consultant staff using validated tools derived from the Objective Structured Assessment of Technical Skills (OSATS) global rating scale, overall performance rating (GRS/OPS) and a procedure specific checklist. The videos were kept valid by maintaining the entire length of the videorecording without splicing from cutting to closing of the skin. Results. Statistical analysis of reliability using interrater agreement (IRA) showed weak concordance at 37% (GRS/OPS) and 40% (Procedure checklist) while intraclass correlation and Cronbach’s alpha showed excellent and high reliability for both the GRS/OPS and procedure checklist. Survey from the residents who used the head mounted action camera (n=8) showed majority agreed/strongly agreed that the action camera was easy to mount (100%) and light and comfortable (62%). Majority of the raters (n=30) agreed/strongly agreed that the videorecording was clear (77%), captured the important parts of the operation (73%), can identify strengths and weakness (93%), and was able to give an objective (87%), well thought (93%) and meaningful feedback (89%). Majority also agreed/strongly agreed video can be used to assess the technical skills of the residents and that it is a practical tool to use (83%, 96%). Conclusion. Video observation of procedural skills is a valid, reliable, and feasible tool for the formative assessment of procedural skills and can be used as an alternative to direct observation to address issues of logistics, single rater limitation, rater bias and provision of meaningful feedback. Recommendation. It is recommended that further studies with bigger population t and that a comparison between direct observation versus a video observation be done.
Thesis (Master in Health Professions Education)
Keywords: Video observation; Assessment -- Video-based.; Assessment -- Formative