Opinion

Measuring competency is a challenge all training programs must address to ensure that their residents can practice independently after graduation. Over the course of training, achievement of competence should be measured to ensure that residents can perform specific actions independently, so called entrustable professional activities (EPA). These activities are complex and heterogeneous, involving the application of knowledge as well as the performance of specific psychomotor skills. In addition to the 6 core competencies established by Accreditation Council for Graduate Medical Education (ACGME), many specialties now have established milestones in training to track the development of specific competencies over the course of training. These milestones describe discrete observable behaviors that correlate with different levels of performance from novice to expert. The Emergency Medicine (EM) Milestones are the current standard that all EM residencies must use to measure progress in resident learning. Residencies are required to report measurements of milestone achievement for each of their residents at 6 month intervals throughout training. Evidence supporting specific milestone level achievement must be obtained from both the clinical learning environment as well as other activities such as simulation training.

Observations in the clinical learning environment, especially in the emergency department where patient volumes can limit the duration of direct observation of residents, is challenging. Simulation-based assessment is an effective method of measuring competency in the performance of specific skills such as central line placement and intubation in addition to the application of clinical knowledge to diagnose and treat specific conditions. Simulation offers the opportunity for indepth extended observation of clinical performance in a riskfree setting. These assessments often involve a checklist-based assessment where specific observable behaviors are measured with the goal of performing all critical actions relevant to the case or situation. While there is ample evidence that checklists of critical actions have a high degree of interobserver reliability, it is unclear whether milestone levels can be applied to simulation-based assessment as an additional or alternative means of measuring performance.

Wittels et al. examined the interobserver agreement of milestone based assessments compared to critical action checklists when used by faculty to measure resident performance during two simulation scenarios. Faculty using assessment rubric based on the EM RRC milestone language had poor inter-rater agreement in rating resident performance. This finding highlights the importance of using checklists of critical actions for reliable assessment of competence, especially when it comes to high stakes assessments. In their current form, the EM milestones related to patient care skills lack the specificity of language required to ensure adequate reliability. Milestone ratings may be used as a pedagogical tool to highlight the next level of performance to a resident learner and this may be useful in providing formative feedback to residents in conjunction with critical action checklists.