Breakout Session
Free Access
Assessing Medical Knowledge of Emergency Medicine Residents
Nikhil Goyal MD,
Amer Aldeen MD,
Katrina Leone MD,
Jonathan S. Ilgen MD, MCR,
Jeremy Branzetti MD,
Chad Kessler MD, MHPE,
Corresponding Author
Nikhil Goyal MD
Department of Emergency Medicine, Henry Ford Hospital, Detroit, MI
Address for correspondence and reprints: Nikhil Goyal, MD; e-mail: [email protected].Search for more papers by this authorAmer Aldeen MD
Department of Emergency Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL
Search for more papers by this authorKatrina Leone MD
Department of Emergency Medicine, Oregon Health & Science University, Portland, OR
Search for more papers by this authorJonathan S. Ilgen MD, MCR
Department of Emergency Medicine, University of Washington, Seattle, WA
Search for more papers by this authorJeremy Branzetti MD
Department of Emergency Medicine, University of Washington, Seattle, WA
Search for more papers by this authorChad Kessler MD, MHPE
Department of Emergency Medicine, University of Illinois–Chicago, Chicago, IL
Search for more papers by this authorNikhil Goyal MD,
Amer Aldeen MD,
Katrina Leone MD,
Jonathan S. Ilgen MD, MCR,
Jeremy Branzetti MD,
Chad Kessler MD, MHPE,
Corresponding Author
Nikhil Goyal MD
Department of Emergency Medicine, Henry Ford Hospital, Detroit, MI
Address for correspondence and reprints: Nikhil Goyal, MD; e-mail: [email protected].Search for more papers by this authorAmer Aldeen MD
Department of Emergency Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL
Search for more papers by this authorKatrina Leone MD
Department of Emergency Medicine, Oregon Health & Science University, Portland, OR
Search for more papers by this authorJonathan S. Ilgen MD, MCR
Department of Emergency Medicine, University of Washington, Seattle, WA
Search for more papers by this authorJeremy Branzetti MD
Department of Emergency Medicine, University of Washington, Seattle, WA
Search for more papers by this authorChad Kessler MD, MHPE
Department of Emergency Medicine, University of Illinois–Chicago, Chicago, IL
Search for more papers by this authorThe list of breakout session participants can be found as the appendix of a related article on page 1486.
This paper reports on a workshop session of the 2012 Academic Emergency Medicine consensus conference, “Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success,” May 9, 2012, Chicago, IL.
The authors have no relevant financial information or potential conflicts of interest to disclose.
Abstract
The Accreditation Council for Graduate Medical Education (ACGME) requires that emergency medicine (EM) residency graduates are competent in the medical knowledge (MK) core competency. EM educators use a number of tools to measure a resident's progress toward this goal; it is not always clear whether these tools provide a valid assessment. A workshop was convened during the 2012 Academic Emergency Medicine consensus conference “Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success” where assessment for each core competency was discussed in detail. This article provides a description of the validity evidence behind current MK assessment tools used in EM and other specialties. Tools in widespread use are discussed, as well as emerging methods that may form valid assessments in the future. Finally, an agenda for future research is proposed to help address gaps in the current understanding of MK assessment.
References
- 1Wagner MJ, Thomas HA Jr. Application of the medical knowledge general competency to emergency medicine. Acad Emerg Med. 2002; 9: 1236–41.
- 2Stahmer SA, Ellison SR, Jubanyik KK, et al. Integrating the core competencies: proceedings from the 2005 Academic Assembly consortium. Acad Emerg Med. 2007; 14: 80–94.
- 3Hobgood C, Promes S, Wang E, Moriarity R, Goyal DG, Council of Emergency Medicine Residency Directors. Outcome assessment in emergency medicine--a beginning: results of the Council of Emergency Medicine Residency Directors (CORD) emergency medicine consensus workgroup on outcome assessment. Acad Emerg Med. 2008; 15: 267–77.
- 4 Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. Common Program Requirements. Available at: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/110emergencymed07012007.pdf. Accessed Nov 3, 2012.
- 5Frank JR, Jabbour M (eds). Report of the CanMEDS Phase IV Working Groups. Ottawa, Canada: The Royal College of Physicians and Surgeons of Canada, 2005.
- 6 2011 EM Model Review Task Force. The 2011 Model of the clinical practice of emergency medicine. Acad Emerg Med. 2012; 19: e19–40.
- 7Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review. Acad Med. 2009; 84: 301–9.
- 8Sherbino J, Bandiera G, Frank JR. Assessing competence in emergency medicine trainees: an overview of effective methodologies. CJEM. 2008; 10: 365–71.
- 9Wenghofer E, Klass D, Abrahamowicz M, et al. Doctor scores on national qualifying examinations predict quality of care in future practice. Med Educ. 2009; 43: 1166–73.
- 10 American Board of Medical Specialties. Research Bibliography. Available at: http://www.certificationmatters.org/research-supports-benefits-of-board-certification/research-bibliography.aspx. Accessed Sep 12, 2012.
- 11Chen J, Rathore SS, Wang Y, Radford MJ, Krumholz HM. Physician board certification and the care and outcomes of elderly patients with acute myocardial infarction. J Gen Intern Med. 2006; 21: 238–44.
- 12Haas JS, Orav EJ, Goldman L. The relationship between physicians' qualifications and experience and the adequacy of prenatal care and low birthweight. Am J Public Health. 1995; 85(8 Pt 1): 1087–91.
- 13Norcini JJ, Lipner RS, Kimball HR. Certifying examination performance and patient outcomes following acute myocardial infarction. Med Educ. 2002; 36: 853–9.
- 14Prystowsky JB, Bordage G, Feinglass JM. Patient outcomes for segmental colon resection according to surgeon's training, certification, and experience. Surgery. 2002; 132: 663–70.
- 15Silber JH, Kennedy SK, Even-Shoshan O, et al. Anesthesiologist board certification and patient outcomes. Anesthesiology. 2002; 96: 1044–52.
- 16Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine. Ann Intern Med. 1989; 110: 719–26.
- 17Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler Publishers, 1998.
- 18Dalton TJ, Korte RC. 2012 In-Training Examination Results. East Lansing, MI: American Board of Emergency Medicine, April 2012.
- 19Frederick RC, Hafner JW, Schaefer TJ, Aldag JC. Outcome measures for emergency medicine residency graduates: do measures of academic and clinical performance during residency training correlate with American Board of Emergency Medicine test performance? Acad Emerg Med. 2011; 18(Suppl 2): S59–64.
- 20Thundiyil JG, Modica RF, Silvestri S, Papa L. Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents? J Emerg Med. 2010; 38: 65–9.
- 21Gene Hern H Jr, Wills C, Alter H, et al. Conference attendance does not correlate with emergency medicine residency in-training examination scores. Acad Emerg Med. 2009; 16(Suppl 2): S63–6.
- 22FitzGerald JD, Wenger NS. Didactic teaching conferences for IM residents: who attends, and is attendance related to medical certifying examination scores? Acad Med. 2003; 78: 84–9.
- 23Margolis MJ, Clauser BE, Winward M, Dillon GF. Validity evidence for USMLE examination cut scores: results of a large-scale survey. Acad Med. 2010; 85(10 Suppl): S93–7.
- 24Sevensma SC, Navarre G, Richards RK. COMLEX-USA and in-service examination scores: tools for evaluating medical knowledge among residents. J Am Osteopathic Assoc. 2008; 108: 713–6.
- 25Beeson MS, Jwayyed S. Development of a specialty-wide web-based medical knowledge assessment tool for resident education. Acad Emerg Med. 2006; 13: 337–40.
- 26Barrows HS, Feltovich PJ. The clinical reasoning process. Med Educ. 1987; 21: 86–91.
- 27Charlin B, Boshuizen HP, Custers EJ, Feltovich PJ. Scripts and clinical reasoning. Med Educ. 2007; 41: 1178–84.
- 28Charlin B, Tardif J, Boshuizen HP. Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Acad Med. 2000; 75: 182–90.
- 29Charlin B, Roy L, Brailovsky C, Goulet F, van der Vleuten C. The Script Concordance Test: a tool to assess the reflective clinician. Teach Learn Med. 2000; 12: 189–95.
- 30Lubarsky S, Charlin B, Cook DA, Chalk C, van der Vleuten CP. Script concordance testing: a review of published validity evidence. Med Educ. 2011; 45: 329–38.
- 31Humbert AJ, Besinger B, Miech EJ. Assessing clinical reasoning skills in scenarios of uncertainty: convergent validity for a Script Concordance Test in an emergency medicine clerkship and residency. Acad Emerg Med. 2011; 18: 627–34.
- 32Goulet F, Jacques A, Gagnon R, Charlin B, Shabah A. Poorly performing physicians: does the Script Concordance Test detect bad clinical reasoning? J Contin Educ Health Prof. 2010; 30: 161–6.
- 33Page G, Bordage G. The Medical Council of Canada's key features project: a more valid written examination of clinical decision-making skills. Acad Med. 1995; 70: 104–10.
- 34Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills. Acad Med. 1995; 70: 194–201.
- 35Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010; 85: 1589–602.
- 36Dillon GF, Clauser BE. Computer-delivered patient simulations in the United States Medical Licensing Examination (USMLE). Simul Healthc. 2009; 4: 30–4.
- 37Clyman S, Melnick D, Clauser BE. Computer-based case simulations from medicine; assessing skills in patient management. In: A Tekian, C McGuire, BE Clauser (eds). Innovative Simulations for Assessing Professional Competence: from Paper-and-pencil to Virtual Reality. Chicago, IL: University of Illinois, Department of Medical Education, 1999, pp 29–41.
- 38Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009; 302: 1316–26.
- 39Hawkins RE, Sumption KF, Gaglione MM, Holmboe ES. The in-training examination in internal medicine: resident perceptions and lack of correlation between resident scores and faculty predictions of resident performance. Am J Med. 1999; 106: 206–10.
- 40 American Board of Internal Medicine. Mini-CEX. Available at: http://www.abim.org/program-directors-administrators/assessment-tools/mini-cex.aspx. Accessed Sep 12, 2012.
- 41Shayne P, Gallahue F, Rinnert S, et al. Reliability of a core competency checklist assessment in the emergency department: the Standardized Direct Observation Assessment Tool. Acad Emerg Med. 2006; 13: 727–32.
- 42Donato AA, Pangaro L, Smith C, et al. Evaluation of a novel assessment form for observing medical residents: a randomised, controlled trial. Med Educ. 2008; 42: 1234–42.
- 43Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial. J Gen Intern Med. 2009; 24: 74–9.
- 44Cook DA, Beckman TJ, Mandrekar JN, Pankratz VS. Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Adv Health Sci Educ Theory Pract. 2010; 15: 633–45.
- 45Short MW, Jorgensen JE, Edwards JA, Blankenship RB, Roth BJ. Assessing intern core competencies with an objective structured clinical examination. J Grad Med Educ. 2009; 1: 30–6.
- 46Ward M, Gruppen L, Regehr G. Measuring self-assessment: current state of the art. Adv Health Sci Educ Theory Pract. 2002; 7: 63–80.
- 47Jones R, Panda M, Desbiens N. Internal medicine residents do not accurately assess their medical knowledge. Adv Health Sci Educ Theory Pract. 2008; 13: 463–8.
- 48Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med. 2005; 80(10 Suppl): S46–54.
- 49Bianchi L, Gallagher EJ, Korte R, Ham HP. Interexaminer agreement on the American Board of Emergency Medicine oral certification examination. Ann Emerg Med. 2003; 41: 859–64.
- 50Maatsch JL. Assessment of clinical competence on the Emergency Medicine Specialty Certification Examination: the validity of examiner ratings of simulated clinical encounters. Ann Emerg Med. 1981; 10: 504–7.
- 51Swing SR. Assessing the ACGME general competencies: general considerations and assessment methods. Acad Emerg Med. 2002; 9: 1278–88.
- 52O'Sullivan PS. Demonstration of Portfolios to Assess Competency of Residents. Final Project Report. Philadelphia, PA: National Board of Medical Examiners, 2002.
