Factors Important to Top Clinical Performance in Emergency Medicine Residency: Results of an Ideation Survey and Delphi Panel

Understanding the development of expertise and top performance has been the subject of considerable research across fields from chess, to sports, to delivering comedy routines.1 A subset of this literature focuses on medicine, specifically assessing factors linked with top clinical performance and learning as well as specific clinical skills.2, 3 As examples, greater practice volume has been linked with improved procedural skills and deliberate practice has been associated with greater clinical skill acquisition.4, 5 By contrast, some factors that should theoretically increase performance have not proven as effective. For example, greater attendance in continuing medical education does not translate to better skill attainment.6

Currently, there is a paucity of literature on determinants of top clinical performance in emergency medicine (EM). Top clinical performance can be defined as superior execution of the functions of an emergency physician in a clinical setting. EM requires specific skills and knowledge to treat conditions from minor lacerations to major trauma and stroke in a complex, dynamic environment.7 EM residency provides focused time for learners to gain competence for independent practice, employing several modes of learning, from book reading, to online resources, to listening to and participating in lectures, and to practicing with simulation in a mentored environment. Residency also teaches and reinforces nonclinical skills, such as professionalism, research, and pedagogic skills.8 While almost all residents gain competence through training, some residents become “top clinical performers” and are distinguished by their superior clinical and organizational skills attained throughout and by the end of their training.

Several factors may differentiate top clinical performance in EM residency such as personal traits, specific behaviors, or, more generally, background prior to entering residency.9, 10 Focused study of factors that predict top performance is important because some skills may be teachable, while others may be more innate. Identifying such factors may have implications for choosing residents, designing educational activities, assessing their performance, and providing feedback on how to improve. Residency is an ideal time to study top clinical performance because, according to theory, training time—termed the cognitive/associative phase according to leading educational researcher K. Anders Ericsson—is a critical inflection point when it comes to final attainment.11 A prior study assessed factors associated with top performance in nine institutions and found that several preresidency factors predicted high rankings at the final semiannual evaluation including grades, test scores, interview performance during the application process, and scholarship.12 To our knowledge, no studies have examined how personal factors, behaviors, and background differentiate top clinical performance during EM residency or which factors used to measure resident performance are most influential for EM educators.

This study is the first in a planned sequence to explore top clinical performance in EM residency. In this study, we explore the role of personal and behavioral factors in top clinical performers in EM residency and how to best predict and measure top clinical performance. We used a two-step process—an ideation survey and a modified Delphi technique to identify factors important to achieve these goals.

Methods

Results

Discussion

We identified a broad universe of potential attributes, traits, background, and ED-specific skills and used an expert panel to prioritize factors felt to be most important to top clinical performance in EM residency and possessed by a high percentage of top clinical performers as they finish their residency training. Several factors fell into the categories of general attributes and personal traits, such as insightfulness and curiosity, while others were ED-specific skills, such as communication, bedside manner, and efficiency that may be more amenable to focused teaching in residency or used for structured assessments of residency.

For general attributes, traits, and skill sets, important factors included good common sense and insightfulness. These attributes are arguably important not just in EM but for all clinicians. Compared to other specialties, EM involves not only answering focused clinical questions and solving isolated problems for patients but also seeing the big picture. General attributes and traits may be more static within an individual—like common sense and insightfulness compared to skills. They also may be assessable prior to entering residency and used as an assessment of success as a clinician.

General skill sets as well as focused skills were also important. These may be more teachable than attributes and traits, especially during the course of residency. In the ED environment, specific skills are needed to be successful as ED patients have complex clinical and psychosocial needs that must be addressed in a time-constrained environment where information is often incomplete.17, 18 To do this effectively, strong problem-solving skills in particular were identified as an important factor. Strong problem-solving skills are particularly important in the United States where acute care delivery is often fragmented, and there is poor availability of access to integrated, longitudinal care settings in many communities.19 Certainly, the absence of good problem-solving skills given the complex nature of ED care would limit a physicians’ ability to perform effectively in this environment.

Being proactive was also seen as focally important. A prior study found that acquiring the breadth of knowledge required for excellence in emergency care requires residents to identify their own learning needs, obtain feedback, and improve their own performance as self-directed learners.20 Therefore, being proactive in learning is a necessary skill to attain expertise. By comparison, important personal traits such as curiosity, grittiness, responsibility, and work ethic, for example, may be less specific to EM. Indeed, grittiness has been shown to be predictive of success in a variety of fields and may similarly be so in EM where learners certainly can encounter challenges at many levels.21

For ED-specific skills and behaviors, many were very specific to the ED—such as the ability to be clinically adaptable and efficient—while others were more general that likely apply to the ED and other settings such as good bedside manner and clinical acumen. Both may be teachable skills during residency training and can be the focus of deliberate practice. There was also a focus on many of the day-to-day skills of ED practice including taking accurate histories and physicals (H&Ps) and having good technical abilities to perform procedures and other complex tasks, which are also amenable to deliberate practice. The application of personal traits to the ED setting—in particular, clinical honesty, “owning” patients, and communications with staff—were seen as important. Clinical honesty is vital in an academic setting where academic emergency physicians trust that what is presented is correct and accurate. However, to some degree, clinical honesty may be teachable but may originate from underlying character, which may be less mutable. In addition, ED physicians must rapidly develop a relationship with patients they have never met before and feel a sense of ownership despite the short-term nature of the relationship. Similar to clinical honesty, ownership may be somewhat teachable, however, and is also related to character. And finally, good communication within the ED and more generally the trait of working well with others were critical, likely given the multitude of stakeholders required for collaboration for effective emergency care. A prior study found that communication and the ability to work within teams were important influencers of learning in emergency care.22 Of the skills identified in our study, communication may be one of the most teachable.

However, several factors that conceptually might be considered important—such as high intelligence and compassion—were not included. This is possibly because these factors may not be differentiators among EM residents because all must have sufficient intelligence and compassion to practice clinically. This confirms the work of many, including McClelland,14 who have found measures of intelligence such as IQ to be less important than other characteristics and learned competencies. In addition, while there were several ideas regarding background factors identified as potentially important, none—including prior work experience or hobbies—were deemed important by the panel or present in most top clinical performers. Yet, it is vital to not characterize these factors that were not ultimately chosen as unimportant. Many of these factors may be important data points when choosing residents to enter a program—i.e., differentiating who successfully complete the program or may have other skills that are valued by the residency, such as research or publication potential—but may be less helpful in determining who will excel clinically.

Notably, the factors chosen through this process bear some overlap with the milestones laid out by the Accreditation Council for Graduate Medical Education for EM. The milestones focus on specific skill attainment for providers of emergency care to be competent.23 However, there were many important differences between the milestones and our list. These differences may originate from different processes for developing our list versus milestones. Milestones were initially identified by an expert advisory panel—not group ideation—and were then validated by asking respondents to assess whether factors were encountered in emergency care practice rather than asking respondents to assess their importance to competence or expertise.24, 25 While milestones are meant to assess the broader of concept of proficiency as well as excellence, several milestones such as participation in patient safety, use of technology, and practice-based performance improvement were not identified as factors that differentiate top clinical performance even in group ideation.

More broadly, when it came to other, nonmilestone ways to measure top clinical performance, isolated measures—such as in-service scores and direct clinical observation—the panel identified limitations. Specifically, in-services scores do not directly test clinical practice skill and feedback from direct clinical observation can be highly variable. There was consensus that examining milestone and other data through the clinical competency committee approach was most effective as it allowed faculty to contextualize all data rather than relying on single data elements in isolation, which may be subject to measurement error. The addition of our work demonstrates that other measurable factors and skills may be more important for excelling within residency (i.e., owning patients, clinical adaptability, clinical honesty), outside of what is present in the milestones. In addition, specific factors—such as taking accurate H&Ps or clinical efficiency, which are included in the milestones but not as isolated skills—may be more important. Top clinical performers in residency just do not achieve the milestones better or faster than their colleagues; they have valued attributes and skills that are not included in the milestones. Focusing on skills that impart excellence rather than competence may suggest that different approaches are needed to impart excellence rather than solely sufficient skills to perform adequately.

Our work also lends to the discussion on how best to identify applicants to EM residency. While several factors were seen as somewhat important by the expert panel, such as standard letters of recommendation, test stores, and direct clinical observation, none of these appeared to differentiate top clinical performance. The sole factor that was seen as important and where nearly all top performers had excelled was in their interview. Similar to the competency committee assessment, interviews provide an opportunity to view aggregate data in the context of a subjective conversation and, therefore, may best help triangulate people who will likely excel or not. However, this should not suggest that other information is unimportant, rather that the interview should be central to assessing the likelihood of residency success.

Limitations

There are several study limitations. First, the initial generation of ideas was conducted solely among participants in the CORD listserv. While many respondents clearly had deep expertise in residency training based on the comprehensiveness of the ideas, many stakeholders in residency education were not included in idea generation including ED nurses and technicians who may have identified other factors. In addition, people who chose to respond to the survey may have had different views than those who did not respond; we could not directly measure response rate or assess for nonresponse bias. However, the purpose of the survey was not to generate representative responses but rather a large volume of ideas, which was achieved.

There were also limitations to the Delphi process. Because the expert panel only included a small group of experts, it is possible that other experts or a larger group of experts may have prioritized the factors differently. We also could have ended with a larger or smaller list had we used different acceptance criteria for importance or for validation (i.e., >80% possessing the factor or >40% possessing the factor). Finally, these factors have not yet been validated by empirical data, and these are solely the opinions of experts. Further research will be required to assess the degree to which these factors actually predict top clinical performance in practice.

Conclusion

In conclusion, we identified 20 factors important to top clinical performance in emergency medicine residency. Some of these are inherent traits but many are teachable skills. Future work is needed to further explore how individuals learn and further develop these factors.