Holistic review for hematology-oncology fellowship applicants: A new paradigm?

Application to hematology-oncology fellowships is a competitive process. In the 2019 National Resident Matching Program (NRMP) match, there were over 800 applicants for just over 600 positions.¹ Specific factors that hematology-oncology fellowship program directors (PDs) take into account when choosing applicants to invite for interviews, and ranking interviewees for the match have not been well-described in the literature. However, evidence from other specialties and personal discussion with several PDs suggests that “traditional” markers of success such as perceived quality of residency program, United States Medical Licensing Examination (USMLE) Step 1 and 2 board scores, academic productivity (measured by number of publications/presentations), and letters of recommendation from Internal Medicine (IM) PDs and subspecialty faculty are common elements used to rank applicants.^{2, 3}

We believe that these “traditional” markers of success leave a gap in the assessment of applicants and, if used in isolation, are not the ideal ways to assess and rank hematology-oncology fellowship applicants. Defining a “high-quality” or “high-ranking” residency program is subjective, and the rank of a program does not necessarily indicate the quality of an individual resident. For example, the Doximity residency ranking system relies on subjective physician opinion of quality to produce a “reputation” value.⁴ A resident who attended a residency program perceived by some application committees as “lower quality” for financial or social reasons (need to be close to family, for example), and was one of the best residents in the program may not be offered an interview or ranked highly on the match list. Whereas, a resident who attended a “top-quality” residency program (possibly at least in part on the basis of attending a “top quality” medical school or having high USMLE board scores), but did not perform well in residency may be offered interviews at top fellowship programs, and even potentially ranked highly on the match list due in part to their residency program's perceived quality. The same constraints apply to use of USMLE board scores, where trainees with socioeconomic advantages may be able to pay for board review courses and/or tutoring, and trainees without these advantages may not score as highly. Additionally, as those from disadvantaged backgrounds may be the very candidates we hope to attract to our programs to increase diversity, equity, and inclusion we are doing these applicants a disservice by focusing entirely on traditional metrics of success. We would also like to acknowledge the disadvantages faced by International Medical Graduates (IMGs) who trained outside the United States and have little chance at being accepted to “top-quality” residency programs. These applicants often take the USMLE examinations much later on in their medical training than American trainees (eg, after completion of medical school and often a home-country internship). Therefore, the use of residency quality and USMLE scores to rank applicants puts IMGs at an inherent disadvantage, no matter their level of skill as physicians.

Additionally, our prior research suggests that specifically for hematology-oncology fellowship programs, ranking of residency program and letters of recommendation indicating a “top” applicant (which are notoriously subjective), are not associated with hematology or oncology in-training examination scores, research productivity, or choice of an academic vs private practice career after fellowship.⁵ While higher USMLE step 1 and 2 scores were associated with higher in-training examination scores, neither were associated with research productivity or career choice (and, we would add, test scores may have little to do with other important milestones such as effective communication). The USMLE examination will become pass/fail as of January 2022, and future studies should examine the effect of this change on fellowship application outcomes. On the one hand the reduced reliance on numeric scores may prompt further consideration of other applicant qualities in a more holistic manner. But on the other hand it has been noted that high USMLE scores may be one of few advantages available to IMGs to set them apart from their competitors. In our research, we also found that IMGs had higher academic productivity, both pre-fellowship and during fellowship, and were more likely to choose academic careers.⁵ As such, we already have data that “traditional” markers of success (residency quality), and country of training are not associated with what we often consider important outcomes as hematology-oncology fellowship programs.

One way that residency training programs have attempted to overcome these challenges is with the Holistic Review process. The Association of American Medical Colleges has defined core principles of the Holistic Review process as (i) emphasizing the importance of individualized consideration of every applicant, (ii) developing mission-driven, diversity-oriented processes, and (iii) encouraging the application of a balanced approach to address the experiences, attributes, competencies, and academic/scholarly metrics (E-A-C-M) of each candidate.⁶ The AAMC urges that all graduate medical education (GME) programs take into account elements of the Holistic Review process when screening and ranking applicants. Additionally, the Alliance for Academic Internal Medicine (AAIM) has developed guidelines for a standardized fellowship letter of recommendation (SLOR), as SLORs have been shown to demonstrate increased reliability as predictors of performance and greater inter-rater reliability, compared to non-standard letters of recommendation.⁷ The AAIM recommends that IM PDs should write SLOR including an assessment of clinical competency achievements, a discussion of scholarly contributions during residency, and details providing “deeper insight and clarity about personal characteristics of the resident,” among other recommendations.⁸

We believe that a Holistic Review process should be considered for hematology-oncology fellowship applicants, taking into account each applicant's individual background, characteristics, experiences, and future career goals. With regards to residency training, we must not simply choose and rank applicants from the “best” IM residency programs but consider why each applicant may have attended their specific IM residency program - were there financial, geographic, social, or other constraints limiting their choice of residency? Were they limited in choice as an IMG or graduate of a smaller/less prestigious medical school? Similarly, with regards to research productivity during residency we must not consider only the number of publications and impact factor, but also what types of research and mentors each applicant may have had access to during their training, along with their relative time for training vs their clinical responsibilities. A resident at a busy, inner-city hospital with almost 100% inpatient training time and few available research mentors cannot be compared to a resident from an academic training program with protected research time and multiple options for mentors. Residents who have overcome incredible odds to publish even one to two papers with little time or guidance may be just as interested in a research career as those with multiple publications in the setting of a residency program with built-in advantages. It is important to ask applicants about challenges faced while doing research and finding mentors, as well as future interest in building a research career in addition to the traditional interview question of “tell me about your research.” Finally, when looking at letters of recommendation, we recommend not placing significant value on how well-known or senior a letter writer is (we often find applicants will ask a renowned hematologist or oncologist at their institution for a recommendation, regardless of how well that person knows the applicant), but instead recommend focusing on the attributes, skills, and personality characteristics discussed in the letter. Simply calling someone a “top resident with a bright future,” for example, says little about their specific skills and attributes. Letters that describe both medical knowledge and also communication skills (eg, a story of the resident interacting with a patient), intellectual curiosity, enthusiasm for the field, and professionalism are most helpful, and we hope that as IM-PDs move to the SLOR format recommended by the AAIM that we will see more such commentary.

Overall, we feel that a Holistic Review process has the potential to improve the hematology-oncology fellowship application process. Taking the time to ask not only “where are you coming from, what have you done, and what do you want to do” but also considering “how and why did you get to where you are, what challenges have you faced along that path, and what do you hope to learn from this in the future” is a broader, more fair, and more inclusive way to screen and rank applicants. Additionally, utilization of a Holistic Review process has associated with significant increases in diversity of medical school interview pools and accepted medical school applicants,^{9, 10} and we hope that implementation in the hematology-oncology fellowship application process will lead to similar improvements in diversity, equity, and inclusion in our future hematology-oncology workforce nationwide. Our own experience with a Holistic Review process has demonstrated the ability to select diverse classes of extremely high-quality, talented, motivated applicants who are incredibly successful during fellowship and after graduation.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

FUNDING INFORMATION

None.