Advertisement

Potential Implications of the New USMLE Step 1 Pass/Fail Format for Diversity Within Radiation Oncology

Open AccessPublished:July 17, 2020DOI:https://doi.org/10.1016/j.adro.2020.07.001
      In March 2019, a collaborative group of medical education stakeholders moved to re-evaluate the scoring system of the United States Medical Licensing Examination (USMLE) exams.
      Committee TIP
      Summary report and preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS).
      These examinations were initially formulated for test takers to score at or above a minimum passing score as an eligibility criteria for state medical licensure.
      Committee TIP
      Summary report and preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS).
      Over the years, the test has become a high-stakes examination, and arguably the single most important component in the residency selection process. After a yearlong deliberation, a decision was made to transition the USMLE Step 1 examination to a pass–fail format.
      USMLE.ORG
      Change to pass/fail score reporting for Step 1.
      There is recent precedent for such a decision, as the Joint Commission on National Dental Examinations transitioned the National Board Dental Examination to a pass–fail format.
      • Bell L.T.
      • Sukotjo C.
      • Yuan J.C.
      • Johnson B.R.
      Applicant selection procedures in endodontic specialty programs in the United States: Program director's perspective.
      This change similarly stemmed from the concern that dental residency programs were using the numerical scores as a differentiator of an applicant’s aptitude, when the test was not psychometrically structured to function as such.
      The hold of the USMLE Step 1 examination on the preclinical curriculum has been paralyzing as medical school educators have undoubtedly felt compelled to model their educational content after high yield concepts on the Step 1 examination. The revised test format will provide new and much needed latitude for the preclinical curriculum, and perhaps allow for the integration of radiation therapy concepts into early medical school education, which have hitherto been sidelined from the syllabus. Perhaps now may be the opportune time for increased calls for curriculum reform.
      One of the concerns raised about transitioning to this new system is the detrimental effect of this change on international medical graduates (IMGs) who have historically relied on a stellar USMLE Step 1 score to gain needed visibility in the application process.
      Committee TIP
      Summary report and preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS).
      ,
      USMLE.ORG
      Change to pass/fail score reporting for Step 1.
      These concerns are valid and also of particular importance to radiation oncology owing to the recent upswing in the matriculation of IMGs into radiation oncology residencies. In reality, the continued use of numerical scoring of USMLE Step 2 Clinical Knowledge (CK) and Step 3 examinations will still provide opportunity for differentiation for IMGs and eliminate the arduous task of performing flawlessly on all 3 standardized examinations.
      Another element discussed by the medical education stakeholders was the concern that the utility of Step 1 scores adversely affects some groups who have historically scored lower on these tests.
      Committee TIP
      Summary report and preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS).
      Multiple studies have shown that women and underrepresented minorities score lower on the USMLE Step 1 examination than their counterparts.
      • Cuddy M.M.
      • Swanson D.B.
      • Clauser B.E.
      A multilevel analysis of examinee gender and USMLE Step 1 performance.
      ,
      • Rubright J.D.
      • Jodoin M.
      • Barone M.A.
      Examining demographics, prior academic performance, and United States Medical Licensing Examination scores.
      In the absence of robust research correlating Step1 achievement to residency performance and future clinical competence, the overreliance on Step 1 as the principal yardstick for residency selection appears problematic. Moreover, the 8-hour, multiple choice format and the attendant high stress introduces unintended confounders such as performance anxiety and attention fatigue that may cloud the intended output of the test.
      • Green M.
      • Angoff N.
      • Encandela J.
      Test anxiety and United States Medical Licensing Examination scores.
      Radiation oncology has traditionally been a competitive specialty that attracts applicants with high USMLE Step 1 scores but also a specialty where representation of underrepresented minorities and women has been disproportionately low.
      • Chapman C.H.
      • Hwang W.T.
      • Deville C.
      Diversity based on race, ethnicity, and sex, of the US radiation oncology physician workforce.
      ,
      • Jang S.
      • Rosenberg S.A.
      • Hullett C.
      • Bradley K.A.
      • Kimple R.J.
      Beyond 'charting outcomes' in the radiation oncology match: Analysis of self-reported applicant data.
      It is unclear how much of this underrepresentation is linked to the prohibitive barrier of high test scores rather than inherent disinterest, lack of exposure, or lack of mentorship. In past years there have often been more applicants than residency positions available in radiation oncology, and it was known that at least some institutions previously used an arbitrary Step 1 score cutoff to prescreen residency applications, and thus did not even review a proportion of applications for interview consideration. This approach could have disadvantaged well-rounded applicants who may have had average or even above average Step 1 scores. It is conceivable that without the barrier of the Step 1 examination, in the future a broader cohort of applicants will explore radiation oncology as a potential specialty. Under this premise, the upcoming change to the USMLE Step 1 scoring system may tilt the balance in favor of racial and gender diversity in our field.
      • McDade W.
      • Vela M.B.
      • Sanchez J.P.
      Anticipating the impact of the USMLE Step 1 pass/fail scoring decision on underrepresented-in-medicine students [E-pub ahead of print].
      However, a recent survey of program directors suggests that fewer than 15% of program directors believe the change in USMLE Step 1 scoring to pass or fail will improve diversity and inclusion.
      • Makhoul A.T.
      • Pontell M.E.
      • Ganesh Kumar N.
      • Drolet B.C.
      Objective measures needed - program directors' perspectives on a pass/fail USMLE Step 1.
      Approximately half of those surveyed feel that IMGs will be at a disadvantage under the new system,
      • Boulet J.R.
      • Pinsky W.W.
      Reporting a pass/fail outcome for USMLE Step 1: Consequences and challenges for international medical graduates.
      and the majority feel the prestige of an applicant’s medical school will become a more important factor in their application. Furthermore, many program directors will place more weight on the numerically scored USMLE Step 2 CK examination and require its completion before residency application. Program directors from several specialties, including radiology and neurosurgery, have expressed their concern about the loss of an objective measure by which to compare residency applicants.
      • Rozenshtein A.
      • Mullins M.E.
      • Marx M.V.
      The USMLE Step 1 pass/fail reporting proposal: The APDR position.
      Under this scenario, the limited window between the completion of third year clerkships and the submission of Step 2 CK scores to residency programs may simply create another high-stakes environment. Students aiming for highly competitive residency programs, or students from smaller medical schools without home radiation oncology departments or high levels of research funding, may opt to take a research year to optimize Step 2 CK examination preparation and to build a stronger research portfolio. The possibility of taking a gap year, with its significant associated costs, may not be a viable option for medical students from less affluent backgrounds. This will invariably disadvantage those from minority groups and those with children or other dependents. Finally, it is conceivable that away-rotations will assume more critical importance in the absence of numeric Step 1 scores. This may also disproportionately affect some minority students and others without socioeconomic privilege whose financial circumstances may place limitations on the possibility of multiple away rotations.
      • Camp C.L.
      • Sousa P.L.
      • Hanssen A.D.
      • et al.
      The cost of getting into orthopedic residency: Analysis of applicant demographics, expenditures, and the value of away rotations.
      Amidst these uncertainties, we cannot lose sight of the new and daunting challenge facing radiation oncology program directors and their admission committees who will need to fundamentally change the way they screen the myriad of applicants they will receive. With most programs interviewing a fraction of their applicant pool, the loss of the simplicity and universality of the Step 1 score will require a more in-depth review of full applications to differentiate among them. Some studies have shown a correlation between medical school shelf exams and the USMLE examinations, which may become appealing to some institutions as a surrogate for board exam scores.
      • Zahn C.M.
      • Saguil A.
      • Artino Jr., A.R.
      • et al.
      Correlation of National Board of Medical Examiners scores with United States Medical Licensing Examination Step 1 and Step 2 scores.
      The Alpha Omega Alpha (AOA) designation, third year clerkship performance, medical school “pedigree,” and letters of recommendation may all assume greater importance in spite of their inherent flaws. In the end, although individual review of all radiation oncology residency applicants will be particularly exacting, it may simply be the right thing to do.
      Although the announced change to pass or fail for Step 1 has startled the medical education community, the new policy will not come into practice until January 1, 2022, at the earliest. This anticipatory period offers a unique opportunity for the creation of a more holistic and innovative approach to medical student evaluation and selection and provides an avenue for the abandonment of the historical approach that oversimplifies an applicant's ability and potential by a reliance on a single component on their application.
      • Pershing S.
      • Co J.P.T.
      • Katznelson L.
      The new USMLE Step 1 paradigm: An opportunity to cultivate diversity of excellence.
      Medical school education leadership should take this opportunity to examine issues such as grade inflation and lengthy “dean’s letters,” which are cumbersome to read and often give little information on an applicant’s fitness for residency. Some in this space are calling for reform.
      • Salari S.
      • Deng F.
      A stepping stone toward necessary change: How the new USMLE Step 1 scoring system could affect the residency application process.
      • Lin G.L.
      • Nwora C.
      • Warton L.
      Pass/fail score reporting for USMLE Step 1: An opportunity to redefine the transition to residency together.
      • Kogan J.R.
      • Hauer K.E.
      Sparking change: How a shift to Step 1 to pass/fail scoring could promote the educational and catalytic effects of assessment in medical education.
      Residency programs also have a role to play in adjusting to the loss of one of our most objective metrics to ensure there are no unintended consequences detrimental to diversity and inclusion efforts. In a powerful piece recently published in the New England Journal of Medicine, Youmans et al
      • Youmans Q.R.
      • Essien U.R.
      • Capers Qt
      A test of diversity - what USMLE pass/fail scoring means for medicine.
      suggest concrete steps that can be taken, including using diverse application reviewers and interviewers, requiring implicit bias training for committee members and teaching faculty, and prioritizing an applicant’s ability to contribute to a program’s cultural competence and diversity.
      Along these lines, we support the creation of a composite applicant scoring system that equally weighs factors of important significance to the future of radiation oncology and for the welfare of our oncology patients. We propose these 7 parameters as a tool for screening initial applicants: (1) research scholarship; (2) academic achievements; (3) demonstrated compassion; (4) commitment to radiation oncology; (5) diversity of perspective, background, and life experiences; (6) interpersonal skills; and (7) demonstrated leadership. Some of these parameters can be gleaned from an applicants’ curriculum vitae, personal statement, or letters of recommendation, but others may be challenging to assess. We may need to consider novel components of the interview process, such as behavioral interviewing techniques and validated assessments of emotional intelligence.
      In summary, we believe the transition to the USMLE Step 1 examination from numerical scoring to a pass/fail system will significantly affect multiple facets of our specialty, chief among which are the restructuring of the residency selection process and the possible disruption of racial and gender norms within the specialty. A paradigm shift by stakeholders that centers on our proposed selection criteria will not only honor the call by the Accreditation Council for Graduate Medical Education (ACGME) to foster diversity, but it will also allow for the selection of well-rounded individuals who will effectively lead our specialty into the future.

      References

        • Committee TIP
        Summary report and preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS).
        (Available at:)
        • USMLE.ORG
        Change to pass/fail score reporting for Step 1.
        (Available at:)
        https://www.usmle.org/incus/
        Date accessed: April 14, 2020
        • Bell L.T.
        • Sukotjo C.
        • Yuan J.C.
        • Johnson B.R.
        Applicant selection procedures in endodontic specialty programs in the United States: Program director's perspective.
        J Endod. 2014; 40: 797-804
        • Cuddy M.M.
        • Swanson D.B.
        • Clauser B.E.
        A multilevel analysis of examinee gender and USMLE Step 1 performance.
        Acad Med. 2008; 83: S58-S62
        • Rubright J.D.
        • Jodoin M.
        • Barone M.A.
        Examining demographics, prior academic performance, and United States Medical Licensing Examination scores.
        Acad Med. 2019; 94: 364-370
        • Green M.
        • Angoff N.
        • Encandela J.
        Test anxiety and United States Medical Licensing Examination scores.
        Clin Teach. 2016; 13: 142-146
        • Chapman C.H.
        • Hwang W.T.
        • Deville C.
        Diversity based on race, ethnicity, and sex, of the US radiation oncology physician workforce.
        Int J Radiat Oncol Biol Phys. 2013; 85: 912-918
        • Jang S.
        • Rosenberg S.A.
        • Hullett C.
        • Bradley K.A.
        • Kimple R.J.
        Beyond 'charting outcomes' in the radiation oncology match: Analysis of self-reported applicant data.
        Med Educ Online. 2018; 23: 1489691
        • McDade W.
        • Vela M.B.
        • Sanchez J.P.
        Anticipating the impact of the USMLE Step 1 pass/fail scoring decision on underrepresented-in-medicine students [E-pub ahead of print].
        Acad Med. 2020; (https://doi.org/10.1097/ACM.0000000000003490. Accessed June 16, 2020)
        • Makhoul A.T.
        • Pontell M.E.
        • Ganesh Kumar N.
        • Drolet B.C.
        Objective measures needed - program directors' perspectives on a pass/fail USMLE Step 1.
        N Engl J Med. 2020; 382: 2389-2392
        • Boulet J.R.
        • Pinsky W.W.
        Reporting a pass/fail outcome for USMLE Step 1: Consequences and challenges for international medical graduates.
        Acad Med. 2020; (https://doi.org/10.1097/ACM.0000000000003534. Accessed June 16, 2020)
        • Rozenshtein A.
        • Mullins M.E.
        • Marx M.V.
        The USMLE Step 1 pass/fail reporting proposal: The APDR position.
        Acad Radiol. 2019; 26: 1400-1402
        • Camp C.L.
        • Sousa P.L.
        • Hanssen A.D.
        • et al.
        The cost of getting into orthopedic residency: Analysis of applicant demographics, expenditures, and the value of away rotations.
        J Surg Educ. 2016; 73: 886-891
        • Zahn C.M.
        • Saguil A.
        • Artino Jr., A.R.
        • et al.
        Correlation of National Board of Medical Examiners scores with United States Medical Licensing Examination Step 1 and Step 2 scores.
        Acad Med. 2012; 87: 1348-1354
        • Pershing S.
        • Co J.P.T.
        • Katznelson L.
        The new USMLE Step 1 paradigm: An opportunity to cultivate diversity of excellence.
        Acad Med. 2020; (https://doi.org/10.1097/ACM.0000000000003512. Accessed June 16, 2020)
        • Salari S.
        • Deng F.
        A stepping stone toward necessary change: How the new USMLE Step 1 scoring system could affect the residency application process.
        Acad Med. 2020; (https://doi.org/10.1097/ACM.0000000000003501. Accessed June 16, 2020)
        • Lin G.L.
        • Nwora C.
        • Warton L.
        Pass/fail score reporting for USMLE Step 1: An opportunity to redefine the transition to residency together.
        Acad Med. 2020; (https://doi.org/10.1097/ACM.0000000000003495. Accessed June 16, 2020)
        • Kogan J.R.
        • Hauer K.E.
        Sparking change: How a shift to Step 1 to pass/fail scoring could promote the educational and catalytic effects of assessment in medical education.
        Acad Med. 2020; (https://doi.org/10.1097/ACM.0000000000003515. Accessed June 16, 2020)
        • Youmans Q.R.
        • Essien U.R.
        • Capers Qt
        A test of diversity - what USMLE pass/fail scoring means for medicine.
        N Engl J Med. 2020; 382: 2393-2395
      1. ACGME Common Program Requirements.
        (Available at:)