Advertisement

Mitigating Bias in Recruitment: Attracting a Diverse, Dynamic Workforce to Sustain the Future of Radiation Oncology

Open AccessPublished:June 14, 2022DOI:https://doi.org/10.1016/j.adro.2022.100977

      Introduction: Diversity and Excellence

      Why do we need a diverse workforce? Diversity allows for improved collective intelligence,1 and can include diversity of professional knowledge, interests, and perspectives. Benefits of workforce diversity include providing different perspectives, better serving patients,2, 3 and improving outcomes.4-6 Diversity is not only about social justice; it is an essential strategy for complex problem solving in clinical care, research, and teaching. However, racial, ethnic, and gender disparities in outcomes remain pervasive in the US health care system.7 Racial biases are integrated into algorithms used for health decisions and money spent on patient populations.8 Radiation oncology is not exempt from this problem- it remains a specialty comprised only of 28% female and 8% under-represented minority physicians (such as Black or Hispanic).6, 9
      This lack of diversity in our field undoubtedly limits our collective potential. In complex systems such as medicine, an individual may only perceive partial truths based on their own unique perspectives and experiences. For example, when trying to solve a clinical problem, a team of radiation oncologists will have more shared and convergent perspectives than a diverse team including radiation, medical, and surgical oncologists. Diversity allows for the effective sharing of distinct expertise repertoires and diverging opinions that contribute to superior problem-solving outcomes. In increasing diversity in the workforce, we can help diversify our perspectives and improve our quality of care.
      This manuscript will focus primarily on strategies to help mitigate unconscious bias. However, conscious bias is an entirely different matter and represents a far greater challenge that is outside the scope of this manuscript. This manuscript is targeted to individuals interested in examining pitfalls in the recruitment and hiring process, and in strategies to mitigate unconscious bias in order to expand workforce diversity.

      Understanding Cognitive Bias in the Recruitment Process

      Cognitive biases come in many forms. They may be best understood as heuristics, or rules of thumb, that make processing our world easier and more efficient. Unfortunately, this efficient thinking comes at a cost, as these cognitive biases can lead to incorrect conclusions or associations. The tendency is to assume that awareness of these cognitive biases will be sufficient to mitigate their harmful effects; however, diversity training or unconscious bias training has failed to reduce levels of bias.5, 10 A belief that awareness is sufficient to mitigate bias will only perpetuate inequities.
      It is helpful to understand why knowledge is not enough, a concept referred to as the GI Joe Phenomenon,11, 12 a play on the television show that closed with the phrase “knowing is half the battle.” It has been asserted that biases may come in two categories, encapsulated biases and attentional biases.12 Encapsulated biases cannot be corrected simply through conscious reflection or rational knowledge alone- by definition, they are cognitively impenetrable. The term “cognitively impenetrable” refers to a cognitive process which is “wired in,” and cannot be altered by changes in an agent's beliefs, desires, or goals.13 In contrast, attentional biases are cognitively penetrable; however, we often fail to attend to the relevant information in order to make better decisions in the moment because of limited attention.
      There are many examples of encapsulated biases (or, perhaps, conscious bias) in recruitment. For example, a name discrimination study found that individuals with names more common in African American culture are significantly less likely to get called back for a job interview that applicants with Caucasian sounding names despite identical resumes,14 and this persists on an analysis done in modern times,15 as well as within science.16 “Lakisha and Jamal” need 8 more years of experience on their resume to get the same interview call backs as “Greg and Emily.” Discrimination is higher at companies with decentralized vs centralized human resources.15 Similar bias has been documented in medicine, with a recent publication describing Black applicants’ confrontation of microaggressions, stereotype threat, tokenism, imposter syndrome, and homophily.17 In radiation oncology, biases affecting those whose gender, race, ethnicity, and other characteristics are underrepresented in medicine (URiM) have also been documented.18-21
      In the average recruitment process, external influences play a surprisingly large role in our, decision-making processes, especially where it relates to hiring. It is usually assumed that unreliability in expert judgement is caused by attentional biases such as fatigue, boredom, and distraction.22, 23 In addition to the random noise provoked by these elements, incidental factors may also introduce systemic biases.
      Incidental factors influence emotions, and emotions influence the decision-making processes. There are three main mechanisms thought to explain how emotions influence judgement, whether incidental or otherwise. First, emotions influence how information is processed, with happy moods inducing more heuristics and sad moods more analytical information processing. For example, subjects in happy moods are more likely to rely on stereotypes in the formation of judgements.24 Emotions provide information that can be misattributed to the wrong cause if the actual one is not salient. For example, respondents interviewed on sunnier days express higher levels of overall happiness.25
      The influence of incidental factors applies to medicine as well, where bias has been documented in medical school application review. Research has shown that applicants’ academic attributes are weighted more heavily on cloudier days and non-academic attributes are weighted more heavily on sunnier days. Changes in cloud coverage can increase a candidate's predicted probability of admission by an average of up to 12%.26
      Similar biases have been documented in the resident application process. In radiology resident selection, facially attractive and non-obese applicants have a higher chance of getting interviews than their peers. Obesity and attractiveness were found to be as influential in applicant selection for interview as traditional medical school performance metrics.27 Photo score, although objectively recognized to have no value on predicting candidate performance, has been shown to have the same impact on interview invitation as AOA membership.28
      Academia, medicine, and radiation oncology are not immune from bias and the influence of incidental factors on decision making. In order to help mitigate this bias, we must consider newer methods for candidate selection, including holistic review, structured interviews, and global redesign of the recruitment process.

      Overcoming Bias in Recruitment: Achieving Holistic Candidate Review

      Many forms of bias are prevalent in the residency selection process. In radiation oncology, while resident physician numbers are increasing across the country, the number of URiM radiation oncologists remains stagnant.20 It is clear that over-reliance on one single metric can limit diversity in radiation oncology.29 Holistic candidate review can be a tool to improve healthcare workforce diversity.
      Holistic candidate review is defined by the Association of American Medical Colleges (AAMC) as a flexible, individualized way of assessing an applicant's capabilities, by which balanced consideration is given to experiences, attributes, competencies, and academic or scholarly metrics, and how the individual might contribute value to the institution or program's mission and goals.30
      In a holistic recruitment and selection process, selection criteria should be broad-based, linked to institutional mission and goals, and promote diversity and inclusion as essential to excellence. A balance of experiences, attributes, competencies, and metrics (EACM) should be reviewed. Balanced EACM should be applied equitably across the entire candidate pool and are used to assess applicants with the intent of creating richly diverse interview and selection pool. Performance data show that certain EACMs are linked to a likelihood of success as a resident.
      Screeners and reviewers should consider how each applicant might contribute to the institution or program's learning environment and practice of medicine. Furthermore, reviewers should weigh and balance the range of criteria needed to achieve the outcomes desired by the program or institution.
      Barriers to implementing the holistic selection process for resident selection include insufficient knowledge on how to implement the holistic candidate review into the residency selection process and fear/concern for lowering standards for residency selection.
      The holistic review process allows the selection committee to find diverse applicants with valuable skills to contribute and enhance a program. Holistic review does not ignore academic metrics, it adds to it. In fact, when general surgery programs implement a holistic applicant review process, the proportion of ranked and matched female and URiM applicants increased, while USMLE step 1 scores remained unchanged.31
      Rubrics can allow for the implementation of holistic candidate review with scoring tools that list important criteria. Examples of holistic rubric categories include traditional items such as USMLE step 1 score, medical school grades, research, and publications. Additional items to consider include past paid employment, persistence/grit, commitment to service, strength of character, interpersonal skills, and capacity for growth.32
      For holistic applicant review to be successful, buy-in is required from department leadership. Time and commitment are also required to thoroughly review an applicant.  Assessing many of the EACM criteria requires careful reading of personal statements, letters of recommendation, and curriculum vitae. It is important to prospectively document recruitment goals and choose a committed and diverse selection committee. Because review committees are strongest when they themselves are diverse, institutions should value the effort these activities require, as otherwise there is the risk of inadvertently overburdening certain individuals who are the only representatives of a particular group and are therefore asked to serve on many selection committees, referred to as “minority tax.”33 Be sure to provide clear instructions to reviewers and promote an environment where everyone can voice their opinions.34
      In summary, holistic review can provide a systematic approach to residency selection. The goal is to increase diversity in residency training programs with the ultimate goal to create learning environments and teams that are more effective and representative of the general population.

      Overcoming Bias in Recruitment: Implementing a structured interview

      A mainstay of the residency recruitment process remains the applicant interview. Traditional unstructured interviews, where interviewers ask questions and evaluate responses at their own individual discretion, are susceptible to biases and the influence of information unrelated to the position.35 Structured interviews incorporate standardized components into content and evaluation aspects of the interview process; for example, candidates may be all asked the same standardized job-focused questions (content) and evaluated on the same competency metrics with a standardized rating scale (evaluation). Structured interviews been found to improve the reliability and validity of candidate evaluations, as well as the perception of fairness.35, 36 Further advantages of structured interviews in medical and dental contexts include reliable predictions of performance, including in patient care and clinical performance.37, 38
      A structured interview focused on job-related factors can be developed by defining competencies relevant to successful job performance, creating standardized questions designed to assess candidates’ proficiency in these competencies, and developing standardized rating scales to evaluate the level of competency demonstrated through candidate responses.
      The first step of identifying characteristics that lead to successful performance in residency and future practice and categorizing them into well-defined competencies can be performed in conjunction with determining EACM criteria for holistic review. These competencies can include aspects of professionalism and non-cognitive skills such as teamwork, adaptability, conscientiousness, communication, stress tolerance, work commitment, and integrity.37 The ACGME core competency domains can also serve as a starting point for determining specific competencies relevant for successful performance.39
      Standardized interview questions can take the form of behavioral questions (which ask for examples of past behavior in work-relevant situations) or situational questions (which ask for responses to hypothetical job-related situations) to elicit responses indicative of future behavior.35, 36 These questions should be carefully designed to prompt responses that will allow assessment of candidates’ proficiency in the specific competencies defined in the first step. Scripting questions focused on these competencies focuses the interview process on factors relevant to job performance and limits the gathering and influence of unrelated personal information that can trigger biases. Training for interviewers may be useful when beginning this process.
      Interviews can be evaluated in a standardized manner through the development of rating scales. Rating scales can be developed to evaluate responses for individual questions, specific competencies, or overall suitability. Anchored rating scales, which provide descriptive examples or definitions for each score value, make it easier for interviewers to understand the scoring criteria and rate candidates consistently.35-37
      After interviews, it is important to assess whether implementation of the interview has met intended goals: for example, whether the questions as scripted prompted responses providing relevant information on key competencies, if the rating scales have worked as intended, or if any signs of systemic bias can still be seen in evaluations. This assessment can then be used to adjust and iteratively improve the structured interview process on an annual basis. Each program must determine not only how to evaluate their candidates, but also how to represent themselves, as well, within the structured interview process. For instance, one might divide 20% of the interviewers (or 20% of their interviews) as less evaluative. These individuals or interview sessions might focus on being of service to the candidate, following an unstructured format, and providing information, but holding less influence in the rank list process in recognition of the lesser value of this sort of interaction.

      Applying Best Practices to Hiring and Recruitment: Lesson in Faculty Hiring from The Committee on Strategies and Tactics for Recruiting to Improve Diversity and Excellence (STRIDE)

      The STRIDE initiative at the xxx has compiled evidence regarding both the need to combat bias in faculty hiring processes and the skills necessary to apply evidence-based strategies to mitigate bias in the four stages of a search process.40 STRIDE is grounded in the recognition that recruiting colleagues is one of the most important things we do, having a diverse and excellent faculty is central to our success as an institution, and we should approach recruiting in a scholarly way. STRIDE training emphasizes that diversity and excellence go hand-in hand, and research can provide insights about improving faculty search processes through evidence-based processes. 41-48
      Abundant evidence demonstrates that a diverse faculty can provide positive role models and mentors for the diverse community of learners, and socially and intellectually diverse teams make better decisions. Therefore, excellent institutions are committed to taking steps to create a diverse faculty. Unfortunately, key challenges exist. The first of these are schemas based on race, ethnicity, gender, or other identities. Certain assumptions or expectations about groups influence our judgments of them; these are also known as stereotypes. We naturally take cognitive short-cuts; this allows rapid processing of information but is susceptible to errors. Conscious bias exists at many levels of leadership. These challenges are ubiquitous. We all, regardless of the social group to which we belong, perceive and treat people differently based on the social groups to which they belong.46, 49-52 Unfortunately, certain pressures commonly present in faculty searches can increase the reliance on schemas, including stress from competing tasks, time pressure, ambiguity or incomplete information, and lack of critical mass. Numerous studies, often using a blinded randomized CV design, whereby the only factor varied relates to the identity of the applicant, reveal evaluation bias: favoring or disfavoring others based on schemas held about their group.53 Bias can affect many groups: racial and ethnic minorities, women, women parents, LGBTQIA+ people, people with disabilities, immigrants, those from less prestigious institutions, and those working outside the “center” of a discipline.14, 53-63
      This bias “habit” may be able to be broken through awareness of when it can happen, understanding of its consequences, and use of effective strategies to reduce its impact outside of mere awareness.64 Although many would argue appropriately that diversity training is less effective, effective techniques such as exposure limitation can be employed by blinding applicant photos. Such strategies also include ways to develop the applicant pool, including leveraging social media, establishing relationships with promising junior faculty elsewhere, widening the pool to a broader set of institutions, and recruiting year-round at conferences and meetings. Job postings should use language known to draw diverse and excellent applicants. They should avoid narrow specification of areas of expertise and recruit from subfields with diversity. In radiation oncology, this might mean interviewing someone with research expertise or clinical passion for things outside disease site specific domains- like financial toxicity, transgender issues, or healthcare disparities. Search committees should solicit information about candidates’ commitment to diversity, equity, and inclusion. Search committees should themselves be diverse, although care should be taken to value and recognize service in such roles so that this does not inadvertently lead to a 'minority tax.'  These committees should solicit information about candidates’ commitment to diversity, equity, and inclusion. In many situations, it may be that such intentional recruitment requires greater effort on the behalf of those hiring in order to attract talent that previously was not represented.
      Efforts are necessary to mitigate evaluation biases during the search process, including discussing and defining evaluation categories and criteria in advance, reading candidate dossiers carefully, making evidence of job-relevant qualifications central to the candidate deliberations, and delaying global evaluations and summary ranking to acknowledge uncertainty.46, 51, 65, 66 In particular, committee members should consider carefully how letters of recommendation influence their decisions, given evidence of how schemas can impact such letters.67-71
      When candidates visit, it is important to create an environment that elicits the best performance from all candidates. This includes attention to providing helpful information, considering accessibility and other needs, being thoughtful about environmental cues, and facilitating positive interactions. This includes reconsidering whether a wall of portraits of individuals all from a single demographic group seems appropriate. Special care must be taken in situations of virtual interviews, which can expose candidates’ personal lives in ways that can activate bias.
      Interviews should aim to evaluate qualifications that are relevant to a faculty position – questions about matters that are not job-relevant (e.g., family status, sexual orientation) must not be asked by the search committee. Such questions are also often illegal. Moreover, exploring non-job-relevant criteria will confound evaluations and is also likely to drive away the candidate.
      Ultimate decisions should reflect deliberation over prespecified criteria. Best practices involve postponing global rankings, aiming first for an unranked list of candidates the institution would be most happy to hire. This is because ranking cements positions before discussion takes place, with an “anchoring” or “focusing” effect. Especially important is focusing not on “fit,” but rather on fitting the criteria identified at the start of the search.
      Building a culture of search excellence is an integral step in the creation of a culture in which new faculty will thrive, succeed, and choose to remain.

      Conclusion

      In conclusion, diversity is a critical consideration in the field of radiation oncology, which has been lagging behind many other specialties. Mitigating bias in recruitment can help improve the diversity of the radiation oncology workforce. Understanding cognitive biases, achieving holistic candidate review, implementing structured interviews, and re-envisioning the recruitment process can help overcome bias in recruitment. We hope this publication helped detail recruitment and hiring pitfall and offered strategies to combat bias and expand radiation oncology workforce diversity.

      Declaration of interests

      Jagsi reports a relationship with National Institute of Health that includes: funding grants. Jagsi reports a relationship with Doris Duke Charitable Foundation that includes: funding grants. Jagsi reports a relationship with The Greenwall Foundation that includes: funding grants. Jagsi reports a relationship with Susan G Komen Breast Cancer Foundation that includes: funding grants. Jagsi reports a relationship with Blue Cross Blue Shield of Michigan that includes: funding grants. Jagsi reports a relationship with Genetech Biotech Co Ltd that includes: funding grants. Jagsi reports a relationship with Sherinian and Hasso that includes: paid expert testimony. Jagsi reports a relationship with Dressman Benzinger LaVelle PSC that includes: paid expert testimony. Jagsi reports a relationship with kleinbard LLC that includes: paid expert testimony. Jagsi reports a relationship with American Society for Radiation Oncology that includes: board membership. Jagsi reports a relationship with Equity quotient that includes: equity or stocks. Mailhot reports a relationship with Ptcog that includes: funding grants. Mailhot reports a relationship with Conquer Cancer Foundation that includes: funding grants. Mailhot reports a relationship with live like bella pediatric research initiative that includes: funding grants. Brown reports a relationship with Varian Medical Systems Inc that includes: speaking and lecture fees. Brown reports a relationship with National Institutes of Health that includes: funding grants. Brown reports a relationship with Radiation Oncology Institute that includes: funding grants. Brown reports a relationship with Radiation Oncology Education Collaborative Study Group that includes: board membership. Juang reports a relationship with American Association of Physicists in Medicine that includes: board membership. Mattes reports a relationship with Radiation Oncology Institute that includes: funding grants. Mattes reports a relationship with New Jersey Health Foundation that includes: funding grants. Mattes reports a relationship with Bristol-Myers Squibb Foundation that includes: funding grants. Mattes reports a relationship with American Society for Radiation Oncology that includes: board membership. Pinnix reports a relationship with Merck & Co Inc that includes: funding grants. Pinnix reports a relationship with Association for Directors of Radiation Oncology Programs that includes: board membership. Pinnix reports a relationship with SWOG that includes: board membership. Evans reports a relationship with Clarity patient safety organization that includes: speaking and lecture fees. Evans reports a relationship with University of Michigan that includes: speaking and lecture fees. Evans reports a relationship with Yale University that includes: travel reimbursement. Co-author has given many lectures. She is a former member of the Board of Directors of ASCO and co chair of ASTRO's ethics committee. She has stock options. - RJ Co-author is the chair of the Radiation Oncology Education Collaborative Study Group, and is involved with Rad Onc Questions LLC. -DG Co-author holds a leadership position in ASTRO's Committee on Health Equity, Diversity, and Inclusion. - MM
      Table 1
      Table 1Examples of seemingly benign recruiting practices which harbor bias
      Example CasesPotential Biases Involved in this approach
      A residency selection committee is tasked with reviewing 134 applications. The program director and associate program director screen out applicants that reported a prior felony conviction, scored <225 on USMLE Step 1, attended osteopathic or foreign medical schools, are already in residency and are applying from another specialty, or received at least two “Passes” on clerkships during M3 year.

      They then rank the remaining applicants using an Excel formula that includes numeric scores for Step 1, clerkship grades, MSPE keyword, AOA status, whether the applicant took a year off for research, number of publications, rating of personal statement, and caliber of letter of recommendation writers from audition electives. To be objective, they invite only the remaining top 40 applicants from the Excel spreadsheet for interviews.

      Lack of holistic review introduces lots of biases. Standardized tests have implicit biases within them, personal statements contain bias, as do letters of recommendation, and MSPE.69, 72, 73

      Favoring taking a year off for research selects for a more socioeconomically advantaged applicant. There is no allowance for the richness of life experiences that may make a candidate an excellent RO.
      On residency interview day, a residency program director is interviewing 20 applicants. She reviewed the applications and made a point to read every personal statement, review all MSPEs, and even reviewed hobbies. During her interviews she asks each applicant about something she “found interesting” in their application. The program director is an avid SCUBA diver and has a particularly great chat with an applicant who has spent many vacations SCUBA diving around the globe.

      After the interview day the program director made sure that all the applicants were invited to an optional after-interview cocktail hour at a local bar and grill to meet the residents and faculty. She finds the applicant from earlier in the day and continues their discussion about SCUBA diving.

      The “found interesting” conversation starter is highly likely to involve discussion about traits not related to their competency and qualifications to become a radiation oncologist, as in this situation, which left to “mirror-tocracy” (hiring those like you) and homophily, to the exclusion of other candidates.

      Some applicants may not drink alcohol for religious, health, or other reasons and the “cocktail hour” may artificially select for applicants that are comfortable in this environment leading to homophily.
      A two-physician radiation oncology practice has grown busier as the town's population has grown. The two physicians are men in their 50s. They decide to hire a third physician. One of the physicians calls a friend from residency at an academic institution, to ask if there are any good residents graduating. The friend says, “Oh, we have a great guy finishing residency this year who you should interview.” The partners interview the senior resident and make him a job offer shortly thereafter.

      The lack of a larger search limits the potential for diversity in their group (i.e., decreased diversity of perspectives), and does not ensure they get the best candidate for the job.

      References

      1. Woolley AW, Chabris CF, Pentland A, Hashmi N, Malone TW. Evidence for a collective intelligence factor in the performance of human groups. Science. Oct 29 2010;330(6004):686-8. doi:10.1126/science.1193147
      2. Shen MJ, Peterson EB, Costas-Muniz R, et al. The Effects of Race and Racial Concordance on Patient-Physician Communication: A Systematic Review of the Literature. J Racial Ethn Health Disparities. Feb 2018;5(1):117-140. doi:10.1007/s40615-017-0350-4
      3. Alberto CK, Kemmick Pintor J, Martinez-Donate A, Tabb LP, Langellier B, Stimpson JP. Association of Maternal-Clinician Ethnic Concordance With Latinx Youth Receipt of Family-Centered Care. JAMA Netw Open. Nov 1 2021;4(11):e2133857. doi:10.1001/jamanetworkopen.2021.33857
      4. Wallis CJ, Ravi B, Coburn N, Nam RK, Detsky AS, Satkunasivam R. Comparison of postoperative outcomes among patients treated by male and female surgeons: a population based matched cohort study. BMJ. Oct 10 2017;359:j4366. doi:10.1136/bmj.j4366
      5. Kalev A, Dobbin F, Kelly E, et al. Best practices or best guesses? Assessing the efficacy of corporate affirmative action and diversity policies. AMERICAN SOCIOLOGICAL REVIEW. 09/01 2006;2:71589-617.
      6. Washington C, Ahmed AA, Cruickshank I, Chapman CH, Thomas CR, Deville C. Representation Trends of Underrepresented Minority Physicians in the US Radiation Oncology (RO) Workforce. International Journal of Radiation Oncology*Biology*Physics. 2019;105(1):S66. doi:10.1016/j.ijrobp.2019.06.513
      7. Evans MK, Rosenbaum L, Malina D, Morrissey S, Rubin EJ. Diagnosing and Treating Systemic Racism. N Engl J Med. Jul 16 2020;383(3):274-276. doi:10.1056/NEJMe2021693
      8. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. Oct 25 2019;366(6464):447-453. doi:10.1126/science.aax2342
      9. Odei B, Kahn J, Holliday EB, et al. Where Are the Women in Radiation Oncology? A Cross-Sectional Multi-Specialty Comparative Analysis. Adv Radiat Oncol. Sep-Oct 2021;6(5):100735. doi:10.1016/j.adro.2021.100735
      10. Paluck EL, Green DP. Prejudice reduction: what works? A review and assessment of research and practice. Annu Rev Psychol. 2009;60:339-67. doi:10.1146/annurev.psych.60.110707.163607
      11. Santos L, Gendler T. Knowing is Half the Battle. https://www.edge.org/response-detail/25436
      12. Kristal AS SL. G.I. Joe Phenomena: Understanding the Limits of Metacognitive Awareness on Debiasing. Harvard Business School Working Paper. 2021:21-84.
      13. Dawson MRW. Cognitive Impenetrability. In: Vonk J, Shackelford T, eds. Encyclopedia of Animal Cognition and Behavior. Springer International Publishing; 2017:1-3.
      14. Bertrand M, Mullainathan S. Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. American Economic Review. 2004;94(4):991-1013. doi:10.1257/0002828042002561
      15. Kline PM, Rose EK, Walters CR. Systemic discrimination among large US employers. 2021.
      16. Eaton AA, Saunders JF, Jacobson RK, West K. How Gender and Race Stereotypes Impact the Advancement of Scholars in STEM: Professors’ Biased Evaluations of Physics and Biology Post-Doctoral Candidates. Sex Roles. 2020/02/01 2020;82(3):127-141. doi:10.1007/s11199-019-01052-w
      17. Ellis J, Otugo O, Landry A, Landry A. Interviewed while Black. N Engl J Med. Dec 17 2020;383(25):2401-2404. doi:10.1056/NEJMp2023999
      18. Diaz DA, Suneja G, Jagsi R, et al. Mitigating Implicit Bias in Radiation Oncology. Adv Radiat Oncol. Sep-Oct 2021;6(5):100738. doi:10.1016/j.adro.2021.100738
      19. Foster CC, Hasan Y, Son CH, McCall AR. Linearly Accelerating Toward Gender Equity in Radiation Oncology. Int J Radiat Oncol Biol Phys. Aug 1 2019;104(5):974-978. doi:10.1016/j.ijrobp.2019.03.037
      20. Deville C, Jr., Cruickshank I, Jr., Chapman CH, et al. I Can't Breathe: The Continued Disproportionate Exclusion of Black Physicians in the United States Radiation Oncology Workforce. Int J Radiat Oncol Biol Phys. Nov 15 2020;108(4):856-863. doi:10.1016/j.ijrobp.2020.07.015
      22. Dawes RM, Faust D, Meehl PE. Clinical versus actuarial judgment. Science. Mar 31 1989;243(4899):1668-74. doi:10.1126/science.2648573
      23. Danziger S, Levav J, Avnaim-Pesso L. Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences. 2011/04/26 2011;108(17):6889-6892. doi:10.1073/pnas.1018033108
      24. Bodenhausen G, Kramer G, Süsser K. Happiness and Stereotypic Thinking in Social Judgment. Journal of Personality and Social Psychology. 04/01 1994;66:621-632. doi:10.1037/0022-3514.66.4.621
      25. Schwarz N, Clore GL. Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states. Journal of Personality and Social Psychology. 1983;45(3):513-523. doi:10.1037/0022-3514.45.3.513
      26. Simonsohn U. Clouds make nerds look good: Field evidence of the impact of incidental factors on decision making. Journal of Behavioral Decision Making. 04/01 2007;20:143-152. doi:10.1002/bdm.545
      27. Maxfield CM, Thorpe MP, Desser TS, et al. Bias in Radiology Resident Selection: Do We Discriminate Against the Obese and Unattractive? Acad Med. Nov 2019;94(11):1774-1780. doi:10.1097/ACM.0000000000002813
      28. Kassam AF, Cortez AR, Winer LK, et al. Swipe right for surgical residency: Exploring the unconscious bias in resident selection. Surgery. Oct 2020;168(4):724-729. doi:10.1016/j.surg.2020.05.029
      29. Fernandez C, Lopez BL, Kushner M, Leiby BE, Den RB. Overemphasis of Step 1 Scores May Affect Application Pool Diversity in Radiation Oncology. Pract Radiat Oncol. Jan - Feb 2020;10(1):e3-e7. doi:10.1016/j.prro.2019.07.009
      31. Nehemiah A, Roberts SE, Song Y, et al. Looking Beyond the Numbers: Increasing Diversity and Inclusion Through Holistic Review in General Surgery Recruitment. J Surg Educ. May-Jun 2021;78(3):763-769. doi:10.1016/j.jsurg.2020.08.048
      32. DeBenedectis CM, Heitkamp DE, England E, et al. A Program Director's Guide to Cultivating Diversity and Inclusion in Radiology Residency Recruitment. Acad Radiol. Jun 2020;27(6):864-867. doi:10.1016/j.acra.2019.07.027
      33. Rodríguez JE, Campbell KM, Pololi LH. Addressing disparities in academic medicine: what of the minority tax? BMC Medical Education. 2015/02/01 2015;15(1):6. doi:10.1186/s12909-015-0290-9
      34. Sungar WG, Angerhofer C, McCormick T, et al. Implementation of holistic review into emergency medicine residency application screening to improve recruitment of underrepresented in medicine applicants. AEM Educ Train. Sep 2021;5(Suppl 1):S10-S18. doi:10.1002/aet2.10662
      35. Levashina J, Hartwell CJ, Morgeson FP, Campion MA. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature. https://doi.org/10.1111/peps.12052. Personnel Psychology. 2014/03/01 2014;67(1):241-293. doi:https://doi.org/10.1111/peps.12052
      36. AAMC. Best Practices for Conducting Residency Program Interviews. https://www.aamc.org/media/44746/download
      37. Marcus-Blank B, Dahlke JA, Braman JP, et al. Predicting Performance of First-Year Residents: Correlations Between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study. Acad Med. Mar 2019;94(3):378-387. doi:10.1097/ACM.0000000000002429
      38. Poole A, Catano VM, Cunningham DP. Predicting performance in Canadian dental schools: the new CDA structured interview, a new personality assessment, and the DAT. J Dent Educ. May 2007;71(5):664-76.
      40. xxx
      41. Ellison SF, Greenbaum J, Mullin WP. Diversity, Social Goods Provision, and Performance in the Firm. SSRN Electronic Journal. 2010;doi:10.2139/ssrn.1670167
      42. Ely RJ, Thomas DA. Cultural Diversity at Work: The Effects of Diversity Perspectives on Work Group Processes and Outcomes. Administrative Science Quarterly. 2001;46(2):229-273. doi:10.2307/2667087
      43. Hale G, Regev T. Gender ratios at top PhD programs in economics. Economics of Education Review. 2014;41:55-70. doi:10.1016/j.econedurev.2014.03.007
      44. Page SE, Project Muse. The Difference How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies - New Edition. Princeton University Press; 2007:1 online resource. https://yale.idm.oclc.org/login?URL=https://muse.jhu.edu/book/61855/
      45. Sommers SR. On racial diversity and group decision making: Identifying multiple effects of racial composition on jury deliberations. Journal of Personality and Social Psychology. 2006;90(4):597-612. doi:10.1037/0022-3514.90.4.597
      46. Stewart AJ, Valian V. An inclusive academy: achieving diversity and excellence.1 online resource (xxxi, 491 pages). https://yale.idm.oclc.org/login?URL=https://doi.org/10.7551/mitpress/9766.001.0001?locatt=mode:legacy
      47. Dennehy TC, Dasgupta N. Female peer mentors early in college increase women's positive academic experiences and retention in engineering. Proc Natl Acad Sci U S A. Jun 6 2017;114(23):5964-5969. doi:10.1073/pnas.1613117114
      48. West J, Carrell, Schott, Page, Marianne. Sex and Science: How Professor Gender Perpetuates the Gender Gap. The Quarterly Journal of Economics. 2009;125:1101-1144. doi:10.3386/w14959
      49. Bobo L, Kluegel J, Smith R. Laissez-faire Racism: The Crystallization of a Kinder, Gentler, Antiblack Ideology. 1997:93-120.
      50. Gaertner SL, Dovidio JF. Reducing intergroup bias: The common ingroup identity model.
      51. Kahneman D. Thinking, fast and slow. 1st ed. Farrar, Straus and Giroux; 2011:1 online resource.
      52. Valian V. Why so slow?: the advancement of women. MIT Press,; 1998:1 online resource (xvii, 401 pages). https://yale.idm.oclc.org/login?URL=https://doi.org/10.7551/mitpress/7235.001.0001?locatt=mode:legacy
      53. Eaton AA, Saunders JF, Jacobson RK, West K. How Gender and Race Stereotypes Impact the Advancement of Scholars in STEM: Professors’ Biased Evaluations of Physics and Biology Post-Doctoral Candidates. Sex Roles. 2020;82(3-4):127-141. doi:10.1007/s11199-019-01052-w
      54. Ameri M, Schur L, Adya M, Bentley F, McKay P, Kruse D. The Disability Employment Puzzle: A Field Experiment on Employer Hiring Behavior. Working paper series (National Bureau of Economic Research). 09/01 2015;doi:10.3386/w21560
      55. Correll J, Park B, Judd CM, Wittenbrink B. The influence of stereotypes on decisions to shoot. European Journal of Social Psychology. 2007;37(6):1102-1117. doi:10.1002/ejsp.450
      56. Clauset A, Arbesman S, Larremore Daniel B. Systematic inequality and hierarchy in faculty hiring networks. Science Advances. 1(1):e1400005. doi:10.1126/sciadv.1400005
      57. Kang SK, Decelles KA, Tilcsik A, Jun S. Whitened Résumés. Administrative Science Quarterly. 2016;61(3):469-502. doi:10.1177/0001839216639577
      58. Moss-Racusin CA, Dovidio JF, Brescoll VL, Graham MJ, Handelsman J. Science faculty's subtle gender biases favor male students. Proceedings of the National Academy of Sciences. 2012;109(41):16474-16479. doi:10.1073/pnas.1211286109
      59. Tilcsik A. Pride and prejudice: Employment discrimination against openly gay men in the United States. American Journal of Sociology. 2011;117(2):586-626. doi:10.1086/661653
      60. Oreopoulos P. Why Do Skilled Immigrants Struggle in the Labor Market? A Field Experiment with Thirteen Thousand Resumes. American Economic Journal: Economic Policy. 2011;3(4):148-171. doi:10.1257/pol.3.4.148
      61. Settles IH, Warner LR, Buchanan NT, Jones MK. Understanding psychology's resistance to intersectionality theory using a framework of epistemic exclusion and invisibility. Journal of Social Issues. 2020;76(4):796-813. doi:10.1111/josi.12403
      62. Way SF, Morgan AC, Larremore DB, Clauset A. Productivity, prominence, and the effects of academic environment. Proceedings of the National Academy of Sciences. 2019;116(22):10729-10733. doi:10.1073/pnas.1817431116
      63. Weichselbaumer D. Sexual orientation discrimination in hiring. Labour Economics. 2003;10(6):629-642. doi:10.1016/s0927-5371(03)00074-5
      64. Carnes M, Devine PG, Baier Manwell L, et al. The effect of an intervention to break the gender bias habit for faculty at one institution: a cluster randomized, controlled trial. Acad Med. Feb 2015;90(2):221-30. doi:10.1097/ACM.0000000000000552
      65. Bauer CC, Baltes BB. Sex Roles. 2002;47(9/10):465-476. doi:10.1023/a:1021652527696
      66. Sensoy O, Diangelo R. "We Are All for Diversity, but ... ": How Faculty Hiring Committees Reproduce Whiteness and Practical Suggestions for How They Can Change. Harvard Educational Review. Win 2017;87(4):557-580. doi:10.17763/1943-5045-87.4.557
      67. Dutt K, Pfaff DL, Bernstein AF, Dillard JS, Block CJ. Gender differences in recommendation letters for postdoctoral fellowships in geoscience. Nature Geoscience. 2016;9(11):805-808. doi:10.1038/ngeo2819
      68. Madera JM, Hebl MR, Dial H, Martin R, Valian V. Raising Doubt in Letters of Recommendation for Academia: Gender Differences and Their Impact. Journal of Business and Psychology. 2019;34(3):287-303. doi:10.1007/s10869-018-9541-1
      69. Ross DA, Boatright D, Nunez-Smith M, Jordan A, Chekroud A, Moore EZ. Differences in words used to describe racial and gender groups in Medical Student Performance Evaluations. PLOS ONE. 2017;12(8):e0181659. doi:10.1371/journal.pone.0181659
      70. Schmader T, Whitehead J, Wysocki VH. A Linguistic Comparison of Letters of Recommendation for Male and Female Chemistry and Biochemistry Job Applicants. Sex Roles. 2007;57(7-8):509-514. doi:10.1007/s11199-007-9291-4
      71. Trix F, Psenka C. Exploring the Color of Glass: Letters of Recommendation for Female and Male Medical Faculty. Discourse & Society. 2003;14(2):191-220. doi:10.1177/0957926503014002277
      72. Turrentine FE, Dreisbach CN, St Ivany AR, Hanks JB, Schroen AT. Influence of Gender on Surgical Residency Applicants' Recommendation Letters. J Am Coll Surg. Apr 2019;228(4):356-365 e3. doi:10.1016/j.jamcollsurg.2018.12.020
      73. White BAA, Sadoski M, Thomas S, Shabahang M. Is the Evaluation of the Personal Statement a Reliable Component of the General Surgery Residency Application? Journal of Surgical Education. 2012/05/01/ 2012;69(3):340-343. doi:https://doi.org/10.1016/j.jsurg.2011.12.003