Advertisement

Prospective clinical evaluation of integrating a radiation anatomist for contouring in routine radiation treatment planning

Open AccessPublished:July 28, 2022DOI:https://doi.org/10.1016/j.adro.2022.101009

      ABSTRACT

      Purpose

      A radiation anatomist was trained and integrated into clinical practice at our multi-site academic center. The primary objective of this quality improvement study was to determine if a radiation anatomist improves the quality of organ at risk (OAR) contours, and secondary impact on efficiency in the treatment planning process.

      Methods and Materials

      From March to August 2020, all patients undergoing CT-based radiation planning at 2 clinics at XXX were assigned every other to either: a) OAR contouring by a radiation anatomist (intervention), or b) contouring by the treating physician (standard of care, SOC). Blinded dosimetrists reported OAR contour quality using a 3-point scoring system based on a common clinical trial protocol deviation scale (1-Acceptable, 2-Minor deviation, 3-Major deviation). Physicians reported time spent contouring for all cases. Analyses included Fisher's exact test and multivariable ordinal logistic regression.

      Results

      There were 249 cases with data available for the primary endpoint (66% response rate). The mean OAR quality rating was 1.1±0.4 for the intervention group and 1.4±0.7 for the SOC group (p<0.001), with subset analysis showing a significant difference for gastrointestinal cases (n=49; p<0.001). Time from simulation to contour approval was reduced from 3 days (IQR 1-6) in the control group to 2 days (IQR 1-5) in the intervention group (p = 0.007). Both physicians and dosimetrists self-reported decreased contouring time in the intervention group compared to control, with decreases of 8 min (17%) (p<0.001) and 5 min (50%) (p=0.002), respectively. Qualitative comments most often indicated edits required to bowel contours (n=14).

      Conclusions

      These findings support improvements in both OAR contour quality and workflow efficiency with implementation of a radiation anatomist in routine practice. Findings could also inform development of autosegmentation by identifying disease sites and specific OARs contributing to low clinical efficiency. Future research is needed to determine the potential impact of reduced physician time spent contouring OARs on burnout.

      INTRODUCTION

      Precise delineation of organs at risk (OARs) is essential to optimal quality and safety of radiation treatment plans. Multiple studies have shown that inter-observer variation in the delineation of OARs exists in disease sites including central nervous system, head and neck, thorax, and pelvis [1-4]. In gastrointestinal cancers, variation in OAR contouring has been associated with worse dosimetry and increased clinical toxicity [5, 6]. Poor quality OARs have also been noted to be limitations in clinical trial efforts in radiation oncology[7]. To date, efforts to standardize normal tissue contouring have focused on dissemination of consensus guidelines (including atlases) by several organizations and institutions, which have become increasingly available as radiation planning has become more complex[8]. However, variation persists[9].
      Normal structure delineation is also time consuming, providing a potential rate-limiting step in the treatment planning process. A study conducted by the German Society of Radiation Oncology has shown that manual segmentation is the most time-consuming task for physicians who, on average, spend 74 minutes contouring for each patient [10]. One strategy to decrease time spent contouring is autosegmentation[11]. However, most validated algorithms still require additional modification by clinical experts, and therefore, have only contributed modest time savings[12-14].
      Therefore, there is an apparent need to identify additional strategies to standardize OAR contour quality while improving efficiency in the treatment planning process. Our institution trained a full-time radiation anatomist to contour OARs for our regional network. We hypothesized that a dedicated radiation anatomist would improve consistency of OAR contours while reducing physician workload and not prolonging time to contour approval.

      METHODS AND MATERIALS

      Study Design

      This was an observational quality improvement study in which data was prospectively collected as part of routine care and approved by the Institutional Review Board at XXX for retrospective analysis. From March to August 2020, all patients who received radiation treatment requiring CT-based planning (i.e. 3D conformal radiation therapy, intensity-modulated radiation therapy (IMRT) and stereotactic body radiation therapy (SBRT)) at 2 regional clinics within our institutional network were included. Treatments for all disease sites were included except for prostate due to an ongoing autosegmentation evaluation [19]. Using an “every other” binning process (i.e. alternating cases between the two study groups), we assigned patients to have OARs contoured by either the treating physician (standard of care, SOC) or a radiation anatomist (intervention). The treating radiation oncologists included in this study were a median of 7 years out from medical school (range 6-14 years). Contour quality, as defined below, was selected as the primary endpoint to highlight the potential patient-centered clinical impact of implementing an anatomist in routine practice. Physician efficiency and time to contour approval were included as secondary endpoints.

      Radiation Anatomist Onboarding

      The radiation anatomist previously received a Master's in Anatomy and Physiology but had no prior knowledge of cross-sectional imaging. The anatomist's onboarding involved 1) developing a set of anonymized cases previously contoured by physicians for self-directed practice simulation with immediate contour comparison, 2) referencing consensus guideline atlases[8], 3) receiving case-specific feedback from physicians via an ARIA 17.0 (Varian Medical Systems, Inc., Palo Alto, CA) task with one multiple choice question using the protocol deviation scale and one open-ended section for comments, and 4) reviewing difficult cases and questions with a radiologist. Training was structured such that one disease site at a time was added to the case list in routine practice, and was considered complete at 8 months when the anatomist was contouring all disease sites with consistently high ratings from the physician scoring task. Once onboarded, the radiation anatomist contours on average 50 cases per week and has 10% time allotted to education, QA, and research, including autosegmentation development.

      OAR Contour Workflow and Assessment

      The overall workflow is outlined in the schema in Figure 1. For every eligible simulation scan sent to the contouring platform, MIM 7.1.6. (MIM Software Inc., Cleveland, OH), the radiation anatomist loaded and saved a session prior to any contouring to blind the evaluating dosimetrists to who performed the initial OAR contours. A physician survey was assigned (via ARIA task) for every eligible patient during the study, which included self-reported time spent contouring and, for the intervention group only, a question regarding OAR quality rating using the 3-point protocol deviation scale.
      Figure 1
      Figure 1Study schema and randomization process for standard-of-care and intervention study arms.
      Given evidence for its association with clinical outcomes[15], the contour quality scoring system commonly used in the clinical trial setting to assign protocol deviations was selected whereby “1” signified OAR contours were acceptable with no edits, “2” signified OAR contours were acceptable with minor edits not likely to affect the treatment plan, and “3” signified OAR contours were unacceptable and major edits were made that would likely affect the treatment plan.
      Subsequently, physicians were allowed to make edits to the anatomist-generated OARs for quality assurance. In both study groups, once OARs have been approved by the physicians, they were rated by blinded dosimetrists using the same 3-point protocol deviation scoring system, via second short survey (ARIA task). If contours were rated as “2” or “3”, the survey prompted the dosimetrist to provide additional comments, which were qualitatively reviewed for themes. Dosimetrists also reported their total time spent making contour edits in preparation for planning. One training session was held with all dosimetrists to review the scoring criteria and clarifications were answered ad hoc by the study team.

      Time to Contour Approval

      Time to contour approval (which initiates the treatment planning process) is a known rate limiting step in the time to treatment sequence. To measure this secondary endpoint, timestamps from MIM Software were collected retrospectively for the point at which the case was ready to contour (after simulation) and upon MD contour approval.

      Statistical Analysis and Sample Size

      The primary outcome of interest was OAR contour quality, which we analyzed as a categorical variable. Inter-rater reliability between physician and dosimetrist's ratings of radiation anatomist-contoured OARs was calculated using the joint agreement probability method. Means and standard deviations for OAR quality score are reported and Fischer's exact test was used to compare radiation anatomist and physician OAR contour ratings. Given that OAR quality rating is an ordinal categorical variable with three levels, an ordinal logistic regression model was constructed to estimate the effect of various predictors on the outcome of interest, OAR quality rating, assuming proportional odds. We verified that the assumption of proportional odds (i.e. odds ratio is the similar across all levels of OAR quality scoring) was reasonable. The primary independent variable of interest was who performed initial OAR contouring, radiation anatomist (intervention) or MD (standard of care). Other independent variables included treatment technique and disease site. Wilcoxon rank sum testing was used to assess differences in time spent contouring and time to contour approval between standard-of-care and intervention groups. A two-sided p value <0.05 was set as a threshold significance, with the exception of Bonferroni correction applied for subset analyses with an adjusted two-sided p value of 0.005. All statistical analysis was performed using RStudio (R foundation for Statistical Computing, Vienna, Austria). Qualitative comments from physicians and dosimetrists were organized thematically.
      To determine study sample size, we reviewed available retrospective data (n=88) which suggested an average score (mean±SD) of 1.50±0.37 for physician OAR contours and 1.15±0.35 for anatomist contours. Sample size calculation (for comparing two means from two independent samples with α=0.05 and β=0.8), suggested only 36 patients would be required. In order to further assess patient subsets by anatomic disease sites of greatest interest (including H&N, GI, lung, metastases, spine), we scaled the study times 5 for a total of 180. To account for potential dropout due to ineligiblility, we aimed to include 200 patients.

      RESULTS

      Study Cohort Characteristics

      A total of 379 eligible patients undergoing radiation treatment were included in our study cohort as detailed in Figure 1. Of those, 183 (48%) CT simulation scans were sent to the treating physician to contour OARs per standard of care, while 196 (52%) CT simulation scans were allocated to the intervention group for the radiation anatomist to contour OARs. In the SOC group, OAR contours were performed by nine physicians across two regional clinics, with each physician treating a median of 18 patients (IQR 14-23) during the 6-month study period. A single radiation anatomist contoured the OARs for all patients in the intervention group. The list of disease sites most represented in the patient cohort included gastrointestinal (n=63, 17%), metastatic disease (n=67, 18%), head and neck (n=53, 14%), lung (n= 49, 13%), breast (n=46, 12%), brain (n=40, 11%), and spine (n=28, 7%). Table 1 shows the patient characteristics in each group, with no significant differences in clinical location, disease sites treated, and radiation treatment techniques.
      Table 1Study cohort characteristics.
      VariableAnatomist, n = 196
      Statistics presented: n (%)
      MD, n = 183
      Statistics presented: n (%)
      p value
      Statistcal tests performed: chi-square test of independence
      Treatment Location>0.99
       Clinical Site 175 (38%)70 (38%)
       Clinical site 2121 (62%)113 (62%)
      Disease site0.31
       Brain23 (12%)17 (9.3%)
       Breast23 (12%)23 (13%)
       GI40 (20%)23 (13%)
       H&N21 (11%)32 (17%)
       Lung23 (12%)26 (14%)
       Mets33 (17%)34 (19%)
       Other
      GU, GYN, Sarcoma, Lymphoma, and Skin
      19 (9.7%)14 (7.7%)
       Spine14 (7.1%)14 (7.7%)
      RT Technique0.53
       3DCRT30 (15%)36 (20%)
       IMRT/VMAT107 (55%)96 (52%)
       SRS/SBRT59 (30%)51 (28%)
      1 Statistics presented: n (%)
      2 Statistcal tests performed: chi-square test of independence
      3 GU, GYN, Sarcoma, Lymphoma, and Skin

      Contour Quality Scoring

      Comparison between physician and dosimetrist rating of OARs contoured by radiation anatomist showed high inter-rater agreement probability (88%), suggesting physicians did not frequently adjust OARs generated by the radiation anatomist and that dosimetrist rating of OARs is a reasonable proxy to assess contour quality. Dosimetrist report of OAR contour quality was available for 249 patient scans with 63% response rate and 68% response rate in the SOC and intervention groups, respectively. 87% of OAR contour sets in the intervention group received a quality score of 1 (acceptable) compared to 65% of OAR contour sets in the SOC group (Figure 1). For all disease sites combined, the mean OAR quality score was 1.1±0.4 for the intervention group (anatomist) and 1.4±0.7 for the SOC group (p<0.001), indicating on average higher quality OAR contouring by an anatomist than a physician. Subset analysis by disease site showed that OARs contoured by radiation anatomist were rated higher in quality than OARs contoured by physicians for gastrointestinal cases (p<0.001) (Table 2). On multivariable analysis, an ordinal logistic regression model controlling for radiation treatment technique and disease site estimated that OARs contoured by MD are 3.9 times more likely to be rated either a 2 or 3, indicating lower OAR quality, compared to contours by a radiation anatomist (p<0.001) (Table 3). Review of dosimetrist open-ended comments with scores of 2 and 3 suggested bowel was the most common structure requiring edits (n=14). Variation in OAR contour quality was noted among physicians (see Supplemental Figure).
      Table 2Subset analysis of OAR quality ratings between SOC and anatomist groups, by disease site.
      Disease Site (n)OAR RatingMeanp value
      Statistical test performed: Fischer's exact test
      123
      OAR contoured byn (%)n (%)n (%)
      All (n=249)<0.001
      indicates statistical significance (p < 0.005)
       MD75 (65%)30 (26%)10 (9%)1.44
       Anatomist117 (87%)15 (11%)2 (2%)1.14
      Head and Neck (n=40)0.17
       MD12 (50%)9 (38%)3 (13%)1.63
       Anatomist13 (81%)3 (19%)0 (0%)1.19
      Lung (n=35)0.10
       MD17 (100%)001.00
       Anatomist14 (78%)4 (22%)01.22
      GI (n=49)<0.001
      indicates statistical significance (p < 0.005)
       MD10 (56%)7 (39%)1 (6%)1.50
       Anatomist30 (97%)1 (3%)01.03
      Mets (n=44)0.39
       MD16 (76%)3 (14%)2 (10%)1.33
       Anatomist21 (91%)1 (4%)1 (4%)1.13
      Spine (n=23)0.03
       MD5 (45%)4 (36%)2 (18%)1.73
       Anatomist11 (92%)01 (8%)1.17
      1 Statistical test performed: Fischer's exact test
      low asterisk indicates statistical significance (p < 0.005)
      Table 3Factors associated with OAR contour quality on multivariable ordinal logistic regression
      VariableOdds Ratio95% Confidence Intervalp Value
      Contoured by

      Radiation anatomist

      MD


      Ref

      3.91




      1.87-6.73




      <0.001
      Treatment Technique
      Treatment technique is a dichotomized variable of 3D conformal or VMAT/IMRT and SRS/SBRT


      3DRT

      Other


      Ref

      0.41




      0.02-8.07




      0.09
      Disease Site
      Disease site is a dichotomized variable; Other include brain, breast, GU, GYN, lung, metastatic, sarcoma, lymphoma and skin.


      GI, H&N and Spine

      Other


      Ref

      1.45




      0.77-2.71




      0.24
      1 Treatment technique is a dichotomized variable of 3D conformal or VMAT/IMRT and SRS/SBRT
      2 Disease site is a dichotomized variable; Other include brain, breast, GU, GYN, lung, metastatic, sarcoma, lymphoma and skin.
      Figure 2
      Figure 2
      Figure 2OAR quality score frequency in standard of care (SOC) and radiation anatomist intervention groups.

      Time Spent Contouring

      On average, physicians reported a median of 33 minutes spent contouring (including both targets and OARs) in the SOC group (n=114, IQR 25-45 minutes) versus a median of 25 minutes spent contouring in the intervention group (n=87, IQR 18-30), reflecting an 8-minute (17% relative) reduction (p<0.001) (Figure 3). Similarly, dosimetrists in the intervention group reported a median of 5 minutes (n=39, IQR 0-12) spent contouring versus 10 minutes in the SOC group (n=48, IQR 5-15), for a 5-minute (50%) reduction (p=0.002). For specific disease sites, the largest reduction in physician time spent contouring was observed for spine cases (primarily SBRT), with physicians reporting a median time of 45 min in the SOC group and 25 minutes in the intervention group (n=18, p=0.02), for a reduction of 20 minutes (44%).
      Figure 3
      Figure 3Time spent contouring by physician in standard of care (without anatomist) and intervention (with anatomist) groups.

      Time to Contour Approval

      The average time to contour approval was 3 days (n= IQR 1-6) for SOC versus 2 days (n= IQR 1-5) for the intervention group, which is a significant reduction in time (p = 0.007).

      DISCUSSION

      To our knowledge, this is the first study to describe and evaluate the implementation of a radiation anatomist for OAR contouring in routine radiation oncology practice, which confirmed improvements in both quality and efficiency in the treatment planning process. The role of the radiation anatomist, at minimum, appears to reduce OAR contour variation and reduce time to contour approval. During routine care, an anatomist provides physicians with accurate OAR contours, which decreases their overall time spent contouring and as a result, reduces total time to contour approval. Physician time savings and shorter time to contour approval could have additional benefits of reducing physician burnout and improving patient experience, but this warrants further investigation. We provide considerations for the integration of an anatomist with autosegmentation efforts as advancements are rapidly occurring in this space.
      To comprehensively assess the quality of OAR contours in this study, we used a clinical trial protocol system for recording deviations. Previous data have shown that radiotherapy protocol deviations are an independent predictor of worse clinical outcomes, including an increased risk of treatment failure and overall mortality[7]. Specifically, quality of bowel contours has been demonstrated to directly correlate with gastrointestinal toxicity; a retrospective quality assurance analysis of the Radiation Oncology Group (RTOG) 0411 phase II study for locally advanced pancreatic cancer reported increased incidence of grade 3 toxicity for patients with radiotherapy protocol deviations compared to guideline-concordant plans[16]. Gastrointestinal and spine cases experienced the most benefit of having a radiation anatomist, most likely due to the cumbersome and complex nature of bowel contouring. Although we were underpowered to assess differences in less common disease sites such as gynecologic and genitourinary (due to planned omission of prostate, as described below), it seems reasonable to extrapolate these findings, and focus radiation anatomist effort on these disease sites in the absence of useful autosegmentation tools.
      Efforts to reduce physician time spent contouring are of high priority to improve efficiency in treatment planning, particularly as the burden of 3-dimensional and highly conformal treatment planning increases. While most studies to date evaluate autosegmentation, herein we evaluate manual segmentation by an anatomist to augment this approach as automated algorithms mature. We found time savings with radiation anatomist contouring OARs resulted in an average of 8 minutes per case (and up to 20 minutes for spine (mostly SBRT) cases), for a total of 17% relatively time savings on average. This is comparable to 9 minutes noted in a prior randomized controlled study of OAR autosegmentation in head and neck cancer[13]. At the time of the current study, our institution was testing autosegmentation for prostate only radiation, so these patients were excluded. Our prostate study and a similar rectal cancer study, ultimately showed ∼ 30% reduction in physician time savings[12]. In our study, dosimetrists often reported bowel as an OAR that required editing (n = 14). This is corroborated by a systematic review evaluating three different commercial software solutions for autosegmentation that highlighted the rectum and bowel as volumes needing the most manual corrections by physicians[17]. Although autosegmenting OARs has been shown to save time, gaps still exist for some disease sites and OARs. The current study helps identify areas of high priority for improving efficiency in the clinical workflow. Such technological work can be done alongside a radiation anatomist, as is now the practice at our institution.
      While it is evident autosegmentation algorithms decrease contouring time, prior studies have not demonstrated significant improvements in overall efficiency in the planning process, as in the current study. A recent study quantifying the resources required for radiation pre-treatment tasks highlighted increased workload in the last decade with patient population increased by 45% while time required to complete these tasks increased by 150%. Meanwhile, the staffing levels only increased by 29% in the same period [18]. Therefore, human resource utilization may be needed to meet the substantial demands associated with technologic changes, and initiatives such as a radiation anatomist should therefore be considered. The cost-effectiveness of this program can be estimated from physician time savings (8 minutes per case) and radiation anatomist case volume (50 cases per week) for approximately 0.2 physician full-time equivalent. Alternatively, time spent contouring OARs for just 3 complex cases (20 minutes per spine case) would equal the typical amount of time for one new consult. Importantly, reducing physician time spent contouring should facilitate more focus on meaningful clinical and academic activities, and evidence suggests that physician burnout is correlated with not operating at the top of one's license [19].
      Data presented in this study is likely relevant to radiation oncology practices broadly, given the inclusion of all disease sites and the persistence of physician contouring in routine practice. A recent twitter poll (n=232) found that OARs are most often contoured by physicians (48%) followed by dosimetrists (42%), autosegmentation (6%), and least commonly by anatomists (3%)[20]. In justifying the hiring of a new position, our department has considered additional opportunities for anatomists to contribute through quality assurance practices (chart rounds and contour review), adaptive planning, and professional education (including trainees and new physician and dosimetry staff). Additionally, radiation anatomist-generated OARs will likely improve quality of training datasets for autosegmentation algorithms development.

      Limitations

      This study has several limitations. First, an “every other” patient assignment is not technically a randomization, although it provides a pragmatic approach to identifying a comparison (control) group that accounts for variation over time and equally distributing the cases between the study groups. Second, the subjective quality scoring was performed by dosimetrists, which was selected for pragmatic purposes to allow for blinded review. Related, the presence of a 2-step editing process itself in the intervention group may inherently improve the quality of OAR contours. Nonetheless, inter-rater agreement was high between OAR ratings by physicians and dosimetrists when both available in the intervention arm, reducing the likelihood that significant edits were made by the physician. Future work should evaluate dosimetric impacts of contouring errors. Third, while self-reported time spent contouring may be subject to recall bias, it is a reliable pragmatic approach compared to a controlled timed setting [11, 13]. Fourth, this data represents the experience of a small group of physicians within a single academic institution with a single anatomist, potentially limiting generalizability. However, it was conducted at two community-based regional practice sites, to ensure distribution of diseases treated were comparable to a typical radiation oncology practice. Finally, the study size potentially limited our power to detect disease-specific differences in quality and time savings in subset analyses. We decided a priori to analyze disease sites with 10 or more patients per study arm to reduce underpowered analyses and limit multiple testing.

      CONCLUSIONS

      This study quantifies improvements in both quality and workflow efficiency with the implementation of a radiation anatomist for contouring OARs. This study took place in community-based practice sites in an academic network, making this information potentially applicable to routine radiation oncology practice. The greatest benefits of anatomist contouring were in bowel contouring, coinciding with evidence from clinical trial QA suggesting quality of bowel contours may affect clinical outcomes. In addition to immediate clinical workflow advantages, consistent radiation anatomist-generated OAR contours may facilitate development of useful and accurate autosegmentation algorithms and standardization for patients treated on trial. Removing the burden of OAR contouring could reduce physician burnout by allowing physicians to practice at their level of training, but this warrants further investigation. Additional opportunities for radiation anatomists may include educating radiation professionals in contouring complex OARs and peer review QA, specifically contour-specific chart rounds.

      Declaration of interests

      Erin Gillespie reports financial support was provided by Radiological Society of North America. EFG is a co-founder of eContour.org

      REFERENCES

      1. Rosewall, T., et al., The effect of delineation method and observer variability on bladder dose-volume histograms for prostate intensity modulated radiotherapy. Radiother Oncol, 2011. 101(3): p. 479-85.
      2. Nelms, B.E., et al., Variations in the contouring of organs at risk: test case from a patient with oropharyngeal cancer. Int J Radiat Oncol Biol Phys, 2012. 82(1): p. 368-78.
      3. McCall, R., et al., Anatomical contouring variability in thoracic organs at risk. Med Dosim, 2016. 41(4): p. 344-350.
      4. Spoelstra, F.O.B., et al., Variations in Target Volume Definition for Postoperative Radiotherapy in Stage III Non–Small-Cell Lung Cancer: Analysis of an International Contouring Study. International Journal of Radiation Oncology*Biology*Physics, 2010. 76(4): p. 1106-1113.
      5. Fuller, C.D., et al., Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting. Int J Radiat Oncol Biol Phys, 2011. 79(2): p. 481-9.
      6. Abrams, R.A., et al., Failure to adhere to protocol specified radiation therapy guidelines was associated with decreased survival in RTOG 9704–a phase III trial of adjuvant chemotherapy and chemoradiotherapy for patients with resected adenocarcinoma of the pancreas. Int J Radiat Oncol Biol Phys, 2012. 82(2): p. 809-16.
      7. Weber, D.C., et al., QA makes a clinical trial stronger: evidence-based medicine in radiation therapy. Radiother Oncol, 2012. 105(1): p. 4-8.
      8. Lin, D., et al., A Systematic Review of Contouring Guidelines in Radiation Oncology: Analysis of Frequency, Methodology, and Delivery of Consensus Recommendations. Int J Radiat Oncol Biol Phys, 2020. 107(4): p. 827-835.
      9. van Mourik, A.M., et al., Multiinstitutional study on target volume delineation variation in breast radiotherapy in the presence of guidelines. Radiother Oncol, 2010. 94(3): p. 286-91.
      10. Vorwerk, H., et al., Protection of quality and innovation in radiation oncology: The prospective multicenter trial the German Society of Radiation Oncology (DEGRO-QUIRO study). Strahlentherapie und Onkologie, 2014. 190(5): p. 433-443.
      11. Cardenas, C.E., et al., Advances in Auto-Segmentation. Semin Radiat Oncol, 2019. 29(3): p. 185-197.
      12. Cha, E., et al., Clinical implementation of deep learning contour autosegmentation for prostate radiotherapy. Radiother Oncol, 2021. 159: p. 1-7.
      13. Walker, G.V., et al., Prospective randomized double-blind study of atlas-based organ-at-risk autosegmentation-assisted radiation planning in head and neck cancer. Radiother Oncol, 2014. 112(3): p. 321-5.
      14. Yang, J., et al., Automatic contouring of brachial plexus using a multi-atlas approach for lung cancer radiation therapy. Pract Radiat Oncol, 2013. 3(4): p. e139-47.
      15. Sherer, M.V., et al., Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: A critical review. Radiother Oncol, 2021. 160: p. 185-191.
      16. Schwartz, G.K., et al., Randomized phase II trial evaluating two paclitaxel and cisplatin-containing chemoradiation regimens as adjuvant therapy in resected gastric cancer (RTOG-0114). J Clin Oncol, 2009. 27(12): p. 1956-62.
      17. La Macchia, M., et al., Systematic evaluation of three different commercial software solutions for automatic segmentation for adaptive therapy in head-and-neck, prostate and pleural cancer. Radiat Oncol, 2012. 7: p. 160.
      18. Thind, K., et al., Increasing Demand on Human Capital and Resource Utilization in Radiation Therapy: The Past Decade. Int J Radiat Oncol Biol Phys, 2022. 112(2): p. 457-462.
      19. West, C.P., L.N. Dyrbye, and T.D. Shanafelt, Physician burnout: contributors, consequences and solutions. J Intern Med, 2018. 283(6): p. 516-529.
      20. Tsai, J.C.C. Hi #radonc folks! Curious about how each dept handles target delineations of OARs. [Poll attached] [Tweet]. Twitter 2021; Available from: https://twitter.com/CJTsaiMDPhD/status/1387372616462241794?s=20.

      Appendix. Supplementary materials