If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Improving Quality Metrics in Radiation Oncology: Implementation of Pretreatment Peer Review for Stereotactic Body Radiation Therapy in Patients with Thoracic Cancer
Traditional peer reviews occur weekly, and can take place up to 1 week after the start of treatment. The American Society for Radiation Oncology peer-review white paper identified stereotactic body radiation therapy (SBRT) as a high priority for contour/plan review before the start of treatment, considering both the rapid-dose falloff and short treatment course. Yet, peer-review goals for SBRT must also balance physician time demands and the desire to avoid routine treatment delays that would occur in the setting of a 100% pretreatment (pre-Tx) review compliance requirement or prolonging the standard treatment planning timeline. Herein, we report on our pilot experience of a pre-Tx peer review of thoracic SBRT cases.
Methods and Materials
From March 2020 to August 2021, patients undergoing thoracic SBRT were identified for pre-Tx review, and placed on a quality checklist. We implemented twice-weekly meetings for detailed pre-Tx review of organ-at-risk/target contours and dose constraints in the treatment planning system for SBRT cases. Our quality metric goal was to peer review ≥90% of SBRT cases before exceeding 25% of the dose delivered. We used a statistical process control chart with sigma limits (ie, standard deviations [SDs]) to access compliance rates with pre-Tx review implementation.
Results
We identified 252 patients treated with SBRT to 294 lung nodules. When comparing pre-Tx review completion from initial rollout to full implementation, our rates improved from 19% to 79% (ie, from 1 sigma limit [SDs]) below to >2 sigma limits (SDs) above. Additionally, early completion of any form of contour/plan review (defined as any pre-Tx or standard review completed before exceeding 25% of the dose delivered) increased from 67% to 85% (March 2020-November 2020) to 76% to 94% (December 2020-August 2021).
Conclusions
We successfully implemented a sustainable workflow for detailed pre-Tx contour/plan review for thoracic SBRT cases in the context of twice-weekly disease site-specific peer-review meetings. We reached our quality improvement objective to peer review ≥90% of SBRT cases before exceeding 25% of the dose delivered. This process was feasible to conduct in an integrated network of sites across our system.
Introduction
Peer review is well established as an important aspect of quality assurance in radiation oncology. The peer-review process typically involves review of case-specific qualitative decisions that include a general assessment of the therapeutic rationale, target, and organ-at-risk (OAR) volumes; prescription dose and fractionation; treatment dosimetric plans; and patient setup. The goal of a peer-review process is to optimize treatment by making recommendations for minor or major changes to the treatment plan when appropriate.
The peer-review group consists of a multidisciplinary team, including ≥1 radiation oncologists who are not the prescribing physician. These multidisciplinary sessions may improve quality by identifying potential errors or opportunities for plan improvement consistent with best practice guidelines, as well as providing continuing education for participants.
The peer-review process is traditionally practiced in the context of a weekly meeting, with the review occurring after treatment has started so that all aspects of the plan, including setup imaging, are available. Peer review usually occurs within the first week of treatment start. Yet, the opportunity to optimize the treatment plan is generally the greatest early in the treatment planning process, and the impact of detecting an anomaly or significant error is the greatest when detected before or early in the treatment course.
In addition, the longitudinal follow up of a prospective peer-review evaluation over the course of 5 years demonstrated that 75% of physicians improved in accuracy of target delineation, OAR contours, and prescription.
and central assessments of radiation protocol compliance are often a component of technical quality in clinical trials.
The American Society for Radiation Oncology (ASTRO) peer-review white paper provides guidance to prioritize aspects of a peer review based on the type of treatment.
For example, the white paper identifies stereotactic radiation therapy (SBRT) as a high priority for review of target and OAR contours and dosimetric assessment before treatment start. SBRT uses short, high dose-per-fraction regimens with a rapid dose falloff and resultant elevated risk of normal tissue complications in the setting of inaccuracies. Weekly reviews may come after most or all of the treatment has been delivered due to the short duration of treatment. Retrospective analyses of stereotactic radiation peer review have demonstrated a significant proportion of cases that had a recommended change ranging from 22.3% to 60% of treatments.
The impact of peer review of volume delineation in stereotactic body radiation therapy planning for primary lung cancer: A multicenter quality assurance study.
American Society for Radiation Oncology. Safety is no accident. Available at: https://www.astro.or g/Patient-Care-and-Research/Patient-Safety/Safety-is-no-Accident. Accessed May 15, 2022.
that endorse the inclusion of a peer review process as part of a quality program make recommendations for prioritization, they generally do not specify a requirement for pretreatment (pre-Tx) case review, meeting frequency, or other aspects of peer review. This is, in part, due to the recognition of the tremendous resources that are necessary for peer review, as well as the need to determine a realistic and feasible balance between pre-Tx review for every case without routinely delaying care or prolonging the standard treatment planning time versus the logistical difficulties in achieving this. In other words, peer-review goals for SBRT must also balance physician time demands and the desire to avoid prolonged treatment planning windows as a result of logistical or staffing challenges. Thus, the optimal process is likely to vary from department to department.
A detailed pre-Tx review of stereotactic cases has not been adopted yet into standard practice. The rate of pre-Tx review ranged from 38.6% to 56% from surveys of radiation-oncology physicians and treatment centers.
and the percentage of cases undergoing high-quality pre-Tx reviews (including detailed 3-dimensional review of contours and isodose lines directly within the treatment planning system) is likely even lower than reported.
The feasibility of pre-Tx review has been established in community centers with daily contour rounds or peer-review meetings up to 4 times per week.
However, this approach is not feasible in a large academic setting without compromising disease-site expertise in peer-review meetings, which we view as a key benefit to our patients. Thus, our objective was not to demonstrate the benefits of pre-Tx review, which have already been clearly established,
but focus on the feasibility of pre-Tx review in the context of a large, regionally expansive academic center while maintaining disease-site specific expertise.
With these considerations in mind, the goal of our initiative was to develop a high-quality peer-review workflow to identify priority cases most likely to benefit from peer review before plan initiation, also known as pre-Tx review, to allow time for clinicians to evaluate and modify contours and/or plans as necessary without extending the treatment planning timeline. If major changes are proposed to a radiation plan, a short delay in treatment may be recommended at the discretion of the peer-review group. Importantly, we wanted to maintain disease-site expertise for peer-review meetings, which is feasible using video conferencing in our large academic department spanning multiple clinical campuses. We aimed to create a process guideline for the implementation and feasibility of pre-Tx peer review activities for appropriately selected cases within a multisite radiation oncology department.
Methods and Materials
Local context
Our radiation oncology department consists of 6 clinical sites across a relatively large geographic footprint spanning >50 miles. Alignment of quality standards across the campuses has been a department priority,
and a transition to disease site-specific peer review across campuses was identified as one means to this standardization in 2019. At that time, video-based meetings dedicated to disease site-specific peer review inclusive of all campuses were initiated. Five peer review groups were created as follows: Breast and sarcoma, thoracic and gastrointestinal, central nervous system and head and neck, pediatrics, and genitourinary and gynecology.
According to our traditional process, groups meet once per week and review cases according to a checklist of items (Table 1), with cases placed on an electronic medical record (EMR) peer-review quality checklist (QCL) by radiation therapists after initial setup imaging is complete. Cases are reviewed at the next disease-site appropriate peer-review meeting, which is generally no more than 5 business days later. Due to case volume and the need for efficiency, treatment plan review consists of selected contour and isodose representations in a PDF format within the radiation-oncology EMR, as well as review of dose-volume histograms and a dose-objective table.
Table 1Timeline with gradual implementation of pretreatment review workflow for stereotactic body radiation therapy cases across disease sites
Standard peer-review checklist before implementation
Adapted peer-review checklist after implementation
In 2019, our department was called upon by our School of Medicine's Institute for Patient Safety and Quality to identify a department-specific metric that would best reflect the quality of care delivered to our patients, and we identified the peer-review process as a key measure of quality, and selected this as a high-priority quality improvement objective.
we identified pediatric, proton, and stereotactic cases as our highest priority cases for pre-Tx contour and plan review. When selecting the optimal metric, we considered our desire to maintain disease-site expertise in our peer-review meetings, time constraints limiting the number of weekly meetings that would be feasible, and the goal to avoid extended treatment planning windows that would be necessary to ensure 100% compliance.
We based our quality metric on the Canadian Partnership for Quality Radiation therapy (RT) guideline that all curative intent cases undergo radiation-oncologist peer review of volumes and dosimetry ideally before the start of treatment in all cases or, if not possible, before exceeding 25% of the total prescribed dose delivered.
With these in mind, we developed the following peer-review based quality metric: 90% of pediatric cases undergo pre-Tx contour and plan review; 90% of proton cases undergo pre-Tx contour and plan review; and 90% of stereotactic cases (defined as ≤5 fractions and dose >5 Gy per fraction) undergo contour and plan review before exceeding 25% of the dose delivered.
Our thoracic peer-review group volunteered to be an early adopter of the revised approach beginning in March 2020, with the long-term goal of streamlining the process in this group and subsequently rolling the process out system-wide. We will discuss the evaluation of pre-Tx review for stereotactic cases, which is common in early stage lung cancer by our thoracic malignancy group.
Intervention
The thoracic peer-review group initiated a second weekly pre-Tx peer-review meeting in addition to the previously existing weekly peer-review meeting. The pre-Tx planning conference is specifically designed to include at a minimum 2 physicians with expertise in thoracic malignancies in addition to the attending physician of the case to be reviewed. Other members of the RT team, including physics, dosimetry, clinical/physics residents, and therapists, are encouraged to attend whenever possible. In our integrated network, 5 to 6 attending physicians who treat patients with thoracic malignancies are available for participation. Cases are identified for pre-Tx review at the time of simulation based on the criteria listed previously, and placed on the relevant peer-review QCL as pre-Tx review requested by the simulation radiation therapist. All other cases continue on the routine pathway, and are not placed on the pre-Tx peer-review checklist until after initial treatment imaging is complete. If the treating physician changes the plan after simulation to an approach requiring pre-Tx review, the case may be requested to be placed on the pre-Tx review checklist at any point before the initiation of treatment.
At the pre-Tx planning conference, each selected case is introduced with a brief case history and patient identifier by the resident or attending physician. Target volumes and OARs are reviewed directly within the treatment planning system and compared with diagnostic imaging when appropriate. Image fusions and breath-hold or 4-dimensional computed tomography scans are included in the review at this step for applicable cases. Cases are eligible for review as soon as contours are completed. The treatment plan, including isodose lines, dose volume histograms, and a table of dosimetric objectives and doses achieved, are reviewed if complete.
If the treatment plan is not available, only contours are reviewed, and the plan is reviewed at the next available meeting. In this scenario, the QCL is not complete until the plan is complete, but a note is made that the contours have already been checked to avoid redundancy at its final review. During the review, any team member can inquire about the technical or clinical aspects of the plan for quality assurance and education. If the plan is satisfactory based on group consensus, a plan is designated as approved. Alternatively, the plan may be returned to the attending physician for additional changes (minor or major), and brought back to the group for a second review at the discretion of the group.
Once the contours and plan are approved, the QCL item is completed, and the case is automatically placed on the standard peer-review checklist with the notation “contours and plan reviewed; image review only”. Then, the setup images are reviewed during the next scheduled peer-review meeting after treatment starts. If a pre-Tx review QCL item is not completed, the case is placed on the standard peer-review list after setup imaging is completed. This was a quality improvement project designed to develop optimal workflows; thus, cases were not delayed if the pre-Tx review was not completed.
Our video conferencing technology has evolved to allow for individual screensharing and the ability to have the treatment planning system open for all cases; therefore, we moved to reviewing both traditional and pre-Tx review cases at both meetings to improve our ability to achieve our goals of pre-Tx review. Beginning in September 2020, our department moved the review of pathology and informed consent paperwork out of the multidisciplinary peer-review meeting and to be completed offline by radiation therapists to enable more in-depth reviews during the allocated time.
Data collection
Following institutional review board approval, we retrospectively identified 1398 unique patients scheduled for thoracic/gastrointestinal peer review at our institution between March 2020 and August 2021. Of the 1398 patients, 252 unique patients met our criteria for pre-Tx review in the thoracic group, and were treated with SBRT techniques to 294 separate lung nodules. Thirty-four patients (of 252) had SBRT to multiple (2-5) lung nodules. Using computer algorithms developed in-house with Visual Studio C++, SQL, and MATLAB, variables extracted from the EMR database included prescription dose and fractionation regimens, SBRT site/location, treatment schedule, and actual dose and fractions delivered. We also recorded the dates of pre-Tx peer-review QCL completion and dates of standard peer-review meetings.
Performance measures
Our goal is to assess our performance in incorporating pre-Tx peer review for patients treated with the SBRT technique to ≥1 pulmonary nodules over the course of the multiphase rollout of this new process. We evaluated our performance in 6 blocks of time, comprised of 3 months each (block 1: 03/2020-05/2020; block 2: 06/2020-08/2020; block 3: 09/2020-11/2020; block 4: 12/2020-02/2021; block 5: 03/2021-05/2021; and block 6: 06/2021-08/2021) using the predefined metrics. For pre-Tx peer reviews, we analyzed the percentage of radiation plans with pre-Tx peer review scheduled (% pre-Tx review scheduled) and the percentage of radiation plans with pre-Tx peer review completed before initiation of RT (% pre-Tx review completed before RT start). We also evaluated metrics on radiation plans with any form of peer review (pre-Tx or standard peer review). These metrics included the percentage of radiation plans with any form of peer review completed before exceeding 25% of the radiation dose delivered (% pre-Tx or standard peer review completed before exceeding 25% of dose delivered) and the percentage of radiation plans with any form of peer review completed by the end of RT (% pre-Tx or standard peer review completed by end of treatment).
Statistical methods
We decided to adopt a process-control methodology with sigma limits (ie, standard deviations [SDs]) to examine rates of adherence to pre-Tx review using SPC Software (QI Macros, version 2020; KnowWare International, Inc, Denver, Colo). A control chart is a traditional statistical tool to assess quality control and aid in decision making. Rhe chart was initially developed in the 1920s to provide information about the reliability of a manufacturing process. The control limits were generally evaluated ±3 sigma limits (ie, SDs). If a process is considered stable and in control, we assume that any variations are likely due to chance. On the other hand, if a process is considered unstable and out of control, we assume there are significant variations introduced to the process behavior over time.
Results
We identified 252 unique patients treated with radiation to 294 separate lung nodules using SBRT techniques. There were no significant differences in patient characteristics (ie, age distribution, sex, race, and ethnicity) in the pre- and postintervention groups. These cases were identified from the QCL generated for peer review, and crossed checked to confirm the treatment type.
Overall, we observed a consistent, gradual improvement in compliance rates for scheduling pre-Tx peer reviews from 38% of eligible cases (15 of 39; block 1) to 100% (50 of 50; block 6; Table 2). We also observed a similar trend in improving compliance rates for completing pre-Tx peer reviews before RT start from 18% (7 of 39; block 1) to 74% (37 of 50; block 6). A sustained shift in the process was apparent on the control chart, indicating a clear relationship between the implementation of pre-Tx peer review scheduling (Fig. 1A) and increase in percentage of pre-Tx review completed before treatment (Fig. 1B). When comparing monthly pre-Tx review scheduling from initial rollout to full implementation, our scheduling rates moved from >3 sigma limits (ie, SDs) below to >2 sigma limits (ie, SDs) above (from 19%-100%). When comparing monthly pre-Tx review completion before RT, our completion rates moved from 1 sigma limit (ie, SDs) below to >2 sigma limits (ie, SDs) above (from 19%-79%).
Table 2Overall peer review metrics
Period
SBRT cases, n
Pre-Tx review scheduled, % (n/N)
Pre-Tx review completed before RT start, % (n/N)
Pre-Tx or standard peer review completed before exceeding 25% of dose delivered, % (n/N)
Pre-Tx or standard peer review completed by end of treatment, % (n/N)
We identified 252 unique patients who were scheduled for thoracic peer review at our institution between March 2020 and August 2021 for definitive radiation to 294 separate lung nodules using SBRT. Our goal is to assess our performance in incorporating pre-Tx peer review for thoracic SBRT cases. We evaluated our performance in 3-month quarters using the following metrics: Percentage of radiation plans with pre-Tx peer review scheduled (% pre-Tx review scheduled), percentage of radiation plans with pre-Tx peer review completed before RT start (% pre-Tx review completed before RT start), percentage of radiation plans with any form of peer review completed before exceeding 25% of radiation dose delivered (% pre-Tx or standard peer review completed before exceeding 25% of dose delivered), and percentage of radiation plans with any form of peer review completed by end of RT (% pre-Tx or standard peer review completed by end of treatment). Percentage refers to % of total number of SBRT cases that meet SBRT prereview guidelines.
Figure 1Statistical process control chart, demonstrating initial rollout and monthly compliance rates for pretreatment peer review (percentage), with percentage of pretreatment peer review, (A) scheduled out of all eligible stereotactic body radiation therapy cases within each month, and (B) completed before start of radiation therapy. In the control chart, the control limits were generally evaluated ±3 sigma limits (ie, standard deviations [SDs]). If a process is considered stable and in control, we assume that any variations are likely due to chance. If a process is considered unstable and out of control, we assume that there are significant variations introduced to the process behavior over time. When comparing monthly pretreatment review scheduling from initial rollout to full implementation, our compliance rates moved from >3 sigma limits (ie, SDs) below to >2 sigma limits (ie, SDs) above (19%-100%). When comparing monthly pretreatment review completion before radiation therapy, our compliance rates moved from 1 sigma limit (ie, SDs) below to >2 sigma limits (ie, SDs) above (19%-79%). Data were collected from electronic medical records, and manually verified.
We also improved our compliance rates for any form of peer review (pre-Tx or standard peer review) completed before exceeding 25% of the SBRT dose delivered, from 67% to 85% in blocks 1 to 3 to 76% to 94% in blocks 4 to 6 (Table 2). This met our quality metric goal, which was to peer review ≥90% of SBRT cases before exceeding 25% of the dose delivered.
Discussion
RT planning can involve significant interprovider variation. Discrepancies can be observed in multiple aspects of RT planning, such as dose prescription, target volume delineation, beam arrangement, planning methods, target coverage, critical-organ dose limits, and patient positioning. Peer review is an effective strategy to evaluate plan quality and safety as recommended by professional organizations such as ASTRO.
The specific approach to the peer review process varies across institutions. Many practices have continued the traditional format of once-weekly meetings, but others have implemented a daily morning review meeting, which most or all physicians are expected to attend.
Given the number of physicians at our large academic practice, combined with the fact that many of our physicians have conflicting commitments (eg, tumor boards, multidisciplinary clinic, and protected research time on selected days), we did not feel that a daily contour rounds approach was optimal for our group.
We also desired to maintain disease site-specific expertise, which would have to be compromised if all types of plans were reviewed on all days. Finally, we recognize that some treatment plans will derive greater benefit from a detailed pre-Tx review than others, in alignment with the ASTRO white paper prioritization recommendations. Based on these considerations, we arrived at the approach of scheduling pre-Tx contour and plan reviews for selected cases, including pediatric, proton, and stereotactic cases. Because of the number of physicians at the practice, satisfactory attendance can generally be accomplished with physicians committing 1 to 2 hours per week to peer review, although we acknowledge this can be a challenge for physicians who treat several disease types.
This initiative illustrates a framework of the technical implementation of prospective peer review of high priority stereotactic cases within a multisite radiation oncology department. We chose to evaluate stereotactic treatment planning given the high conformality and short treatment courses where evaluation of target definition and planning directives are critical before or very early after treatment initiation.
By block 6 of the implementation period, 74% of stereotactic cases had a pre-Tx review of contours and plans completed before treatment start. Furthermore, 94% of cases had either a pre-Tx or standard peer review completed before exceeding 25% of the dose delivered, which means we were able to demonstrate feasibility and meet our quality metric goal of peer review ≥90% of SBRT cases before exceeding 25% of the dose delivered. To achieve this, we aimed to create a safety culture in the workplace and ensure a blame-free environment where all stakeholders feel comfortable advocating for patient safety. Multiple studies (including the ASTRO Safety is No Accidence reference guide) have found that teamwork and leadership that reduces hierarchical behavior and encourages a blame-free environment increases the prioritization of safety.
American Society for Radiation Oncology. Safety is no accident. Available at: https://www.astro.or g/Patient-Care-and-Research/Patient-Safety/Safety-is-no-Accident. Accessed May 15, 2022.
To achieve this safety culture, we promoted the peer-review concept among key stakeholders, with buy-in from leadership among the RT, dosimetry, physics, and physician working groups. With support from these key stakeholders, we proceeded to incorporate pre-Tx peer reviews into the local workflow during the rollout (ie, implementation) process.
Several features about the infrastructure of our workflow are important to highlight. The system currently relies entirely on a standardized process for the identification of cases for pre-Tx review and completion of the relevant QCL item. Thus, this process is somewhat automated, without directed oversight, to ensure that cases are planned on a timeframe that relates to the meeting schedule. Our radiation oncology EMR relies on a QCL system rather than custom-built care pathways, and this process could easily be tailored or improved in the context of different EMRs. We also felt strongly that we did not want to extend our standard treatment planning timeline, which would result in treatment delays, to meet this new, elevated standard that exceeds standard practice in most departments, which is part of the rationale for the metric that we selected at <100% compliance. With these constraints in mind, the significant improvement in our ability to perform early peer review in SBRT cases is encouraging to note, but also sobering that even with twice weekly meetings, the overall percent of cases fully prereviewed before treatment was <80%.
In review of our column % pre-Tx review scheduled (Table 2) and statistical process control chart (Fig. 1A), our rollout of the workflow to identify and schedule the correct types of cases for pre-Tx review clearly was effective, with steady improvement over the period reviewed. This primarily involved training of simulation therapists, who were responsible for placing appropriate cases on the QCL checklist. This training was carried out in phases across our 6 clinical sites, which accounts for the gradual uptake of the new process. Our ability to actually complete pre-Tx reviews before treatment start also substantially improved over the course of the implementation, as shown in % pre-Tx review completed before RT start and % pre-Tx or standard peer review completed before exceeding 25% of the RT dose delivered (Table 2). Early percent completion of peer review (defined by completion of peer review before exceeding 25% of dose delivered) also improved over the implementation period, with a notable uptick in the most recent quarter. However, the changes in this metric were not as pronounced, showing that even at baseline, a majority of SBRT cases already met this benchmark. In addition, continued clinician feedback and attendance in these meetings are crucial to its success, because a full discussion of clinical factors and recommendations require the prescribing physician to be present.
However, before inferring that adopting this system did not have a profound impact on the 25% metric, we also substantially increased the depth of review for cases selected for prereview in addition to simply completing pre-Tx reviews by conducting a more detailed review of contours and plans within the treatment planning system rather than reviews of screen capture of treatment plans. There is little value in a dose-table or dose-volume histogram that reflects the achievement of established dosimetric objectives if the contours identifying those structures are not contoured or not contoured accurately.
Also, the data suggest that important errors may go undetected in a traditional peer-review format, possibly due to decreased participation, lack of nonphysician participation and time for detailed plan evaluation, or the wallpaper effect (ie, cognitive bias that occurs when repetitive decision making promotes a tendency to favor the status quo).
Thus, we believe we have enhanced the value of peer review by selecting out those cases that are most likely to benefit from a more detailed review.
Conclusions
In the future, we plan to investigate the impact of pre-Tx reviews on clinical parameters (eg, rate of minor and major changes to a radiation plan). We also plan to solicit and document feedback from multidisciplinary staff, and assess how often changes are implemented to the peer-review process based on these recommendations. Finally, we plan to quantify the impact of aforementioned changes to the peer-review process, and conduct stakeholder interviews to better understand and address any potential barriers to implementation.
References
Hoopes DJ
Johnstone PA
Chapin PS
et al.
Practice patterns for peer review in radiation oncology.
The impact of peer review of volume delineation in stereotactic body radiation therapy planning for primary lung cancer: A multicenter quality assurance study.
American Society for Radiation Oncology. Safety is no accident. Available at: https://www.astro.or g/Patient-Care-and-Research/Patient-Safety/Safety-is-no-Accident. Accessed May 15, 2022.
Sources of support: The authors acknowledge R01 CA237005 for funding support.
Disclosures: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Data sharing statement: Research data are stored in an institutional repository, and will be shared upon request to the corresponding author.
Not repeated if prereview was done.
00110-5&id=gr1.jpg){kind=link}