Human-like intelligent automatic treatment planning of head and neck cancer radiation therapy

VTP is a neural network trained to operate the TPS, in lieu of a human planner, to define and refine dosimetric constraints and generate high-quality plans. Our group previously introduced a hierarchical VTP (HieVTP) to address the treatment planning task, emulating the hierarchical decision-making used by human planners (Shen et al 2021b). HieVTP was structured into three sub-networks: Structure-Net, Parameter-Net, and Action-Net, serving the roles of making decisions at various levels—structure, parameter, and adjustment action. They were sequentially applied each time HieVTP interacted with the TPS. Specifically, Structure-Net utilized the dose-volume histograms (DVHs) of the plan to identify structures in need of improvement. Subsequently, Parameter-Net selected a treatment planning parameter (TPP, e.g. priority, dose limit, or volume constraint) that had the most impact on the dose of the structure, considering all associated TPPs. Finally, Action-Net determined the specific adjustment for this TPP, which entailed increasing or decreasing its value.

In this study, for the purpose of improving training efficiency, we reduced the complexity of this HieVTP in two aspects. The first one was combining Structure-Net with Parameter-Net. The Parameter-Net decided on one of the eight TPPs to adjust, and subsequently, the Action-Net determined its adjustment direction.

Another aspect was to input the VTP with a vector containing plan quality scores, as opposed to the DVHs in our previous studies, because these scores directly indicate plan quality, making it straightforward for the VTP to extract relevant information to make decisions for plan quality improvement. The list of scores followed the ProKnow scoring system (ProKnow Systems, Elekta, Sanford, FL, USA), as we applied our VTP to the 2023 AAMD Planning Challenge that employed this metric for plan quality (American Association of Medical Dosimetrists 2023). The ProKnow scoring system is a commercial software product designed to offer a fair platform for evaluating plan quality. It is a sum of a number of scores, each representing plan quality from a certain aspect (Nelms et al 2012). For the context of H&N planning, the evaluation criteria on plan quality are listed in the supplement material (table S1).

With these considerations in mind, the input to the VTP consisted of 21 values, corresponding to various dosimetric criteria. Both sub-networks process data through a sequence of three fully connected layers, with each layer followed by a Leaky ReLU action function with a slope of 0.1. Following the final activation function, Parameter-Net predicted the improvement in plan quality associated with adjusting the TPPs of each structure. This prediction informed the creation of a structure-coded vector, which served as an indicator for the selected structure. The structure-coded vector maintained the specific score of the chosen structure while setting all other scores to zero, preserving the same dimension as the original input. Subsequently, this vector was concatenated with the original 21 ProKnow scores. The combined vector was then passed into the Action-Net to predict the improvement in plan quality corresponding to each parameter adjustment action. The detailed architecture of the VTP is presented in figure 1. Each parameter was allowed for two adjustment directions (increasing or decreasing). The adjustment magnitude for TPPs was empirically selected. We opted to increase the priority value by 6 if there was an improvement over an empirically chosen criterion of 20% in the plan score during the last TPP adjustment or reduce the adjustment size to 3 if the plan score change was less than 20%, deemed minor.

Zoom In Zoom Out Reset image size

Figure 2 outlines the automated workflow of VTP for H&N cancer RT treatment planning within the Eclipse TPS (Varian Medical System, Palo Alto, CA, USA). The treatment planning process by the VTP was initiated through the Eclipse Scripting Application Programming Interface (ESAPI). It first autonomously configured the treatment planning initialization process, including tasks such as specifying the isocenter, defining beam fields, prescribing the dose, and creating auxiliary structures that aid in shaping the dose distribution. Following this, VTP introduced TPPs, such as dosimetric objectives for each structure to formulate the optimization problem within the TPS. Plan optimization and dose calculations were then carried out in the background. Once the optimization phase produced a plan, VTP collected the plan data and evaluated the plan quality. In the case that the stopping criteria were not met, VTP proceeded to determine adjustments to the TPPs for the subsequent planning iteration and start a new optimization cycle. This iterative planning process continues until a high-quality plan is achieved.

Zoom In Zoom Out Reset image size

The training process of the VTP followed the Q-learning framework (Watkins and Dayan 1992). We present it briefly here for completeness. Readers interested in details can refer to our previous publications (Shen et al 2021b).

In Q-learning, the training process determines the optimal action-value function

$$\begin{align} Q^*\left(s,a\right) = \max_{\pi}\left[r^l+{\gamma}r^{l+1}+{\gamma}^2r^{l+2}+\cdots|s^l = s,a^l = a,{\pi}\right],\end{align} \tag{ 1 }$$

where $Q^*(s, a)$ indicates the expected cumulative rewards associated with taking action a (the decision related to TPP adjustment policy) in state s (represented by the vector of ProKnow scores). It quantifies the desirability of action a in the context of state s. s^l and a^l represent the state and action at the lth TPP adjustment step, respectively. r^l denotes the reward obtained at the lth step, which is determined by a predefined reward function related to clinical objectives, such as the ProKnow scoring system. A positive reward is achieved when the action applied to the state leads to improved plan quality with the updated TPP. The term $\pi = P(a|s)$ refers to the policy of TPP adjustments, signifying the selection of action a based on the observed state s.

In DRL, the action-value function $Q^*(s,a)$ is represented by deep neural networks as described in the previous subsection. It is obtained by a training process including an iterative process based on the VTP's experiences with TPP adjustments and interactions with the TPS. The primary goal of training is to determine the optimal policy for VTP to adjust the TPP of the selected structure to maximize the expected cumulative reward. The update of the Q-learning action-value function can be expressed in the following manner, where α is the learning rate to control the weight given to the new information obtained from the current experience.

$$\begin{align} Q^*\left(s,a\right) \leftarrow Q^*\left(s,a\right)+\alpha\left[r^l+\gamma\max\left(Q\left(s^l,a^l\right)\right)-Q\left(s,a\right)\right].\end{align} \tag{ 2 }$$

A prerequisite for VTP is to quantitatively define H&N cancer plan quality. In this study, we incorporated the ProKnow scoring system, which served as the plan evaluation criteria during the 2023 AAMD Plan Study. ProKnow scoring system includes a series of dosimetric metrics of the structure of interests (American Association of Medical Dosimetrists 2023). The detailed computation methods of all metrics were referred to the planning guidance provided by AAMD, which can be found in supplementary material table S1. The final score $\psi(d)$ as a function of the dose distribution of a plan d was determined by the summation of the scores across all these metrics, resulting in a plan score that ranges from 0 to 150, where a higher score corresponds to superior plan quality.

Based on the plan quality score, the reward function to guide the learning process in DRL was naturally defined as the change in plan quality for a TPP adjustment, namely $r = \psi(d^{^{\prime}}) - \psi(d)$, where dose d and d' are dose distributions before and after the TPP adjustment. A positive reward signifies an enhancement in plan quality, whereas a negative reward indicates the opposite.

Our previous work has established the basis for training with hierarchical DRL by incorporating a multi-level decision-making hierarchy that enabled VTP to learn and strategize across multiple levels. We employed the same training strategy with modifications to adapt to the two-level decision-making process of VTP in this study.

Let's denote Parameter-Net as $S(s,p;\theta^*_S)$ and Action-Net as $A(s,p,a;\theta^*_A)$. Here, $\theta_S^*$ and $\theta_A^*$ represent the optimal network parameters to be determined from the training process, s is the state of the treatment plan (21 ProKnow scores) to input to the networks, p is the set of TPP to adjust, and a the action to adjust the TPP. Once $S(s,p;\theta^*_S)$ is known, the policy to decide the TPP $p^*$ to adjust for the plan state s is achieved by the one that maximizes it, i.e. $p^* = \max_pS(s,p;\theta^*_S)$. After that, based on the selected TPP, the Action-Net then decides on one of two actions $a^*$ to adjust by selecting the one that maximizes the function, i.e. $a^* = \max_aA(s,p^*,a;\theta^*_A)$.

In terms of training, we alternatively trained the two networks $S(s,p;\theta_S)$ and $A(s,p,a;\theta_A)$, each time fixing one while training the other. When $S(s,p;\theta^*_S)$ is fixed, $A(s,p,a;\theta_A)$ can be updated with the standard Q-learning algorithm to solve the problem

$$\begin{align} \min_{\theta_A}\left[\|A\left(s,p,a;\theta_A\right)-\left(r+\gamma \max S\left(s^{^{\prime}},a;\theta_S\right)\right) \|^2_2\right].\end{align} \tag{ 3 }$$

With the updated $A(s,p,a;\theta_A)$ fixed, $S(s,p;\theta_S)$ can subsequently be trained as a conventional supervised learning problem by solving

$$\begin{align} \min_{\theta_S}\left[\|S\left(s,p;\theta_S\right)-\max_p A\left(d,p,a;\theta_A\right) \|^2_2\right].\end{align} \tag{ 4 }$$

We trained the VTP based on the single patient case in the 2023 AAMD Planning Challenge. Throughout the training process, VTP engaged with the Eclipse TPS for treatment planning and collecting data for training. At each iteration of TPP adjustment, we incorporated an ε-greedy algorithm to introduce stochasticity. This involved VTP making a random selection among all possible actions with a probability of ε, and a probability of $(1-\epsilon)$ to choose actions based on VTP's learned strategies. The value of ε gradually diminished at a rate defined as $\epsilon = \max(0.1, 0.99/\mathrm{EPI})$, where $\mathrm{EPI}$ is the index of episodes. The ε-greedy strategy allowed VTP to explore diverse state-action pairs without biasing towards prior experiences. In each training iteration, a tuple $(s, a, r, s^{^{\prime}})$, including the state s, action a, reward r, and the next state s', was collected and stored in a pool. Experience replay strategy was employed to sample data from this pool to update network parameters. Random sampling ensured uncorrelated experiences for updating Q-values, reduced the risk of overfitting, and encouraged exploration, allowing VTP to learn from both successes and failures and refine its strategies over time.

It is worth noting that, in our previous developments on prostate SBRT (Shen et al 2021b), we built the VTP trained with an in-house TPS with an inverse planning optimization engine similar to Eclipse TPS. This approach was chosen due to the inefficiencies associated with direct VTP interaction with Eclipse TPS during the training process. We successfully trained the VTP under this setup, which was able to autonomously create high-quality treatment plans using the Eclipse TPS. However, when it comes to the H&N cancer case, we observed the limitation of this approach due to slight dose disparities between the two TPSs. To address these issues, in the current study, we seamlessly integrate VTP with Eclipse TPS, facilitating direct training using it. As such, the state s and s', as well as the reward r, were all computed based on dose distribution computed by the Eclipse dose engine, mitigating the issue of mismatch in dose calculations between the TPS used in training and application of VTP.

The VTP was constructed using Python with Pytorch on a desktop workstation with two Nvidia Quadro RTX 5000 GPUs on a computer equipped with a CPU of 26 cores and 64 GB of host memory. We interfaced the VTP with Eclipse TPS v16.0 using ESAPI to enable the fully automated planning workflow and collection of training data.

Our VTP participated in the 2023 AAMD Planning Challenge. Organized by the AAMD, a two-phase plan study was undertaken in 2023 for medical dosimetrists to develop an adaptive RT treatment plan for a patient with H&N cancer (American Association of Medical Dosimetrists 2023). This patient, a 34-year-old male, was diagnosed with poorly differentiated squamous cell carcinoma of the left retromolar trigone and positive lymph nodes. The patient's treatment journey began with surgery, followed by post-operative concurrent chemo/RT.

During the RT course, the patient received external beam treatment with a 6 MV photon beam, and the prescription dose was 210 cGy per fraction, totaling 30 fractions. The contours were created by the physicians and dosimetrists at Mary Bird Perkins Cancer Center. The treatment plan included four levels for different PTVs at 63 Gy, 60 Gy, 57 Gy, and 54 Gy with the aim of achieving at least 95% coverage for PTV 63 Gy and PTV 60 Gy, and at least 90% coverage for PTV 57 Gy and PTV 54 Gy. This patient presents a substantial overlap between targets and OARs, which introduces complexities in the planning process. Specifically, PTV 54 Gy intersects with the oral cavity, pharyngeal constrictor, both parotid glands and esophagus. PTV 57 Gy exhibits overlap with the pharyngeal constrictor, left parotid gland, and brachial plexus. PTV 60 Gy overlaps with the oral cavity, left parotid gland, and pharyngeal constrictor. Finally, PTV 63 Gy shows an overlap with the pharyngeal constrictor and brachial plexus.

Notably, as shown in figure 3, by Week 4 of the RT course, the patient experienced a significant weight loss of 16.5%, which led to an increase of 2 cm in the source-to-surface distance on the patient's left side and 0.8–1 cm on the patient's right side. In response to this change, the RT team decided to implement an adaptive treatment plan, which necessitated acquiring a new CT scan and adjusting the PTVs. Hence, the purpose of the Planning Challenge was twofold. The first was to develop a high-quality plan for the initial anatomy, and the second was to, based on this knowledge, to develop an adaptive plan for the changed anatomy to meet dosimetric requirements in an efficient manner. The plan quality in this Challenge was evaluated using the ProKnow scoring system presented in section 2.2.2.

Zoom In Zoom Out Reset image size

We assessed the planning performance of the developed VTP on the both initial and adaptive phases as part of the AAMD Planning Challenge. As we were participating in the Planning Challenge, we predefined a set of reasonable TPPs for VTP as the initialization of the TPP adjustment process. For the adaptive plan, we used the TPPs of the final plan in the initial phase as the starting point. Planning time was measured as the active time spent on planning operations from beam placement to plan completion.

In order to further gauge the VTP's applicability and generalizability to managing clinically realistic cases, we extended our testing to include 20 clinical cases for patients who had undergone H&N cancer RT treatment at our institution between 2018 and 2023. Patient characteristics are listed in supplementary material table S2. The patient cohort was chosen primarily based on the prescription dose of 63 Gy in 30 fractions, following the setting in the AAMD planning challenge case. We did not consider primary tumor location or extension in patient selection. For each case, we used VTP to generate a plan. Unlike the AAMD case that included four dose levels (63, 60, 57, and 54Gy), patients at our institution are typically treated with 2–3 target dose levels. Other than the highest dose (63Gy) to the primary tumor, target dose levels may be similar to, but not exactly match, the AAMD case. For a target with different prescription dose level from the AAMD case, the fixed optimization parameters in table 1 with the closest prescription dose were chosen. For a fair comparison, each plan generated by VTP was normalized to achieve the same PTV coverage as their respective clinical plans. In our analysis, we compared the ProKnow scores between the plans generated by VTP and the scores from the clinical plans. Our assessment of plan scores and dosimetric metrics utilized a non-inferiority test with the null hypothesis that VTP is inferior to human planners at a significance level of $p^* = 0.05$. We also assessed the planning time of VTP, measuring its active operation duration from beam placement to plan completion. Additionally, we offered insights into the decision-making capabilities of VTP by contrasting the plan quality in its initial, intermediate, and final steps along the treatment planning process, thus highlighting the enhancements made during the process.

VTP successfully generated a high-quality H&N cancer VMAT plan for the initial phase of the 2023 AAMD Plan Study, utilizing Eclipse TPS. Figure 4(a) presents the dose distributions across six axial slices from the top to the bottom of the targets and figure 4(b) displays the DVHs of the resulting plan. Remarkably, the VTP effectively balanced PTV and OARs during the plan optimization process as the plan maintained PTV coverage and spared the dose to OARs. For a comprehensive overview of the final VTP-generated plan, we presented the ProKnow scores in figure 4(c). The VTP-generated plan achieved full compliance with all clinical metrics outlined with a cumulative score of 139.08 out of 150. Our VTP plan was ranked the 21^st among 149 human-submitted plans with scores 127.32 ± 13.73 (American Association of Medical Dosimetrists 2023). This achievement demonstrated VTP's competence in treatment planning, particularly considering the complexity of the H&N cancer case, where human planners can employ more auxiliary structures to refine dose distribution.

Zoom In Zoom Out Reset image size

During the RT treatment course, the patient's substantial weight loss necessitated a reevaluation and adaptation of the treatment plan to accommodate the anatomical changes. Figure 5 presents dose distributions in six axial and DVHs of the adaptive plan for the changed patient anatomy. Similar to the process generating the initial plan, the VTP demonstrated remarkable proficiency in preserving adequate coverage of PTVs while effectively minimizing radiation dose to OARs. The primary goal of the adaptive phase of this Challenge is to expedite the patient's treatment with the new plan while striving to achieve the dosimetric goals with the utmost precision. The specific minimally required dosimetric metrics are detailed in figure 5(c). Notably, VTP generated a plan that fulfilled all the specified requirements within 15 minutes via a fully automated planning process. VTP achieved the $1\mathrm{st}$ position in the adaptive phase competition for the shortest planning time with plans meeting the plan quality requirements among all the participants. The average planning time of all the participants was 2.62 ± 2.24 h (American Association of Medical Dosimetrists 2023). The first author of this paper also participated in the Challenge as an AAMD-certified medical dosimetrist with 2.5 years of experience and achieved a plan meeting all requirements in 30 min of planning time.

Zoom In Zoom Out Reset image size

We first illustrate VTP's decision-making behaviors of operating the Eclipse TPS in a representative example case. In figure 6, we present the details of the planning process driven by the VTP to operate the TPS to improve the plan scores from 132 to 139.41 at various planning steps. Through strategic adjustments of TPPs, VTP effectively improved the scores of various metrics. For instance, by emphasizing the importance of the left parotid gland (TPP₂), right parotid gland (TPP₄) and pharyngeal constrictor (TPP₇) in the plan optimization, VTP enhanced the scores of $V_\mathrm{ParotidL}[30Gy](\%)$, $D_\mathrm{ParotidL}[mean](Gy)$, $V_\mathrm{ParotidR}[30Gy](\%)$, and $D_\mathrm{Constrictors}[mean](Gy)$ by 0.75, 0.66, 0.38, and 1.19, respectively. Additionally, VTP demonstrated intelligence to make improvements by decreasing the priority of the planning structure. For example, VTP reduced the priority of the oral cavity (TPP₅). As the oral cavity intersects with PTV volumes, this adjustment increased scores of $V_\mathrm{PTV}[54Gy](\%)$, $V_\mathrm{PTV}[57Gy](\%)$, and $V_\mathrm{PTV}[60Gy](\%)$ by 2.48, 0.95, and 0.09, without negatively impacting the score of $D_\mathrm{Oral}[\mathrm{mean}](Gy)$. Note that this entire process was autonomously completed by the VTP without human intervention. It hence indicated the decision-making capability in handling the challenging problem of H&N cancer treatment planning.

Zoom In Zoom Out Reset image size

To demonstrate the model's effectiveness in treatment planning, we conducted an assessment of VTP involving 20 clinical cases of patients who received treatments at our institution.

Figure 7 compares dose distributions at four axial levels, plan scores, and DVHs between VTP-generated and human-generated plans for a representative patient case. The VTP-generated plan presented superior dose sparing of OARs, including the brainstem, spinal cord, parotid glands, oral cavity, esophagus, and pharyngeal constrictor, without sacrificing the PTV coverage. The VTP-generated plan achieved a total plan score of 130.86, significantly outperforming the clinical plan, which scored 108.29. It is worthwhile to point out that the VTP-generated plan had lower homogeneity with larger hot spots in PTV60 and PTV57, which may be ascribed to the fact that the ProKnow criteria did not include this requirement, and hence VTP was not trained to achieve this. In contrast, the human planner prioritized homogeneous doses across all PTVs. Despite achieving a higher ProKnow score and better OAR sparing, the hotspots in the lower dose PTVs may not be clinically acceptable.

Zoom In Zoom Out Reset image size

Figure 8 summarizes the ProKnow scores for 18 clinically relevant dosimetric metrics measuring various aspects of plan quality, as well as total plan score, and MUs. We also performed a non-inferior statistical test for each metric to compare them. Statistical significance was considered with a threshold of $p^* = 0.05$. On the whole, VTP achieved an average plan score of $125.33\pm11.12$, in contrast to clinical plans scored an average of $117.76\pm13.56$ (p = 0.069). This indicated that the quality of VTP-generated plans was slightly better than those generated by human planners, although the improvement was not found to be statistically significant.

Zoom In Zoom Out Reset image size

Overall, there was a trend that scores of plans generated by VTP were higher than the corresponding counterparts in the clinical plans. VTP outperformed human planners statistically significantly in several specific metrics, including $D_\mathrm{cord}[0.03cc]$ (p = 0.014), $D_\mathrm{Oral}[\mathrm{mean}]$ (p = 0.041), $D_\mathrm{Esophagus}[\mathrm{mean}]$ (p = 0.045), and $D_\mathrm{Constrictors}[\mathrm{mean}]$ (p = 0.035). Specifically, VTP-generated plans received scores of $4.97\,\pm\,0.12$, $5.00\,\pm\,0.01.56$, $4.90\,\pm\,0.32$, and $3.15\,\pm\,0.51$ for these four metrics, respectively. In contrast, clinical plans scored at $4.59\,\pm\,0.40$, $4.69\,\pm\,0.50$, $3.93\,\pm\,1.98$, and $4.21\,\pm\,1.05$.

VTP-generated plans exhibited higher MUs, with an average of $795.35\,\pm\,125.18$ MU per fraction, compared to human-generated plans with $607.43\,\pm\,108.16$ MU (p = 0.002). This implies that VTP-generated plans tend to have more intensity modulations and greater complexity. Considering that the prescription of the highest dose level was 210 cGy per fraction (6300 cGy in 30 fractions), the modulation factors defined as the ratio between MUs and prescription dose were ${\sim}3.8$ for VTP plans, and ${\sim}2.9$ for clinical plans, both lower than the empirically acceptable threshold of 4.0 to warrant deliverability of the plans.

The automated planning process proved efficient, producing plans in an average of $91.46\pm15.14$ minutes. VTP's decision-making was nearly instantaneous, with the majority of planning time spent on Eclipse's optimization process.