A benchmarking method to evaluate the accuracy of a commercial proton monte carlo pencil beam scanning treatment planning system

Abstract AcurosPT is a Monte Carlo algorithm in the Eclipse 13.7 treatment planning system, which is designed to provide rapid and accurate dose calculations for proton therapy. Computational run‐time in minimized by simplifying or eliminating less significant physics processes. In this article, the accuracy of AcurosPT was benchmarked against both measurement and an independent MC calculation, TOPAS. Such a method can be applied to any new MC calculation for the detection of potential inaccuracies. To validate multiple Coulomb scattering (MCS) which affects primary beam broadening, single spot profiles in a Solidwater® phantom were compared for beams of five selected proton energies between AcurosPT, measurement and TOPAS. The spot Gaussian sigma in AcurosPT was found to increase faster with depth than both measurement and TOPAS, suggesting that the MCS algorithm in AcurosPT overestimates the scattering effect. To validate AcurosPT modeling of the halo component beyond primary beam broadening, field size factors (FSF) were compared for multi‐spot profiles measured in a water phantom. The FSF for small field sizes were found to disagree with measurement, with the disagreement increasing with depth. Conversely, TOPAS simulations of the same FSF consistently agreed with measurement to within 1.5%. The disagreement in absolute dose between AcurosPT and measurement was smaller than 2% at the mid‐range depth of multi‐energy beams. While AcurosPT calculates acceptable dose distributions for typical clinical beams, users are cautioned of potentially larger errors at distal depths due to overestimated MCS and halo implementation.


| INTRODUCTION
For pencil beam scanning (PBS), Monte-Carlo (MC) based treatment planning systems (TPSs) [1][2][3] can potentially calculate dose distributions more accurately than those based on analytical algorithms. 4 Full MC calculations, such as Geant4 5 and MCNPX2.7, 6 often take hours to achieve satisfactory statistical uncertainty for PBS plans, which is unacceptable for clinical application. A fast MC calculation module, AcurosPT with simplified radiation transport, has been benchmarked with MCNPX 2.7 and released in a commercial TPS, Eclipse 13.7 (Varian Medical Systems, Palo Alto, CA). Like Acur-osXB, 7 AcurosPT uses a form of Fokker-Planck approximation for multiple Coulomb scattering (MCS). 8 In contrast with AcurosXB, however, which is a deterministic algorithm for coupled photon and electron transport problems, AcurosPT calculates dose distributions by direct MC simulation of PBS proton transport. A parallel implementation as well as several approximations enables AcurosPT to calculate the dose for a typical PBS treatment plan within a few minutes, yielding 2% statistical uncertainty of the target dose.
To achieve the required MC dose calculation accuracy, both the proton source and radiation transport mechanisms must be appropriately implemented so that proton spot profiles can be correctly calculated from primary and halo regions for multi-spot PBS plans. 9,10 On one hand, primary spot profiles are affected by the choice of MCS mechanism. 11,12 Conversely, the halo is predominantly caused by large angle scattering and nuclear interactions within the phantom, in addition to that from the nozzle (i.e. proton source). The halo can affect the output of proton beams by up to 10% depending on field size, proton energy and depth. 9,10,13 Therefore, in commissioning a new MC calculation module, it is essential to evaluate clinically relevant scenarios for both single and multiple spot profiles.
AcurosPT utilizes a simplified radiation transport to improve calculation efficiency. AcurosPT was internally benchmarked for agreement with MCNPX 2.7, as there was no measurement data available to the TPS vendor during the development of new MC calculation module. In that regard, Stankovskiy et al. 14 have reported underestimation of Bragg peak curves using fast Monte Carlo and benchmarking with MCNPX. 15 reported that calculated spot profiles are larger in MCNPX than in Geant4, while Sawakuchi et al. 16 have reported the overestimation of spot profiles in MCNPX compared with measurement.
The modeling of radiation transport affects the dosimetric agreement for most proton energies. For low proton energies (below 150 MeV), Lin et al. 17 pointed out that for some treatment nozzles, the halo from the nozzle can be more dominant than that in phantom. The treatment nozzle halo can be modeled in Monte Carlo based on nozzle design 16 or using experimental data at nozzle exit. 18 These user defined parameters are vital to the dosimetric agreement for low proton energies. Prior publications have stressed the importance of modeling radiation transport in phantom and proton source.
In this paper, we assess the accuracy of the AcurosPT dose calculation algorithm and the validity of the proton source by benchmarking AcurosPT against both measurement and an independent Monte Carlo dose calculation 19,20 TOPAS, 21 that is based on Geant4. This benchmark process is broadly applicable and can be utilized to identify potential inaccuracies in any newly released MC-based TPS during commissioning.

| METHODS AND MATERIALS
The IBA dedicated PBS nozzle and its design used in this study have been previously described. 17,22 The nozzle was characterized in the Eclipse TPS with the AcurosPT 13.7 calculation algorithm, which uses interaction data from MCNPX2.7. AcurosPT 13.7 is a Monte Carlo algorithm designed to provide rapid and accurate dose calculations for proton radiotherapy problems by simplifying or eliminating less significant physics processes to keep the computational run-time to a minimum. AcurosPT simplifies radiation transport into four cate- concern over the scintillator's energy-dependent response, 28 Lynx measurements were limited to relative dose distributions that involve minimal energy variation.
Absolute output and depth doses were validated in a water phantom, depth by depth, using a MatriXX PT â , which utilizes a new two-dimensional ionization chamber array designed specifically for the high dose rates in PBS delivery. 29 Agreement of field size factor (FSF) calculations with measurements, where FSF is defined to be the output at the center of a given square field size † relative to that for a field size of 96 mm, was used to quantify the accuracy of the halo characterization 25 in AcurosPT for different nominal proton energies. Absolute depth dose and profile measurements were validated for multi-energy beams RxMyFz, to reduce the measurement uncertainty along the depth direction. FSF analysis at selected proton energies aids in the interpretation of radiation transport inaccuracy related to proton energy. AcurosPT and measurement is greater than 99% using gamma criteria of 1 mm/1% for the pass rate, for all the Bragg peaks except some marginal 4% failures for the lowest 100 MeV beam ( Fig. 2(a)), validating the derived energy spread in Fig. 1(a). The TOPAS simulation is indistinguishable from measurement and therefore omitted from the figure. In contrast, Fig. 2(b) indicates that AcurosPT calculates too large a spot sigma, which increases with depth in comparison to measurement, leading to the largest disagreement near the end of the proton range. The disagreement can be up to 15% (maximum 1 mm) larger than measurement, which is larger than the measurement uncertainty. Allowing AcurosPT's spot sigma to be smaller than measurement at phantom entrance, this disagreement at all depths is comparable, however, to the tolerance of 10% or 1 mm recommended by 32 and 15% observed by, 33 and smaller than the 20% day-to-day variation specified by our proton therapy treatment delivery vendor. In contrast, the spot sigma calculated by TOPAS followed measurement much better than AcurosPT and was well within the measurement uncertainty.  AcurosPT indicates that the residual disagreement between Acur-osPT and measurement over the buildup or ripple/uniform regions are more likely due to measurement uncertainties rather than calculation inaccuracy in AcurosPT.

| RESULTS
Since the overestimation of spot sigma in AcurosPT is larger for more distal depths of high energy proton beams, the penumbra of a multi-energy beam has a larger difference at a mid-SOBP depth of   Fig. 4(b) also show that the distance-to-agreement of profiles at 95% and 5% of the uniform dose can exceed 2 mm with dose difference within 2%.

| CONCLUSION
In this article, we describe a benchmark method to detect potential radiation transport/proton source inaccuracies in a commercial MC TPS using measurement and an independent MC calculation. Using such a method, we detect the MCS and halo overestimation in Acur-