Dear Editor,

We have read Cruz and Alexander’s comments regarding our manuscript titled “Evaluating Crown Fire Rate of Spread Predictions from Physics-Based Models” [1] and appreciate the opportunity to respond to their comments. In our original manuscript [1], we presented an evaluation of crown fire rate of spread predictions from two physics-based wildland fire behavior models: FIRETEC and the Wildland Urban Interface Fire Dynamics Simulator (WFDS). Our approach, as outlined in our original paper, relied upon using a previously published dataset of real-world crown fires, hereafter referred to as AC06 [2], and previously published fire rate of spread predictions. Following recommendations for model performance assessment proposed in Rykiel [3], we examined the proportion of simulated crown fire rates of spread that fell within a 95th percentile prediction interval we developed using the data in AC06. We also provided several discussion points about the challenges associated with evaluating physics-based model performance, potential sources of error/disagreement, and future research needs. Our manuscript built upon an existing body of work and offered insights based on one of many potential approaches that can be used to assess physics-based model performance.

In their comment on our manuscript, Cruz and Alexander suggest that: (1) our lack of understanding regarding the development of and uncertainties associated with the data reported in AC06 led us to misconstrue the usefulness of the dataset, and (2) that our approach is flawed and of limited utility, leading us to make a number of unfounded conclusions. Regarding [1], they supply several clarifying statements concerning the methods used in the development of the AC06 dataset and reiterate that these assumptions were both necessary and critical for their original purpose of independently evaluating an operationally-used crown fire rate of spread model. We agree with Cruz and Alexander that these assumptions were necessary and critical for them to evaluate an empirically derived crown fire model and that the dataset is indeed of value. We did not in any way intend to devalue the AC06 dataset. In fact, we believe that AC06 is one of the best available datasets at this time, which is why we chose to use it in our manuscript.

Cruz and Alexander suggest that if we had a better understanding of the assumptions used in the development of the AC06 data set, we would or could have chosen to perform an analysis based on direct model comparisons and that this approach would have yielded greater insight. We agree with Cruz and Alexander that direct point-to-point comparisons are an ideal approach to assess model performance. However, we disagree with them on the ability for the AC06 dataset to support such comparisons for physics-based models, given the number of required model input parameters and boundary conditions that are either unknown or unreported in the dataset. The absence of input and boundary conditions in AC06 needed to parametrize the two models we used is not a criticism, but rather a reflection of the fact that this dataset was not developed for the intended purpose of validating physics-based model predictions. To conduct the direct-point-to-point comparison suggested by Cruz and Alexander would have required us to assign values for all missing and unknown input and boundary conditions not reported in the ACO6 dataset. For example, the AC06 dataset reports three pieces of information regarding the fuels complex for each fire: mean canopy bulk density, estimated fine fuel moisture for surface fuels, and the major fuel type; however the physics-based models we evaluated require estimates of a number of basic surface fuel properties (e.g. the fuel load, bulk density, surface area to volume ratio), as well as additional information on the properties of the individual overstory trees (e.g. number of trees per hectare, crown base height, tree height, location etc..). Although Cruz and Alexander’s assertion that we could have simply made a series of assumptions to estimate all the missing input data seems logical, this approach is not recommended due to the substantial number of unknown or uncertain inputs needed for model parametrization [e.g., 4]. It is our belief that had this approach been taken, it would result in large predictive uncertainties, given the range of plausible input values, that would severely limit our ability to assess the degree to which the model accurately represents the real world. For this reason, we remain confident that our approach is appropriate given the lack of detail regarding numerous required input parameters within the AC06 dataset. Although we believe that the wildland fire science community should continue to use established data sets such as AC06 in model performance studies, we also believe that advancements in physics-based modeling evaluation are most likely to occur through the initiation of new “validation experiments”. These experiments will ideally be co-designed by modelers and experimentalists to supply parameter and uncertainty estimates for all key information required to initialize the models and ultimately allow for multiple approaches and lines of evidence to be used to assess model performance [5].

Finally, in their comment Cruz and Alexander suggest that the interpretation of our findings are not justified at least in part, based on our model assessment criteria. Given that there is a wide range of views on models, model uses, and no single agreed-upon performance criteria within the wildland fire science community, it is not surprising that there is an equally wide set of opinions and thoughts regarding the best approach and criteria to use during model assessment. Our main conclusions that “overall 86% of all simulated ROS values using FIRETEC and WFDS fell within the 95% prediction interval of the empirical data” is but one measure used to characterize the models’ prediction performance. In our manuscript we clearly stated what performance criteria we used and suggested that any end user should judge both our criteria and the model performance relative to their purpose. We appreciate that Cruz and Alexander provided a clear example where a more refined approach to model performance assessment than we conducted would be required to assist in decision making. However, it is important to recognize that the inability of our approach to provide this level of assessment does not imply a lack of usefulness of the models to represent reality within some limits of uncertainty. Given the limited data available in AC06 to initialize, bound and evaluate physics-based models (e.g., initial atmosphere conditions, type and distribution of vegetation, location and spread rate along a fire’s perimeter), we stand behind our approach, choice of performance criteria, and interpretation.