Fig. 7
Model results for the Quercus kelloggii Ryan and Amman (R-A) model. Q. kelloggii provides an illustrative example of the problem with the R-A model exhibited for many Quercus species: the main damage variable—percentage crown volume scorched (CVS)—has a weak relationship with observed mortality. (A) Locations of fires occurring from 1981 to 2016 within the USA from which data to evaluate the model was sampled. Fire locations are plotted over the species’ range (green polygons). BTcoef = specific specific bark thickness coefficient. (B) The bi-plot shows where the observations used to evaluate models (orange points) fall within the species’ bioclimatic niche space (black points) in terms of temperature (x-axis) and precipitation (y-axis). (A) and (B) Q. kelloggii was relatively well sampled across its range and environmental niche, particularly compared to other Quercus species. (C) Model evaluation summary statistics including the AUC (area under the receiver operator characteristic curve) at 0.5 threshold for determining mortality, and confidence intervals (CI) around the AUC. Model evaluation statistics include accuracy, sensitivity (Sens.), specificity (Spec.), positive predicted values (PPV), and negative predictive values (NPV), summarized over a range of probability thresholds (0.1 to 0.9; rows), with the commonly used threshold of 0.5 shown in bold. At a 0.5 threshold, the model had very high sensitivity (0.96), and high NPV (0.80), but low accuracy (0.49), low specificity (0.10), and relatively low PPV (0.49). Warmer colors indicate higher values. The top three bold rows show model performance metrics for the “best” threshold, which optimizes sensitivity and specificity, the best threshold with sensitivity >0.8, and the best threshold with specificity >0.8. (D) The distributions of defense (diameter at breast height [DBH], as an interpretable representation of bark thickness) and injury (crown volume scorch) variables used in the model are shown with bi-plots. Box plots in the margins of (D) show median (bar), interquartile range (IQR; box; 25th and 75th percentiles), and whiskers show the minimum and maximum values that do not exceed a 1.5 × IQR. Like all the Quercus species, the sample was unbalanced, with more live tress (n = 219) than dead (n = 184) and the average CVS was higher for live trees than dead trees. (E) and (F) Assessment of species-level error comparing the predicted probability of mortality using a 0.5 threshold (Pm; orange points show values and shading shows range) and the observed proportion of trees or stems killed (gray points) within binned observations of the primary injury variables (E), and the DBH (F). (E) The lack of a relationship between CVS and mortality is apparent in this panel, which shows that model predictions and observed levels of mortality diverge as crown scorch increases, with no relationship between observed mortality and CVS level except when CVS >80%. (F) The proportion of observed mortality in smaller-diameter trees is much lower than the model predicts. Qualitative ratings of data quality, model performance, and direction or error in model predictions are listed at the bottom of the figure