Assessing Accuracy of Manually-mapped Wildfire Perimeters in Topographically Dissected Areas

Abstract

Accurate mapping of wildfires is critical to fire management. Technological advances in remote sensing and Geographic Information Systems (GIS) over the last decade have been widely incorporated into wildfire mapping and management, but neither have been assessed for accuracy nor compared to established manual methods. Since Landsat-based mapping of wildfires will soon replace manual mapping methods, this type of comparison is critical to understanding the strengths of each method. Landsat ETM+ imagery was classified to create fire perimeter maps for 53 fires in Nevada, USA. These maps were then assessed for agreement with published, manually mapped fire perimeters. Published perimeters were found to correlate poorly to remotely sensed fire perimeters, and significantly overestimated area burned (p ≤ 0.05) by an average of 18 percent. Mapping disagreement was then correlated to a measure of topographic roughness at four spatial scales to determine whether increasing terrain complexity was a factor in increased disagreement. Mapping disagreement showed a significant positive correlation (r = 0.57) to topographic roughness. For fire research spanning multiple decades, these results indicate that it may be difficult to utilize fire perimeter data sets comprising both satellite-derived and manually mapped perimeters because the two data sets are significantly different.

Introduction

Advances in Geographic Information Systems (GIS) and spatial analysis of remotely sensed data have greatly improved a variety of land management applications (Franklin et al. 2000). Wildfire management has benefited enormously from spatial technologies, particularly given the inherent risk of working around wildfires and the difficulties in acquiring in situ data (Ambrosia et al. 1997, Lentile et al. 2006). Integration of spatial technologies, however, requires periodic reassessment to determine the level of accuracy and efficiency achieved using current methodologies (Congalton 1999).

Mapping and measuring of wildfire perimeters and area burned has evolved considerably since the early 20th century. All active wildfires that have suppression personnel present are usually mapped at least once per day (http://geomac.usgs.gov). This process assists fire managers in determining their resource needs and daily assignments. Additionally, fire perimeters need to be mapped as rapidly and efficiently as possible following the fire to begin Burned Area Emergency Rehabilitation (BAER) efforts. Currently, most fire perimeters are mapped in one of two ways. The primary method utilizes a Global Positioning System (GPS) mounted on a helicopter, where the pilot obtains boundary georeference points by flying the burn perimeter. On fires where a helicopter is not available, fire managers walk the burn perimeter or use infrared photography. Once the perimeter is mapped, the area burned each day is calculated using a GIS tool for planar area calculation (GAO 2003).

To map the perimeter of a wildfire accurately, either the pilot of the helicopter or ground personnel must follow the burning edge exactly using a GPS. This is difficult for several reasons. For the pilot, the difficulty lies in the need to maintain a safe flying altitude and dealing with low visibility as a result of smoke, heavy vegetation cover, and shadow effects. If aerial reconnaissance is not used, ground-based mapping of the fire edge is difficult due to the challenges of following burned edges in rough terrain and the non-uniform manner in which wildfires burn across the landscape. Due to these challenges, two potential sources of mapping error arise: detection and delineation of unburned islands, and accurate delineation of fire boundaries. First, on most wildfires there are islands of unburned vegetation scattered throughout the burned area, ranging from only a few isolated trees to areas encompassing hundreds of hectares. These islands are often not mapped because of safety concerns or the sheer impracticality of delineating numerous small patches by helicopter or on the ground (see Figure 3 as an example). Additionally, there is inherent subjectivity in deciding the minimum mapping unit for delineating unburned islands of various sizes. The second general source of error concerns mapping of the fire perimeter. Delineation of the burn perimeter is highly subjective since this boundary is itself a patchy, convoluted “fuzzy edge” that is difficult to define when on the ground, let alone flying overhead in a helicopter. Safety concerns may also contribute to boundary mapping error since in extreme terrain it can be unsafe to stick to the true fire perimeter, and more prudent to include some unburned areas by taking a different access route.

An alternative option to GPS mapping uses remotely sensed data to delineate fire edges. On a daily basis, this is accomplished using aerial infrared photographs captured before dawn to locate active fire areas, or “hot spots.” On a coarser spatiotemporal scale, space-borne sensors with infrared bands can provide data that have been used extensively for BAER analysis of burn severity over the last decade (Lentile et al. 2006). The satellite platforms with the most useful spatiotemporal resolution include Landsat (30-m pixels, 16 day revisit cycle) and SPOT (20-m pixels, 26 day revisit cycle). The change in infrared and red reflectance between burned and unburned vegetation is quantified as the differenced Normalized Burn Ratio (dNBR) to empirically gauge the level of burn severity across a burned area (Key 2005). Just as the manual mapping methods have associated potential sources of error, the ability of remotely sensed methods to adequately capture areas of low burn severity in some regions has been questioned by many (Cocke et al. 2005, Epting et al. 2005, Holden et al. 2005). Remotely sensed burn severity mapping depends upon the ability of the sensor to see the burned area, and in regions and vegetation types where an unburned overstory canopy occludes a low severity understory burn, the sensor may not detect significant change, and low severity burns may be classified as unburned (Cocke et al. 2005). In some soil types, changes in reflectance and brightness may also distort the ability to discriminate burned versus unburned areas (Chafer et al. 2004). Perhaps the greatest mechanism for error in delineating burn severity, however, lies in the variability of solar angle and shadow effects during image acquisition. As noted in two Australian studies, a low sun angle during image acquisition results in misclassification of burned areas, particularly in regions that are topographically complex, both from shadowing effects and from reduced or highly variable solar intensity depending on the surface aspect and albedo (Hammill and Bradstock 2006, Walz et al. 2007). Much of the misclassification in these cases occurs in the low and moderate burn severity areas, with some burned areas classifies as unburned, which is problematic for delineation of fire perimeters since areas misclassified as unburned areas would be excluded. Holden et al. (2005) noted, however, that despite the potential sources of error associated with deriving burn severity from Landsat imagery, accuracy of perimeter delineation should be highest in areas of high burn severity, and Chafer et al. (2004) noted that discrimination of burned areas is easier in xeric regions based on soil reflectivity. Since the study region assessed here is xeric and most fires burn entirely at high severity (USDI 2000), the potential for error is significantly reduced.

Despite the potential drawbacks of spaceborne derived burn severity, remotely sensed mapping methods will soon be the standard for mapping large fires in the U.S. The U.S. Geological Survey (USGS) is amidst a multi-year project to create a historic fire atlas for all fires since 1984, of greater than 400 ha in the western U.S. and 200 ha in the eastern U.S. The Monitoring Trends in Burn Severity (MTBS) project, as it is known, will utilize dNBR to produce both fire severity and fire perimeter maps (Eidenshink 2006). This reassessment of historical Landsat imagery will provide a new large-fire database for the U.S. and has implications for trend analyses that utilize the current large-fire databases such as fire patterns (Rollins et al. 2001), fire and climate relationships (Westerling et al. 2006), and land-cover change studies (Rollins et al. 2002). It is uncertain how the accuracy of the MTBS database will compare to the current regional large-fire databases (e.g. Brown et al. 2002), which Holden et al. (2005) found to have mapping errors of greater than 20% for two fires in New Mexico, USA. It is critical to understand what kind of disagreement potentially exists between fire perimeter maps produced by the two methods, however, since research across multiple decades (e.g., Minnich 1983) will potentially be comparing perimeters created utilizing the two different methods.

Because MTBS methods will be the standard for mapping fires in the future, and because our study area fires burned at high severity in xeric grass, shrub, and woodland communities, we assumed for the purposes of this study that Landsat-based fire mapping methods are more accurate than manual methods and described disagreement between the two methods as error on the side of manual mapping methods. The objectives of this study were to: 1) use remotely sensed (Landsat ETM+) imagery (the same imagery being used for MTBS) to assess the disagreement (described hereafter as error) with wildfire perimeter mapping conducted using traditional manual methods; and 2) determine if topographic roughness is a factor in the level of mapping error. We hypothesized that increased topographic complexity would correlate positively to increased error in manually mapped fire perimeters, since flatter terrain is conducive to better visibility and reduced concerns for safety on the part of the helicopter pilots and on-the-ground personnel.

Methods

Wildfires were selected for analysis from the Nevada Bureau of Land Management (BLM) published fire perimeters for the 1999 and 2000 fire seasons based on two criteria: 1) availability of cloud-free Landsat 7 ETM+ scenes within 90 days of the fire occurrence, and 2) a published burned area between 1,000 ha and 40,000 ha. Archival data were acquired from the Intermountain Region Digital Archive Image Center at Utah State University (http://earth.gis.usu.edu). In total, 53 fires were analyzed, all in northern Nevada (Figure 1).

Figure 1

Locations of 53 fires analyzed in Nevada.

Processing Imagery

Fire perimeters were mapped from 30-m Landsat 7 ETM+ data that had been Level 10 “Terrain Corrected” for the National Landsat Archive Production System, and so had been both geometrically and radiometrically rectified. For each fire, an NBR image was created to improve detection of burned vegetation (Key 2005). NBR delineates burned area using a ratio of two short-wave infrared bands, Band 4 (0.76 µm to 0.90 µm) and Band 7 (2.08 µm to 2.35 µm) in the difference equation, Equation 1:

$${\rm{NBR}}\;({\rm{x}}) = {{({\rm{Band}}\;4 - {\rm{Band}}\;7)} \over {({\rm{Band}}\;4 + {\rm{Band}}\;7)}}$$

(1)

A 3 × 3 low pass filter was used to remove single cell island artifacts for each fire and an unsupervised classification was performed on the filtered NBR image for each fire to delineate burned and unburned areas. Between two and five classes were identified, depending on the image. A raster-to-feature transformation was then used to create a fire perimeter (Figure 2). Each post-processing perimeter was then overlapped with the fire perimeter polygons published by the BLM. For each pair of fire maps, we calculated the percent of area in agreement, the percent area mapped as burned but not actually burned (i.e., error of commission), and the percent area actually burned but not mapped as burned (i.e., error of omission) (Figure 3).

Figure 2

Classification of fires using (from l – r) NBR, unsupervised classification, and a raster to feature transformation.

Figure 3

Percent agreement (A) calculated based on overlap of BLM-mapped polygons and the post-processing polygons from remotely sensed data, with error of commission (B) where it did not burn and error of omission (C) where it did burn but was not mapped as such.

Assessing Topographic Roughness

For this study, four measures of topographic roughness (TR), also known as terrain roughness, were created for each fire to assess the influence of TR for mapping accuracy at multiple spatial scales. The Jenness TR measure (Jenness 2004) calculates TR as the ratio of surface area to planar area, a measure that was also used by Guyette and Dey (2000) in their assessment of topographic roughness on potential wildfire intensity. This ratio, however, estimates TR at the scale of the entire fire. To address the issue of topographic roughness over multiple spatial scales, we created maps of standard deviation of elevation from the 30-m DEM using the Focal Statistics tool in ArcToolbox 9.1. Three standard deviation filters of sizes 3 × 3, 25 × 25, and 75 × 75 were applied across the region, and a standard deviation raster map extracted for each fire for each of the three sizes. The median values of standard deviation were reported for each map, constituting the remaining three values of TR for each fire. We also correlated size of fire to mapping accuracy to determine whether larger fires were more difficult to map accurately.

The three different filter sizes for the focal statistics calculation were chosen to correspond to varying scales of topographic roughness on a landscape. The 3 × 3 filter (90 m × 90 m in dimension) captures the local topographic roughness characterized by stream channels and other erosion features. The 25 × 25 filter (750 m × 750 m) captures mass-wasting events, toe slopes, and other high-resolution geomorphic features. The 75 × 75 filter (2250 m × 2250 m) captures the topographic complexity of a section of mountain range, including canyons, ridges, valleys and the transitions from foothills to montane, multiple canyons and ridges; i.e., the lowest-resolution landscape features.

Statistical Methods

A paired Student’s t-test was used to assess significant differences in area burned between the published map perimeters and the postprocessing perimeters from the imagery, with a confidence level of 95% (p ≤ 0.05 alpha error). To test whether the error in mapping was a function of TR, we calculated a Pearson correlation coefficient to correlate percent agreement, percent omission, and percent commission in mapping to each of the four values of TR. We also correlated the three error percentages to area burned to determine whether mapping accuracy is associated with fire size.

Results

Mapping Fire Perimeters

Percent agreement between published and Landsat-derived fire perimeters ranged from 40% to 93%, with a mean of 76%. Errors of omission ranged from 0% to 45%, with a mean error bias of 5%, while errors of commission ranged from 6% to 60%, with a mean error bias of 18%. There was a significant difference between published area burned and the Landsat-derived fire area burned (t = 4.42, d.f. = 52, p = 0.0001), with a range of 2% to 63% total change in area, and a mean of 17% (Figure 4). Two fires produced severe outliers (3.0 × Inter Quartile Range) evident in the error box plots for percent omission and total change. On the 1999 Eugene incident, the high error of omission is attributed to the entire eastern section of the fire not being mapped, although it is unclear why this section (which appears in the imagery as fairly flat terrain along an alluvial fan) was not mapped. The 1999 Piney fire was a rangeland fire occurring near a road system. The suppression tactics included initiating a burnout operation along the road, but the wind direction changed and did not carry the main body of the fire in the direction of the burnout. As a result of this, the burnout section was not mapped as part of the fire.

Figure 4

Range of variability in three fire mapping agreement categories and total change in area, shown by box plots, with mild (1.5 × IQR) outliers indicated by squares and severe (3.0 × IQR) outliers indicated by circles. IQR = interquartile range.

Of the three categories of agreement (agreement, commission, and omission), percent commission showed the strongest correlations to topographic roughness (Table 1). Percent commission was significantly and positively correlated to all four values of TR (p ≤ 0.01 alpha error), with the strongest correlation (r = 0.570) to TR25. The correlations between percent agreement and TR were negative, and weaker than for errors of commission, but still significant at the 99% confidence level, and with the strongest correlations also against TR25. Errors of omission were not significantly correlated to any TR category. Area burned was also not significantly correlated to any of the agreement levels.

Table 1 Pearson’s r correlation coefficient for total area burned and each of the four TR values to mapping agreement, omission, and commission values. Significant relationships are indicated with an asterisk (*).

Discussion

The results indicate that the level of disagreement between wildfire perimeter mapping methods was significant, and that the level of error in manual mapping significantly increased in areas of higher TR. This is consistent with the hypothesis that manual mapping errors can be attributed to the difficulties in mapping associated with rougher terrain. Since area burned was not significantly correlated to mapping agreement or error, the size of the fire did not alter the level of accuracy in mapping the fire perimeter.

Areas of greater terrain roughness were prone to increased manual mapping error on the side of commission. There are at least two explanations for this. First, much of the area falsely classified as burned consists of small island polygons. As described in the introduction, unburned islands within a burned area are almost always included in manually mapped fire perimeters, since it is unsafe to map these locations during or even immediately following a fire, and more efficient to simply include them in the burned area. This is visually consistent with the imagery and the published fire perimeters for this study; there are several cases where a published perimeter skirts the base of a slope or canyon instead of following the fingers of burned area that lay on ridges or in canyons. Since errors of omission were not significantly correlated to TR, the weaker significant correlation between TR and overall mapping agreement can be attributed primarily to over-mapping in the areas of higher TR.

The scale of topographic roughness that had the greatest impact on both percent agreement and commission error was the TR25, or 750 m, level (Table 1). Since all three scales of TR had similar significant correlations, however, this indicates that error is independent of scale.

As previously discussed, there has been much debate over the accuracy of NBR at delineating burned area for forest vegetation and where burn severity is low or there is rapid regeneration of vegetation (Cocke et al. 2005, Epting et al. 2005, Holden et al. 2005). However, the 1999 and 2000 fires in Nevada burned primarily at high severity in grass and shrub ecotypes, meaning that the Landsat-based NBR method for delineating area burned is essentially detecting a conversion from vegetated to non-vegetated landscape in this study (USDI 2000). Additionally, the timing of the image acquisition, which has been noted as fairly critical in other studies (Holden et al. 2005, Hammill and Bradstock 2006), was ideal in this study, with post-fire imagery acquired prior to the fall rains and any revegetation of the burned area. Along these lines, this study might be characterized as ideal for burned area delineation with Landsat imagery, particularly because the high number of cloud-free days in Nevada makes it an optimal location for acquiring cloud-free imagery on a regular basis. Other regions and ecotypes, which see longer time periods between optimal Landsat imagery acquisition due to cloud cover, or which have a higher mix of burn severity leading to classification errors, may yield examples where manual mapping is more accurate and more timely than imagery-based. Additionally, utilization of other types of satellite imagery, such as the Moderate Resolution Imaging Spectroradiometer (MODIS) (e.g., Walz et al. 2007), or other indices of burn severity (reviewed in Epting et al. 2005) may reveal different levels of agreement between manual and remotely sensed mapping methods.

Implications for Land Management and Fire Research

For land management purposes, there are numerous ramifications associated with incorrect mapping of wildfire perimeters. Economically, wildfire perimeters are utilized to allocate resources for fire suppression efforts, as well as to aid rehabilitation efforts. Millions of dollars are spent each year rehabilitating landscapes after wildfires, and methods for rehabilitation are chosen based on costs per unit area. Additionally, fire budgets for subsequent years are estimated from previous annual area-burned totals. Overestimation of area burned is problematic from a funding appropriations standpoint, and also from a scientific standpoint. Published perimeters and associated estimates of area burned are regularly used by the scientific community for a variety of wildfire research questions (Rollins et al. 2002, Westerling et al. 2006). Especially problematic is the apparent bias in fire area calculation associated with surface roughness of the terrain. Error in fire boundary delineation is not randomly distributed among different study regions.

Remotely sensed data are captured by a variety of satellites each day, and the spatial and temporal resolution of these data continue to improve. While the current Landsat ETM+ sensor captures data for a location only once every 16 days, other sensors [e.g., SPOT, Airborne Visible/Infrared Imaging Spectrometer (AVIRIS), MODIS] with infrared bands can be used to gather the information required for wildfire analysis on a daily basis (van Wagtendonk et al. 2004, Walz et al. 2007). Additionally, these data are acquired at less risk to personnel engaged in mapping burn perimeters. The MTBS project will not only streamline the methodology for creating and cataloguing wildfire perimeters, but will also simplify and speed up the process of image acquisition such that imagery-derived perimeters can be utilized in the post-fire rehabilitation period. While there are still concerns as to the accuracy of NBR-based methods for mapping wildfire burn severity, the production of the national historical fire atlas will provide a more accurate set of fire maps for wildfire research and land management needs. Our findings suggest that future land managers and researchers utilizing manually mapped perimeters and the MTBS atlas data need to be aware of the significant overestimation of area burned in manual mapping methods.

Conclusions

Remotely sensed data analysis of 53 wildfires showed that fire perimeter mapping error can be significant using field and helicopter based methods, and that the error of commission is likely the primary contributor to overall error. The error of commission increased significantly with increased terrain complexity at all spatial scales, suggesting that land managers have difficulty mapping fire edges correctly in the roughest terrain, and inadvertently overestimate the area burned. These errors are problematic for land managers and researchers who use published fire perimeters and area burned databases (including the federal historic wildland fire database) for research and information to support land management applications. The MTBS project being undertaken by USGS will provide a more reliable source for fire perimeter and area burned data. Availability of GPS and remote sensing technology for high-precision delineation and mapping of earth surface features opens new horizons for careful monitoring of key landscape perturbations such as result from wildfire. However, as for any transition from older to newer technologies, there has been a necessary period of error and adjustment.