Assessment of k-Nearest Neighbor and Random Forest Classifiers for Mapping Forest Fire Areas in Central Portugal Using Landsat-8, Sentinel-2, and Terra Imagery

Abstract

Forest fires threaten the population’s health, biomass, and biodiversity, intensifying the desertification processes and causing temporary damage to conservation areas. Remote sensing has been used to detect, map, and monitor areas that are affected by forest fires due to the fact that the different areas burned by a fire have similar spectral characteristics. This study analyzes the performance of the k-Nearest Neighbor (kNN) and Random Forest (RF) classifiers for the classification of an area that is affected by fires in central Portugal. For that, image data from Landsat-8, Sentinel-2, and Terra satellites and the peculiarities of each of these platforms with the support of Jeffries–Matusita (JM) separability statistics were analyzed. The event under study was a 93.40 km² fire that occurred on 20 July 2019 and was located in the districts of Santarém and Castelo Branco. The results showed that the problems of spectral mixing, registration date, and those associated with the spatial resolution of the sensors were the main factors that led to commission errors with variation between 1% and 15.7% and omission errors between 8.8% and 20%. The classifiers, which performed well, were assessed using the receiver operating characteristic (ROC) curve method, generating maps that were compared based on the areas under the curves (AUC). All of the AUC were greater than 0.88 and the Overall Accuracy (OA) ranged from 89 to 93%. The classification methods that were based on the kNN and RF algorithms showed satisfactory results.

1. Introduction

Forests are subject to a variety of disturbances, which are strongly influenced by climate change and human activities [1]. Forest disturbance due to fires is a major challenge for forest management in various ecosystems due to the loss of life and infrastructure, emissions of greenhouse gases, degradation, soil erosion, and the destruction of species, biomass, and biodiversity [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]. According to the Intergovernmental Panel on Climate Change (IPCC), climate change tends to increase the risks of major fires on Earth.

Accurate information that is related to the impact of fire on the environment is a key factor in quantifying the consequences of fires on the landscape, planning and monitoring restoration and recovery activities, and providing relevant data for understanding the dynamics of fire, serving as a basis for future monitoring [31]. After a fire, detailed and rapid knowledge of the level of damage and its spatial distribution are the first desirable information. Accurate and complete data on fire sites and burned areas are important for a variety of applications, including quantifying trends and patterns of occurrences in a variety of natural and social systems [32,33,34,35,36,37,38,39,40,41].

The understanding of fire regimes and forest recovery patterns in different environmental and climatic conditions improves the management of sustainable forests, facilitating the process of forest resilience, according to Chu and Guo [42].

In the last decades, the use of remote sensing has allowed unprecedented advances in mapping fire dynamics, mainly to locate the occurrence of fire in time and space, and to quantify the total extent of the burned area. Several remote sensing studies have been carried out to map burned areas on a global and regional scale [10,12,38,39,43,44,45,46,47,48,49,50,51,52,53,54,55,56]. In particular, some authors have studied burned areas in Portugal using remote sensing techniques by [12,47,49,51,52,53,57,58,59].

The availability of well-calibrated global remote sensing data since the late 1990s has enabled the production of a variety of global and multi-annual products for burned areas, which are now freely available [60]. Several of these products are based on data from orbital sensor systems with different spatial resolutions (coarse, medium, and high), such as: Operational Land Imager (OLI)/Landsat-8, MultiSpectral Instrument (MSI)/Sentinel-2, Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER)/Terra, or Moderate Resolution Image Spectroradiometer (MODIS)/Terra. According to Libonati et al. [61], the development of a precise algorithm to detect changes in surfaces that are caused by fires on a global scale is hampered by the complexity, diversity, and high number of biomes involved. The limitations of estimating burned areas, on a global scale, can be reduced with the development of algorithms that consider characteristics, such as vegetation type, soil, and climate, and where validation and calibration exercises are less complex to implement [61].

Mapping burned areas using remote sensing techniques is based on post-fire changes due to the burns [57]. The approaches include supervised and unsupervised classification techniques at the pixel level. The quality of the classification of the natural environment is associated with the precision and reliability derived from satellite data, which are determined by the classification algorithm. This involves the image resolution (pixel, window, or segment size) that is used in the classification process. To evaluate the classifiers and obtain thematic precision, it is necessary to take the different classes of forest identified into account [62]. In the last decades, non-parametric methods, algorithms that are based on machine learning (MLAs), have gained great attention from applications based on remote sensing [63,64], although some of them, such as the k-Nearest Neighbor (kNN), have been used since the 1950’s [65,66,67,68,69,70,71]. MLAs have become widely accepted as evidenced by their use in mapping burned areas [44,46,72]. They perform well in situations that involve category prediction of spatially dispersed training data and are especially useful when the process under investigation is complex and/or represented by a high-dimensional input space [73].

In recent years, Landsat, Sentinel-2, and Terra data have been used in conjunction with MLAs to distinguish and map fires in different types of biomes, anthropogenic types of land use (including plantations), and degraded forests ([61,74,75]). Many of the classification algorithms have been compared with standard products from burned areas and active fires derived from satellite data, such as MCD64A1 [76], MCD14DL [75], Landsat Burn Area [77], or Fire_cci [78].

MLAs have also been implemented in satellite data to map fires, examine spectral properties, accurately delineate the area affected by the fire [79], analyze fire severity [72], and carry out precision analysis of the product [43,61]. Some of the most common MLAs for classifying and mapping burned areas include support vector machines (SVM), kNN, and Random Forest (RF) [80,81]. RF, for example, allows for integrating data from different scales and sources, which explains its wide use in many mapping applications based on satellite images [72]. In particular, several studies show the RF potential that is applied to satellite images for the detection of forest fires [82,83,84,85,86,87,88].

The ability of MLAs to distinguish and map different forest types, which have suffered varying levels of fire severity and their consequences across the planet, needs to be further assessed by different orbital sensors. This will support conservation management, being able to serve in places of different territorial extension. However, it should be noted, that there are few published studies on the performance of kNN and RF using different orbital platforms in areas burned by fire at the local scale, especially in Portugal [81,89,90,91,92].

In this work, the feasibility of kNN and RF classification algorithms to map areas that are burned by forest fires in a region of native pine vegetation in the municipalities of Santarém and Castelo Branco (central Portugal) is evaluated using Landsat-8, Sentinel-2, and Terra satellite data. The main aims are: (i) to examine the effectiveness of different remote sensing data sources for delineating the area affected by the fire; (ii) to compare, while considering the advantages and limitations of the sensors used, the performance of two MLAs (kNN and RF) that are commonly used to delineate and map forests that suffered fires; and, (iii) to evaluate the structural and spectral properties of the burned area and its influence on the classification.

We found that no significant differences in the burned area are obtained with each algorithm for each image sensor. The classifications carried out using both kNN and RF algorithms mapped the burned areas with high accuracy for the different sensors, regardless of the spatial resolutions and the spectral characteristics of each source data.

2. Materials and Methods

2.1. Study Area

Portugal is characterized by a mild Mediterranean climate with climatic variability, involving droughts and desertification in the southern sector, according to Miranda et al. [93]. The majority of burned areas in Portugal (80%) are due to fires, which occur in a small number of summer days (10%) when the atmospheric circulation forms a prominent ridge over the Iberian Peninsula with a strong flow to the south [94].

The study area (Figure 1) covers a 93.4 km² fire that occurred on 20 July 2019 in the districts of Santarém and Castelo Branco (central Portugal). In this area, the vegetation of maritime pine and microclimate predominate with prolonged summers, having very limited rainfall. High temperatures reduce the moisture content of forest fuels, often resulting in large fires when combined with strong winds [95].

According to Nunes et al. [96], who analyzed a set of 506 fires that occurred in Portugal in 1991, large fires (greater than 1500 ha) mainly occur in posts of Pinus pinaster, Eucalyptus globulus Labill., and Eucalyptus/Pine trees mixture, and later by bush. On the other hand, as these types of vegetation are sowers, which respond to fire through the rapid dispersion of seeds, post-fire regeneration in the central region of Portugal will crucially depend on the destruction of seeds that are present on the soil surface during the fire episode [97]. Therefore, it can be predicted that the magnitude of fire damage will play an important role in the dynamics of vegetation in this region.

2.2. Data and Image Processing

In this work, the following multisensor satellite images fully covering the study area, including Landsat-8, Sentinel-2, and Terra, as well as their spectral bands, were selected and used to discriminate the area that is affected by the fire in the pixel distribution format on the digital number (DN) scale:

(i) A Landsat-8 scene acquired on 1 August 2019 by the OLI sensor (LC08_L1TP_203033_20190801_20190801, orbit/point: 203/033) with a spatial resolution of 30 m obtained from the Earth Resources Observation and Science Center of the US Geological Survey (USGS) [98]. This is a product of level 1T (corrected terrain) and adjusted with the solar angle with the processing steps described in [99].

(ii) A Sentinel-2 scene acquired on 3 August 2019 through the cloudless MSI sensor (S2A_MSIL1C_20190803T112121_N0208_R037_T29SND_20190803T132806) with 20 m spatial resolution obtained from the European Space Agency (Copernicus Open Access Hub). It is a Level 1C Top of Atmosphere (TOA) Reflectance product, which includes radiometric and geometric corrections (UTM projection with Geodetic Reference System WGS84), together with orthorectification [100].

(iii) For Terra satellite, one scene acquired on 25 July 2019 by the ASTER sensor. It is a cloud-free 1T level product with 15 m spatial resolution obtained from the USGS EROS Center [98]. For ASTER, unfortunately, shortwave infrared (SWIR) bands were not available for the study region, as they are no longer usable since 2008.

(iv) Additionally, for Terra satellite, one scene was acquired on 28 July 2019 by the MODIS sensor using the surface reflectance product (product MOD09A1). We also used the MODIS Terra MOD09A1 (Version 6) product from the Oak Ridge National Laboratory’s Distributed Active Archive Center (ORNL DAAC) (Global Subset Tool: MODIS/VIIRS Land Products: https://modis.ornl.gov/cgi-bin/MODIS/global/subset.pl (accessed on 14 February 2021)). This product, with 500 m spatial resolution, provides spectral surface reflectance of the MODIS 1–7 Terra bands corrected for atmospheric conditions (for example, gases, aerosols, and Rayleigh scattering) at eight-days interval. For each pixel, a value is selected from all acquisitions within the eight-day compounding period. The criteria for choosing the pixel include cloud and solar zenith. When several acquisitions meet the criteria, the pixel with the minimum value of channel 3 (blue) is used [101].

Table 1. Landsat-8/Operational Land Imager (OLI), Sentinel-2/MultiSpectral Instrument (MSI), Terra/Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER), and Terra/Moderate Resolution Image Spectroradiometer (MODIS) spectral band numbers, wavelength ranges (λ), and spatial resolutions used in this study.

Band	OLI			MSI			ASTER			MODIS
	λ (µm)	Res.	(m)	λ (µm)	Res.	(m)	λ (µm)	Res.	(m)	λ (µm)	Res.	(m)
B1	–				–		0.52–0.60	Green	15	–
B2	0.45–0.51	Blue	30	0.45–0.52	Blue	10	0.63–0.69	Red	15	–
B3	0.53–0.59	Green	30	0.54–0.57	Green	10	0.78–0.86	NIR	15	0.45–0.47	Blue	500
B4	0.64–0.67	Red	30	0.65–0.68	Red	10	–			0.54–0.56	Green	500
B5	0.85–0.88	NIR	30	0.69–0.71	Red edge	20	–			1.23–1.25	NIR2	500
B6	1.57–1.65	SWIR1	30	0.73–0.74	Red edge	20	–			1.62–1.65	SWIR1	500
B7	2.11–2.29	SWIR2	30	0.77–0.79	Red edge	20	–			2.10–2.15	SWIR2	500
B8	–			0.78–0.89	NIR	10	–			–
B11	–			1.56–1.65	SWIR	20	–			–
B12	–			2.10–2.28	SWIR	20	–			–

summarizes the bands that were used in this study for the different sensors. In the case of MSI, an image composition with all bands (10 and 20 m) was performed, resulting in a product of 20 m of Ground Sampling Distance (GSD).

2.2.1. Flowchart

Figure 2 summarizes the classification scheme and analysis followed in this work. Fire area classification methods using kNN and RF algorithms were used to explain the effects of different satellite images on both classifiers.

The workflow for the supervised classification of burned vegetation using kNN and RF algorithms was implemented with multispectral images from Landsat 8/OLI, Sentinel-2/MSI, and Terra (ASTER/MODIS) through training samples using photointerpretation features. The classification accuracy was determined making use of validation data and the results obtained from the analysis of the classification parameters using the generated confusion matrices. After image composition, the procedure includes the following steps: training samples, spectral separability analysis, classification with kNN and RF algorithms, validation, and accuracy analysis.

2.2.2. Training Samples

The initialization of a supervised classification process requires composite images and training samples (polygons). The sample polygons selected in the composite images are used to obtain the burned and unburned areas class descriptors. The training areas were randomly collected, with 30 polygons with an area of 65 km² for each class, respecting the separation limits that are based on the ICNF burned area product.

2.2.3. Separability Analysis

The purpose of the separability analysis was to evaluate the spectral separability in all of the bands used in the classification of burned and unburned areas and contribute, for instance, to the decision of which bands have greater classification properties in supervised classification algorithms. The separability of each pair between classes can be quantitatively measured by the average distance between the class density distributions of the pairs or histograms of the values of each band [75]. The Jeffries–Matusita (JM) distance is one of the most widely criterion used in remote sensing in the field of pattern recognition and feature selection. In comparison with other separability indices, JM distance has been suggested as more reliable in separability measures, and also more suitable for less homogeneous main classes [102]. Therefore, we chose the JM distance to indicate the separability between the burned and unburned vegetation. It is calculated according to Equation (1), as [103]:

$$J{M}_{ij} = \sqrt{2\left(1-e^{-B}\right)}$$

(1)

where B is the Bhattacharyya distance given by Equation (2), as:

$$B_{ij}=\frac{1}{8}{\left({\mu }_j-{\mu }_i\right)}^T{\left[\frac{{\Sigma }_i+{\Sigma }_j}{2}\right]}^{-1}\left({\mu }_j-{\mu }_i\right)+\frac{1}{2}ln\frac{\frac{\left|{\Sigma }_i+{\Sigma }_j\right|}{2}}{\sqrt{\left|{\Sigma }_i\right|\left|{\Sigma }_j\right|}}$$

(2)

For classes i and j, μ is the mean vector of the reflectance values and Σ is the variance-covariance matrix. Previous research has shown that the JM distance can provide a more accurate classification than other distance measures, such as the Euclidean distance or divergence [104]. It ranges between 0 (completely inseparable) and 2 (completely separable) [102].

2.2.4. kNN Classifier

The kNN algorithm proposed by Aha et al. [105] is an instance-based learning method that classifies elements based on the closest k training samples in the resource space. These data play important roles in spatial forecasting, in addition to being the main adjustment parameter of the kNN algorithm. kNN is a common classification tool used in remote sensing data mining applications [63,105], and it is widely used for mapping burned areas [106,107]. kNN is a non-parametric MLA, which makes no assumptions regarding the main data set. This is important when classifying processes of change in territory, such as floods and fires, for which there is little or no prior knowledge of data distribution. In kNN, the pixel whose class is unknown is a member of a class, as described by its spectrally closest neighbors whose class identities are recognized. Figure 3 shows the scheme of the kNN algorithm.

Initially, the parameter k, which represents the number of closest neighbors, must be selected. This parameter will direct the number of neighbors. In the case of k = 5 in a binary group, the five closest points are identified by the Euclidean distance. In this way, through the shortest distance between the existing k neighbors, which is, the point to be classified and all points in the data set, it is possible to know which class is most similar to. Thus, the classification is completed, and the unknown point is classified. The parameter k plays an important role in the performance of the kNN, being the main kNN adjustment parameter. In this study, we tested different k values (5 to 20) to select the ideal parameter for the kNN classifier based on the lowest estimate of the Root Mean Square Error (RMSE), using different subsets of data. However, in previous studies, as in Cariou et al. [108] and Noi and Kappas [63], it was revealed that this is not the only criterion for selecting an appropriate k value because a small or large k value has characteristics that are suitable for each case. We used SNAP (Sentinel Application Software, ESA) software for this classification.

2.2.5. RF Classifier

The RF algorithm is based on the creation of several decision trees, combining them to obtain a more accurate and stable forecast. According to Rodríguez-Galiano et al. [109], the RF algorithm has advantages in remote sensing area, as it generates an internal unbiased estimate of the generalization that is represented by the Out of Bag (OOB) error, which is a way of validating the RF model. Therefore, it is relatively robust for outliers and noise, in addition to being computationally lighter than other tree set methods. The RF is trained using bootstrap aggregation, where each new tree is adjusted based on a bootstrap sample from the training observations. OOB is the average error for each calculated tree using predictions from trees that do not contain it in their respective bootstrap sample. This allows for the RF classifier to be adjusted and validated while being trained [110].

The Information Gain Rate criterion [111] and the Gini Index [112] are the attribute selection measures most frequently used to induce the decision tree. We chose the Gini Index, which measures the impurity of an attribute in relation to the classes. For a given T training set, it randomly selects a case (pixel) and determines the class that it belongs to.

In this work, the RF classification was tested for 10 to 400 trees for the set of images composed for each sensor. One-third of the training number of trees was used to test the error that is associated with the predictions, the above-mentioned OOB error. In RF, the parameter MTRY, the optimal trees at each node, controls the number of variables available to split at each node of a tree [113]. In this study, a default value was used as provided by the SNAP software.

2.2.6. Validation and Accuracy Analysis

The validation of remote sensing data is generally based on measurements that were obtained in field campaigns, which are seen as a reference on site. In many cases, the validation process is carried out by remote sensing products provided by official institutions or by sensors with high spectral or spatial resolutions. In this work, the validation product that was used as a reference was the 2019 annual burned area of the atlas provided by the National Institute for Conservation of Nature and Forests (ICNF) of Portugal.

The data were made available on the website http://www.icnf.pt/ (accessed on 14 February 2021) in an ESRI shapefile format covering the entire national territory through the representation of polygons from areas that are affected by fires, coupled with information such as area, date, duration, and the cause that started the fire. The elaboration of the national mapping of the burned areas through the compilation of all geospatial files comes from semiautomatic classification processes using Landsat, Sentinel, or other satellite images [114].

The quality of a given thematic map that is derived from remote sensing data is generally assessed by systematic comparison with other maps also derived from remote sensing [6]. Quality assessment is generally carried out based on verification measures derived from confusion matrices [115]. The choice of validation methods and objectives must be guided by the end use of the products. The cross-tabulation approach is the most common way to assess thematic accuracy. In this context, the comparison and analysis of the quality of the burned area maps that were obtained by the kNN and RF classifications in the different tested sensors were carried out.

The burned area polygon that was obtained by the ICNF map was used as a spatial reference in this study. The pixel-based analysis was based on a confusion matrix (

Table 2. Confusion matrix between the reference product and the burned/unburned classified areas.

		Reference Map (True Class)
		Burned	Unburned	Total
Classified Product	Burned	TP	FN	TP + FN
	Unburned	FP	TN	FP + TN
	Total	TP + FP	FN + TN	TP + FN + FP + TN

). Following the terminology that was presented by Fawcett [116], the reference data (true class) will be referred to as positive or negative (burned or unburned). If the instance is positive (burned) and classified as positive (burned), it will be counted as a true positive (TP); if it is classified as negative (unburned), it will be counted as false positive (FP). On the other hand, if the instance is negative (unburned) and it is classified as negative (unburned), it will be counted as true negative (TN); if it is classified as positive (burned), it will be counted as false negative (FN) (Table 2).

The confusion matrices aim to determine the probability of detection of burned areas in the different sizes of fractions of this area at the study site. This explains the error inherent in the burned areas due to the difference between the reference product and the resolutions between the sensor images. According to Cohen [117], the classification methods are evaluated while using statistical parameters, such as the Omission Error (OE), Commission Error (CE), Overall Accuracy (OA), and Dice Coefficient (DC).

OE is related to the producer’s accuracy, which is, when a pixel is classified as unburned area being really burned area. CE is related to the user’s accuracy, which is, when a pixel is attributed to a class of burned area to which it does not really belong. OA is defined as the fraction of pixels correctly classified as burned or unburned [61]. Finally, DC is a measure of similarity between the classifier and reference map in terms of the number of common burned pixels.

OE and CE vary on a reverse scale of (0–100%), where the lowest values indicate the best estimates. For OA and DC, on the contrary, the largest values indicate the best estimates.

2.2.7. ROC Curve and AUC

The ROC curve has been used in studies of burned areas analysis to verify the general performance of classifiers and models. The ROC curve and a useful statistic calculated from it, the area under the curve (AUC), are mainly used to compare diagnostic tests and act as a performance measure for classifying binary data. The AUC value, as in Equation (3), shows the success rate of the model through the analysis of the training data set and its forecast rate for the tested data set.

$$\mathrm{AUC} = \frac{\sum \mathrm{TP}+\sum \mathrm{TN}}{\mathrm{M}+\mathrm{N}}$$

(3)

where M and N are the total number of pixels in the burned and unburned areas. An AUC value that is close to 1 indicates a better performance. An AUC value of 1 indicates a perfect model, while an AUC value of 0 indicates a poor performance model. Between these values, the model performance is classified as excellent (0.9–1), very good (0.8–0.9), good (0.7–0.8), medium (0.6–0.7), and poor (0–0.6).

3. Results

3.1. Spectral Separability Analysis

Table 3. Jeffries–Matusita (JM) separability values and band for the OLI, MSI, ASTER, and MODIS sensor bands used in the classification.

Band	JM Separability
	OLI		MSI		ASTER		MODIS
B1	–		–		0.02	Green	–
B2	0.31	Blue	0.39	Blue	0.07	Red	–
B3	0.18	Green	0.19	Green	1.84	NIR	0.15	Blue
B4	0.26	Red	0.36	Red	–		0.53	Green
B5	1.91	NIR	0.45	Red edge	–		1.65	NIR2
B6	0.24	SWIR1	1.82	Red edge	–		0.50	SWIR1
B7	0.70	SWIR2	1.83	Red edge	–		0.75	SWIR2
B8	–		1.75	NIR	–		–
B11	–		0.14	SWIR	–		–
B12	–		0.81	SWIR	–		–

summarizes the JM separability values at the study site, where the burned and unburned pixels were analyzed for each spectral band used between the OLI, MSI, ASTER, and MODIS sensors.

In general, less separability is observed for the visible bands in all sensors, mainly for the bands B1 and B2 for ASTER, and especially in the green range for OLI, MSI, and ASTER, where the bands presented low separability values, with the exception of MODIS, which presented slightly greater separability in this range.

The near infrared (NIR) is the spectral region where the sign of recent fire scars is the strongest, being generally considered to be the best spectral region for detection and mapping burned areas [118] and, therefore, of crucial contribution to image digital classification processes. This is seen in the results of Table 3 with the high values of separability in all sensors, even with some existing spectral and spatial resolution disparities. In addition, the results corroborate the spectral resolution of the sensors, where the thinner infrared range of MODIS and OLI (Table 1) ensured greater separability, very different from the sparse range of MSI and ASTER, even with a slight difference in the spatial resolution and methods of pre-processing.

In the visible–NIR transition bands, there was high separability, as shown in bands B6 = 1.82 and B7 = 1.83 for MSI sensor, except in band B5 = 0.45. However, the band B5 presented low separability, because it is closer to the red band in relation to bands B6 and B7.

The short-wavelength infrared SWIR bands showed low JM separability values.

3.2. kNN Training

In this study, we tested different k values (5 to 20) to select the ideal kNN classifier parameter for each set of images. The lowest RMSE value was used as a criterion to select the best k parameter. Thus, despite the low RMSE, from Figure 4 we can see that, after tests, the k parameter was set to 5. It shows that, the lower the value of k, the higher the accuracy of the classification.

3.3. RF Training

Figure 5 shows the distribution of OOB errors for a different number of trees from 10 to 400. It is observed that the classification error between the sensors in the same tree does not change significantly. However, with the increase in the number of trees, the error decreases considerably. In this study, we used the number of trees that had the lowest OOB error. It can be seen that 400 is the best value for trees. One of the advantages of using the RF classifier is its versatility with the processing time, and this can be verified in this work. The classification performed with 10 trees took 10 s, while for 400 trees it took two minutes, a moderately acceptable time interval.

3.4. Burned Area Analysis

Figure 6 and

Table 4. Size of the burned area obtained with each classifier for each sensor as well as the size of the burned area in the reference map (INCF), differences between the areas obtained with each classifier, and errors when compared with the reference map together to the percentage that they represent with respect to the reference area.

Classifiers	Area (km2)
	OLI	MSI	ASTER	MODIS	ICNF (Reference Map)
kNN	75.39	76.95	88.02	45.99	93.40
RF	76.36	78.38	89.37	45.63
$\left|\mathrm{kNN}-\mathrm{RF}\right|$	0.97	1.43	1.35	0.36
$\mathrm{kNN}-\mathrm{ICNF}$	–18.01 (19.3%)	–16.45 (17.6%)	–5.38 (5.8%)	–47.41 (50.8%)
$\mathrm{RF}-\mathrm{ICNF}$	–17.04 (18.2%)	–15.02 (16.1%)	–4.03 (4.3%)	–47.77 (51.1%)

show the pixel distribution and size of the burned area for the classification provided by the different sensors with both kNN and RF algorithms. The finer spatial resolution of OLI, MSI, and ASTER showed a burned area with greater spatial detail, but with less density of features. In turn, the map that was generated by MODIS presented, as expected, a burned area with less detail at the edges and a high distribution of overestimated features within the burned area. When comparing the classifiers, the maps visually showed no significant differences with variations in the burned areas ranging between 0.36 and 1.43 km², with the lowest differences being for MODIS (0.36 km²) and the largest for MSI (1.43 km²). However, they presented important errors in the total burned area when compared to the ICNF reference map. The errors in the total burned area are not constant, ranging between 4.3% and 51.1% (Table 4), and being the difference sensitive to the technical specifications of the images.

3.5. Classification Errors

Figure 7 shows the spatial distribution of the OE and CE for the classification of burned and unburned areas from OLI, MSI, ASTER, and MODIS sensors using the kNN and RF algorithms.

It is observed that, in general, all of the classifications have low CE more frequently within the perimeter that is affected by the fire, although, for ASTER, there is a significant presence of missing mixing pixels and CE outside the burned area (Figure 7e,f).

For ASTER images, the classifications present the smallest OE, with a spatial distribution of 8.73 km² of areas with missing pixels for kNN and 8.19 km² for RF. In contrast, despite the lower spatial resolution of MODIS, there was a moderate frequency of missing pixels within the burned area when compared to the other sensors, which decreased the sensors OE reaching ~13–14 km². It is more evident in the upper border, as shown in Figure 7g,h, the place of transition between burned and unburned areas, which, in turn, is more susceptible to errors that are caused by low spatial resolution.

3.6. Overall Accuracy (OA)

The differences in areas that were classified as burned in our classifications and the reference map were the lowest for ASTER (4 km²) and the highest for MODIS (47 km²).

This result is consistent for the images with better spatial resolution and greater proximity to the date of the reference product, such as OLI, MSI, and ASTER, resulting in a stable thematic quality. When the time interval between the data is too long, it is difficult to know exactly what period the pixel finally extracted from the image refers to. This statement is disconnected from the results that were obtained by the MODIS sensor, which, despite the proximity of the day of the burning occurrence, its spatial resolution, and its eight-days compaction form, was an important factor as mentioned above.

In terms of algorithms, RF was the classification method that presented the smallest error in the total burned area in relation to the ICFN reference area with values of the order of 4 to 17 km² for the finer spatial resolution sensors (Table 4) and good estimates of OA and DC, as can be seen in

Table 5. Values of OA and Dice Coefficient (DC) for the products generated by kNN and RF classifiers in the different sensors used.

Classifiers	Parameters	OLI	MSI	ASTER	MODIS
kNN	OA (%)	92.95	93.09	93.62	89.83
	DC	0.88	0.88	0.93	0.85
RF	OA (%)	93.24	93.35	93.52	89.45
	DC	0.89	0.89	0.93	0.84

.

The results show that the classification based on kNN and RF for the different sensors mapped the burned area with a very high accuracy (OA > 89% and DC > 0.8) and without significant variations in the computed OA and DC values for all of the sensors.

3.7. Algorithms Errors

A ROC curve analysis was performed to graphically assess the sensitivity and specificity of the classifications carried out. From the analysis of Figure 8, it can be seen that, as the score point increases, the discriminating power also increases, which is, the curve is closer to the upper left corner and, consequently, a greater area is obtained below the ROC curve. In both classifiers, the largest value was recorded for ASTER and the lowest for MODIS, corroborating the results obtained by the OE and CE.

4. Discussion

This study assesses the application of an automatic methodology for mapping burned areas in Portugal through the supervised classification algorithms kNN and RF using multispectral satellite images of different technical specifications. The integration of the use of these images increases the temporal accuracy of imaging a target that is susceptible to extreme events, which often require intense monitoring.

In this study we show, in detail, the quality, errors, and incompatibilities in the classification of a burned area at a local scale, which, in turn, can be used to explain phenomena of non-resistance (edge effects, unexpected artifacts, or underestimation related to low intensity fires) that are often propagated or masked when applied on a continental or global scale. This is widely discussed in Randerson et al. [99] who observed an underestimation of 4–15% of the burned area missing in global products, slightly below the 30% that is normally assumed. This underestimation occurs due to the absence or small overlap in the detection of small fires (<270 ha) derived from different global burned area products. Such problems are also found in Nogueira et al. [119], Chuvieco et al. [120], and Roteta et al. [56]). Therefore, our analysis demonstrates the importance of accurate mapping of a burned area at a local scale, which still remains the most accurate base of reference data in protocols for validations of global burned area, after evaluation by photointerpretation [121,122] or in the field [61].

4.1. Separability Analysis

The errors that were found in the classification of burned areas were caused by several factors, one of which was the spectral similarity of burned areas with other surface elements, mainly darker bodies, in addition to the technical disparities of the kNN and RF classifiers. However, the spatial accuracy of the images was the most important agent in reducing the performance of the products. This behavior can be seen in the maps that are generated by MODIS sensor, due to its coarse spatial resolution.

The assessment of the ability to detect burned areas was performed using the JM separability index in the different bands (Equations (1) and (2) and Table 3) and the results of the confusion matrices represented by OE, CE, OA, DC, and AUC (Figure 7 and Figure 8 and Table 5).

In agreement with previous studies [123,124,125], less separability is observed for the visible bands in all sensors in our results, mainly for the bands B1 and B2 for ASTER [126] and especially in the green range. This occurred because forest fires affect the leaf structure and photosynthetic capacity. They also decrease the green pigment of the leaf (chlorophyll) and increase the brown-yellow pigment (carotenoids, pheophytin, and xanthophyll) [124]. In the visible-NIR transition bands, there was high separability corroborating the studies conducted by Fernández-Manso et al. [127]. The authors proved that recent fires in healthy vegetation show a characteristic increase in the reflectance from red to NIR, associated with variations in chlorophyll content.

The analysis was able to show good discrimination of the burned areas. This approach improved the spatial homogeneity of the affected areas (even if random) of the classification thresholds, as shown by the high values of AUC (>0.88), reducing the dependence on having information on land cover, usually used in automatic burned area algorithms. Although it is important to emphasize that the lack of information on land use for adapting the algorithms behavior can imply the recurrence of systematic errors, increasing the uncertainty of the final burned area classification, as shown in Figure 9. As already mentioned, we note the presence of features that presented spectral behavior that was similar to the burned area (for example, low reflectance values in the NIR), which can be caused by topography shadows and changes in land cover not associated with fires, such as very humid soils. Therefore, it is recommended to take special care in regions where these characteristics and events occur close to the area that is affected by the fire, in addition to controlling the photointerpretation with the size of the samples of interest, especially in applications with sensors of different spatial and spectral resolutions [128]. Thus, as a future study in the study area, assessing the separability for different classes of land use and the influence of sample size may be a good alternative.

4.2. Validation Product

The reference mapping that is generated by ICNF proves to be quite efficient in the generation of geospatial data, providing a database that is rich in accurate information of the burned areas throughout the national territory and of open access. However, here we list some advantages and limitations of the product, based on a visual comparison after the classification process. Initially, we emphasize the thoroughness of the delimitation at the edges of the burned area that is generated by the ICNF product, in front of a complex landscape, where the study area is inserted. This could be proven in both product classifiers that are generated by the MODIS sensor, as expected, with a high frequency of omission and commission pixels at the edges of the burned area and in urban areas, as shown in Figure 7. However, according to Mouillot et al. [45], OE and CE found at the edges of burned areas cannot be strictly seen as false or omitted alarms. For a given level of CE and OE, it is acceptable as long as both are similar. We can also see that the influence of the spatial validation product provided by ICNF was crucial for the errors that are shown in Figure 7, since this product is based on the 10 m resolution bands of MSI [129], thus causing the lowest error estimates for the ASTER sensor (15 m) due to its greater proximity to spatial detail.

Even with the absence of the blue and SWIR bands in the ASTER images, this sensor showed the highest accuracy parameters in both of the classifiers, although the good results found in OLI and MSI can be attributed to the use of these bands. Therefore, the 15 m GSD of ASTER was responsible for this good performance, although its proximity to the date of the reference mapping must also be taken into account.

4.3. kNN and RF Classifiers

The RF algorithm presented the highest quality values of the classification among all of the sensors and a greater stability in relation to the data change in the attribution of burned and unburned classes. This result corroborates the low complexity of its application, low cost of time, and memory. Despite the variations found in the OOB error with the number of trees, this parameter may not be very relevant in binary classifications, since the use of two classes reduces the voting options of each set of trees in the data set. In general, empirically, the error in the classification with this algorithm depends on the strength of the individual tree and the correlation between two trees in the forest. Strength can be interpreted as a performance measure for each tree. Increasing the correlation increases the error rate of the forest, and increasing the strength of an individual tree decreases the error rate of the forest, since a tree with a low error rate is a strong classifier. On the other side, reducing the number of selected random attributes reduces correlation and strength [85,130]. In our study, we selected 400 trees. In several studies of buried areas by the RF classifier, the largest number of trees commonly used ranges from 100 to 1500 [83,84,131].

However, kNN, even with accuracy values very close to the RF, mainly in the AUC parameter, has a direct relationship with the k parameter, time, and memory. Once a k value is given, more training samples are needed to improve the performance, but more time and storage memory were needed. In this study, the k value that was based on the RMSE was used. Therefore, the disparities found in the quality of the mapping of this classifier can be attributed to other parameters not tested here. The values are consistent with the studies conducted by Meng et al. [132]. The value of k may not present significant differences in relation to the final result of the mapping; however, this value directly influences the processing time. It is worth mentioning that, for k = 5, the processing time was 0.56 min, while, for k = 20, the time was approximately 70 min. This time interval depends significantly on the resources of the computer used and are common for kNN classifications, depending on the size and composition of the data set [133]. Blanzieri and Melgani [134] show that the best values of k were found empirically below k = 5 using SAR data, which could be explained by the image filtering applied to the true soil homogeneity. This indicates that the decrease in k is associated with the registration of optically active elements in the images. This statement is also related to the location of the pixels to be classified in relation to the training samples. When the k neighboring pixels are close enough, the precision will naturally tend to the value very close to the sample pixel set, consisting of a decrease in time and error.

4.4. Accuracy Analysis

It is observed that, in general, all of the classifications have low CE more frequently within the perimeter that is affected by the fire, although, for ASTER, there is a significant presence of missing mixing pixels and CE outside the burned area. This behavior may be related to the confusion of the classifiers in distinguishing between burned areas and dark soils with little vegetation. As said, this was quite evident in the classification with ASTER images, since, for this sensor, only three bands (green, red, and NIR) were used in the classification, which is, less resources for feature detection, which also favored the increase of false alarms pixels in relation to the other sensors for both classification algorithms, as also found in [59]. This is spectrally true while taking the results found with OLI and MSI sensors into account because of NIR and SWIR bands used in the classification probably influenced the presence of low CE. These channels strongly reflect the spectral signal of change detection in the vegetation state in addition to having high separability between burned and unburned areas, as shown in Table 3 and in several works [46,135,136,137], who also used this region of the spectrum for the separation of burned areas obtaining satisfactory results. According to Lambin et al. [138], reflectance generally decreases in the NIR range after the fire event due to the removal of vegetation retained by water due to the fire. The decrease in brightness is more substantive than in the visible, which makes the NIR range more suitable for discriminating burned areas. The low CE for OLI, MSI, and ASTER can be attributed to the higher spatial resolution, since this condition improves the performance of classifying algorithms mainly in places with homogeneous and more compact distribution of the burned area [31,139]. For the classifications that were performed with MODIS, the largest CE of the data series were observed with pixels well distributed throughout the affected perimeter. In this case, the low spatial resolution of this sensor was the main cause of the errors, causing a high frequency of underestimated pixels inside and outside the burned area.

OE, being represented by pixels mistakenly classified as unburned areas, presented significant and well distributed values on the maps, with emphasis on the east sector of the burned area in both classifiers. These errors are related to the high frequency of pixels referring to small urban centers that are inserted in the burned area, which, in turn, were correctly classified as unburned areas, but, due to problems of pixels spectrally mixed at the edges of these features, there was a high presence of pixels of burned areas omitted from their assignment in the classification. This problem was also found in [46,140], who showed moderate performance in mapping burned areas in optically complex locations, caused by ambiguity problems in the classification and spectral mixing.

In the western sector, the same problem occurred, but, more frequently, because, in addition to the housing polygons, agricultural areas also caused confusion in the classifiers. This directly influenced the results of the spatial distribution of the missing pixels in burned areas, where both of the sensors presented area variations between 17 and 18 km² for kNN classifier and between 16 and 17 km² for RF, which is, a high frequency of pixels incorrectly classified as unburned areas.

It is observed that for ASTER images, the classifications presented the smallest OE, with a spatial distribution of 8.73 km² of areas with missing pixels for kNN and 8.19 km² for RF. These values were already expected, since this sensor has the best spatial resolution of the set of images and, consequently, reduced spectral mixing problems, even using only visible bands. In addition, the use of ASTER images limited the overestimation of the burned areas due to the pixel size, most suitable for classifying unburned areas that are inserted in the investigated fire polygon [141].

In contrast, despite the lower spatial resolution of MODIS, there was a moderate frequency of missing pixels within the burned area when compared to the other sensors, which decreased the sensors OE reaching ~13–14 km². It is more evident in the upper border, as shown in Figure 9g,h, the place of transition between burned and unburned areas, which, in turn, is more susceptible to errors that are caused by low spatial resolution. Another influencing factor can be explained by the process of creating the image composition of the MODIS sensor with the acquisition of the best pixel within the eight-day period. The result generates an image with moderate quality once some information is lost.

For the classifications that are generated by the best spatial resolution sensors (OLI, MSI, and ASTER), errors were found in the different elements of land use in the study area, for example, in the products generated by the OLI and MSI scenes. We detected a high frequency of OE on the main highways that cut the area that is affected by the fire, especially the highways N2 and N244 (Figure 9a,b,k,l), thus showing the limitation of the ICNF product in the detection of burned areas in these characteristics. In addition, these errors were also found in the ASTER images, but more frequently in the areas of pasture and agriculture with approximately ~0.06–0.1 km² (Figure 9g,h). However, the reference product proved to be advantageous in the classification of areas of soil degradation in kNN and RF classifiers, erroneously classifying these areas as burned areas, as shown in Figure 9i,j.

Finally, kNN and RF classifiers were not efficient in differentiating water bodies and burned areas in all sensors, causing several CE pixels, as shown in Figure 9c–f. This result is in accordance with Roy et al. [74], Palomino-Ángel et al. [142], and Shimabukuro et al. [143], who reported classification errors in burned areas caused by the spectral similarity with water bodies.

4.5. OA and Algorithms Errors

Overall, the classifications present good estimates of OA and DC. These OA values are also related to the correct classification of unburned areas and, for this reason, particular attention needs to be paid to this parameter, not using it as the only thematic quality parameter [61]. The high DC values, as summarized in Table 5, show a good performance in continuous adherence with the reference data for the class of presence of burned area, even when considering the sensitivity of this parameter to false alarms and missing pixels shown in the maps of Figure 7.

Although Tanase et al. [144], in studies of burned areas in Tropical Africa, suggested that temporarily short sample units may underestimate the accuracy of the detection of burned areas, Schroeder et al. [145] showed, in their studies in the Brazilian Amazon, that the date of the imaging must be as close as possible with respect to the spatial reference data, which may have intensified the OE or increased areas with different time on the hour scale. The methods of detecting changes based on the application of temporal metrics to assess sudden variations in the pixel signature of moderate and coarse resolution sensors are gaining importance as better-quality satellite data sets become available [146,147].

In general, ASTER presented the highest values of OA and DC in relation to the values of the other sensors, because its spatial resolution may have a greater influence in detecting the details of fire scars. MODIS sensor showed the lowest values of OA and DC of all the sensors, being, however, large values. These data are important, as they show that even the low spatial accuracy of MODIS in relation to the reference map as well as OE and CE greater than 10% did not drastically decrease the estimates of OA and DC, because, with both classifiers and sensors, the maps were considered to be excellent according to Cohen’s classification [117]. The same behavior was seen in Lanorte et al. [141], who showed, in applications of ASTER and MODIS sensors in burned areas in southern Italy, that these data were efficient in allowing the detection of burned areas and discriminating the severity of the fire.

The OLI and MSI sensors did not show significant variations in OA and DC, displaying MSI the best results, which is attributed to the low OE that was made in the classification. An identical result was found in [71,148], who reiterated that the reason why the classification provided by Sentinel-2 is more accurate than by Landsat 8 is due to the higher spatial resolution of Sentinel-2 images. Because of this, the burned areas obtained with the classification process on Landsat-8 may have been overestimated. Other studies following this approach also found similar OA values, for example, 90% in Axel [149], 79.2% in Liu et al. [75], 95% in Libonati et al. [61], 94.7% in Zhang et al. [150], 99% in Alonso-Cañas and Chuvieco [46], and 96% in Roy et al. [74].

It is worth noting that both of the classifiers require that choices be made by the modeler concerning numerous parameters under different performances. In general, the classifiers based on kNN and RF brought high quality in the classification of burned areas with AUC values above 0.88, DC above 76%, and OA above 89%, in addition to the ability to process data efficiently and enable parallel training of the same samples in different orbital data sets.

Therefore, the results show a statistically significant ROC curve with an AUC varying between 0.88 and 0.94 for both algorithms, showing that, even in the case of supervised classifications, approximately 90% of the burned areas were well classified by the algorithms in the different sensors. This result agrees with the initial study by Chou et al. [151], where the classification improvement was significant when accounting for spatial autocorrelation in logistic fire probability models in Southern California. Likewise, Siljander [152] found values of AUC in the order of 0.86–0.94, indicating that the fire classification models that were responsible for the spatial distribution of the affected areas showed themselves to be superior in the estimate of burned area on a regional scale when compared with products of global scale burning. In addition, Dlamini [153] found high precision with AUCs of 0.94 and 0.97 in models of Bayesian networks for data of active fire and burned area in ASTER images, respectively. The author also stressed the validity of the Bayesian networks and that the probability estimation based on the data from the burned area can estimate the fire risk a little better than from the active fire data.

5. Conclusions

Based on kNN and RF classifiers and using Landsat-8, Sentinel-2, and Terra imagery, a methodology for assessing their performance in the classification of burned areas in a forest fire occurred in central Portugal is proposed. The main conclusions are as follows:

(i) Less separability is observed for the visible and SWIR bands in all sensors, particularly in the green range, and high separability for NIR region.

(ii) For kNN classification algorithm, k = 5 was found as the best parameter. In the same line, for RF, 400 trees were selected as an optimal value.

(iii) No significant differences in the burned areas that were obtained with each classifier for each sensor were found.

(iv) When compared with ICNF validation data, the lower errors in the total burned area were found in the classifications that were performed with ASTER and the largest errors with MODIS.

(v) Contrary to expectation, the classification that was performed by OLI had greater precision but lower accuracy when compared to MSI. In general, high precision and accuracy were found in the classifications.

(vi) The lowest CE (<5%) were found in the classifications carried out with kNN and RF in OLI, MSI, and ASTER, and large CE, of the order of 15%, with MODIS, with a significant presence in ASTER outside the burned areas. Related to OE, significant and well distributed values were found in all sensors (8–20%), with emphasis on the eastern sector of the burned area, being the low values for ASTER.

(vii) The classification that was based on kNN and RF for the different sensors mapped the burned area with a very high accuracy (OA > 89% and DC > 0.8). The results show a statistically significant ROC curve with an AUC varying between 0.88 and 0.94 for both classifiers, showing that, even in the case of supervised classifications, approximately 90% of the burned areas were well classified by the algorithms in the different sensors.

It is possible to observe that the visible, intermediate, and SWIR bands showed low values of separability, which corresponds to the results that were found by Pereira et al. [118], who stated that the spectral changes induced by fire in the SWIR are similar to those in the visible range, since the burned areas are generally more reflective than green vegetation, but darker than vegetation predominantly in savannas during the dry season. It is important to note that the SWIR band has the advantage of having low interference with atmospheric scattering during the scene recording process. Following this premise, there may be no significant reduction in the spectral contrast of the surface in the images, consequently resulting in increased separability indices. However, this behavior was not observed in our experiments.

This methodology can be useful for mapping the burned areas in regions of native vegetation and the improvement of methods for monitoring the burned areas in Portugal, in addition to assisting in the management of fire in the region and estimating the impacts that are generated by it. The availability of detailed information on the spatial and temporal distributions of the burned areas is currently crucial. Therefore, the applied method makes it possible to survey the scars of fires while using geospatial data with the greatest possible accuracy, assisting in the maintenance of an information bank, serving not only the management of the territory, but also the comparison with related future events.

In general, the errors that were found in both kNN and RF classifiers can also be related to the creation of very heterogeneous objects, even in a region with a predominance of sparse vegetation. Despite the similar results of OE and CE and the differences in the processing of each algorithm, it was shown that the spectral resolution and, especially the spatial resolution, is a more important factor in the process of classification of burned areas. OE and CE are directly linked to the burned areas used as reference mapping, as product incompatibility can generate low generalization capacity and, consequently, OE and CE close to 100% as found in Lizundia-Laiola et al. [154].

Finally, this study opens up the possibility of using multiple Earth Observation data to assess environmental disturbances, increasing the range of possibilities for implementing these data when, for example, there is no scene or a specific band for a given period or problems with cloud cover.