3.2. Performance Evaluation Metrics
Results of fire detection are assessed and compared using two performance accuracies, user’s accuracy (a measure of the commission error) and producer’s accuracy (a measure of the omission error), given by
where the
stands for the cardinality of a set
S, ‘
’ is the number of fire pixels reported by the reference data product, namely MODIS MOD14/MYD14 and EUMETSAT FIR, ‘reported
fire pixels’ is the number of pixels reported by the designed (proposed) fire detection algorithm (per analysis method), and ‘correctly reported
fire pixels’ is the number of
fire pixels according to the reference data products correctly reported (classified) by the designed algorithms. The performance of the fire detection method is also assessed through two performance curves, a graph of user’s accuracy against producer’s accuracy (also known as Precision-Recall curve) and ROC curve. The Precision-Recall curve is sensitive to both class skews and varying unequal/asymmetric misclassification (error) costs (i.e., when the cost of misclassifying a sample from one class is much higher than the cost of misclassifying a sample from the other class) while the ROC curve is insensitive to both [
44], and the class skews are evident in a fire detection application where the true instances of
non-fire class occur much more often than the true instances of
fire class. However, due to a high number of
non-fire pixels compared to
fire pixels, the Precision-Recall curves visually expose clearer the difference between the detection methods, in a given FAR range of interest, that is not noticeable in the ROC space [
45]. The pixel reported as
fire pixel by the validation fire products but is reported by the proposed fire detection method as a
non-fire pixel or or has
(expected) background brightness temperature or has
observation (i.e., instant of completely missing data [missed pixels]), is counted as a missed detection. If an MSG-SEVIRI pixel containing more than one MODIS fire pixel is not reported as a
fire pixel by a proposed fire detection algorithm, then all MODIS MOD14/MYD14 fire pixels belonging to the MSG pixel are missed (i.e., counted only under ‘
fire pixels’), while if reported as a
fire pixel, all MODIS fire pixels belonging to the MSG pixel are detected (i.e., counted under ‘correctly reported
fire pixels’ and ‘
fire pixels’). The process of deriving the best admissible CFAR at which a fire detection system can operate can be conducted during a phase that can be referred to as the ‘detection calibration phase’ using an available validation set [reference data] (in this study, the validation set is made solely of MODIS MOD14/MYD14 and EUMETSAT FIR reports). The operational CFAR (or the best CFAR for a detection method according to the reference data) can be determined based on different criteria. This study determines the operational CFAR, as a FAR, that minimizes an error function
J, given by
where
and
are the prior probability that a
fire condition and a
non-fire condition, respectively, will occur – they represent the proportion of correct
fire or
non-fire pixels to the sum of all pixels, and can be referred to as
fire or
non-fire true class distribution, respectively.
and
are representations of class skews.
and
are the misclassification cost of omission and cost of commission, respectively.
is the measured FAR while the variable
is the design FAR (known also as the nominal or the estimated FAR). Noting that the detection cost function
given by Equation (
7) assumes that the cost of correct classifications is zero, and
accounts of class skew [
46].
3.3. Fire Detection Performance
User’s accuracy (
U) and producer’s accuracy (
P) computed per estimated FAR are shown in
Figure 1a and
Figure 1b, respectively. The two accuracies are derived from the fire results of the proposed fire detection algorithms in the study area for the period of 31 July 2007 to 4 August 2007. The validation reference data are made of a combination of EUMETSAT FIR, and MODIS MOD14/MYD14 reports. MODIS detection time is rounded off to the nearest MSG-SEVIRI detection time (detection time is
ahead of Level 1.5 observation time).
performs better than other detection methods in terms of
U (gives less commission errors) except at the low estimated FARs. However, the low estimated FAR region was not considered during the design of the CFAR detector (the estimation of FAR starts from
to
as indicated in
Section 2.2). For the estimated FARs in the low FAR region (i.e., high threshold values),
can produce a lot of false alarms given that the assimilation window of 4D-Var is more likely to contain more missed fire samples.
performs worse than both
and
in terms of
U. At high estimated FAR, all methods give similar results in terms of
U. Overall,
performs better than other methods in terms of
P (low omission error or high probability of detection), and at some estimated FARs, it performs more like
.
Figure 2a illustrates the Precision-Recall curve (i.e.,
U vs.
P) for a single test set in which all data for period 31 July 2007–4 August 2007 are included, with validation reference data comprising of the combination of MODIS MOD14/MYD14 and EUMETSAT FIR products, and MODIS detection time rounded to the nearest SEVIRI detection time. The performance of the detection methods when any of the three analysis is implemented is fairly modest (no method with a Precision-Recall above the shown diagonal/random classifier) and is not good due to a high number of false alarms according to the reference data products.
shows a better performance compared to other detection methods, as
dominates in Precision-Recall space, it does also dominate in ROC space as shown in ROC curves in
Figure 2b.
A single operational FAR, when the analysis is implemented with any of the three methods considered in this study, is derived using the error (cost) function
given by Equation (
7). This error function considers both commission and omission errors and is a function of the estimated FAR. It is assumed in this study that
and the error function simplifies then to
.
Figure 2c depicts a resulting
for a single test set made of data spanning the period 31 July 2007–4 August 2007 with a reference dataset consisting of a combination of MODIS MOD14/MYD14 and EUMETSAT FIR products, and MODIS detection time is rounded to the nearest SEVIRI detection time. The optimal design FAR point is found among discrete points ranging from
to
(with a base-10 logarithmic step of
).
gives the lowest error compared with other detection methods, and its optimal point
FAR gives the operational FAR. The
FAR point is closer to the optima of
and
, and the error difference (
) from
point to the optima of
and
is
and
, respectively.
When each detection not confirmed by EUMETSAT FIR product is confirmed by any MODIS MOD14/MYD14 report in
before or after the detection time, the number of reported false alarms according to the validation reference data (the combination of MODIS MOD14/MYD14 and EUMETSAT FIR ) is reduced for all detection methods as shown in
Figure 3, and the reported overall performance is then increased. In this case under consideration, fire events confirmed by the same MODIS MOD14/MYD14 report are counted as one ‘correctly reported fire’ during computation of both
P and
U, and the number of ‘fire pixels’ is the sum of the number of EUMETSAT FIR reports and the number found by subtracting the number of MODIS MOD14/MYD14 fire reports to the number of times one or more EUMETSAT FIR fire reports are found in the interval of
around a MODIS MOD14/MYD14 report (i.e., 1 to 3 fire events). The results depict an instance the validation products are used to validate a whole fire course on a pixel location and not just a fire event at a given inspection time
k on a given pixel location.
Results of a comparison between
,
and
detection methods at
FAR when only MODIS MOD14/MYD14 fire product is taken in the validation reference data (and MODIS detection time rounded to the nearest SEVIRI detection time) are depicted in
Table 3. The number of true positives, according to a MODIS MOD14/MYD14 product, is presented (the number of true positives in
Table 3, is presented in terms of the number of MODIS fire pixels). From the results presented in
Table 3, we can conclude that
performs better than other detection methods (throughout the 5 days) in terms of detection, according to MODIS MOD14/MYD14 product.
and
at
FAR achieve a true positive rate of
and
, respectively, of fire events reported in MODIS MOD14/MYD14 product. A moderate portion of MODIS MOD14/MYD14 reports is detected by
and
methods, and the accuracy is restricted given that small and cool fires might not be detected due to the low spatial resolution of the SEVIRI compared to the MODIS resolution.
Table 4 summarizes results for
(under ‘4D-Var-CD’ columns),
(under ‘EnKF-CD’ columns) and
(under ‘SIR-CD’ columns) at
design FAR with reference data being both EUMETSAT FIR and MODIS MOD14/MYD14 products (‘FIR’ column presents the number of EUMETSAT FIR reports and ‘
’ column portrays the number of MODIS MOD14/MYD14 reports not reported by EUMETSAT FIR ). For each of
,
, and
detection methods,
Table 4 lists the number of true positives, according to EUMETSAT FIR product (in ‘
’ column). Among the remaining false positives of the proposed detection method, according to EUMETSAT FIR product, true positives according to MODIS MOD14/MYD14 are extracted (listed in ‘
’ column), and false positives according to both EUMETSAT FIR and MODIS MOD14/MYD14 are presented in ‘
’ column. A significant portion of false alarms, according to EUMETSAT FIR, are reduced when MODIS MOD14/MYD14 product is introduced in the validation reference dataset. Comparing true positive results presented in
Table 3 for MODIS MOD14/MYD14 and true positive results presented in
Table 4, the proposed fire detection algorithms detect MODIS MOD14/MYD14 fire reports at a lesser extent than they detect EUMETSAT FIR fire reports. Because, as mentioned above, of the small and cool fires that might not be detected due to the low spatial resolution of SEVIRI compared to MODIS. The degree of omission of EUMETSAT FIR fire reports by the proposed algorithms mainly depends on the level of the design FAR, and in this section, the design FAR is taken to be
. However, the omission of EUMETSAT FIR reports at a given design FAR highly materializes while reporting fire events during times of extreme fires, and omission error arises in case the fire temperature dynamics at certain times are not far from the expected background temperature dynamics. Results of
Table 4 show that
and
perform relatively similar and better than
detection method.
detects EUMETSAT FIR reports at
, and detects the combination of EUMETSAT FIR fire events and MODIS MOD14/MYD14 fire events not reported by EUMETSAT FIR at
.
detects EUMETSAT FIR reports at
, and detects the combination of EUMETSAT FIR fire events and MODIS MOD14/MYD14 fire events not reported by EUMETSAT FIR at
. All proposed detection methods report a significant number of false alarms according to MODIS MOD14/MYD14 and EUMETSAT FIR, and detect EUMETSAT FIR reports relatively well.
Figure 4a,b present number of fire events at each time of a day (where times are expressed in terms of local solar time) for MODIS MOD14/MYD14 product and EUMETSAT FIR product, respectively. The values of producer’s accuracy (or true positive rate) of the proposed detection algorithms at each time of a day are presented in
Figure 4c,d for cases the validation reference dataset is made of MODIS MOD14/MYD14 product only and EUMETSAT FIR product only, respectively. Fires recorded by EUMETSAT FIR are significantly detected by
and
detection methods at
FAR with
outperforming other methods in detection of MODIS night fires. Their detection performance is reduced from around
(i.e., the time of the maximum thermal radiation) to around
(onset of the nighttime temperature decay) with respect to EUMETSAT FIR and MODIS MOD14/MYD14 results, and such performance can be viewed to be more associated with a high number of fire events reported by both MODIS MOD14/MYD14 and EUMETSAT FIR products in the time range. During time of a high number of fire events, a proposed algorithm can miss a fire event reported by the fire products but can report a fire before or after the fire product reports. Furthermore, the observed temperature in MSG
spectral channel of an MSG pixel identified by MODIS to be affected by fire, might not deviate significantly from its expected background temperature.
has an overall low detection, especially during the nighttime—the time of high fluctuations in brightness temperatures due to different coupled land–atmosphere phenomena, such as orographic forcing effect. In the afternoon, its low detection is more explained by the DTC linearization from around
to around
(or likewise, based on the approximate observation operator given by Equation (S8)). The detection rate of MODIS MOD14/MYD14 report is low compared to the detection rate of EUMETSAT FIR reports, and
and
perform better in detecting MODIS MOD14/MYD14 reports compared to
at the 4 MODIS overpass times (
is better than all detection methods during the night). Based on the MODIS MOD14/MYD14 report, all detection methods certainly detect both daytime and nighttime fires, with
to a lower extent. Noting that MODIS detection time is rounded to the nearest SEVIRI detection time, and the number of ‘fire pixels’ is counted in terms of MODIS pixels to determine the true positive rate (
P), according to MODIS MOD14/MYD14 product.
The empirical distribution of MSG
brightness temperatures for MODIS MOD14/MYD14 reported fire pixels and EUMETSAT FIR reported fire pixels are shown in
Figure 5a,b, respectively. MODIS detection time is rounded off to the nearest MSG-SEVIRI detection time. For MODIS MOD14/MYD14 reported fire pixels, the temperature value of an MSG pixel containing the centers of a number of MODIS fire pixels at a given time step
k is counted as many times as the number of the centers of the MODIS pixels. For fire detection results of
method, producer’s accuracy for each MSG brightness temperature value is shown in
Figure 5c,d when the validation set is made of MODIS MOD14/MYD14 product and EUMETSAT FIR product, respectively.
is able to detect cooler fire compared to EUMETSAT FIR, down to approximately a brightness temperature value of
(with
), as shown in
Figure 5c when only MODIS MOD14/MYD14 product is accommodated in the validation set. Some very hot fires are missed by
, especially at the SEVIRI’s saturation point (which can be attributed to the linear approximation of the observation function and the data assimilation window that does not contain only clean samples). For fire detection results of the
method, producer’s accuracy for each brightness temperature value is shown in
Figure 5g,h when the validation set is made of MODIS MOD14/MYD14 product and EUMETSAT FIR product, respectively.
reports cold fires more frequently than
, and can detect fires with brightness temperatures below
(with
) as shown in
Figure 5g. For fire detection results of
method, producer’s accuracy for each brightness temperature value is shown in
Figure 5e,f when the validation set is made of MODIS MOD14/MYD14 product and EUMETSAT FIR product, respectively. Similar to
and
methods, the
method is able to detect cooler fire compared to EUMETSAT FIR, even for fires with a brightness temperature value below
(with
) as shown in
Figure 5e when only the MODIS MOD14/MYD14 product makes the validation set.
An example of MSG
images at three MSG timestamps on 3 August 2007 is illustrated in
Table 5 to portray detection reports of
,
, and
against both MODIS MOD14/MYD14 (MODIS detection time is rounded off to the nearest MSG-SEVIRI detection time) and EUMETSAT FIR reports. In addition, the detection reports of MODIS MOD14/MYD14 and EUMETSAT FIR at the three timestamps are provided to present and describe MODIS MOD14/MYD14 and EUMETSAT FIR products. The table provides Level 1.5 observation time (expressed in Coordinated Universal Time (UTC)) in ‘Time’ column, the pseudocolor versions of the MSG
images in ‘Pseudocolor image’ column to better visualize the brightness temperature distribution in the images, and MODIS MOD14/MYD14 and EUMETSAT FIR fire reports overlaid on the MSG
images in the ‘MOD14/MYD14 & FIR’ column. A great number of MODIS MOD14/MYD14 reports are not reported by EUMETSAT FIR. Essentially, since in this study, we assume that a MODIS pixel belongs to an MSG pixel into which its center falls, if a burning part of the MODIS pixel location is in a different MSG pixel, some reports would be missed by a fire detection method that uses MSG-SEVIRI data. The table also presents detection reports of
,
and
methods on the three images, overlaid on the corresponding MSG
images and placed under ‘4D-Var’, ‘EnKF’ and ‘SIR’ columns, respectively.
and
detect all EUMETSAT FIR reports while
missed, at 08:12 a.m. (corresponds to the detection time 08:00 a.m.) and at 12:42 p.m. (corresponds to the detection time 12:30 p.m.), 4 fire events and 22 fire events, respectively.
and
detect a great number of MODIS MOD14/MYD14 reports with
detecting less at all the three timestamps consistently. The results of
contain a great number of false alarms according to MODIS MOD14/MYD14 and EUMETSAT FIR products, compared to both
and
, and
has less false alarms.