Satellite Image Pansharpening Using a Hybrid Approach for Object-Based Image Analysis

Abstract

Intensity-Hue-Saturation (IHS), Brovey Transform (BT), and Smoothing-Filter-Based-Intensity Modulation (SFIM) algorithms were used to pansharpen GeoEye-1 imagery. The pansharpened images were then segmented in Berkeley Image Seg using a wide range of segmentation parameters, and the spatial and spectral accuracy of image segments was measured. We found that pansharpening algorithms that preserve more of the spatial information of the higher resolution panchromatic image band (i.e., IHS and BT) led to more spatially-accurate segmentations, while pansharpening algorithms that minimize the distortion of spectral information of the lower resolution multispectral image bands (i.e., SFIM) led to more spectrally-accurate image segments. Based on these findings, we developed a new IHS-SFIM combination approach, specifically for object-based image analysis (OBIA), which combined the better spatial information of IHS and the more accurate spectral information of SFIM to produce image segments with very high spatial and spectral accuracy.

1. Introduction

Recent years have seen an increase in the number of satellites that acquire high spatial resolution imagery. The latest generation of these satellites (e.g., GeoEye-1, Worldview-2, Pléiades 1A) acquire imagery at a very high spatial resolution for a panchromatic (PAN) band (around 0.5 m) and at a slightly lower resolution (around 2 m) for several multispectral (MS) bands. For these types of remote sensing data that contain PAN and MS bands of different spatial resolutions, image fusion methods referred to as “pansharpening” methods are often performed to increase the resolution of the MS bands using the PAN band [1]. Pansharpening allows for the high level of spatial information in the PAN band to be combined with the more detailed spectral information in the MS bands. A large number of pansharpening algorithms have been developed, and [2] provides an overview and comparison of many of the frequently-used algorithms. In general, some pansharpening algorithms, such as Intensity-Hue-Saturation (IHS) transformation [3] and the Brovey Transform (BT) [4] preserve almost all of the spatial details in the PAN image but distort the MS information to some degree [5,6], while others like the Smoothing-Filter-Based Intensity Modulation (SFIM) [7] preserve more accurate MS information at the cost of some reduction in spatial information [6]. In addition, some pansharpening methods incorporate an adjustable filter or parameter that allows users to control this tradeoff between color distortion and spatial resolution enhancement (e.g., [6,7,8,9,10]).

A large number of metrics have been developed to evaluate the spatial and/or spectral quality of different pansharpening techniques, and [2] review the most commonly-used metrics. These evaluation metrics all involve pixel-based calculations, but in many cases high resolution images are classified using an object-based image analysis (OBIA) approach rather than a pixel-based approach due to the higher classification accuracy often achieved by the OBIA approach [11,12,13]. In the OBIA approach, an image is first segmented into homogeneous regions called “image segments” or “image objects”, and then classified using the spectral (e.g., mean, standard deviation) and non-spectral (e.g., size, shape, etc.) attributes of these image segments [14]. Since classification variables in the OBIA approach are calculated for image segments rather than individual pixels, it should therefore be more appropriate to evaluate the quality of pansharpening algorithms using segment-based evaluation metrics rather than pixel-based metrics if the pansharpened image is intended for OBIA. These segment-based evaluation metrics should take into account the spatial and spectral accuracy of image segments.

One common method to estimate the spatial accuracy of image segments is to calculate the similarity in position, size, and/or shape between image segments and manually-digitized reference polygons [15]. These types of segment-based spatial accuracy measures have been used in past remote sensing studies for optimizing segmentation parameters [16] and for comparing different segmentation algorithms [17], but to our knowledge no studies have compared the spatial accuracy of segmentations using different pansharpening algorithms to see which lead to more spatially-accurate segmentations. In addition, while a large number of segment-based spatial accuracy metrics exist, there is a lack of segment-based spectral accuracy metrics, probably because it is assumed that segments with a higher spatial accuracy should also have more accurate spectral information. This is likely to be true for a set of segmentations of a single image. However, to compare segmentations produced by multiple pansharpening algorithms (and thus multiple images with different levels of spectral distortion), it is necessary to include a spectral accuracy measure because a more spatially-accurate segmentation may not necessarily be more spectrally-accurate.

In this study, we evaluated the effect that three commonly-used pansharpening techniques—IHS, BT, and SFIM—had on the spatial and spectral quality of image segmentations. As previously mentioned, IHS and BT are both excellent at preserving spatial details from the PAN image, while SFIM minimizes the distortion of spectral information from the MS image, so a comparison should be useful to see which characteristic, if either, is most useful for OBIA. For each pansharpened image, a wide range of segmentation parameters were tested, and the spatial accuracy of segments was calculated for each segmentation parameter. Like past studies, we assumed that, for an individual pansharpened image, segmentations with higher spatial accuracy should also be more spectrally-accurate (and thus did not calculate spectral accuracy for every segmentation parameter). However, once the most spatially-accurate segmentations of each pansharpened image were identified, we calculated the spectral accuracy of these segmentations to allow for comparisons of spectral distortion to be made among the three algorithms. We anticipated that IHS and BT pansharpening would produce more spatially-accurate segmentations since they better preserve spatial information, while SFIM would better preserve the spectral information of image segments. In past research, pixel-based evaluation metrics have indicated that the IHS, BT, and SFIM algorithms work well in combination [6], so another objective of this study was to develop a hybrid approach specifically for OBIA that can incorporate both the rich spatial information of IHS and BT and the accurate spectral information of SFIM.

2. Pansharpening Equations

IHS is one of the most commonly-used pansharpening methods in remote sensing due to its efficiency and ease of implementation [5]. The original IHS pansharpening algorithm was designed for only three spectral bands [2], but [18] provides an efficient n-band IHS-like algorithm. Using this IHS-like algorithm for a 4-band image with blue (B), green (G), red (R), and near infrared (NIR) spectral bands, pansharpened MS values are calculated as

(1-1)

(1-2)

(1-3)

where B'_ihs is the pansharpened value for the blue band, B is the digital number (DN) value of the pixel in the original MS band (radiance or reflectance values can be used alternatively), and PAN is the DN of the pixel in the PAN image. α₁ to α₄ can be determined based on the spectral response curve of a sensor’s PAN and MS bands [6]. Alternatively, a simple average can be used in place of α₁ to α₄, i.e., (B + G + R + NIR)/4), or α₁ to α₄ can be determined through multivariate regression [18,19]. Since the MS bands are used for calculation of δ_i, the theoretical spatial resolution of the sharpened image is not exactly equal to that of the PAN image, but satisfactory resolution can be achieved in practice [6]. We chose to test IHS pansharpening in this study for its ability to preserve the spatial information of the PAN image, which may be beneficial for image segmentation purposes.

BT is another fast and efficient pansharpening method that has been used in many remote sensing studies [5]. Like IHS, BT was also originally developed for 3-band imagery, but an n-band BT-like algorithm is given in [8]. For a 4-band image, it can be calculated as

(2)

where B'_bt is the pansharpened value for band B and I is calculated as in Equations (1) to (3). This BT algorithm is similar to the “naïve” mode of the Hyperspherical Color Sharpening (HCS) algorithm recently proposed for pansharpening WorldView-2 imagery [6]. BT is also similar to IHS in that it preserves almost all of the spatial information of the PAN image while distorting the MS information. However, IHS pansharpening is known to result in saturation compression, while BT results in saturation stretch [6]. We chose to test BT in addition to IHS to assess which property, if either, resulted in higher image segmentation quality.

As the third pansharpening algorithm in this study, we chose to test SFIM due to its capability to minimize the distortion of MS information (unlike IHS and BT). Also unlike IHS and BT, it can be applied to n-bands without requiring α₁ to α₄. Instead, a smoothed version of the PAN image, often obtained using a 7 × 7 mean filter [6], is required. For a 4-band image, SFIM can be calculated as

(3)

where B'_sfim is the pansharpened value for band B and PAN_smooth is the DN value of the pixel in the PAN band after the 7 × 7 mean filter has been applied. Thus, any change in the spectral information of the MS bands is caused solely by PAN / PAN_smooth. This SFIM equation is very similar to the “smart mode” of HCS [6].

3. Methods

3.1. Study Area and Data

For this study, GeoEye-1 imagery was acquired for a residential area and a forested area in Ishikawa Prefecture, Japan. The imagery consisted of four 2 m resolution MS bands (B, G, R, and NIR bands) and a 0.5 m PAN band. After orthorectifying the images using the provided Rational Polynomial Coefficients (RPCs) and a 5 m resolution LIDAR-derived digital elevation model (DEM), they were pansharpened using Equations 1 to 3. To calculate I in Equations (1) to (3) for IHS and BT pansharpening, α₁ to α₄ values (0.343, 0.376, 0.181, and 0.1) were determined based on the spectral response of GeoEye-1’s spectral bands (values obtained from [6]). False color composites of the original and pansharpened images of the residential and forested study areas are shown Figure 1 and Figure 2, respectively, to allow for a visual comparison of results.

Figure 1. GeoEye-1 image (NIR, R, G bands) of (a) the residential study area. (b) Intensity-Hue-Saturation (IHS), (c) Brovey Transform (BT), and (d) Smoothing-Filter-Based-Intensity Modulation (SFIM) pansharpened images. Yellow polygons in (a) delineate reference tree polygons and black polygons delineate reference building polygons. A standard deviation contrast stretch of 2.0 was used for display purposes.

Figure 1. GeoEye-1 image (NIR, R, G bands) of (a) the residential study area. (b) Intensity-Hue-Saturation (IHS), (c) Brovey Transform (BT), and (d) Smoothing-Filter-Based-Intensity Modulation (SFIM) pansharpened images. Yellow polygons in (a) delineate reference tree polygons and black polygons delineate reference building polygons. A standard deviation contrast stretch of 2.0 was used for display purposes.

Figure 2. GeoEye-1 image (NIR, R, G bands) of (a).the forested study area, (b) IHS, (c) BT, and (d) SFIM pansharpened images. Yellow polygons in (a) delineate reference polygons of damaged or killed trees. A standard deviation contrast stretch of 3.0 was used for display purposes.

Figure 2. GeoEye-1 image (NIR, R, G bands) of (a).the forested study area, (b) IHS, (c) BT, and (d) SFIM pansharpened images. Yellow polygons in (a) delineate reference polygons of damaged or killed trees. A standard deviation contrast stretch of 3.0 was used for display purposes.

3.2. Digitizing Reference Polygons of Land Cover Objects of Interest

For each study area, we digitized reference polygons of specific land cover objects of interest, which were used to evaluate the spatial accuracy of image segmentations. The boundaries of these reference polygons closely matched the boundaries of the objects of interest based on detailed visual analysis. For the residential study area, we digitized 30 polygons of individual trees to assess how well each pansharpening algorithm allowed for small features to be segmented, and 30 polygons of buildings to test how well each algorithm allowed for larger features with distinctive shapes to be segmented (reference polygons are shown in Figure 1(a)). In the forested study area, we digitized 30 polygons of damaged or killed Oak trees to test how well each pansharpening algorithm allowed for specific targets to be detected in a vegetation-dominated landscape (reference polygons are shown in Figure 2(a)). These Oak trees were severely damaged or killed due to mass attacks by ambrosia beetles (Platypus quercivorus) carrying the fungus Raffaelea quercivora [20]. Early detection of attacked trees is important to prevent further expansion of Platypus quercivorus [21], so a more accurate segmentation of damaged trees may lead to more accurate detection using OBIA techniques.

3.3. Image Segmentation

Image segmentation was done in BerkeleyImageSeg (BIS; http://www.imageseg.com). BIS’s segmentation algorithm is based on the region-merging approach described in [11], making it computationally similar to eCognition’s (http://www.ecognition.com) “multi-resolution segmentation” algorithm [11] that has been used in many remote sensing studies. Any difference between BIS’s and eCognition’s segmentation algorithms is therefore due to proprietary implementation details not publically available [16]. There are three user-defined parameters for segmentation: a “Threshold” parameter that controls the relative size of segments, a “Shape” parameter that controls the relative amounts of spectral and spatial information used in the segmentation process, and a “Compactness” parameter that controls how smooth vs. how jagged segment boundaries are [16].

For delineation of individual trees in the residential study area, we tested Threshold parameters from 5 to 40 at a step of 5, and Shape and Compactness parameters of 0.1, 0.3, 0.5, 0.7, and 0.9 for segmenting the pansharpened images (total of 200 segmentations for each pansharpened image). As a baseline for comparison purposes, we also segmented the original PAN image using these segmentation parameters. For delineation of buildings in the residential study area, we tested Threshold parameters of 40–100 at a step of 5 and the same Shape and Compactness parameters used for individual trees (total of 325 segmentations for each pansharpened image). Finally, for delineation of damaged trees in the forested study area, we tested Threshold, Shape, and Compactness parameters equivalent to those used for extracting trees in the residential area. The optimal parameters for segmenting each of these types of land cover were determined quantitatively using the methods described in Section 3.4. As shown in Section 4.1., the most spatially-accurate segmentations were not obtained using the highest or lowest Threshold parameter values tested. For this reason, we did not further expand the parameter search to include lower or higher Threshold parameters.

3.4. Calculating the Spatial Accuracy of Image Segments

The spatial accuracy of each image segmentation was assessed using the D metric (D) [16], which combines measures of oversegmentation (i.e., segments smaller than the land cover objects of interest) and undersegmentation (i.e., segments larger than the land cover objects of interest) into a single value that estimates closeness to an ideal segmentation result (i.e., a perfect match between the reference polygons and image segments). Oversegmentation (OverSeg_ij) is defined as

(4)

where area(x_i ∩ y_j) is the area of the geographic intersection of reference polygon x_i and image segment y_j. Y_i^* is the subset of segments relevant to reference polygon x_i(i.e., segments for which either: the centroid of x_i is in y_j, the centroid of y_j is in x_i, > 0.5, or > 0.5. OverSeg_ij values range from 0 to 1, with lower values indicating less oversegmentation. Undersegmentation (UnderSeg_ij) is defined as

(5)

UnderSeg_ij also ranges from 0–1, and lower values indicate less undersegmentation. D combines these measures of over- and under-segmentation using root mean square error (RMSE), and is given by

(6)

Lower D values indicate segmentations with a higher spatial accuracy (i.e., less over- and under-segmentation). D was chosen to measure the spatial accuracy of segments in this study due to its good performance in [16] (it frequently indicated an optimal segmentation equivalent or similar to the optimal segmentation indicated by many other metrics) and its speed and ease of implementation (it can be automatically calculated in BIS).

3.5. Calculating the Spectral Accuracy of Image Segments

Once the most spatially-accurate segmentations of each pansharpened image were identified (using D), their spectral accuracy was measured by comparing the spectral characteristics of image segments before and after pansharpening. First, pixels in the original MS bands were upsampled to 0.5 m resolution to match the pixels in the pansharpened images. Then, the mean DN values from the original and MS bands were calculated for each image segment polygon. Once image segments contained their original and pansharpened spectral values, the spectral accuracy of the segmentation was measured, band by band, by calculating the RMSE and average bias (BIAS) of the original and pansharpened values. RMSE was calculated as

(7)

where Pansharp_i is the mean DN value in a pansharpened band for segment i, Original_i is the mean DN value in the original MS band for segment i, and n is the total number of image segments. For evaluation of spectral distortion using pixel-based methods, RMSE has been considered as a good metric for spectrally-homogeneous regions in an image [22]. Since image segments represent relatively homogeneous image regions, calculating RMSE at the segment level (i.e., comparing the mean values of the original MS and pansharpened MS bands for each segment) should provide a relatively accurate estimate of spectral quality. However, the mean spectral values of a segment are expected to change to some degree due to pansharpening, and in some cases this change may not be due to color distortion but instead indicate useful spectral information added by pansharpening. Thus, the segmentation with the lowest RMSE may not always indicate the segmentation with the highest spectral accuracy. On the other hand, if the mean spectral values of some segments increase, the mean spectral values of other segments should decrease by roughly the same amount if there is little distortion (since pansharpening should not drastically change the radiometric properties of the image as a whole). Thus, an error measure that takes into account the direction of errors is also important for assessing spectral accuracy. For this study, we calculated average bias (BIAS) for this purpose. BIAS is calculated as

(8)

BIAS values range from −∞ to ∞. BIAS values near to 0 indicate little over- or underestimation of the spectral values of segments due to pansharpening, making BIAS another good indicator of the spectral accuracy of a segmentation.

4. Results and Discussion

4.1. Spatial and Spectral Accuracy of Image Segments

As was originally anticipated, use of the pansharpening algorithms which preserved more of the spatial information from the PAN band (i.e., IHS and BT) led to more spatially-accurate image segmentations. D values of the most accurate segmentations for each pansharpened image are shown in Table 1. From this table, it is clear that all of the pansharpened images tended to produce more spatially-accurate segmentations than the original PAN image, indicating that the addition of MS information increased spatial accuracy. Of the three pansharpening algorithms, IHS produced the most spatially-accurate segments for all land cover objects of interest, while SFIM produced the least spatially-accurate segments. The worse spatial accuracy of SFIM was likely due to the fact that it degraded the spatial resolution of the PAN image, causing the edges of objects in the pansharpened image to be blurry (leading to less accurate segment boundaries). IHS and BT suffered from spectral distortion, but they seemed to at least preserve sufficient local contrast (i.e., adjacent land cover objects were still relatively easy to distinguish from one another), so this spectral distortion had less of an effect on the spatial accuracy of segmentation than the spatial information degradation of SFIM. The better performance of IHS over BT in all cases suggests that the saturation compression effect of IHS had less of a negative impact than the saturation stretch effect of BT for segmentation purposes.

Table 1. D Metric values (D) of the most spatially-accurate segmentations (i.e., those with the lowest D value) for each pansharpening method, and the Threshold, Shape, and Compactness parameters that produced these segmentations. Lower D indicates a more spatially-accurate segmentation. The pansharpening method with the most accurate segmentation for each land cover of interest is highlighted in gray.

Land Cover of Interest (Study Area)	Pansharpening Method	D	Threshold	Shape	Compactness
Trees (Residential Area)	PAN (No pansharpening)	0.7666	15	0.5	0.7
	IHS	0.7156	10	0.9	0.9
	BT	0.7224	15	0.5	0.1
	SFIM	0.7598	15	0.3	0.3
Buildings (Residential Area)	PAN (No pansharpening)	0.653	85	0.7	0.9
	IHS	0.6362	65	0.9	0.9
	BT	0.654	60	0.9	0.9
	SFIM	0.6705	45	0.9	0.7
Damaged Oak Trees (Forested Area)	PAN (No pansharpening)	0.7594	20	0.7	0.9
	IHS	0.5856	15	0.9	0.5
	BT	0.6115	20	0.5	0.5
	SFIM	0.6299	10	0.9	0.9

Table 1. D Metric values (D) of the most spatially-accurate segmentations (i.e., those with the lowest D value) for each pansharpening method, and the Threshold, Shape, and Compactness parameters that produced these segmentations. Lower D indicates a more spatially-accurate segmentation. The pansharpening method with the most accurate segmentation for each land cover of interest is highlighted in gray.

Land Cover of Interest (Study Area)	Pansharpening Method	D	Threshold	Shape	Compactness
Trees (Residential Area)	PAN (No pansharpening)	0.7666	15	0.5	0.7
	IHS	0.7156	10	0.9	0.9
	BT	0.7224	15	0.5	0.1
	SFIM	0.7598	15	0.3	0.3
Buildings (Residential Area)	PAN (No pansharpening)	0.653	85	0.7	0.9
	IHS	0.6362	65	0.9	0.9
	BT	0.654	60	0.9	0.9
	SFIM	0.6705	45	0.9	0.7
Damaged Oak Trees (Forested Area)	PAN (No pansharpening)	0.7594	20	0.7	0.9
	IHS	0.5856	15	0.9	0.5
	BT	0.6115	20	0.5	0.5
	SFIM	0.6299	10	0.9	0.9

Next, we assessed the spectral accuracy of the most spatially-accurate segmentations of each pansharpened image. As shown in Table 2, for all land cover objects of interest, segmentations of the IHS and BT imagery had high RMSE values and BIAS values well below 0. This indicates that these two pansharpening methods produced segments with a low spectral accuracy, and that they tended to significantly reduce the DN values of image segments. SFIM imagery, on the other hand, produced segmentations with a much lower RMSE and BIAS values much closer to 0, indicating a higher spectral accuracy.

Table 2. RMSE and BIAS of the most spatially-accurate segmentations for each pansharpening method. The pansharpening method with the highest spectral accuracy (i.e., lowest RMSE and BIAS) is highlighted in gray.

Land Cover of Interest (Study Area)	Pansharpening Method	B RMSE	G RMSE	R RMSE	NIR RMSE	B BIAS	G BIAS	R BIAS	NIR BIAS
Trees (Residential Area)	IHS	115.8	115.8	115.8	115.8	−79.7	−79.7	−79.7	−79.7
	BT	136.8	102.6	85.5	179.2	−93.3	−67.5	−48.8	−128.7
	SFIM	102.5	81.1	74.3	125.0	10.6	8.6	8.5	9.6
Buildings (Residential Area)	IHS	107.9	107.9	107.9	107.9	−70.7	−70.7	−70.7	−70.7
	BT	129.7	94.9	76.1	144.2	−90.9	−64.1	−46.3	−106.9
	SFIM	95.5	77.9	71.5	103.5	28.5	23.2	21.6	29.6
Damaged Oak Trees (Forested Area)	IHS	75.8	75.8	75.8	75.8	−70.5	−70.5	−70.5	−70.5
	BT	90.8	60.8	29.5	190.0	−81.8	−55.0	−26.8	−174.0
	SFIM	39.2	26.4	12.7	85.8	−7.9	−5.3	−2.8	−14.7

Table 2. RMSE and BIAS of the most spatially-accurate segmentations for each pansharpening method. The pansharpening method with the highest spectral accuracy (i.e., lowest RMSE and BIAS) is highlighted in gray.

Land Cover of Interest (Study Area)	Pansharpening Method	B RMSE	G RMSE	R RMSE	NIR RMSE	B BIAS	G BIAS	R BIAS	NIR BIAS
Trees (Residential Area)	IHS	115.8	115.8	115.8	115.8	−79.7	−79.7	−79.7	−79.7
	BT	136.8	102.6	85.5	179.2	−93.3	−67.5	−48.8	−128.7
	SFIM	102.5	81.1	74.3	125.0	10.6	8.6	8.5	9.6
Buildings (Residential Area)	IHS	107.9	107.9	107.9	107.9	−70.7	−70.7	−70.7	−70.7
	BT	129.7	94.9	76.1	144.2	−90.9	−64.1	−46.3	−106.9
	SFIM	95.5	77.9	71.5	103.5	28.5	23.2	21.6	29.6
Damaged Oak Trees (Forested Area)	IHS	75.8	75.8	75.8	75.8	−70.5	−70.5	−70.5	−70.5
	BT	90.8	60.8	29.5	190.0	−81.8	−55.0	−26.8	−174.0
	SFIM	39.2	26.4	12.7	85.8	−7.9	−5.3	−2.8	−14.7

4.2. IHS-SFIM Combination Approach

Since IHS led to very spatially-accurate image segments and SFIM led to very spectrally-accurate image segments, in this section we propose a new IHS-SFIM hybrid approach that combines the benefits of both pansharpening algorithms to produce a final segmented image with a high spatial and spectral accuracy. To combine the spatially-accurate IHS segment boundaries with the spectrally-accurate information of the SFIM pansharpened imagery, we developed a simple two-step process, shown in Figure 3. First, the segment polygons from the most accurate segmentations of the IHS pansharpened imagery were overlaid onto the SFIM imagery. For the residential study area, the segmentation with the highest spatial accuracy for single trees and the segmentation with the highest spatial accuracy for buildings were overlaid onto the SFIM image, and for the forested study area the segmentation with the highest accuracy for damaged/dead trees was overlaid onto the SFIM image. Next, the mean spectral values for each image segment were extracted from the SFIM image bands, and these mean values replaced the spectral values derived from the IHS image. Thus, the final segmentations consisted of segment boundaries from the IHS image segmentation and spectral information from the SFIM imagery.

Figure 3. True color IHS pansharpened image of: (a) a subset of the forested study area, (b) the most spatially-accurate segmentation of the IHS image, and (c) image segments from (b) overlaid on the SFIM pansharpened image to extract more accurate mean spectral values for the segments. Brown areas in the images show trees severely damaged or killed by Raffaelea quercivora. False color imagery (NIR, R, G) used in place of the true color imagery in (d–f) for visualization purposes. White-colored areas show stressed, damaged, or dead trees, which are clearer in (f).

Figure 3. True color IHS pansharpened image of: (a) a subset of the forested study area, (b) the most spatially-accurate segmentation of the IHS image, and (c) image segments from (b) overlaid on the SFIM pansharpened image to extract more accurate mean spectral values for the segments. Brown areas in the images show trees severely damaged or killed by Raffaelea quercivora. False color imagery (NIR, R, G) used in place of the true color imagery in (d–f) for visualization purposes. White-colored areas show stressed, damaged, or dead trees, which are clearer in (f).

The spectral accuracies of these final segmentations, shown in Table 3, were much higher than the spectral accuracies of the original IHS segmentations in Table 2. For the most spatially-accurate segmentation of single trees in the residential area (i.e., the IHS segmentation with Threshold, Shape, and Compactness parameters of 10, 0.9, and 0.9), use of the IHS-SFIM hybrid approach led to a relative reduction in RMSE values for the B, G, R, and NIR spectral bands of 19.6% (from 115.8 to 93.0), 37.1% (from 115.8 to 72.8), 42.0% (from 115.8 to 67.2) and 4.7% (from 115.8 to 110.4), respectively. BIAS for each of the four spectral bands was decreased by 86.6%, 89.1%, 89.3%, and 86.7%, respectively. For the most spatially-accurate segmentation of buildings in the residential area, RMSE was reduced by 20.1%, 33.9%, 37.3%, and 13.3%, and BIAS decreased by 63.9%, 69.5%, 70.0%, and 58.6%. Finally, for the most spatially-accurate segmentation of insect-damaged trees in the forested study area, RMSE was reduced by 61.1%, 74.0%, 87.3%, and 18.3%, and BIAS decreased by 88.1%, 92.1%, 95.7%, and 79.6%. These results indicate that, for both study areas and all three land cover types of interest, the proposed IHS-SFIM approach was able to achieve the same spatial accuracy as the most spatially-accurate segmentations (i.e., the IHS segmentations in Table 1), while significantly increasing the spectral accuracy of these segmentations (using SFIM spectral information).

For practical applications like image classification, the spectral and spatial information of image segments in one or more of these final segmentations could be incorporated for analysis. For example, to detect insect-damaged trees in the forested study area, the spectral (mean values for each band) and spatial attributes (e.g., texture, size, shape, etc.) of image segments in the most accurate segmentation could be used as classification variables. On the other hand, a multi-scale classification approach similar to [23,24] may be preferable for mapping trees and buildings in the residential area since trees and buildings are much different in terms of size and shape.

Table 3. RMSE and BIAS of each spectral band using the proposed IHS-SFIM combination approach. The values are similar to those of SFIM in Table 2, and have D Metric values equal to those of IHS in Table 1.

Land Cover of Interest (Study Area)	Pansharpening Method	B RMSE	G RMSE	R RMSE	NIR RMSE	B BIAS	G BIAS	R BIAS	NIR BIAS
Trees (Residential Area)	IHS-SFIM	93.0	72.8	67.2	110.4	10.7	8.7	8.5	10.6
Buildings (Residential Area)	IHS-SFIM	85.3	71.3	67.7	93.5	25.5	21.5	21.2	29.3
Damaged Oak Trees (Forested Area)	IHS-SFIM	29.5	19.7	9.6	61.9	−8.4	−5.6	−3.0	−14.4

Table 3. RMSE and BIAS of each spectral band using the proposed IHS-SFIM combination approach. The values are similar to those of SFIM in Table 2, and have D Metric values equal to those of IHS in Table 1.

Land Cover of Interest (Study Area)	Pansharpening Method	B RMSE	G RMSE	R RMSE	NIR RMSE	B BIAS	G BIAS	R BIAS	NIR BIAS
Trees (Residential Area)	IHS-SFIM	93.0	72.8	67.2	110.4	10.7	8.7	8.5	10.6
Buildings (Residential Area)	IHS-SFIM	85.3	71.3	67.7	93.5	25.5	21.5	21.2	29.3
Damaged Oak Trees (Forested Area)	IHS-SFIM	29.5	19.7	9.6	61.9	−8.4	−5.6	−3.0	−14.4

The proposed hybrid approach was used in this study for combining the positive characteristics of two different pansharpening algorithms, but it may also be useful to apply this approach for pansharpening algorithms with adjustable parameters or filters (e.g., [6,7,8,9,10]) that control the tradeoff between spatial and spectral information preservation. For example, segment boundaries can be generated from images pansharpened using parameters/filters that maximize spatial information preservation, and the spectral information of these segments can be derived from images pansharpened using parameters/filters that minimize spectral distortion.

5. Conclusions

In this study, we compared the effects of IHS, BT, and SFIM pansharpening on the spatial and spectral accuracy of image segmentation and proposed a new IHS-SFIM hybrid approach based on the results of these comparisons. IHS and BT tend to preserve the spatial information of the PAN band while distorting the spectral information of the MS bands, while SFIM preserves accurate spectral information from the MS bands while losing some information from the PAN band. We found that IHS and BT pansharpening led to more spatially-accurate segmentation (with IHS producing the most spatially-accurate segmentations), indicating that spatial information preservation was most useful for segmentation purposes. On the other hand, SFIM pansharpening led to segments with more accurate spectral information, which may be more important for applications such as image classification, change detection, or the extraction of vegetation biophysical parameters.

To combine the high spatial accuracy of IHS segments with the high spectral accuracy of SFIM imagery, we proposed a hybrid approach developed specifically for OBIA that involves (i) overlaying the segment boundaries from the IHS image segmentation onto a SFIM image and (ii) deriving the spectral values for image segments (mean DN for each spectral band) from the SFIM imagery. Based on our segment-based calculations of spatial and spectral accuracy, the proposed approach led to higher spatial and spectral accuracy of image segments than the use of a single pansharpening algorithm alone. Since a hybrid approach including two pansharpening methods produced the best results in this study, we recommend users planning to process images with PAN and MS bands using OBIA to pansharpen the imagery themselves (rather than purchase pansharpened imagery directly from the image vendor) so that they can incorporate multiple pansharpening methods for their analysis. It should be noted that we only tested D for measuring the spatial accuracy of image segments, and RMSE and BIAS for measuring the spectral accuracy of image segments. In future studies it may be beneficial to include additional spatial and spectral accuracy measures to see if our findings remain consistent. Finally, since image classification often follows image segmentation in OBIA, future studies are needed to quantitatively assess the impact that different pansharpening algorithms and our proposed hybrid approach have on classification accuracy.