Image Segmentation and Object-Based Image Analysis for Environmental Monitoring: Recent Areas of Interest, Researchers’ Views on the Future Priorities

Abstract

Image segmentation and geographic object-based image analysis (GEOBIA) were proposed around the turn of the century as a means to analyze high-spatial-resolution remote sensing images. Since then, object-based approaches have been used to analyze a wide range of images for numerous applications. In this Editorial, we present some highlights of image segmentation and GEOBIA research from the last two years (2018–2019), including a Special Issue published in the journal Remote Sensing. As a final contribution of this special issue, we have shared the views of 45 other researchers (corresponding authors of published papers on GEOBIA in 2018–2019) on the current state and future priorities of this field, gathered through an online survey. Most researchers surveyed acknowledged that image segmentation/GEOBIA approaches have achieved a high level of maturity, although the need for more free user-friendly software and tools, further automation, better integration with new machine-learning approaches (including deep learning), and more suitable accuracy assessment methods was frequently pointed out.

1. Introduction

Image segmentation and (geographic) object-based image analysis (GEOBIA [1], or simply OBIA), have been utilized in remote sensing for around two decades now [2]. Image segmentation is the first step of GEOBIA, and involves the partitioning of an image into relatively homogeneous regions, i.e., “image segments” or “image objects” [3]. These image segments serve as the base unit for further analysis, e.g., image classification or change detection, using the spectral/spatial/contextual attributes of the segments. Image segmentation is a fundamental issue in GEOBIA research, as the quality of segmentation results often affects the accuracy of subsequent analysis (e.g., land-use/land-cover classification accuracy).

Originally, GEOBIA was proposed as a way to incorporate contextual information for high-spatial-resolution image classification, which was necessary because the pixels in these images tend to be smaller than the real-world features intended to be mapped [2,3]. Since then, it has been used to analyze images having a wide range of spatial resolutions and from various types of sensors (e.g., multispectral, hyperspectral, synthetic aperture radar). The first major review of this topic was conducted in 2010 [4], and since then several others have been undertaken [5,6,7].

In this Editorial, we share some highlights of GEOBIA research over the last two years (2018–2019), including a Special Issue on the topic in the journal Remote Sensing. We also present 45 researchers’ responses to an online questionnaire on the current state and future priorities of GEOBIA research.

2. Highlights from 2018–2019

2.1. Research Topics of Interest

From a search of the Scopus database (title/keyword/abstract search for papers containing the term “object-based image analysis”), we identified 369 journal articles published on the topic of GEOBIA over the last two years (2018–2019). From these articles, we attempted to highlight some topics of significant recent interest based on the text in the papers’ titles/keywords/abstracts. High-frequency terms from the text were identified using Citespace software [8], and after filtering out several overly general terms (e.g., “object”, “based”, “image”, “analysis”, “remote sensing”, and “resolution”), a wordcloud map (Figure 1) was generated to allow for a visualization of the frequently-used terms (larger words in the figure were more frequently used). In Figure 1, mapping and segmentation can be seen as the most frequent areas of interest overall, which is perhaps unsurprising. The types of applications GEOBIA was most frequently used to support can be seen as forestry, vegetation, wetland, and urban area analysis. Classification algorithms that were of significant interest included decision trees (which are often incorporated in ensemble algorithms like random forests [9]) as well as support vector machines [10]. Finally, the most frequent remote sensing datasets of interest included Landsat images, synthetic aperture radar (SAR) data, Worldview images, Sentinel images, images from UAVs/other airborne optical sensors, and Lidar data. This frequent interest in moderate spatial resolution imagery (e.g., Landsat and Sentinel) as well as SAR/Lidar data suggests that GEOBIA has moved beyond its initial sole focus on high-spatial-resolution optical data.

As another way of looking at the recent areas of interest in GEOBIA research, we also identified the papers that were most frequently cited in these 369 articles (

Table 1. Ten most frequently cited papers in recent articles on GEOBIA (based on an analysis of 369 articles published in Scopus indexed journals from 2018–2019), and the focus of each paper.

Paper Title	# of Times Cited	Year of Publication	Focus of Paper
Object based image analysis for remote sensing [4]	74	2010	Review
Per-pixel vs. object-based classification of urban land cover extraction using high spatial resolution imagery [15]	39	2011	Image classification
Unsupervised image segmentation evaluation and refinement using a multi-scale approach [12]	34	2011	Segmentation parameter selection
Geographic object-based image analysis–towards a new paradigm [11]	34	2014	Review
A review of supervised object-based land-cover image classification [16]	30	2017	Image classification
Change detection from remotely sensed images: From pixel-based to object-based approaches [7]	23	2013	Change detection
An assessment of the effectiveness of a random forest classifier for land-cover classification [18]	22	2012	Image classification
Automated parameterisation for multi-scale image segmentation on multiple layers [13]	22	2014	Segmentation parameter selection
Discrepancy measures for selecting optimal combination of parameter values in object-based image analysis [14]	20	2012	Segmentation parameter selection
Training set size, scale, and features in Geographic Object-Based Image Analysis of very high resolution unmanned aerial vehicle imagery [17]	20	2015	Image classification

). Aside from review articles covering the field as a whole [4,11], the remainder of the 10 most frequently cited papers all dealt with image segmentation parameter selection [12,13,14] or image classification [15,16,17,18]/change detection [7]. This is similar to the result of the title/keyword/abstract text analysis, and indicates that the general areas of interest within GEOBIA are still related to image segmentation and classification/mapping of land-use/land-cover objects of interest.

2.2. Special Issue of Remote Sensing on “Image Segmen Tation for Environmental Monitoring”

In December 2019, a Special Issue on the topic of GEOBIA was published in Remote Sensing, entitled “Image segmentation for environmental monitoring”. The eight papers published in the special issue were largely representative of the current topics of interest within GEOBIA, covering image segmentation algorithm development [19,20] and segmentation parameter optimization strategies [21,22] as well as object-based image classification [23,24,25] and image fusion [26] methods.

On the topic of image segmentation algorithm development, Tang et al. [19] proposed a nonparametric clustering-based segmentation approach called the edge dependent Chinese restaurant process (EDCRP) method, which utilizes both spectral and spatial information for segmentation, and has the benefit of automatically determining the appropriate number of segments to generate. The EDCRP method was found to produce more accurate segmentation results than several other state-of-the-art methods, although it was more computationally intensive. On the other hand, Shepherd et al. [20] proposed a fast clustering-based approach which uses k-means clustering to generate initial clusters of pixels, followed by a local elimination procedure to aggregate small clusters of pixels until a predefined minimum mapping unit size is met. The high speed and scalability of this approach allowed it to be used to segment a mosaic image of the entire continent of Australia at 30m resolution. Notably, a downloadable tool for implementing this method was made available by the authors at https://www.rsgislib.org/.

On the topic of image segmentation parameter selection/optimization, Georganos et al. [21] and Xiao et al. [22] both developed new methods for local (as opposed to global) optimization of segmentation parameters. Georganos et al. [21] approached the problem by first sub-dividing a study area image into smaller sub-regions, and then performing parameter optimization for each of these sub-regions separately. On the other hand, Xiao et al. [22] first identified globally-optimal segmentation parameters, and then refined this initial segmentation to better delineate different types of urban greenery, by utilizing local information (mean pixel values and standard deviation values within each initial segment). Both of these local approaches were found to outperform global segmentation parameter optimization approaches.

On the topic of object-based image classification, Roodposhti et al. [24] developed a robust rule-based ensemble framework (dictionary of trusted rules, or DoTRules) based on mean-shift segmentation. The approach was tested on three common hyperspectral image benchmark datasets, and found to outperform other ensemble classifiers and support vector machines in many cases. Samat et al. [23] mapped vegetation types in an arid landscape, utilizing an object-based morphological profile method (“extended object-guided morphological profile”) to extract contextual features and ensemble algorithms for classification. Finally, Lu et al. [25] applied popular deep learning and transfer learning methods in an object-based image analysis framework to detect landslides in UAV images.

On the last topic, image fusion, Radoux et al. [26] focused on the topic of ecotope mapping using a GEOBIA workflow and multisensor remote sensing data. They found that fusion of aerial optical imagery (blue, green, red, and near-infrared bands) and Lidar topographic data (digital height model and hillshade maps) improved the automated delineation of ecotopes (the smallest ecologically distinct features in a landscape classification system [26]).

We were delighted to receive many high quality papers for this special issue, and would like to sincerely thank all of the authors who submitted their work.

3. Researchers’ Views on the Current Status and Future Priorities of GEOBIA

As a final effort of this Special Issue, we disseminated an online questionnaire to the corresponding authors of journal articles published on the topic in the last two years (i.e., the corresponding authors of the 369 journal articles we found in Scopus), and compiled all of the authors’ responses (Table S1).

Table 2. Questions asked in online survey on image segmentation and GEOBIA.

Question	Format of Response
Q1: How many years have you been using image segmentation and GEOBIA approaches for remote sensing image analysis?	Numerical (1–20)
Q2: How often do you currently use image segmentation/GEOBIA approaches for remote sensing image analysis, compared to other approaches?	Multiple choice
Q3: What topic(s) are, in your opinion, currently NOT receiving sufficient research attention within the field of image segmentation and GEOBIA? (Check all that apply)	Selected from a list (selecting “Other” allows a free response)
Q4: What types of environments are, in your opinion, currently NOT receiving enough research attention within the field of image segmentation and GEOBIA? (Check all that apply)	Selected from a list (selecting “Other” allows a free response)
Q5: On a scale from 1–10, how mature do you believe the current image segmentation and GEOBIA approaches are for remote sensing image analysis?	Numerical score between 1 (“They are still at a very early stage of development”) and 10 (“They are already good enough, and little-to-no further improvements are required”).
Q6: What do you feel is the biggest remaining weakness of the current image segmentation/GEOBIA approaches? (Up to ~100 words)	Free response
Q7: What, in your opinion, should be a priority for image segmentation and GEOBIA research over the next 5–10 years for the field to further mature? (Up to ~100 words)	Free response

shows the questions asked in the survey.

Invitations to participate in the survey were sent by email in March 2020, and we received 45 responses in total. The number of years that the respondents had been using GEOBIA approaches (Q1) ranged from 1–20, with an average of 7.18 years (Figure 2). Around half (46%) of the respondents reported that they used GEOBIA approaches more frequently than other remote sensing image analysis approaches, and another 40% used them about as frequently as other approaches (Q2) (Figure 3). The responses to these two questions suggest that survey respondents were generally quite experienced in the use of GEOBIA.

Among the topics within GEOBIA that were currently not receiving sufficient research attention (Q3), object-based accuracy assessment was the most frequently noted (by 22 respondents), followed by big image data analysis (indicated by 19 respondents), and multi-sensor/multi-temporal data fusion (indicated by 17 respondents) (Figure 4). The latter two topics may be particularly important in the context of the growing archives of free high and moderate spatial resolution satellite data provided by different countries’ space programs. Among the types of environments that were currently not receiving sufficient research attention (Q4), post-disaster areas was the most frequently indicated (by 18 respondents), followed by coastal areas (indicated by 14 respondents) (Figure 5). Interestingly, urban/built-up areas were least frequently indicated for this question, suggesting a potential oversaturation of urban GEOBIA studies. Finally, in response to Q5, the majority of respondents perceived the current image segmentation and GEOBIA approaches as already having received a relatively high level of maturity (i.e., value of 7 or 8 on a scale from 1 (“They are still at a very early stage of development”) to 10 (“They are already good enough, and little-to-no further improvements are required.”)) (Figure 6). That said, several remaining weaknesses of GEOBIA were pointed out in response to Q6.

The replies to the free response questions on the biggest remaining weaknesses (Q6) and future priorities (Q7) of image segmentation and GEOBIA research are all included in Table S1, and intended to serve as the respondents’ anonymous messages to the GEOBIA community. Various views were expressed in response to these two questions, but some common responses were that there is a need for:

• More free (and user-friendly) GEOBIA software and tools;
• Further automation of the segmentation process (especially the parameter setting process);
• More efficient algorithms for handling large image datasets (e.g., for regional/global scale analyses, hyperspectral image analysis, or time-series image analysis);
• Better integration of GEOBIA with deep learning methods as well as 3-D image data;
• More suitable/more standardized accuracy assessment methods.

Some of the other views expressed were unique and quite thought provoking. One interesting response to Q7 stressed the need for greater inclusiveness and creativity, as “Right now the domain as a whole is very centrally controlled by a few people who have clout, and there should be more room for creative ideas.” Another interesting response to Q7 was that GEOBIA research should put more attention on “Detecting individual animals from high spatial resolution imagery”. Most GEOBIA research to date has focused on detection of land features or artificial features of interest, but expanding its applicability to animal monitoring could help broaden interest in GEOBIA. Although there is not space to highlight all of the other responses to the survey (see Table S1), we hope they can provide some general ideas for future GEOBIA research.

To conclude this Special Issue Editorial, we would like to again express our sincere thanks to all of the authors who submitted their work, and to all of the researchers who responded to our questionnaire survey. Much has been accomplished in the first two decades of GEOBIA research, and we look forward to the new developments the next two will bring!