Next Article in Journal
Revisiting the Role of Institutions in Transformative Contexts: Institutional Change and Conflicts
Next Article in Special Issue
Estimation of Ecological Connectivity in a City Based on Land Cover and Urban Habitat Maps
Previous Article in Journal
Soil Quality Assessment Based on a Minimum Data Set: A Case Study of a County in the Typical River Delta Wetlands
Previous Article in Special Issue
Analysis of Environmental Purification Effect of Riparian Forest with Poplar Trees for Ecological Watershed Management: A Case Study in the Floodplain of the Dam Reservoir in Korea
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Comparison of Fuzzy AHP and AHP in Multicriteria Inventory Classification While Planning Green Infrastructure for Resilient Stream Ecosystems

1
Department of Forestry and landscape Architecture, Graduate School, Konkuk University, Seoul 05029, Korea
2
Department of Forestry and Landscape Architecture, Konkuk University, Seoul 05029, Korea
3
Life and Environment Research Institute Sanghuh College of Life Sciences, Konkuk University, Seoul 05029, Korea
*
Author to whom correspondence should be addressed.
Sustainability 2020, 12(21), 9035; https://doi.org/10.3390/su12219035
Submission received: 6 October 2020 / Revised: 25 October 2020 / Accepted: 26 October 2020 / Published: 30 October 2020

Abstract

:
As climate change-induced weather variability increases, various green infrastructure plans, such as stream ecosystems, have been studied to overcome ecological and environmental problems arising from extreme weather events; however, our understanding of them and their resilience to extreme weather events is considerably limited. This study proposes a multicriteria inventory classification while planning green infrastructure for resilient stream ecosystems under extreme weather events. Literature reviews, expert surveys, and reliability/validity analyses were used to enlist indicators for this classification. The analytic hierarchy process (AHP) and fuzzy analytic hierarchy process (fuzzy AHP) were used to compute the weights and ranks of indicators for identifying critical indices while planning green infrastructure. The AHP and fuzzy AHP analysis suggested that meteorological phenomena and disasters, hydraulic characteristic of streams, land use/geographic characteristics, and experience/damage restoration were important factors. High weights were attributed to aquatic ecology, potentially vulnerable areas, population, topography, and heat waves. The weights and ranks attributed by AHP and fuzzy AHP varied slightly, but the indicator groups with high and low weights were the same; hence, primary indicators to be considered while planning green infrastructure for resilient stream ecosystems could be suggested. These results could be used as a preliminary analysis in establishing countermeasures against climate change or in distributing budgets for green infrastructure plans.

1. Introduction

Stream ecosystems are particularly vulnerable to climate change because they are directly affected by extreme weather events caused by climate change and spatiotemporal variability [1]. Extreme weather events, such as drought, flooding, and heat waves lead to fluctuations in water temperature and flow rates in stream ecosystems, resulting in physical, chemical, and biological disturbances in the stream ecosystems [2,3,4]. Furthermore, extreme weather events affect environmental systems that include stream ecosystems; consequently, the entire environmental system become less resilient [5].
To reduce the damages and disturbances in an environmental system including streams, the concept of resilience needs to be considered. First introduced by Holling [6], in terms of climate change and ecology, “resilience” was defined as the ability to absorb disturbances caused by climate change or to self-restructure to continue a prevailing state of transformation into another stable state [7,8]. An important way of increasing resilience and reducing the effects of climate change is by developing green infrastructure. Green infrastructure can help in strengthening networks between greenspaces and using natural-based accesses while planning such infrastructure to improve stream ecosystems [9,10]. In addition, the structure and function of stream ecosystems could be restored, which would help in improving water quality and storm water storage capacity and reducing the effects of meteorological disasters, ultimately making stream ecosystems more resilient [11,12]. The resilience of stream ecosystems is an abstract concept that cannot be directly observed or measured; hence, substitute indicators and variables must be used to determine its attributes [13,14]. Stream ecosystems are complex systems affected by social, physical, and biological factors; hence, when determining the indicators for the resilience of stream ecosystems, it is necessary to consider the social–ecological system of a region [15,16].
A social–ecological system is one that encompasses social and natural elements and connects society, economy, and the environment [12]. In green infrastructure, the interaction between social–ecological systems helps identify their vulnerabilities and clarify the indicators that affect them [17,18]. Vugteveen et al. [17] used a social–ecological system to derive potential indicators for the integrated management of the Dutch Wadden Sea region, while Flynn and Davidson [12] recognized indicators affecting storm water by using a social–ecological system for storm water management through green infrastructure planning. In addition to determining the indicators that consider the multifunctional aspects of ecology, society, and economy, determining the weights and ranks of each indicator is also important. The weights and ranks could be used as a basis to appropriately allocate budgets when planning green infrastructure and to prepare proactive plans for areas with weak resilience against climate change effects [19]. Kotzee and Reyers [20] used social–ecological system-based indicators whose weights were used to assess flood resilience and provide a basis for disaster policy and planning. Kim et al. [21] assessed the degree of vulnerability of water management to climate change by using vulnerability indicators and their weights, helping identify areas to invest in, for preparing against climate change.
A representative method considering weights and ranks is the analytic hierarchy process (AHP). This method has been widely used in determining evaluation indicators to cope with climate change because it helps relatively represent the influence of indicators, compare each alternative, and examine the suitability of judgment [19,22,23,24,25]. However, the AHP has raised challenges in terms of language ambiguity and uncertainty in the decision-making process. The fuzzy AHP, which was introduced to solve this problem, is helping derive weights and priorities of indicators in various fields, such as economics, safety engineering, disaster risk, and water resource management planning [26,27,28]. In addition, the weights and ranks of various indicators obtained through the AHP and fuzzy AHP were compared to support decision makers in selecting optimal alternatives and indicators [29,30,31,32]. The AHP is one of the best ways to make a determination within a complex criteria structure at various levels. The fuzzy AHP is a synthetic extension of the AHP method, given the ambiguity of decision making [33]. Thus, this study presented a comparative analysis of AHP and fuzzy AHP for multicriteria inventory classification. In addition, previous studies have also dealt with vulnerability assessment and the level of resilience in terms of climate change, respectively. Therefore, it has the distinction of being a paper combining green infrastructure planning and vulnerability assessments in order to improve the durability of a stream ecosystem.
Therefore, this study aimed to establish a multicriteria inventory classification for consideration while planning green infrastructure for a resilient stream ecosystem in response to climate change. It also aimed to establish the weights and ranks of various indicators considered in the multicriteria inventory classification and compared the results obtained through the AHP and fuzzy AHP.

2. Literature Review

Streams are a critical aspect of green infrastructure, which not only maintain biodiversity but also provide social and economic benefits [34,35]. However, several climate change induced extreme weather events, including heat waves, droughts, floods, and typhoons, threaten the resilience of stream ecosystems. Previous studies on green infrastructure planning have been conducted to improve the resilience of stream ecosystems in response to climate change. For example, Environmental Protection Agency (EPA) and Yaoze Liu [36,37] presented low-impact development to improve urban water circulation and reduce the effects of floods and storm water, while Han et al. [38] suggested facilities such as rooftop greening and artificial wetlands to control runoff. Beckingham [18] suggested providing ecological services, dampening storm water flows, and reducing pollutants through storm water ponds built using the green infrastructure best management practice method. Chao Mei [39] conducted an integrated assessment based on a storm water management model and life cycle cost analysis to mitigate flooding in urbanized watersheds and identified the most effective green infrastructure strategy. However, studies on green infrastructure planning for stream ecosystems have mainly focused on disaster prevention and hydrologic systems; hence, research concerning water environments and aquatic ecology were found to be relatively scarce [40].
Studies have also been conducted to develop vulnerability assessment indicators to improve resilience to extreme weather events caused by climate change. In the marine sector, indicators were selected to evaluate vulnerability in the fisheries sector [23], resilience in marine regions [41,42,43], and the effects of climate change on marine organisms [44]. In the agricultural sector, indicators were selected to evaluate the vulnerability to agricultural production infrastructure and productivity [45,46], vulnerability to rice pests [47], and resilience of African agriculture to climate change [48]. In addition, an index was developed based on spatial information to evaluate the vulnerability of forest ecosystems to climate change [49,50], while in the urban sector, the vulnerability to heat waves and urban thermal environments was assessed, and an improvement policy was proposed [24,51,52]. Studies on the effects of climate change on water management have focused on biological elements [53,54,55], disaster prevention [19,56], and water resources [57,58,59]. However, research on the effects of climate change on the water quality and aquatic ecosystems of streams is still in a nascent stage [21,60]. With climate change induced extreme weather phenomena, resilience indicators for stream ecosystems which suggested for evaluating vulnerability to climate change are related to social system and ecological system directly and indirectly [21,22,60,61,62,63,64,65,66]. The indicators that help evaluate the frequency and intensity of meteorological disasters, social indicators such as population, and those that play a role in mitigating disasters have been suggested [61,62,63,64]. Kim et al. and Kim et al. [21,58] researched the vulnerability of water management and presented indicators for areas directly affected by disasters, and for physical property and human damage caused by disasters. Park et al. and Koh and Kim [22,59] suggested indicators to evaluate the level of stable supply of water resources and access to clean water, and to evaluate economic resources and administrative capabilities when responding to disasters. Han et al. and Myung et al. [65,66] suggested indicators related to social infrastructure that play a pivotal role in maintaining and recovering urban functions during meteorological disasters. Although Ministry of Environment (MOE) and Ryu et al. [67,68] is not a study related to evaluating vulnerability, it is referred to when selecting an indicator because it represents the structure, hydraulic characteristics, and ecological health of streams, and presents social, economic, and environmental indicators that affect streams.
In order to develop a resilient stream ecosystem in response to climate change, previous studies were identified to understand the type of green infrastructure plans required for improving the resilience of stream ecosystems, and to select indicators to evaluate their vulnerability and resilience. Although studies to address climate change have been conducted in various fields, green infrastructure planning and vulnerability assessments have mainly focused on physical factors and disaster prevention. Hence, understanding the ecological, social, and environmental aspects of stream ecosystems was a fundamental step. Therefore, by establishing evaluation indicators through a literature review, we aimed to establish a multicriteria inventory classification for consideration while planning green infrastructure for resilient stream ecosystems based on a social–ecological system that helps encompass the complexity of stream ecosystems.

3. Research Methodology

3.1. AHP

This study proposes the use of AHP and fuzzy AHP to determine the weights of indicators in a multicriteria inventory classification. The AHP was devised by Saaty [69] to make the best choice through weights and ranks, considering various alternatives and complex criteria (Table 1). The AHP helps simplify criteria by clarifying the overall goal and organizing complex criteria into a hierarchical structure, such as criteria, sub-criteria, etc. [70]. In addition, the relative weights of the alternatives of each criterion and their overall ranks can be easily identified through a pairwise comparison [71]. The AHP is widely used in various fields requiring decision-making processes and also for weighting of vulnerability assessments of climate change [23,24,72].
The first step of the AHP involves establishing a hierarchical structure of decision-making problems. A comprehensive goal is established at the top level; thereafter, a connection is made between the top and bottom elements, i.e., the interrelated decision elements to achieve the overall goal. The second step involves a pairwise comparison of the decision elements. To obtain the weight of the decision element, we can ask how much i is preferred for element j compared to each element in the hierarchy consisting of n elements using a nine-point scale. The nine-point scale is as proposed by Saaty [69] based on Miller’s experimental results, wherein the human short-term information processing ability is determined as 7 ± 2 and is most often employed when the AHP method is applied [73]. a i j = 1 means i and j are equally important, a i j = 7 means i is very strongly more important than j , and a i j = k signifies that a j i = 1 / k . The result of comparing elements i and j can be obtained with the comparison matrix A = ( a i j ); i ,   j = 1,2, ···, n.
A =   [ 1 a 1 n a n 1 1 ]
The third step involves deleting inconsistent values by verifying consistency. The consistency index (CI) and consistency ratio (CR) are used to verify whether respondents consistently respond to pairwise comparison questions. For example, if the respondents answer that i is more important than j , and j is mor important than k , then for i and k , the respondents should answer that i is more important than k . The more consistent the responses, the smaller is the CI; and if the CR is ≤0.1, it is determined to be consistent. Criteria with CR < 0.1 and < 0.2 are considered reasonable and acceptable, respectively [71]. Here, the CR was set below 0.2 considering that the AHP hierarchical structure consisted of two layers; hence, there are numerous pairwise comparison elements. The CI and CR can be obtained by Equations (2) and (3), where λ m a x is the principal eigenvalue, and RI is a random index.
C I =   λ m a x n n 1  
C R =   C I R I
Finally, the vectors of weights are calculated by Saaty’s eigenvector. After comparison matrices are composed, the eigenvector is normalized by Equation (4). Then, the weights are calculated by Equation (5).
a i j =   a i j j = 1 n a i j
w i j =   j = 1 n a i j n

3.2. Fuzzy AHP

The AHP method has many positive aspects; however, it has a limitation. It is difficult for respondents to clearly determine their preferences as a single value, and the standard of the accepted value could differ depending on the respondent, even if the same value is obtained [74,75]. A methodology was required to compensate for the ambiguity and uncertainty of the response due to the linguistic limitations of the AHP, and a fuzzy AHP emerged [75,76]. Fuzzy AHP, developed by Laarhoven and Pedrycz [77], is an addition of the existing fuzzy theory to the AHP method to compensate for the disadvantages of AHP. The fuzzy AHP is expressed as an interval using a triangular fuzzy scale rather than a specific value using a nine-point scale to reflect the uncertainty of the selection of respondents [78]. The execution of the fuzzy AHP is outlined here. First, as in the AHP, a hierarchical structure is established and a pairwise comparison is performed; Table 2 shows the scale used for the pairwise comparison [78,79].
Second, through Equations (6)–(9), the value of the fuzzy synthetic extent S i = ( l i ,   m i ,   u i )   for the i th object can be calculated as follows:
S i =   j = 1 m M i j   ×     [ i = 1 n j = 1 m M i j ] 1
s .   t   j = 1 m M i j   =     ( j = 1 m l i j ,   j = 1 m m i j ,   j = 1 m u i j )
i = 1 n j = 1 m M i j =     ( i = 1 m l i j ,   i = 1 m m i j ,   i = 1 m u i j )
[ i = 1 n j = 1 m M i j ] 1 =     (   1 i = 1 n u i j ,   1 i = 1 n m i j ,   1 i = 1 n l i j )
Third, the degree of possibility between two fuzzy synthetic extent values is compared. The degree of probability that S j =   ( l j ,   m j ,   u j ) S i   =   ( l i ,   m i ,   u i ) is obtained as follows:
V ( S 2   S 1 ) = h g t ( S 2   S 1 ) =   μ ( d ) =   { = 1 ,                                                                       i f   m 2   m 1 = 0 ,                                                                       i f       l 1     u 2 = l 1 u 2 ( m 2 u 2 ) ( m 1 l 1 ) ,                                           o t h e r w i s e
Fourth, the degree of possibility for a convex fuzzy number to be greater than “k” convex fuzzy numbers can be calculated as follows:
V ( S i S 1 ,   S 2 ,   ,   S k ) =   { V [   ( S i S 1 ) a n d ( S i S 2 ) a n d ( S i S k )   ] min V ( S i S j ) ,       i = 1 ,   2 ,   3 ,   ,   k                                  
Finally, it is assumed that d ( i )   =   m i n   V ( S i > = S k ) . Then, the weight vector is given by W = ( d ( 1 ) ,   d ( 2 ) ,   ,   d ( i ) ) T . Through normalization, the normalized weight vectors are W = ( d ( 1 ) ,   d ( 2 ) ,   ,   d ( i ) ) T , where W is a non-fuzzy number.

3.3. Methodology Framework

This study aimed to propose a multicriteria inventory classification method on green infrastructure planning for resilient stream ecosystem in response to extreme weather events. In addition, this study examined the relative weights and priorities of classified indicator groups. From previous studies, we extracted 116 potential indicators assessing vulnerability and resilience of stream ecosystems for establishing green infrastructure in response to extreme climate events. To focus on establishing the indicators for a multicriteria inventory classification and calculating their weights for a resilient stream ecosystem, the methodology was systematic/structured. Here, the methodology framework was formulated in three primary stages (Figure 1).
Stage 1: Establishment of the multicriteria inventory classification, which included: (a) gathering potential evaluation indicators from prior studies; (b) classifying the evaluation indicators based on various criteria; and (c) confirming the reliability and validity of the evaluation indicators through a pilot test for the main survey.
A reliability and factor analysis was used for the pilot test, and irrelevant indicators were deleted to refine the classification of the evaluation indicators [80,81]. The respondents consisted of 22 researchers with a master’s degree or higher in aquatic ecosystem, resilience, and landscape studies. They were asked questions about the importance of each evaluation indicator, which were scored based on the most used Likert five-point scale. Multiple on-line surveys were performed for the multiple expert groups, including government agencies, engineers, and college educators, during the period of 1 January–31 March in 2020.
Stage 2: Weights and ranks were assigned to the indicators being considered in the multicriteria inventory classification, which included the following: (a) establishing a hierarchical structure and (b) analyzing pairwise comparisons using AHP and fuzzy AHP.
The indicators for a resilient stream ecosystem were confirmed through stage 1, and the main survey was conducted to obtain their weights and ranks in the multicriteria inventory classification. The main survey was divided into socio-economic and physio-environmental categories, with 110 and 83 respondents, respectively, who each held a master’s degree or higher in aquatic ecosystem, resilience, water ecology, and landscape studies. The respondents were selected using the purposive or judgment sampling method to intentionally select a sample suitable for the research subject, and the snowball sampling method was used to expand the number of samples based on the recommendation of the respondents [82].
Stage 3: The results of the AHP and fuzzy AHP were compared; this included a comparison of the differences in ranking.

4. Results

4.1. Establishment of the Multicriteria Classification

4.1.1. Gathering Potential Evaluation Indicators from Prior Studies

Previous studies for gathering evaluation indicators were reviewed using Google Scholar, Web of Science, Scopus and Korea Citation Index (KCI). Keywords included in the literature review were “resilience,” “vulnerability,” “stream ecosystem,” “green infrastructure,” and “climate change.” Searches were carried out for studies published between 2000 and 2019. In total, 105 research papers and reports were reviewed to enlist indicators for evaluating vulnerability or resilience to climate change, improvement plans considering climate change effects, and for categorizing vulnerable areas. Research that was unrelated to the resilience of stream ecosystems was excluded. Thus, 410 indicators were derived, and 116 potential indicators were selected by excluding indicators having duplicate meanings and those irrelevant to stream ecosystems.

4.1.2. Classifying Evaluation Indicators Based on “Classification Criteria”

Based on the social–ecological systems concerning green infrastructure, the indicators for a resilient stream ecosystem were also divided into socio-economic and physio-environmental categories, and the criteria for the classification of these categories were set as shown by Table 3.
The evaluation indicators were organized based on the “evaluation indicator classification criteria,” and were classified into physio-environmental and socio-economic categories. Additionally, “evaluation indicator classification criteria” and the evaluation indicators classified accordingly were grouped into division, subdivision, and evaluation indicators to help identify a stream ecosystem and to establish a hierarchical structure for a pairwise comparison. The physio-environmental sector was divided into water quality, topography, biological factors, hydraulic characteristics of streams, meteorological phenomena, and meteorological disasters, while the socio-economic sector was divided into population, land use/geographic characteristics, economy/system, infrastructure, and experience/damage restoration. The results of the multicriteria inventory classification, composed of the indicators classified based on various criteria and a detailed outcome of factor and reliability analyses, are showed in Appendix A and Appendix B. Table 4 and Table 5 show the summarized results of factor and reliability analyses.

4.1.3. Confirming the Reliability and Validity of Evaluation Indicators

The pilot test was conducted based on the multicriteria inventory classification to confirm the reliability and validity of the evaluation indicators and to establish their importance. In the pilot test, 21 of the 22 participants responded, and there were 0–3 missing responses for each indicators, depending on its type. The Cronbach’s a   values for both the physio-environmental and socio-economic sectors indicated that the indicators were reliable (Table 4 and Table 5). The number of evaluation indicators was reduced by excluding those having communalities < 0.4 and low factor loading values through the factor analysis. In addition, indicators with considerably low importance or those that could not be clearly measured were also excluded; thereafter, 85 evaluation indicators were shortlisted.

4.2. Assigning Weights and Ranks to Indicators Considered in the Multicriteria Inventory Classification

4.2.1. Establishing a Hierarchical Structure

For the main survey, an AHP hierarchical structure was established through the division and subdivision of the evaluation indicators that were reliable and valid. As shown in Figure 2, the top layer of the hierarchical structure highlighted the primary objectives of the study, the first tier had the divisions as judgment criteria, and the second tier had the subdivisions as an alternative. A pairwise comparison of 11 divisions of the judgment criteria hierarchy and 25 subdivisions of the alternative hierarchy was performed based on the AHP hierarchical structure. We received responses from 83 and 110 experts in the physio-environmental and socio-economic sectors, respectively. Respondents with a CR ≥ 0.2 were excluded from the analysis. Consequently, 74, 56, 60, 59, 98, 87, and 76 respondents for the first tier, hydraulic characteristics, meteorological phenomena, meteorological disasters, land use/geographic characteristics and economy/system, infrastructure, and experience/damage restoration, respectively, were found to have a CR value ≤ 0.2, and weights were derived using only their responses.

4.2.2. Analyzing Pairwise Comparison Results through AHP and Fuzzy AHP

AHP analysis: Table 6 shows the weights and rankings obtained through the AHP. The outcome for the first tier showed that meteorological disasters, land use/geographic characteristics, hydraulic characteristics of streams, and experience/damage restoration could be considered important factors that affect the resilience of a stream ecosystem to cope with extreme weather phenomena, as they ranked high, having weights > 0.1 points. Additionally, meteorological phenomena, biological factor, and infrastructure were found to have high relative importance, as compared to economy/system, population, and topography.
Considering the global weights, aquatic ecology demonstrated the highest importance, followed by potentially vulnerable areas, population, topography, heat waves, and floods, which demonstrated high importance in that order; land use, heavy rain, and runoff demonstrated an upper-middle rank. Policy capacity, hydraulic characteristics, and restoration capacity were ranked together, while vulnerable areas, water quality assessment, water supply facilities, and wastewater were ranked in the middle. Extent of damage, precipitation, economy, and groundwater demonstrated a similar low importance, followed by road/industry and industrial facilities; evacuation demonstrated the lowest importance.
Fuzzy AHP analysis: Table 7 shows the weights and ranking of indicators obtained through the fuzzy AHP. Meteorological disasters, hydraulic characteristics of streams, and land use/geographic characteristics ranked the highest in the first tier and were considered important factors affecting the goal of the hierarchical structure. Meteorological phenomena, experience/damage restoration, biological factors, and infrastructure ranked in the middle; however, water quality, population, topography, and economy/system had low weights.
Considering global weights, aquatic ecology was the most important, followed by population, topography, and potentially vulnerable areas. Heat waves, water quality assessment, runoff, policy capacity, land use, and hydraulic characteristics demonstrated an upper-middle rank, while floods, heavy rain, wastewater, restoration capacity, water supply facilities, and vulnerable areas ranked in the middle. Economy, typhoons, precipitation, road/industry, extent of damage, droughts, groundwater, and evacuation ranked lower, while industrial infrastructure had the lowest weight.

4.3. Comparison between the AHP and Fuzzy AHP Results

The AHP and fuzzy AHP methods were used to analyze the relative importance of the indicators in the proposed multicriteria inventory classification to make stream ecosystems more resilient. On comparing the outcomes of the two weighting analysis methods for the first tier, the ranks were not significantly different, except for those of the meteorological disasters, biological factors, and infrastructure. The ranks of hydraulic characteristics of streams, land use/geographic characteristics, meteorological phenomena, and experience/damage restoration changed using the fuzzy AHP. Economy/system had the lowest weight; consequently, the ranking of water quality, population, and topography rose by one step. Considering the weights, those of the top ranking indicators using the AHP (meteorological disasters, land use/geographic characteristics, hydraulic characteristics of streams and experience/damage restoration), were slightly lower than in the fuzzy AHP, whereas those of the other divisions were higher. The ranks of indicators in the second tier using the AHP and fuzzy AHP were the same, except for those of meteorological phenomena and infrastructure. Using the fuzzy AHP, typhoons under meteorological phenomena ranked higher than precipitation, while evacuation under infrastructure ranked higher than industrial facilities. Overall, all the rankings were different, except for those of aquatic ecology, heat waves, water supply facilities, and precipitation.
Using the AHP and fuzzy AHP, the weights and ranks of indicators slightly varied, but the indicator groups with high and low weights remained the same. In the first tier, the groups that demonstrated high weights included meteorological disasters, land use/geographic characteristics, hydraulic characteristics of streams, experience/damage restoration, meteorological phenomena, biological factors, and infrastructure, while the groups that demonstrated low weights included water quality and population, followed by topography. In addition, in the overall result, the groups demonstrating high weights included aquatic ecology, potentially vulnerable areas, population, topography, and heat waves, while the group demonstrating low weights included typhoons, precipitation, road/industry, extent of damage, droughts, groundwater, evacuation, and industrial facilities.

5. Discussion

To improve the resilience of stream ecosystems, a multicriteria inventory classification was proposed considering green infrastructure, and weights and ranks of indicators were derived using the AHP and fuzzy AHP. The differences in the outcomes through the AHP and fuzzy AHP could have arisen because of the variations in the analysis of the decisions of respondents. The AHP expressed the opinions of respondents as single numbers, which significantly affected the overall result, whereas fuzzy AHP displayed the opinions of respondents as triangular fuzzy numbers, taking into account the ambiguity of languages and uncertainties arising through human judgment [28,31,78]. Although the ranks and weights of indicators using the AHP and fuzzy AHP varied due to the above reasons, the high-weighted and low-weighted indicator groups remained the same for the first tier and the final overall result. Considering the same group, it is possible to know the indicators that could be first considered when planning green infrastructure to enhance the resilience of stream ecosystems in response to climate change.
Meteorological disasters, hydraulic characteristics of streams, land use/geographic characteristics, meteorological phenomena, and experience/damage restoration showed high weights in both the AHP and fuzzy AHP results for the first tier. In particular, meteorological disasters ranked first in both the methods. Meteorological disasters appear to have been attributed higher weights than other evaluation indicators in previous studies that developed climate change vulnerability assessments; they are indicators that expose social–ecological systems to disturbances, suggesting that they greatly influence the resilience of stream ecosystems [2,19,21,65,84]. The high ranking of land use/geographic characteristics could be attributed to the fact that land use affects stream ecologies and water quality [15], while the geographic characteristics of potentially vulnerable areas directly affects the scale of damage due to disasters [85]; moreover, high weights were attributed to these factors in previous studies [22,84]. Extreme meteorological phenomena cause fluctuations in the hydraulic characteristics of streams [86], in turn causing secondary damage, such as affecting the ecological function of streams [87]. A high weight seems to have been attributed to the hydraulic characteristics of streams because of the effect of basin management and maintenance plans of streams [88]. Considering the division for experience/damage restoration, it was found that the factors related to restoration could be considered important because the weight of the relevant division was high, as an indicator added in regard to resilience, unlike in previous studies. Other divisions (infrastructure, water quality, population, and economy/system) that were relatively low-weighted were indirectly influenced by the high-weighted ones; prior studies demonstrated that the evaluation indicators belonging to these divisions were attributed low weights [19,21,84,89].
Considering global weights, aquatic ecology, potentially vulnerable areas, population, and topography ranked high; aquatic ecology ranked the highest in both the AHP and fuzzy AHP results. The aquatic ecology subdivision includes the evaluation indicators of aquatic health index and lateral/vertical continuity of streams. The aquatic health index has been investigated and evaluated by the Ministry of Environment, considering the trophic diatom index (TDI), benthic macroinvertebrates (BMI), fish assessment index (FAI), habitat and riparian index (HRI), radar vegetation index (RVI), and water quality in streams across Korea for over 10 years. These aquatic health indices continue to be important as they help identify the changing trends in aquatic ecosystems [68]. The lateral/vertical continuity of streams contributes to the movement of energy and materials in a linear stream ecosystem; it is an important factor for the long-term survival of fish and other aquatic life by connecting various habitats [90]. Therefore, it is considered that the aquatic health index and lateral/vertical continuity of a stream indicates the biological and structural characteristics of the stream ecosystem, respectively, which are considered to have high weights.
Population and topography showed the lowest ranking in the first tier; however, they ranked high as global weights. This is because the global weight was calculated with the weight of the second tier set to one, as there was no comparable subdivision for either population or topography in the pairwise comparison of the second tier due to the limitation of the hierarchical structure for the pairwise comparisons. Therefore, in future studies, when planning green infrastructure for resilient stream ecosystems by selecting an actual study site, it is necessary to verify the effects of the weight of the selected evaluation indicators on the overall results and to continue supplementing the weights with the actual applied results.

6. Conclusions

In this study, a multicriteria inventory classification was established to help plan green infrastructure for resilient stream ecosystems to cope with extreme weather phenomena caused by climate change. First, based on the evaluation indicator classification criteria, the evaluation indicators of previous studies that helped conduct assessments of vulnerability toward climate change and the indices that affect stream ecosystems were classified. The classified evaluation indicators were confirmed through reliability and validity tests. Finally, the multicriteria inventory classification was presented considering the physio-environmental and socio-economic sectors. In addition, based on this suggested classification, a survey was conducted with experts to derive the weights and ranks of the various indicators. By analyzing the weights and ranks of the indicators considered in the multicriteria inventory classification, the indicators that could be first considered when planning green infrastructure of a stream ecosystem were grouped.
The comparison between the weights and ranks of the indicators derived using the AHP and fuzzy AHP showed slight variations in their weights and rankings in the first tier. Meteorological disasters, land use/geographic characteristics, hydraulic characteristics of streams, experience/damage restoration, meteorological phenomena, biological factors, and infrastructure received high weights, while water quality, population, economy/system, and topography received relatively low weights. There was also a difference in the indicator weights in the second tier, but their ranks were the same except for those of typhoons and precipitation under the weather phenomena and for evacuation and industrial facilities under infrastructure. In the final overall result, aquatic ecology, population, topography, potentially vulnerable areas, and heat waves received high weights, while typhoons, precipitation, road/industry, extent of damage, droughts, groundwater, evacuation, and industrial facilities received low weights. Although the weights and ranks of indicators varied using the AHP and fuzzy AHP, indicator groups with high and low weights were the same; it was also possible to identify indicators of the social–ecological system that could be mainly considered when planning green infrastructure for resilient stream ecosystems.
By helping derive weights and ranks of indicators, the study could be valuable in suggesting data that could be referred to when establishing countermeasures and distributing budgets for ensuring the resilience of stream ecosystems against climate change. Moreover, unlike existing studies which deal with vulnerability assessments and resilience in the face of extreme weather phenomena, this study benefits from the distinction of combining vulnerability assessment and green infrastructure planning so as to improve the resilience of stream ecosystems. However, a limitation exists in determining the weights and ranks of indicators in the presented multicriteria inventory classification. This is because the weights of indicators in the first tier were used as they were for the indicators having only one subdivision in the second tier, when global weights were assigned; consequently, in the final overall result, the subdivisions that ranked lower then ranked high. Therefore, it is necessary to help clarify the weights and ranks of indicators considered in this classification through additional research on weights. In future research, based on the multicriteria inventory classification suggested here, it is necessary to evaluate the resilience of an actual stream ecosystem and to select regions with weak resilience to help analyze spatiotemporal changes in a stream ecosystem due to extreme weather phenomena. In addition, if the selected vulnerable area is categorized and a response strategy for each category is established through a relationship analysis between the factors of the multicriteria inventory classification, and if a practical method to improve resilience is established by comprehensively analyzing the response strategies for each basin, scale unit, and meteorological phenomena, the resilience of stream ecosystems could be further improved.

Author Contributions

Study design conceptualization, Y.P. and J.L.; writing—original draft preparation, Y.P.; writing—review and editing, S.L. and J.L. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2019R1F1A1063823; 2018R1D1A1B07050805). This paper was also supported by the KU Research Professor Program of Konkuk University.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1. Multicriteria inventory classification of evaluation indicators considering the physio-environmental sector and the outcome of the factor and reliability analyses.
Table A1. Multicriteria inventory classification of evaluation indicators considering the physio-environmental sector and the outcome of the factor and reliability analyses.
DivisionSubdivisionEvaluation IndicatorFactor LoadingCommunalitiesExplained Variances Cronbach s   α Prior Studies
Water qualityWater quality assessmentHydrogen ion concentration (pH)0.7610.84179.8610.831[68,91]
Biological oxygen demand (BOD)0.7380.923[67,68,92]
Chemical oxygen demand (COD)0.6200.857[67,68,92]
Total organic carbon (TOC)0.8400.894[68]
Suspended solids (SS)0.8670.776[68]
Dissolved oxygen concentration (DO)0.6560.808[68]
Total phosphorus (T-P)0.6570.841[67,68,89,92]
Count of coliform group (MPN1/100 mL)0.6520.503[68]
WastewaterWhether certain water pollutants are released 0.750[64]
Water pollution prevention facility capacity0.9520.933[65]
Nonpoint pollution reduction facility capacity0.5750.659[64,92]
TopographyTopographyRiverside width0.8770.76976.8930.696[68]
Transverse structures0.8770.769[68]
Biological factorsAquatic ecologyAquatic health index0.9340.87287.2190.782[67,68]
Stream lateral/vertical continuity0.9340.872[67,68]
Hydraulic characteristics of streamsHydraulic characteristicsStream density of small watersheds0.8940.80876.7830.745[93]
Average river temperature0.6110.561[91]
Flow rate0.8760.914[92]
Flux0.8790.785[92]
RunoffHundred-year frequency of daily direct runoff0.7180.789[84]
Daily maximum runoff0.6280.600[61,94]
Annual maximum runoff0.9280.884[58,62,95]
GroundwaterGroundwater level0.8690.801[59,83,96]
Meteorological phenomenaHeavy rain“Heavy rain” warning level rainfall0.7420.56374.2100.758[66,94]
TyphoonsFrequency of typhoon occurrence0.8570.826[66]
PrecipitationAnnual precipitation0.8830.790[45,62,93,95,96,97]
Average daily precipitation over 80 mm0.8210.834[13,45,95]
Hundred-year frequency of precipitation0.6800.832[84,94]
Maximum precipitation per day0.5610.524[2,13,21,45,61,62,64,65,95,98,99,100]
Average number of consecutive days without precipitation0.8780.785[64]
Evaporation rate0.8620.783[96]
Meteorological disastersFloodsHundred-year flood volume0.7670.58974.2000.725[66]
DroughtsNumber of days under SPI2-1 for three months a year0.9540.711[92]
Number of days under EDI3-1 for three months a year0.9270.413[92]
Heat wavesNumber of days over the highest heat index of 32 °C0.7130.816[25,65]
Average heat index of June–September0.6420.821[25]
Number of tropical nights0.9000.906[2,13,24,25,64,65,99,101]
Number of days under a heat wave0.8720.937[2,13,24,25,65,97,99,101,102]
1 Most Probable Number; 2 Standardized Precipitation Index; 3 Effective Drought Index.

Appendix B

Table A2. Multicriteria inventory classification of evaluation indicators considering the socio-economic sector and the outcome of the factor and reliability analysis.
Table A2. Multicriteria inventory classification of evaluation indicators considering the socio-economic sector and the outcome of the factor and reliability analysis.
DivisionSubdivisionEvaluated IndicatorFactor LoadingCommunalitiesExplained Variances Cronbach s   α Prior Studies
PopulationPopulationPopulation of small watersheds----[2,13,45,47,68,93,96]
Land use/
Geographic characteristics
Land useRatio of urbanization promotion area to the total area0.6690.63572.2170.745[68,103]
Ratio of agricultural area to the total area0.7370.780[2,45,47,68,72,93,96]
Ratio of forest area to the total area0.8760.790[2,45,68,95,100]
Ratio of grassland area to the total area0.8830.881[68]
Ratio of wetland area to the total area0.7690.707[68]
Ratio of bare land area to the total area0.9010.820[68]
Ratio of area under water to the total area0.6010.459[25,68]
Potentially vulnerable areasRatio of occupancy area of the stream site to the total area0.5320.603[13,21,72,103]
Ratio of area below the hundred-year flood-level to the total area0.6900.724[84]
Lowland area under 10 m0.7150.697[21,93,99]
Distance to the stream and residential areas0.9100.881[64,100,104]
Proximity to the inundated area0.8910.795[83]
Percentage of inundated area0.7580.616[58,65,67,100,105]
Economy/SystemEconomyGross regional domestic product (GRDP)0.8310.84475.0730.745[13,21,23,41,65,90,92,93,98,99,103]
Financial independence rate0.9060.821[13,21,23,41,84,93,94,98,99,101,103]
Policy capacityEnvironmental/disaster/safety management and firefighting officials0.7280.645[21,59,98,103]
Cost of stream restoration0.6840.606[67]
Disaster management capacity of local government0.9000.837[84,103]
InfrastructureRoad/industryRoad density0.8520.85183.6920.929[2,13,21,25,58,65,93,94,95,98,103]
Rate of industrialization0.8560.817[13,72,83,89,94,101]
EvacuationNumber of shelters0.7080.927[83]
Information on evacuation routes0.7690.774[83]
Industrial facilitiesArea of water supply facilities0.7600.759[65,72]
Area of electricity supply facilities0.7360.834[65,72]
Area of gas supply facilities0.8930.922[65,72]
Area of heat supply facilities0.8570.894[65,72]
Area of oil storage and oil supply facilities0.7690.698[65,72]
Water supply facilitiesCapacity of flood-control dam and reservoir0.9160.857[21,58,61,83,94]
Capacity of domestic drainage facilities0.7380.900[21,58]
Capacity of rainwater pumping station0.8320.853[58,94,105]
Capacity of storm water storage tank0.6340.935[94,105]
Whether to secure emergency drainage facilities0.6080.841[64]
Sewage distribution rate0.6320.806[2,13,62,84,89,94,101,103]
Water supply distribution rate0.8240.804[62,93,95,101,103]
Water demand per unit area0.7190.884[62,95]
Groundwater availability0.6030.710[62,96,106]
Experience/Damage restorationExtent of damageProperty damage0.5720.61477.8850.875[21,58,94,105]
Total number of victims0.7840.848[21,99,105]
Number of casualties0.7940.883[21,99,103]
Vulnerable areasRatio of flooding area to the total area0.8170.704[58,72]
Ratio of area prone to floods to the total area0.9200.855[99,100,101,103,104,105]
Area habitually under drought0.8060.667[101]
Restoration capacityDamage recovery cost0.8720.783[105]
Number of disaster recovery projects0.9060.837[84]
Number of psychological recovery support projects for disaster victims0.9050.818[107]

References

  1. World Meteorological Organization (WMO). The Global Climate in 2015–2019. Available online: https://library.wmo.int/doc_num.php?explnum_id=9936 (accessed on 28 October 2020).
  2. Lee, K.H.; Chung, E.G.; Kim, K.H.; Yu, J.A.; Lee, E.J. Vulnerability assessment of water quality and aquatic ecosystem to climate change in Korea using proxy variables. J. Korean Soc. Water Environ. 2012, 28, 444–452. [Google Scholar]
  3. Lee, J.K.; Kim, Y.O.; Kang, N.E. Analysis of Climate Change Researches Related to Water. Clim. Chang. Res. 2012, 3, 71–88. [Google Scholar]
  4. Trimmel, H.; Weihs, P.; Leidinger, D.; Formayer, H.; Kalny, G. Can riparian vegetation shade mitigate the expected rise in stream temperatures during heat waves in a pre-alpine river? Hydrol. Earth Syst. Sci. Discuss. 2018, 22, 437–461. [Google Scholar] [CrossRef] [Green Version]
  5. Kundzewicz, Z.W.; Kanae, S.; Seneviratne, S.I.; Handmer, J.; Nicholls, N.; Peduzzi, P.; Mechler, R.; Bouwer, L.M.; Arnell, N.; Mach, K.; et al. Le risque d’inondation et les perspectives de changement climatique mondial et régional. Hydrol. Sci. J. 2014, 59, 1–28. [Google Scholar] [CrossRef] [Green Version]
  6. Holling, C.S. Resilience and stability of ecological systems. Rev. Lit. Arts Am. 1973, 4, 1–23. [Google Scholar] [CrossRef] [Green Version]
  7. Folke, C. Resilience: The emergence of a perspective for social-ecological systems analyses. Glob. Environ. Chang. 2006, 16, 253–267. [Google Scholar] [CrossRef]
  8. Randhir, T.O. Resilience of Watershed Systems to Climate Change. J. Earth Sci. Clim. Chang. 2014, 5, e109. [Google Scholar] [CrossRef] [Green Version]
  9. Artmann, M.; Bastian, O.; Grunewald, K. Using the concepts of green infrastructure and ecosystem services to specify leitbilder for compact and green cities-The example of the landscape plan of Dresden (Germany). Sustainability 2017, 9, 198. [Google Scholar] [CrossRef] [Green Version]
  10. Reisinger, A.J.; Woytowitz, E.; Majcher, E.; Rosi, E.J.; Belt, K.T.; Duncan, J.M.; Kaushal, S.S.; Groffman, P.M. Changes in long-term water quality of Baltimore streams are associated with both gray and green infrastructure. Limnol. Oceanogr. 2019, 64, S60–S76. [Google Scholar] [CrossRef] [Green Version]
  11. Keeley, M.; Koburger, A.; Dolowitz, D.P.; Medearis, D.; Nickel, D.; Shuster, W. Perspectives on the use of green infrastructure for stormwater management in cleveland and milwaukee. Environ. Manag. 2013, 51, 1093–1108. [Google Scholar] [CrossRef] [PubMed]
  12. Flynn, C.D.; Davidson, C.I. Adapting the Social-Ecological System Framework for Urban Stormwater Management: The Case of Green Infrastructure Adoption. Available online: https://www.ecologyandsociety.org/vol21/iss4/art19/ (accessed on 29 October 2020).
  13. Yoo, G.Y.; Kim, I.E. Development and Application of a Climate Change Vulnerability Index; Korea Environment Institute: Seoul, Korea, 2008; ISBN 9788984643215. [Google Scholar]
  14. Jeon, E.Y.; Byun, B.S. A study on the development and application of community resilience evaluation indicators for Responding to climate change. Geogr. J. Korea 2017, 51, 47–58. [Google Scholar]
  15. Hwang, S.A.; Hwang, S.J.; Park, S.R.; Lee, S.W. Examining the relationships between watershed urban land use and stream water quality using linear and generalized additive models. Water 2016, 8, 155. [Google Scholar] [CrossRef] [Green Version]
  16. Kim, J.A.; Lee, S.W.; Hwang, G.S.; Hwang, S.J.; Kim, C.G.; An, K.J. Effects of streamline complexity on the relationships between urban land use and ecological communities in streams. Paddy Water Environ. 2016, 14, 299–312. [Google Scholar] [CrossRef]
  17. Vugteveen, P.; Rouwette, E.; Stouten, H.; van Katwijk, M.M.; Hanssen, L. Developing social-ecological system indicators using group model building. Ocean Coast. Manag. 2015, 109, 29–39. [Google Scholar] [CrossRef] [Green Version]
  18. Beckingham, B.; Callahan, T.; Vulava, V. Stormwater ponds in the southeastern, U.S. coastal plain: Hydrogeology, contaminant fate, and the need for a social-ecological framework. Front. Environ. Sci. 2019, 7, 1–14. [Google Scholar] [CrossRef] [Green Version]
  19. Park, T.S.; Kim, K.M.; Yoon, Y.S.; Lee, S.B. Analysis of Flood Damage Characteristics and Development of Flood Damage Index; Korea research Institute for human settlements: Daejeon, Korea, 2005. [Google Scholar]
  20. Kotzee, I.; Reyers, B. Piloting a social-ecological index for measuring flood resilience: A composite index approach. Ecol. Indic. 2016, 60, 45–53. [Google Scholar] [CrossRef]
  21. Kim, Y.K.; Yoo, J.A.; Chong, E.S. Water management vulnerability assessment considering. J. Clim. Chang. Res. 2012, 3, 1–12. [Google Scholar]
  22. Koh, J.K.; Kim, H.S. A study on local vulnerability assessment to climate change–The case of municipalities of Gyeonggi-Do. J. Environ. Policy Adm. 2010, 18, 79–105. [Google Scholar]
  23. Lee, B.D.; Kim, B.T.; Cho, Y.S. A study on vulnerability assessment to climate change in regional fisheries of Korea. J. Fish. Bus. Adm. 2011, 42, 57–70. [Google Scholar]
  24. Koo, Y.S.; Kim, J.E.; Kim, J.S.; Lee, S.H. Study on the improvement of adaptation ability by vulnerability analysis of heat wave-The case of Busan metropolitan city. J. Korean Reg. Dev. Assoc. 2015, 27, 331–348. [Google Scholar]
  25. Kim, K.; Eum, J.H. Policies for improving thermal environment using vulnerability assessment–A case study of Daegu, Korea. J. Korea Assoc. Geogr. Inf. Stud. 2018, 21, 1–23. [Google Scholar] [CrossRef]
  26. Choi, E.H.; Bae, S.S.; Jee, H.K. Prioritization for water storage increase of Agricultural reservoir using FAHP method. J. Korea Water Resour. Assoc. 2013, 46, 171–182. [Google Scholar] [CrossRef] [Green Version]
  27. Yang, X.L.; Ding, J.H.; Hou, H. Application of a triangular fuzzy AHP approach for flood risk evaluation and response measures analysis. Nat. Hazards 2013, 68, 657–674. [Google Scholar] [CrossRef]
  28. Shin, J.Y.; Park, Y.J.; Kim, T.W. Evaluation of Inland Inundation Risk in Urban Area using Fuzzy AHP. J. Korea Water Resour. Assoc. 2014, 47, 789–799. [Google Scholar] [CrossRef] [Green Version]
  29. Kabir, G.; Ahsan Akhtar Hasin, M. Comparative analysis of ahp and fuzzy ahp models for multicriteria inventory classification. Int. J. Fuzzy Log. Syst. 2011, 1, 1–16. [Google Scholar]
  30. Ishizaka, A. Comparison of fuzzy logic, AHP, FAHP and hybrid fuzzy AHP for new supplier selection and its performance analysis. Int. J. Integr. Supply Manag. 2014, 9, 1–22. [Google Scholar] [CrossRef] [Green Version]
  31. Mosadeghi, R.; Warnken, J.; Tomlinson, R.; Mirfenderesk, H. Comparison of Fuzzy-AHP and AHP in a spatial multi-criteria decision making model for urban land-use planning. Comput. Environ. Urban Syst. 2015, 49, 54–65. [Google Scholar] [CrossRef] [Green Version]
  32. Plebankiewicz, E.; Kubek, D. Multicriteria selection of the building material supplier using AHP and fuzzy AHP. J. Constr. Eng. Manag. 2016, 142, 04015057. [Google Scholar] [CrossRef]
  33. Özdağoğlu, A.; Özdağoğlu, G. Comparison of AHP and Fuzzy AHP for the multi-criteria decision making processes with linguistic evaluations. İstanbul Ticaret Üniversitesi Fen Bilim. Derg. 2007, 6, 65–85. [Google Scholar]
  34. Sander, H.A.; Polasky, S. The value of views and open space: Estimates from a hedonic pricing model for Ramsey County, Minnesota, USA. Land Use Policy 2009, 26, 837–845. [Google Scholar] [CrossRef]
  35. Scott Shafer, C.; Scott, D.; Baker, J.; Winemiller, K. Recreation and amenity values of Urban Stream corridors: Implications for green infrastructure. J. Urban Des. 2013, 18, 478–493. [Google Scholar] [CrossRef]
  36. United States Environmental Protection Agency (USEPA). Reducing Stormwater Costs through Low Impact Development (LID) Strategies and Practices. Available online: https://www.epa.gov/sites/production/files/2015-10/documents/2008_01_02_nps_lid_costs07uments_reducingstormwatercosts-2.pdf (accessed on 29 October 2020).
  37. Liu, Y.; Theller, L.O.; Pijanowski, B.C.; Engel, B.A. Optimal selection and placement of green infrastructure to reduce impacts of land use change and climate change on hydrology and water quality: An application to the Trail Creek Watershed, Indiana. Sci. Total Environ. 2016, 553, 149–163. [Google Scholar] [CrossRef]
  38. Han, M.Y.; Kim, J.K.; Park, S.C. The effect of greening roof on the quantity and quality of rainfall runoff. J. Archit. Inst. Korea Plan. Des. 2003, 19, 279–285. [Google Scholar]
  39. Mei, C.; Liu, J.; Wang, H.; Yang, Z.; Ding, X.; Shao, W. Integrated assessments of green infrastructure for flood mitigation to support robust decision-making for sponge city construction in an urbanized watershed. Sci. Total Environ. 2018, 639, 1394–1407. [Google Scholar] [CrossRef] [PubMed]
  40. Lee, H.M.; You, S.J.; Park, S.; Chon, J.H. A study on categories of green infrastructure to enhance urban resilience. J. Korea Plan. Assoc. 2018, 53, 215–235. [Google Scholar] [CrossRef]
  41. Park, S.M.; Lee, W.K.; Kwon, T.H.; Lee, B.D.; Son, Y.H.; Cho, Y.S. Vulnerabilitiy assessment for ocean to climate change using Spatial information based on GIS. Spat. Inf. Res. 2011, 19, 1–9. [Google Scholar]
  42. McClanahan, T.R.; Donner, S.D.; Maynard, J.A.; MacNeil, M.A.; Graham, N.A.J.; Maina, J.; Baker, A.C.; Beger, M.; Campbell, S.J.; Darling, E.S.; et al. Prioritizing key resilience indicators to support coral Reef management in a Changing climate. PLoS ONE 2012, 7, 1–7. [Google Scholar] [CrossRef]
  43. Davidson, J.L.; van Putten, I.E.; Leith, P.; Nursey-Bray, M.; Madin, E.M.; Holbrook, N.J. Toward Operationalizing Resilience Concepts in Australian Marine Sectors Coping with Climate Change. Available online: https://www.ecologyandsociety.org/vol18/iss3/art4/ (accessed on 29 October 2020).
  44. Moore, S.E.; Huntington, H.P. Arctic marine mammals and climate change: Impacts and resilience. Ecol. Appl. 2008, 18, S157–S165. [Google Scholar] [CrossRef] [Green Version]
  45. Kim, S.J.; Kim, S.M.; Kim, S.M. A study on the vulnerability assessment for agricultural infrastructure using principal component analysis. J. Korean Soc. Agric. Eng. 2013, 55, 31–38. [Google Scholar] [CrossRef]
  46. Douxchamps, S.; Debevec, L.; Giordano, M.; Barron, J. Monitoring and evaluation of climate resilience for agricultural development–A review of currently available tools. World Dev. Perspect. 2017, 5, 10–23. [Google Scholar] [CrossRef]
  47. Kim, M.H.; Bang, H.S.; Na, Y.E.; Kim, M.R.; Oh, Y.J.; Kang, K.K.; Cho, K.J. Vulnerability assessment of rice production by main disease and pest of rice plant to climate change. J. Korea Soc. Environ. Restor. Technol. 2013, 16, 147–157. [Google Scholar] [CrossRef] [Green Version]
  48. Ifejika Speranza, C. Resilient Adaptation to Climate Change in African Agriculture; German Development Institute: Born, Germany, 2010; ISBN 9783889854896. [Google Scholar]
  49. Byun, J.Y.; Lee, W.K.; Choi, S.H.; Oh, S.H.; Yoo, S.J.; Kwon, T.S.; Sung, J.H.; Woo, J.W. Vulnerability assessment for forest ecosystem to climate change based on spatio-temporal information. Korean J. Remote Sens. 2012, 28, 159–169. [Google Scholar] [CrossRef] [Green Version]
  50. Lim, S.J.; Kim, H.J.; Lee, S.H. Forest vulnerability assessment of Pine and pine mushroom using GIS. J. Agric. Life Sci. 2014, 48, 1–11. [Google Scholar] [CrossRef] [Green Version]
  51. Davoudi, S.; Shaw, K.; Haider, L.J.; Quinlan, A.E.; Peterson, G.D.; Wilkinson, C.; Fünfgeld, H.; McEvoy, D.; Porter, L. Resilience: A bridging concept or a dead end? “Reframing” resilience: Challenges for planning theory and practice interacting traps: Resilience assessment of a pasture management system in Northern Afghanistan Urban resilience: What Does it mean in planni. Plan. Theory Pract. 2012, 13, 299–333. [Google Scholar] [CrossRef] [Green Version]
  52. Eum, J.H. Vulnerability assessment to urban thermal environment for spatial planning–A case study of Seoul, Korea. J. Korean Inst. Landsc. Archit. 2016, 44, 109–120. [Google Scholar] [CrossRef]
  53. Durance, I.; Ormerod, S.J. Climate change effects on upland stream macroinvertebrates over a 25-year period. Glob. Chang. Biol. 2007, 13, 942–957. [Google Scholar] [CrossRef]
  54. Bêche, L.A.; Connors, P.G.; Resh, V.H.; Merenlender, A.M. Resilience of fishes and invertebrates to prolonged drought in two California streams. Ecography 2009, 32, 778–788. [Google Scholar] [CrossRef]
  55. Lawrence, J.E.; Lunde, K.B.; Mazor, R.D.; Bêche, L.A.; McElravy, E.P.; Resh, V.H. Long-term macroinvertebrate responses to climate change: Implications for biological assessment in mediterranean-climate streams. J. N. Am. Benthol. Soc. 2010, 29, 1424–1440. [Google Scholar] [CrossRef] [Green Version]
  56. Lee, M.H.; Jung, I.W.; Bae, D.H. Korean flood vulnerability assessment on climate change. J. Korea Water Resour. Assoc. 2011, 44, 653–666. [Google Scholar] [CrossRef] [Green Version]
  57. Muller, M. Adapting to climate change: Water management for urban resilience. Environ. Urban. 2007, 19, 99–113. [Google Scholar] [CrossRef] [Green Version]
  58. Kim, D.E.; Jung, Y.; Park, M.J.; Yoon, J.Y.; Kim, S.D.; Choi, M.H. Vulnerability analysis of water resources considering climate change. J. Wetl. Res. 2011, 13, 25–33. [Google Scholar]
  59. Park, H.S.; Kim, H.J.; Chae, Y.R.; Kim, Y.J. Assessment of water use vulnerability considering climate and socioeconomic changes in Han river watershed. J. Korean Soc. Civ. Eng. 2017, 37, 965–972. [Google Scholar]
  60. Ahn, J.H.; Hong, Y.S.; Kang, H.S.; Han, D.H.; Jang, J.H.; Seo, D.I.; Park, Y.H. Water Quality Management Strategy in the Context of Climate Change; Korea Enviornment Institute: Sejong, Korea, 2010; ISBN 9788984645257. [Google Scholar]
  61. Bae, D.H.; Lee, M.H. Flood vulnerability assessment and case analysis due to climate change. Korea Res. Inst. Hum. Settl. 2010, 20–31. [Google Scholar]
  62. Chung, J.W.; Lee, W.K.; Cui, G.S.; Lee, S.C.; Choi, S.H.; Choi, H.A. Application study of vulnerability assessment models for water resources to climate change by spatial and watershed scales. J. Clim. Chang. Res. 2010, 1, 21–30. [Google Scholar]
  63. Lee, B.W.; Chung, D.H.; Kim, B.C. Relative importance evaluation of advertising agency selection sing AHP. Korean J. Adevert. Public Relat. 2011, 13, 7–30. [Google Scholar]
  64. Seoul National University (SNU). MOE Research on Climate Change Vulnerability Survey/Evaluation Measures Such as Non-Point Pollution Reduction Facilities; SNU: Seoul, Korea, 2014. [Google Scholar]
  65. Han, W.S.; Sim, O.B.; Lee, B.J.; You, J.H. The proposal of evaluation method for local government infrastructure vulnerability relating to climate change driven flood. J. Clim. Chang. Res. 2012, 3, 25–37. [Google Scholar]
  66. Myung, S.J.; Kim, J.Y.; Shin, S.H.; An, B.O. Assessing Vulnerability to Climate Change of the Physical Infrastructure and Developing Adaptation Measures in Korea II; Korea Environment Institute: Seoul, Korea, 2010; ISBN 9788984645196. [Google Scholar]
  67. Ryu, J.N.; Hwang, S.I.; Kim, H.J.; Lee, B.K.; Lee, Y.K.; Jung, A.Y. Sustainability Assessment of Water Infrastructure and Services. Available online: https://library.kei.re.kr:444/dmme/img/001/014/007/%ec%82%ac%ec%97%85_2018_03_%eb%a5%98%ec%9e%ac%eb%82%98_%ed%99%a9%ec%83%81%ec%9d%bc.pdf (accessed on 29 October 2020).
  68. Ministry of Environment. Guide to Diagmosis of Health Deterioration in River Aquatic Ecosystem; MOE: Sejong, Korea, 2019.
  69. Saaty, T.L. A scaling method for priorities in hierarchical structures. J. Math. Psychol. 1977, 15, 234–281. [Google Scholar] [CrossRef]
  70. Srdjevic, B.; Medeiros, Y.D.P. Fuzzy AHP assessment of water management plans. Water Resour. Manag. 2008, 22, 877–894. [Google Scholar] [CrossRef]
  71. Chae, B.G.; Cho, Y.C.; Song, Y.S.; Seo, Y.S. Development of an evaluation chart for landslide susecptivility using the AHP analysis method. J. Eng. Geol. 2009, 19, 99–108. [Google Scholar]
  72. Koh, J.K.; Kim, H.S. A study on Vulnerability Assessment to Climate Change in Gyeonggi-Do; Gyeonggi Research Institute: Gyeonggi, Korea, 2009. [Google Scholar]
  73. Song, K.W.; Lee, Y. Re-scaling for improvement the consistency of the AHP method. Soical Sci. Res. Rev. 2007, 29, 271–288. [Google Scholar]
  74. Cho, S.K.; Lee, J.S. Development of the attributes and their weights for evaluation of amenity of seoul by applying fuzzy-AHP. Seoul Stud. 2006, 7, 1–16. [Google Scholar]
  75. Lee, S. Determination of priority weights under multiattribute decision-making situations: Ahp versus fuzzy ahp. J. Constr. Eng. Manag. 2015, 141, 1–9. [Google Scholar] [CrossRef]
  76. Kutlu, A.C.; Ekmekçioǧlu, M. Fuzzy failure modes and effects analysis by using fuzzy TOPSIS-based fuzzy AHP. Expert Syst. Appl. 2012, 39, 61–67. [Google Scholar] [CrossRef]
  77. Van Laarhoven, J.M.P.; Pedrtycz, W. A fuzzy extension of Saaty’s priority theory. Fuzzy Sets Syst. 1983, 11, 229–241. [Google Scholar] [CrossRef]
  78. Mo, S.W.; Kim, C.B. A Relative Importance Evaulation of the industrial sector according to the FTA using AHP and fuzzy AHP. J. Ind. Econ. Bus. 2012, 25, 1827–1842. [Google Scholar]
  79. Daǧdeviren, M.; Yüksel, I. Developing a fuzzy analytic hierarchy process (AHP) model for behavior-based safety management. Inf. Sci. 2008, 178, 1717–1733. [Google Scholar] [CrossRef]
  80. Chung, D.H.; Yang, H.C. Reliability and validity assessment in 3D video measurement. J. Broadcast Eng. 2012, 17, 49–59. [Google Scholar] [CrossRef]
  81. Young, A.; Sean, P. A beginner’s guide to factor analysis: Focusing on exploratory factory analysis. Tutor. Quant. Methods Psychol. 2013, 9, 79–94. [Google Scholar] [CrossRef]
  82. Taherdoost, H. Sampling methods in research methodology; How to choose a sampling technique for research. Int. J. Acad. Res. Manag. 2016, 5, 18–27. [Google Scholar] [CrossRef]
  83. Balica, S.F.; Douben, N.; Wright, N.G. Flood vulnerability indices at varying spatial scales. Water Sci. Technol. 2009, 60, 2571–2580. [Google Scholar] [CrossRef] [Green Version]
  84. Park, G.J.; Kim, H.S.; Park, M.J.; Kin, S.D.; Choi, M.H.; Yoon, J.Y. Assessment of flood vulnerability considering climate change using entropy method. J. Korean Soc. Hazard Mitig 2011, 1, 263–264. [Google Scholar]
  85. Hwang, Y.J. Application of geographic database for prediction of flood vulnerable area. J. Korean Assoc. Reg. Geogr. 2006, 12, 172–178. [Google Scholar]
  86. Jun, H.D.; Kim, S.J. Analysis of future hydrological cycle considering the impact of climate change and hydraulic structures in Geum River Basin. J. Korean Soc. Hazard Mitig. 2014, 14, 299–309. [Google Scholar] [CrossRef]
  87. Hahm, C.H.; Kim, G.H. A study of ecological flow assesment for environmental development in natural river. J. Korean Soc. Geospat. Inf. Sci. 2010, 18, 47–53. [Google Scholar]
  88. Ahn, S.R.; Lee, Y.J.; Park, G.A.; Kim, S.J. Analysis of fucture land use and climate change impact on stream discharge. J. Korean Soc. Civ. Eng. B 2008, 28, 215–224. [Google Scholar]
  89. Kim, E.S.; Yoon, K.Y.; Lee, S.H. Lake vulnerability assessment. J. Korea Acad. Coop. Soc. 2014, 15, 6877–6883. [Google Scholar] [CrossRef] [Green Version]
  90. Lee, J.W.; Chun, S.H.; Kim, K.H.; Kim, C.W. Flow analysis based on the recovery of lateral connectivity in the River. J. Wetl. Res. 2014, 16, 213–220. [Google Scholar] [CrossRef] [Green Version]
  91. Hare, J.A.; Morrison, W.E.; Nelson, M.W.; Stachura, M.M.; Teeters, E.J.; Griffis, R.B.; Alexander, M.A.; Scott, J.D.; Alade, L.; Bell, R.J.; et al. A vulnerability assessment of fish and invertebrates to climate change on the Northeast, U.S. continental shelf. PLoS ONE 2016, 11, 1–30. [Google Scholar] [CrossRef] [Green Version]
  92. Jo, Y.T.; Kim, H.C.; Kim, G.H.; Nam, K.H.; Kim, H.T. A Study on vulnerability assessment of water quality caused by drought in daegu metropolitan city. J. Korea Soc. Water Environ. 2018, 34, 223–224. [Google Scholar]
  93. Kang, J.Y.; Oh, K.S. Establishing flood vulnerability assessment indicies for climate chagne adaptation and its application: The case of the Seoul metropolitan area. J. Korean Urban Manag. Assoc. 2014, 27, 43–67. [Google Scholar]
  94. Jung, Y.; Choi, M.H. Survey-based approach for hydrological vulnerability indicators due to climate change: Case study of small-scale rivers. J. Am. Water Resour. Assoc. 2012, 48, 256–265. [Google Scholar] [CrossRef]
  95. Oh, S.H.; Lee, W.K.; Byun, J.Y.; Park, S.M.; Kwak, H.B.; Cui, G.S.; Kim, M.I.; Jung, R.S.; Nam, K.J.; Shin, D.H. A study on vulnerability assessment and prioritizing sectors to support adaptation strategy to climate change–Case study of Gangwon province. J. Clim. Chang. Res. 2012, 3, 245–257. [Google Scholar]
  96. Shin, H.J.; Kwon, H.J.; Lee, J.Y.; Lee, J.H.; Park, C.G. Vulnerability evaluation for water supply of irrigation facilities: Focusing on Dangjin-si, Yesan-gun, Cheongyang-gun, South Korea. J. Korean Soc. Agric. Eng. 2018, 60, 33–42. [Google Scholar] [CrossRef]
  97. Kim, C.C.; Kim, T.G. Evaluation on climate change vulnerability of Korea National Parks. Korean, J. Ecol. Environ. 2016, 49, 42–50. [Google Scholar] [CrossRef]
  98. Lee, S.H.; Kang, J.E.; Bae, H.J.; Yoon, D.K. Vulnerability assessment of the air pollution using entropy weights: Focused on ozone. J. Korean Assoc. Reg. Geogr. 2015, 21, 751–763. [Google Scholar]
  99. Lee, J.H. A study on linkage methodology between climate change vulnerability assessment and adapation policy. Korea Resour. Econ. Assoc. Conf. 2016, 235–266. [Google Scholar]
  100. Kang, J.E.; Lee, M.J. Assessment of flood vulnerability to climate change using fuzzy model and GIS in Seoul. J. Korean Assoc. Geogr. Inf. Stud. 2012, 15, 119–136. [Google Scholar] [CrossRef] [Green Version]
  101. Kim, S.M.; Kim, N.E. A study on vulnerability assessment to climate change–Focused on 22 municipalities of JeollaNamDo. J. Korea Assoc. Gov. 2012, 19, 99–123. [Google Scholar] [CrossRef]
  102. Yoo, S.J.; Lee, W.K.; Oh, S.H.; Byun, J.Y. Vulnerability assessment for public health to climate change using Spatio-temporal information based on GIS. J. Korea Spat. Inf. Soc. 2012, 20, 13–24. [Google Scholar] [CrossRef]
  103. Myung, S.J. Cliamte change vulnerability assessment using spatial analysis. Water Futur. 2011, 44, 57–62. [Google Scholar] [CrossRef]
  104. Adelekan, I.O. Vulnerability assessment of an urban flood in Nigeria: Abeokuta flood 2007. Nat. Hazards 2011, 56, 215–231. [Google Scholar] [CrossRef]
  105. Hwang, H.S.; Byun, B.S. Building vulnerability index on climate change: Focused on Seoul metropolitan city. J. Environ. Policy Adm. 2011, 19, 93–119. [Google Scholar]
  106. Sullivan, C.; Meigh, J. Targeting attention on local vulnerabilities using an integrated index approach: The example of the climate vulnerability index. Water Sci. Technol. 2005, 51, 69–78. [Google Scholar] [CrossRef] [PubMed]
  107. Hyun, J.H.; Kim, H.S.; Ahn, Y.J. Examination on Koreas disaster psychological support services based on 912 Gyeongju earthquake experience. J. Community Welf. 2018, 65, 169–201. [Google Scholar] [CrossRef]
Figure 1. Flow diagram of the methodology framework.
Figure 1. Flow diagram of the methodology framework.
Sustainability 12 09035 g001
Figure 2. Analytic hierarchy process (AHP) hierarchical structure.
Figure 2. Analytic hierarchy process (AHP) hierarchical structure.
Sustainability 12 09035 g002
Table 1. Comparison scale for AHP [69].
Table 1. Comparison scale for AHP [69].
ScaleDefinitionExplanation
1Equal importanceTwo criteria contribute equally to the objective
3Moderate importanceJudgment moderately favors one criterion over another
5Strong importanceJudgment strongly favors one criterion over another
7Very strong importanceOne criterion is favored very strongly over another
9Extreme importanceThere is evidence favoring one criterion that is of the highest possible order of affirmation
2, 4, 6, 8Immediate values between those of the above scaleWhen a compromise is required
ReciprocalsCompared to activity ‘b’, if any of the above numbers is assigned to element ‘a’, ‘b’ is the reciprocal of ‘a’
Table 2. Triangular scale for the fuzzy number conversion [79].
Table 2. Triangular scale for the fuzzy number conversion [79].
Fuzzy NumberLinguistic ScaleFuzzy Triangular ScaleReciprocal Fuzzy Triangular Scale
1Equal importance(1, 1, 1)(1, 1, 1)
3Moderate importance(1, 3/2, 2)(1/2, 2/3, 1)
5Strong importance(3/2, 2, 5/2)(2/5, 1/2, 2/3)
7Very strong importance(2, 5/2, 3)(1/3, 2/5, 1/2)
9Extreme importance(5/2, 3, 7/2)(2/7, 1/3, 2/5)
Table 3. Evaluation indicator classification criteria [67,83].
Table 3. Evaluation indicator classification criteria [67,83].
FieldClassificationEvaluation Indicator Classification Criteria
Stream ecosystemSocio-economicPopulation density
Population in flood prone areas
Proximity to river/stream
Past experiences of extreme weather disaster
Preparedness/awareness
Quality of water supply
Quality of energy supply
Warming system
Evacuation routes
Institutional capacity
Emergency services
Shelters
Land use
Regional GDP per capita
Infrastructure management
Dams and storage capacity
Ration of disaster-prone area
Physio-environmentalGroundwater level
Water quality
Health of aquatic organisms
Rainfall index
Recovery time after floods
Evaporation rate
Dikes/levees
Flood duration
River discharge
Flow velocity
Storm surge index
Drought index
Heat wave index
Table 4. Summarized outcome of the factor and reliability analyses considering the physio-environmental sector.
Table 4. Summarized outcome of the factor and reliability analyses considering the physio-environmental sector.
DivisionSubdivisionExplained Variances Cronbach s   a
Water qualityWater quality assessment79.8640.831
Wastewater
TopographyTopography76.8930.696
Biological factorsAquatic ecology87.2190.782
Hydraulic characteristics of streamHydraulic characteristics76.7830.745
Runoff
groundwater
Meteorological phenomenaHeavy rain74.2100.758
Typhoons
Precipitation
Meteorological disastersFloods74.2000.725
Droughts
Heat waves
Table 5. Summarized outcome of the factor and reliability analyses considering the socio-economic sector.
Table 5. Summarized outcome of the factor and reliability analyses considering the socio-economic sector.
DivisionSubdivisionExplained Variances Cronbach s   a
PopulationPopulation--
Land use/Geographic characteristicsLand use72.2170.745
Potentially vulnerable areas
Economy/SystemEconomy75.0730.745
Policy capacity
InfrastructureRoad/Industry83.6920.929
Evacuation
Industrial facilities
Water supply facilities
Experience/Damage restorationExtent of damage77.8850.875
Vulnerable areas
Restoration capacity
Table 6. Outcome of the AHP analysis.
Table 6. Outcome of the AHP analysis.
First TierSecond TierOverall Outcome
DivisionLocal WeightsRankSubdivisionLocal WeightsRankGlobal WeightsRank
Water quality0.0689Water quality assessment0.5410.03614
Wastewater0.4620.03116
Topography0.06311Topography1.00010.0634
Biological factors0.0896Aquatic ecology1.00010.0891
Hydraulic characteristics of streams0.1113Hydraulic characteristics0.37120.04111
Runoff0.40310.0459
Groundwater0.22630.02520
Meteorological phenomena0.0945Heavy rain0.48810.0468
Typhoons0.22830.02222
Precipitation0.28420.02719
Meteorological disasters0.1291Floods0.38620.056
Heat waves0.42910.0555
Droughts0.18530.02421
Population0.06710Population1.00010.0673
Land use/geographic characteristics0.1162Land use0.40720.0477
Potentially vulnerable areas0.59310.0692
Economy/system0.06988Economy0.38720.02718
Policy capacity0.61310.04310
Infrastructure0.0877Road/industry0.23020.0223
Evacuation0.16540.01425
Industrial facilities0.22830.0224
Water supply facilities0.37710.03315
Experience/damage restoration0.1074Extent of damage0.26330.02817
Vulnerable areas0.35320.03813
Restoration capacity0.38410.04112
Table 7. Outcome of the fuzzy AHP analysis.
Table 7. Outcome of the fuzzy AHP analysis.
First TierSecond TierOverall Outcome
DivisionLocal WeightsRankSubdivisionLocal WeightsRankGlobal weightsRank
Water quality0.0818Water quality assessment0.05510.0456
Wastewater0.04520.03713
Topography0.07610Topography1.00010.0763
Biological factors0.0946Aquatic ecology1.00010.0941
Hydraulic characteristics of streams0.1072Hydraulic characteristics0.39820.04210
Runoff0.41110.0447
Groundwater0.19630.02123
Meteorological phenomena0.0974Heavy rains0.39610.03912
Typhoons0.30420.0318
Precipitation0.30130.02919
Meteorological disasters0.1091Floods0.35420.03911
Heat waves0.44010.0485
Droughts0.20630.02222
Population0.0789Population1.00010.0782
Land use/geographic characteristics0.1013Land use0.42120.0429
Potentially vulnerable areas0.57910.0584
Economy/system0.07411Economy0.40420.0317
Policy capacity0.59610.0448
Infrastructure0.0887Road/industry0.31420.02820
Evacuation0.21630.01924
Industrial facilities0.06940.00625
Water supply facilities0.40010.03515
Experience/damage restoration0.0955Extent of damage0.26630.02521
Vulnerable areas0.37520.03614
Restoration capacity0.35910.03416
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Park, Y.; Lee, S.-W.; Lee, J. Comparison of Fuzzy AHP and AHP in Multicriteria Inventory Classification While Planning Green Infrastructure for Resilient Stream Ecosystems. Sustainability 2020, 12, 9035. https://doi.org/10.3390/su12219035

AMA Style

Park Y, Lee S-W, Lee J. Comparison of Fuzzy AHP and AHP in Multicriteria Inventory Classification While Planning Green Infrastructure for Resilient Stream Ecosystems. Sustainability. 2020; 12(21):9035. https://doi.org/10.3390/su12219035

Chicago/Turabian Style

Park, Yujin, Sang-Woo Lee, and Junga Lee. 2020. "Comparison of Fuzzy AHP and AHP in Multicriteria Inventory Classification While Planning Green Infrastructure for Resilient Stream Ecosystems" Sustainability 12, no. 21: 9035. https://doi.org/10.3390/su12219035

APA Style

Park, Y., Lee, S. -W., & Lee, J. (2020). Comparison of Fuzzy AHP and AHP in Multicriteria Inventory Classification While Planning Green Infrastructure for Resilient Stream Ecosystems. Sustainability, 12(21), 9035. https://doi.org/10.3390/su12219035

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop