Next Article in Journal
The Potential of Fibroblast Transdifferentiation to Neuron Using Hydrogels
Previous Article in Journal
Achieving a Sustainable Development Process by Deployment of Solar PV Power in ASEAN: A SWOT Analysis
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

An Integrated Multicriteria Decision-Making Approach for Collection Modes Selection in Remanufacturing Reverse Logistics

1
School of Automobile and Traffic Engineering, Wuhan University of Science and Technology, Wuhan 430065, China
2
School of Computing, Engineering & Maths, University of Brighton, Brighton BN2 4GJ, UK
*
Author to whom correspondence should be addressed.
Processes 2021, 9(4), 631; https://doi.org/10.3390/pr9040631
Submission received: 18 March 2021 / Revised: 1 April 2021 / Accepted: 2 April 2021 / Published: 4 April 2021
(This article belongs to the Section Environmental and Green Processes)

Abstract

:
Reverse logistics (RL) is closely related to remanufacturing and could have a profound impact on the remanufacturing industry. Different from sustainable development which is focused on economy, environment and society, circular economy (CE) puts forward more requirements on the circularity and resource efficiency of manufacturing industry. In order to select the best reverse logistics provider for remanufacturing, a multicriteria decision-making (MCDM) method considering the circular economy is proposed. In this article, a circularity dimension is included in the evaluation criteria. Then, analytic hierarchy process (AHP) is used to calculate the global weights of each criterion, which are used as the parameters in selecting RL providers. Finally, technique for order of preference by similarity to ideal solution (TOPSIS) is applied to rank reverse logistics providers with three different modes. A medium-sized engine manufacturer in China is taken as a case study to validate the applicability and effectiveness of the proposed framework.

1. Introduction

As the third source of profit next to resources and manpower [1], logistics has gained increasing attention in various industries globally. These activities include research on not only forward logistics but also reverse logistics (RL), which has been proved to have a positive impact on productivity and the natural environment [2]. The definition of RL was proposed by Rogers and Tibben-Lembke [3]: “RL is the process of planning, implementing, and controlling the efficient, cost effective flow of raw materials, in process inventory, finished goods and related information from the point of consumption to the point of origin for the purpose of recapturing value or proper disposal”. Companies concentrate more on RL practices in order to gain more profits and be more socially responsible [4]. However, the implementation of RL requires a systematic and well-thought-out strategy; otherwise, it will increase the cost of the enterprise and will not achieve the desired objectives. On the other hand, along with green design and product recovery, RL is a new core aspect of supply chain management [5]. Traditionally, third-party reverse logistics providers are only responsible for transportation tasks. This can no longer meet the demand for sustainability and circularity. An RL that can meet the needs of enterprises in both economic and environmental aspects is needed.
The manufacturing industry is one of the industries that very closely related to RL. A good balance between economic benefits and environmental impacts may be achieved through RL via recycling and remanufacturing. Remanufactured products with high added value can reduce the overall cost of products by 50–60%, and their performance indexes are as good as those of new products, thus significantly improving enterprises’ economic benefits [6]. In fact, most manufacturing enterprises cannot carry out remanufacturing activities well due to the lack of abundant capital and professional technical support, which requires significant investment and increases the economic burden of enterprises at the beginning [7]. In the current market situation, where the original equipment manufacturer (OEM) remanufacturing mode is difficult to implement due to the lack of a professional logistics system, the third-party remanufacturer (TPR) remanufacturing mode has become the mainstream mode due to its economic advantages provided by its efficient and mature logistics network [8]. A precondition of remanufacturing is the collection of waste products. Currently, there are three collection modes, depending on the operating party of the RL provider in the supply chain: third party take-back (TPT), manufacturing take-back (MT) and retailer take-back (RT) [9]. In MT mode, the manufacturer in the forward supply chain completes the process of collecting, disassembly, cleaning, renovation, testing, assembly, etc. and remanufactures the waste products into new products. In contrast, collecting is carried out by retailers/distributors in the forward supply chain in RT mode. Usually, this mode is used for collecting waste products with fewer varieties and small volumes. In TPT mode, the collection task is undertaken by a professional third-party logistics provider. Manufacturers outsource the collecting and part of remanufacturing of waste products to third-party logistics providers by paying outsourcing costs. In this article, a new kind of RL provider based on the concept of value-added service is proposed, in which the third-party logistics providers in TPT mode not only are responsible for the collection of waste products but also provide customized value-added services such as cleaning, damage degree identification and classification based on the OEMs’ needs.
Normally, several alternative providers may be short-listed, the selection criteria are important for the identification of the optimal alternative. Although the evaluation index system for the selection of RL providers is formed by considering the needs of enterprises, most researchers build a complete index system based on an assumption that each criterion is dependent on the others, and their interactions are modeled and optimized to achieve a comprehensive evaluation and selection of alternatives [10]. In recent years, many articles have been published on the topic of “sustainability”, and the circular economy, which is considered to be a subset of sustainability [11], has become a research hotspot. While sustainable development research is focused on the integration of economic and environmental aspects, the cores of circular economy are “reduction, reuse and recycling”, which have more demands on the circularity and resource efficiency of the manufacturing industry [12]. Instead of focusing on the social, economic and environmental dimensions of sustainability, a new index system is proposed in this article taking “circularity” as one of the dimensions according to the circular economy theory.
Many studies have been conducted on RL provider selection, most of which have employed a hybrid approach of fuzzy number or grey theory and analytic hierarchy process (AHP). However, many researchers are skeptical of such an overly broad approach. For example, Saaty, the founder of AHP theory, questioned the scientific and logical nature of fuzzy AHP methods because AHP itself is generated from the fuzzy theory [13], which is difficult to verify. Thus, in this article, fuzzy logic is not adapted in the proposed decision method.
The rest of the article is organized as follows: In Section 2, a literature review and summary of the RL provider research are provided. In Section 3, the framework of the proposed RL provider selection problem is introduced and the composition of the criteria system is described. The framework is applied to a case study in Section 4, and the results are discussed in Section 5. Finally, in Section 6, the conclusions and limitations of this paper are summarized and views on future research are presented.

2. Literature Review

This section is divided into two parts. The first part describes the common methods used in the existing literature for RL provider selection. The second part introduces the criteria classification theory and criteria selection in relevant research.

2.1. RL Provider Selection Methods

RL provider selection is a multicriteria decision-making (MCDM) problem in which a number of alternative providers are evaluated according to a set of criteria [14]. Generally, a hybrid MCDM method, namely two (or more) different decision-making methods combined, is used to calculate the relative weight of evaluation criteria and rank the alternatives accordingly. There is a growing number of studies on RL provider evaluation and selection owning to the increasing recognition of RL in the circular economy. Zhang et al. (2020), in a systematic review of 41 articles on this topic, pointed out that fuzzy theory is useful for decision making since it can quantify the subjective criteria indicators. AHP, as a mature method used to deal with pairwise comparison, has become the most frequently used method in calculating the relative weights between criteria [10]. There are other methods that are also frequently used to calculate weights individually or in combination with other evaluation methods in different contexts, such as analysis network process (ANP) [15,16,17,18], data envelopment analysis (DEA) [19,20,21], interpretive structural modeling (ISM) [22,23,24] and best–worst method (BWM) [25,26,27]. The next step, when ranking alternatives, is to use a variety of ranking methods. The common ones are ViseKriterijumska Optimizacija I Kompromisno Resenje (VIKOR) [26,28,29,30,31,32,33], technique for order of preference by similarity to ideal solution (TOPSIS) [22,34,35,36,37,38] and multiobjective optimization by ratio analysis (MOORA) [39,40,41]. Zarbakhshnia et al. (2020) proposed a novel hybrid MCDM approach integrating fuzzy AHP and grey MOORA and used the MOORA method as the baseline to verify the reliability of the new method [39]. Santos et al. (2019) developed a hybrid Entropy-TOPSIS-F framework to weight the criteria and select the provider with the best environmental performance [34]. Garg et al. (2018) used an integrated BWM–VIKOR framework to evaluate and select outsourcing partners for an electronics company in India [26]. Li et al. (2018) developed an integrated cumulative prospect theory (CPT)-based MCDM approach to identify sustainable third-party reverse logistics providers (3PRLPs) [42]. Zarbakhshnia et al. (2018) proposed a fuzzy stepwise weight assessment ratio analysis (SWARA) to weight the evaluation criteria, based on which the approach of fuzzy complex proportional assessment of alternatives (COPRAS) was applied to rank and select the sustainable 3PRLPs for the automotive industry [43]. Sen et al. (2017) developed a decision support framework based on the dominance measure concept integrated with grey set theory, which can determine the dominance of two alternatives according to a specific criterion [44].
According to the above review, MCDM methods adapted by the combination of several techniques are widely utilized to deal with the problem of RL provider selection. Compared with other approaches, the ease of applicability in pairwise comparisons makes AHP a more useful tool. As the two most widely used ranking methods, both VIKOR and TOPSIS are based on the principle of compromise programming. The former focuses on sorting and selecting a set of alternatives in the presence of conflicting criteria, while the latter has no strict restrictions on data distribution and indicators.

2.2. RL Provider Selection Criteria

Establishing the selection criteria is the first step in RL provider selection [10]. Various evaluation criteria have been utilized depending on applications. Widely applied criteria include sustainability, green supply chain management (GSCM), circular economy (CE) and performance dimensions, as shown in Table 1.
Most articles in the field of sustainability focus on the three pillars of sustainability, namely society, economy and environment, to determine the evaluation criteria. In articles related to GSCM, more attention is paid to the reduction of the negative environmental impact in the transportation and manufacturing process when establishing evaluation criteria. Similarly, attention is also placed on the environment in the topic of CE, but when selecting evaluation criteria, greater importance is placed on the product being environmentally friendly in design for the reuse of end-of-life products. Another commonly used criterion is performance, for which evaluation criteria are established from the perspective of examining various aspects of RL provider capabilities. Some articles do not take the above four factors as the foundation but instead consider the application-specific requirements as the criteria. For instance, Liu et al. (2019) studied the recycling and disposal of used mobile phones. Considering that noises of different frequencies and intensity are emitted during the recycling process, noise pollution was taken as one of the evaluation criteria [25].
By summarizing and analyzing the existing research results, it can be concluded that sustainability is a popular research topic, and many integrated evaluation methods have been combined with fuzzy theory. Compared with existing works, this paper mainly makes the following contributions: (1) A new index system is proposed in this article taking circularity as one of the dimensions according to the circular economy theory. (2) Because the validity of the combination of fuzzy logic and various evaluation methods has not been proven, an MCDM approach consisting of AHP and TOPSIS was applied to select the best RL provider.

3. Methodology

Govindan et al. proposed a two-stage 3PRLP selection model that utilized AHP to identify factors and ANP to select providers [18]. Kumar and Dixit [28] and Liu et al. [29] combined this two-stage model with MCDM and upgraded it to a three-stage approach. This paper applies a three-stage process created through the synthesis of Govindan [18], Kumar and Dixit [28] and Liu et al.’s [29] frameworks for the RL provider selection as shown in Figure 1. In the first stage, the selection criteria for RL providers are determined based on an extensive literature review. In the second stage, AHP is used to calculate the relative weights of criteria. In this stage, the data of rating are collected from social groups through questionnaires via the Internet. In the last stage, the scores of RL providers in all three modes are collected, and then TOPSIS is used to calculate and rank the final scores of each RL provider. The best partner is the one with the highest final score.

3.1. Evaluation Criteria of RL Provider Selection

Based on literature review and according to the Govindan’s [15] application of CE theory in supplier selection and the operation mode of reverse logistics in the remanufacturing industry proposed by Tian et al. [45], a hierarchical structure of RL provider evaluation and selection is established as shown in Figure 2. The attributes contained in the first three dimensions of economy, society and technology are mainly derived from the improvement on the basis of the framework proposed by Tian et al. [45]. Tian et al.’s criteria focus on the evaluation of manufacturing enterprises [45]. By combining his framework with relevant research in the logistics industry [18,22,28,31,38,45,46,47], the criteria that need to be considered when evaluating and selecting RL providers can be determined. The attributes of the CE cluster are determined according to the theory of CE and the processes that may affect the environment when a product is moved in the supply chain. For example, the processes of collection, transportation and dismantling of waste products will all have an impact on the environment [15,45,48,49], which necessitates the consideration of these attributes.
Operating cost (E1) is the expense paid by logistics providers to maintain their normal operation and development. It is related to the operation of enterprises and the operation of equipment, components and facilities. The operating cost composition of RL providers with different modes is different, but a sound operation can improve the efficiency of enterprises [18,45].
RL cost (E2) determines whether remanufacturing activities can bring benefits to enterprises, and it includes transportation cost, inventory cost, inspection cost, packaging cost and material handling cost. The high cost of RL is one of the main obstacles to cost-effective remanufacturing production. Reducing RL cost is a common concern to both manufacturers and logistics companies [22,45].
Source of raw materials (E3) refers to the sources and channels of collecting waste products of logistics companies. Waste products come from a variety of sources. One of the advantages of having a logistics provider for the collection of waste products is that they can benefit the existing logistics network, which makes the collection and transportation process more efficient and cost-effective, thus reducing time and capital costs.
Resource utilization (E4) refers to the ability to compare the value and cost of waste products after collection and to process them in different ways. Waste products are the essential resources of remanufacturing enterprises, and analyzing their effectiveness and utilization is an important step [45]. For RL providers, a preparation step is paramount to analyze and classify waste products and send them to different processing centers [28].
User satisfaction (E5) includes effective communication with customers and partners, service improvement and overall working relationship. For an RL provider, whether communicating with partners or customers, efficient transmission of information is crucial as it can reduce unnecessary economic or time loss and improve the enterprise’s reputation [42]. In addition, it is also necessary to pay attention to the timely upgrade and improvement of services [18].
Quality management (E6) is used to measure integrity and quality after the processing of waste products. The quality of the remanufactured products is the final criterion for the entire remanufacturing process [45]. Although the quality of the remanufactured product should be comparable to that of the new product, more attention should be paid to quality management, such as providing a clear quality level after remanufacturing [31].
Remanufacturing process technology (E7) is the key to product remanufacturing. For RL providers who are mainly responsible for the collection process, it mainly includes some remanufacturing pre-preparation technologies such as disassembly technology, cleaning technology and inspection technology [45].
Excellent RL providers can not only complete basic reverse logistics services according to the requirements of the entrusted enterprises but also provide some value-added services (E8) [38]. Such services may include testing, classification of parts of waste products, remanufacturing of parts that only need to be simply refurbished before entering the secondary market, disposal of waste that is considered not to be worthy of remanufacturing and after-sales service [46,47].
Ecological efficiency (E9) refers to the improvement of the resource transformation efficiency and environmental pollution reduction in the process of product collection through appropriate transportation and processing methods. Transportation activities are one of the main causes of ecological climate change [45]. Therefore, ecological efficiency is one of the evaluation criteria in the process of remanufacturing.
Environmental standards (E10) are used to determine whether RL providers are complying with environmental protection standards for the collection of waste products [15].
Eco-friendly raw materials (E11) refer to the utilization and reuse of recyclable materials in the product remanufacturing process. There is a trend to realize a recyclable product production process [48]. In this process, enterprises need to pay attention to what kind of raw materials are used in the manufacturing/remanufacturing process of products.
In the theory of circular economy, it is necessary to design a product with the minimum environmental degradation effect and the maximum recovery capacity while trying to reduce the cost in the manufacturing process [15]. This is the concept of eco-design (E12). For the current situation where the scale of remanufacturing activities is about to be expanded, the design is one of the main factors affecting remanufacturing [49].
Clean technology (E13) measures the ability of RL providers to process products using the appropriate technology. This criterion not only considers the process of collecting waste products but also includes the distribution processing of products, such as labeling, packaging and assembly procedures [15,50].
The framework consists of three parts: goal (B), cluster (A) and attribute (E). The goal is RL provider selection. The cluster includes economy (A1), society (A2), technology (A3) and CE (A4). The economic cluster includes attributes covering operating cost (E1) and RL cost (E2). The society cluster includes attributes of source of raw materials (E3), resource utilization (E4) and user satisfaction (E5). The technology cluster includes attributes of quality management (E6), remanufacturing process technology (E7) and value-added services (E8). The CE cluster includes attributes of ecological efficiency (E9), environmental standards (E10), eco-friendly raw materials (E11), eco-design (E12) and clean technology (E13).

3.2. Selection Methodology for RL Provider

In order to determine the best RL provider for remanufacturing, an MCDM approach consisting of AHP and TOPSIS is applied. Saaty, the author of the AHP approach, questioned the rationality of the combination of fuzzy set theory and AHP [13] and believed that fuzzifying is a kind of interference without substantially changing the judgments. Therefore, the methods in this paper do not apply fuzzy numbers, and the results of AHP are compared with those of fuzzy AHP.

3.2.1. Analytic Hierarchy Process (AHP)

The basic idea of AHP is to hierarchize the problem to be analyzed (Saaty, 1980) [51]. According to the overall goal to be achieved, the problem is decomposed into different influence factors, which are combined in different levels according to their association and relationship to form a multilevel analysis structure model; finally, the alternatives are compared and ranked. The steps of AHP are as follows (Figure 3) [28,51]:
Step 1: Build a hierarchical structure. The top level indicates the problem to be solved. The second level consists of the decision criteria. The bottom level represents the alternatives.
Step 2: Construction of judgment matrix. In this step, data are usually collected through questionnaires or scores from a panel of experts. Decision-makers are required to make pairwise comparisons between decision dimensions and criteria according to the scale in Table 2. In order to obtain a more detailed attitude difference of the respondents and a more precise rating, this paper uses a 9-point scale.
Step 3: Calculate the geometric mean of each row of the judgment matrix, and normalize it to obtain relative weights of dimensions and criteria, so as to obtain global weights and ranking of each criterion.
Step 4: Conduct the consistency test. Consistency ratio (CR) was calculated to evaluate the consistency of the comparisons. CR is the ratio of consistency index (CI) to random index (RI). CI is obtained as follows:
CI   =    ( λ max N ) ( N 1 )
where λmax is the maximum eigenvalue and N is the size of the matrix. Random index (RI) is obtained from Table 3. It is generally believed that when CR < 0.1, the inconsistency degree of the matrix is within the allowable range and indicates that the result has passed the consistency test. Otherwise, the judgment matrix should be reconstructed to improve the consistency.

3.2.2. TOPSIS

TOPSIS, an analysis method that compares and selects multiple alternatives based on multiple criteria, was firstly proposed by Hwang and Yoon (1981) [52]. Fundamental for TOPSIS is to determine the positive ideal solution and negative ideal solution of each attribute. The positive ideal solution is the optimal solution among the alternatives, and its attribute values reach the best value, while the negative ideal solution is the worst solution. After calculating the Euclidean distance between each scheme, the positive ideal solution, the distance between each scheme and the negative ideal solution, the approximate degree of each alternative to the optimal solution can be obtained, which can be used as the basis for evaluating the merits of the alternatives. The TOPSIS method has many advantages. It has no strict constraints on the data distribution and sample content. It is applicable to the analysis of small samples as well as large systems with multiple evaluation units and indexes. It is flexible and convenient to use and has universal applicability. The steps of TOPSIS are as follows (Figure 4):
Step 1: Obtain and normalize the original decision matrix.
Assuming that there are N criteria and M alternatives, the decision matrix formed by them is
X = [ x 11 x 1 n x m 1 x m n ]
To eliminate the effects of various scales of criteria, the normalized decision matrix (B) is given by
b i j = x i j i = 1 m x i j 2 , i = 1 ,   2 , , m ; j = 1 ,   2 , , n
Step 2: Obtain the weighted normalized matrix.
The weighted normalized matrix (Z) is obtained by
z i j = b i j · w j , i = 1 ,   2 , , m ; j = 1 ,   2 , , n
where w j is the weight of the criterion.
Step 3: Determine the positive ideal solution ( V + ) and the negative ideal solution ( V ) using the following equations:
V + = ( z 1 + , z 2 + , , z n + )   where   z j + =   max   { z i j } , i = 1 ,   2 , , m ; j = 1 ,   2 , , n
V = ( z 1 , z 2 , , z n )   where   z j =   min   { z i j } , i = 1 ,   2 , , m ; j = 1 ,   2 , , n
Step 4: Calculate the distance of each alternative from V + and V using Equations (7) and (8):
D + = j = 1 n ( z i j z j + ) 2 , i = 1 ,   2 , , m
D = j = 1 n ( z i j z j ) 2 , i = 1 ,   2 , , m
Step 5: Obtain the closeness coefficient (CC) of each alternative as follows:
CC i   = D i D i + + D i , i = 1 ,   2 , , m
Step 6: Rank the alternatives.
The alternatives are ranked according to the value of CC. The best alternative has the highest closeness coefficient.

4. Application in Case Study

A case study with data from a Chinese engine manufacturer was conducted to verify the proposed approach. In recent years, increasing attention has been paid to the research and development of automobile remanufacturing in China. Compared with traditional manufacturing, remanufacturing has the characteristics of energy saving and material saving, which are important in the circular economy. With the introduction of relevant policies and the increase in public awareness of environmental issues, more and more automobile enterprises are actively engaging in remanufacturing activities. By June 2020, the motor vehicle population in China has reached 360 million, among which more than 270 million are automobiles [6]. The annual maintenance and replacement parts output value is nearly 1 trillion yuan, which represents a huge potential market for remanufacturing [6]. As an automobile part with high standardization and strong versatility, mainly made of metal materials, the engine is the automotive part that is commonly remanufactured.
For engine remanufacturing, the main process is the dismantling and cleaning of faulty or obsolete engine parts. This is followed by technical processing and transformation, according to the size modification requirements. Finally, after being reassembled and tested, it will be made into a new engine [53]. Before the remanufacturing processes, the reverse logistics of waste products should be completed first, which is a process requiring systematic operation. The company’s production equipment level is above the average level of the same industry in China. At present, its products include remanufactured engines from various brands which have more than ten variations, including Steyr, Cummins and Chaochai 6102, which are used as maintenance parts for the after-sale market. This paper takes the problem of this company as an example and applies the proposed framework to select a suitable RL provider for it.

4.1. Application of AHP

First, an online AHP questionnaire was created according to the hierarchy shown in Figure 2. The questionnaire was sent to 340 respondents between September 14 and October 5, and 200 responses were collected after it was issued. Then, pairwise comparison matrices for each level were obtained according to the collected data. The weights of the four dimensions after data screening and processing are shown in Table 4 (this matrix passed the consistency test). “Technology” had the highest weight, followed by “circularity” and “society” at similar importance, and “economy” was the least important factor.
The weights of each criterion are listed in Table 5. Global weights were obtained by multiplying the relative weights between each criterion and the weights of each dimension. The CR of each matrix was less than 0.1 and could be used for evaluation and selection. Compared with “operating cost (E1)”, “RL cost (E2)” is considered to be a more important indicator. “Resource utilization (E4)” and “quality management (E6)” are considered as the most important indicators in the social dimension and the technical dimension, respectively. “Eco-friendly raw materials (E11)” is the most important criterion and “environmental standards (E10)” is the least important criterion in the circularity dimension.

4.2. Application of TOPSIS

The decision-making team composed of eight experts scored three different modes (based on Figure 2, the three different modes were TPT, MT and RT) against the evaluation criteria, as shown in Table 6. For each mode, better performance for a criterion is reflected by a higher score. The range of the score is 1 to 9. Each final score was based on the sum of the average scores of the eight experts.
After normalization (Table 7), the weight of each index was calculated to obtain the weighted normalized matrix shown in Table 8. The positive ideal solutions V + and negative ideal solutions V are listed at the bottom of the table.
The distance of each alternative was then calculated using Equations (7) and (8). Finally, Equation (9) was used to calculate the closeness coefficient, and the results are shown in Table 9. The higher the closeness coefficient, the higher the alternative is ranked. The results show that TPT is the best solution, and its CCi is 0.7565. MT has the furthest distance from the negative ideal solution, ranking the second.

5. Discussions of Findings

As seen in Table 4, the technology (A3) dimension is ranked the highest in the ranking of dimensions by a large margin. This result indicates that both the technology to be used in each specialized process of remanufacturing the engines and the general technology of processing other parts are considered as the most important criteria in the remanufacturing reverse logistics activities. As seen in Table 5, quality management (E6) has the highest weight among the three criteria under this dimension. The final product of remanufacturing is the product whose quality and performance are the same as (or better than) the new product after the waste product has been processed. Therefore, it is particularly important to manage the quality of the output through professional technology and strict quality control; otherwise, core competitiveness will be lost. The criterion of value-added services (E8) is ranked after E6. Before remanufacturing, the waste products with excessive damage should be screened out. If the RL provider can complete a series of preparatory work steps before dispatching the good-quality waste products, it will save significant time and improve efficiency for manufacturers/remanufacturers. Therefore, this is also an important criterion. In contrast, remanufacturing process technology (E7) is less important for RL providers who are mainly responsible for collection and logistics tasks.
Circularity (A4) is slightly more important than society (A2). The reason may be that the circularity dimension involves further requirements on both the technical and social dimensions, thus becoming more important to decision-makers. Eco-friendly raw materials (E11) ranked first of the five criteria. It makes sense that the primary task of implementing the concept of circular economy is to use materials that have a less negative impact on the environment and to reuse them properly. Next is clean technology (E13) and ecological efficiency (E9). In addition to being environmentally friendly in terms of raw materials, proper methods and technologies should be used in transportation and processing to minimize unnecessary environmental pollution and energy consumption. It is also of significance to promote the concept of industry sustainability and the implementation of a circular economy [54]. Finally, eco-design (E12) and environmental standards (E10) rank last, both of which are less practical and more theoretical.
The purpose of the collection of waste products and remanufacturing is to adhere to the concept of sustainable development and improve the social prestige of companies [55]. For the immature engine remanufacturing industry in China, if a motor vehicle manufacturer is forced by policy or other reasons to carry out remanufacturing activities, it essentially trades the economic benefits for the benefits of other aspects, so in comparison, the impact of the economy (A1) dimension is lower than that of the social dimension under this circumstance. The highest-ranking criterion for the society (A2) dimension is resource utilization (E4). Waste products are the resources of RL providers, and these products have varying availability. The ability to utilize these varying resources can reflect the adequacy of a company’s basic business capacity in addition to transportation. Source of raw materials (E3) and user satisfaction (E5) mainly represent the business scope and operational level of RL providers, making them less important than resource utilization (E4). Regarding the operating cost (E1) and RL cost (E2) under the economy (A1) dimension, RL cost (E2), which can better reflect the capabilities of the main business, is considered to be more important.
As seen in Table 9, the ranking of the alternatives in descending order is TPT, MT and RT. Since the first two have been applied to most cases, it is reasonable for RT to be ranked as last. MT is more suitable for large enterprises that normally have sufficient capital and can afford to invest in remanufacturing systems. For such medium-sized enterprises as the case in this paper, TPT can effectively make up for problems such as a lack of collecting channels, underdeveloped reverse logistics networks and systems with 3PRLPs’ more specialized logistics systems. RL providers that provide strong value-added service capability can also share some of the manufacturing and remanufacturing processes.

6. Conclusions

Remanufacturing can be a risky decision for a manufacturer because it requires financial and technical support and affects the company’s overall operating performance [56]. Choosing a good RL provider is particularly critical, starting with choosing the right take-back mode. This study proposes a new systematic index system and multicriteria decision-making method to select the best RL providers. First, evaluation criteria were established after an evaluation of the role of RL in manufacturing enterprises and a review of literature related to the selection of RL providers. Then, the decision method composed of AHP–TOPSIS was developed and the source data were collected in two different ways. The data used by AHP were obtained from questionnaires. The criteria weights obtained in this step were used as the input of TOPSIS, in which the environmental dimension obtained the highest weight. TOPSIS uses data provided by a panel of experts. The proposed framework was applied to a medium-sized automobile engine manufacturer in China, and the results show that TPT was the best RL provider mode.
The study presented in this paper has some limitations. The use of questionnaire data can compromise the results in certain ways; for example, the data may become irregular or too extreme because of a low level of professionalism of the respondents. This research also does not consider the uncertainty of expert scoring data and actual operation in the real situation due to the fuzzy theory not being introduced. Future research may discuss finding a new way to eliminate uncertainty. In addition, AHP and TOPSIS are methods that are widely applied due to their relatively simple calculation process, and they can easily be combined with other methods to form new approaches (such as connection-degree-based TOPSIS [57]). Future research may focus on the evaluation of various MCDM methods, including various alternative methods such as DEA, ISM, VIKOR and MOORA, to identify the optimal method that can be used to analyze the interrelationships between indicators and to rank criteria and alternatives.

Author Contributions

Conceptualization, Z.L. and X.Z.; writing—original draft preparation, Z.L.; writing—review and editing, Z.L., X.Z., Y.W. and W.Y.; supervision, X.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Natural Science Foundation of China: Energy Efficiency Integrated Optimization of CNC Machining System Driven by Multi-source and On-line Energy Consumption Data Hybrid, grant number [51975432].

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data is contained within the article.

Acknowledgments

The authors acknowledge the support and inspiration of Wuhan University of Science and Technology and the University of Brighton.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Nishizawa, O. Circulation Expenses: An Unknown Third Source of Profits; Koshinesha: Tokyo, Japan, 1970. [Google Scholar]
  2. Kaviani, M.A.; Tavana, M.; Kumar, A.; Michnik, J.; Niknam, R.; de Campos, E.A.R. An integrated framework for evaluating the barriers to successful implementation of reverse logistics in the automotive industry. J. Clean. Prod. 2020, 272, 122714. [Google Scholar] [CrossRef]
  3. Rogers, D.S.; Tibben-Lembke, R. Going Backwards: Reverse Logistics Trends and Practices; Reverse Logistics Executive Council: Reno, NV, USA, 1999. [Google Scholar]
  4. Lai, K.-h.; Wu, S.J.; Wong, C.W.Y. Did reverse logistics practices hit the triple bottom line of Chinese manufacturers? Int. J. Prod. Econ. 2013, 146, 106–117. [Google Scholar] [CrossRef]
  5. Jayaram, J.; Avittathur, B. Green supply chains: A perspective from an emerging economy. Int. J. Prod. Econ. 2015, 164, 234–244. [Google Scholar] [CrossRef]
  6. National Development and Reform Commission. Remanufacturing of Auto Parts: The “First Year” Has Arrived, and the Future Can Be Expected; National Development and Reform Commission: Beijing, China, 2020.
  7. Cao, J.; Chen, X.; Zhang, X.; Gao, Y.; Zhang, X.; Kumar, S. Overview of remanufacturing industry in China: Government policies, enterprise, and public awareness. J. Clean. Prod. 2020, 242, 118450. [Google Scholar] [CrossRef]
  8. Zhang, Y.; Chen, W.; Mi, Y. Third-party remanufacturing mode selection for competitive closed-loop supply chain based on evolutionary game theory. J. Clean. Prod. 2020, 263, 121305. [Google Scholar] [CrossRef]
  9. Wang, H.; Jiang, Z.; Zhang, H.; Wang, Y.; Yang, Y.; Li, Y. An integrated MCDM approach considering demands-matching for reverse logistics. J. Clean. Prod. 2019, 208, 199–210. [Google Scholar] [CrossRef]
  10. Zhang, X.; Li, Z.; Wang, Y. A Review of the Criteria and Methods of Reverse Logistics Provider Selection. Processes 2020, 8, 705. [Google Scholar] [CrossRef]
  11. Schöggl, J.-P.; Stumpf, L.; Baumgartner, R.J. The narrative of sustainability and circular economy—A longitudinal review of two decades of research. Resour. Conserv. Recycl. 2020, 163, 105073. [Google Scholar] [CrossRef]
  12. Pieroni, M.P.P.; McAloone, T.C.; Pigosso, D.C.A. Circular economy business model innovation: Sectorial patterns within manufacturing companies. J. Clean. Prod. 2020, 286, 124921. [Google Scholar] [CrossRef]
  13. Saaty, T.L.; Tran, L.T. On the invalidity of fuzzifying numerical judgments in the Analytic Hierarchy Process. MComM 2007, 46, 962–975. [Google Scholar] [CrossRef]
  14. Haeri, S.A.S.; Rezaei, J. A grey-based green provider selection model for uncertain environments. J. Clean. Prod. 2019, 221, 768–784. [Google Scholar] [CrossRef]
  15. Govindan, K.; Mina, H.; Esmaeili, A.; Gholami-Zanjani, S.M. An Integrated Hybrid Approach for Circular provider selection and Closed loop Supply Chain Network Design under Uncertainty. J. Clean. Prod. 2020, 242, 118317. [Google Scholar] [CrossRef]
  16. Tosarkani, B.M.; Amin, S.H. A multi-objective model to configure an electronic reverse logistics network and third party selection. J. Clean. Prod. 2018, 198, 662–682. [Google Scholar] [CrossRef]
  17. Tavana, M.; Zareinejad, M.; Santos-Arteaga, F.J.; Kaviani, M.A. A conceptual analytic network model for evaluating and selecting third-party reverse logistics providers. Int. J. Adv. Manuf. Technol. 2016, 86, 1705–1721. [Google Scholar] [CrossRef]
  18. Govindan, K.; Sarkis, J.; Palaniappan, M. An analytic network process-based multicriteria decision making model for a reverse supply chain. Int. J. Adv. Manuf. Technol. 2013, 68, 863–880. [Google Scholar] [CrossRef]
  19. Zarbakhshnia, N.; Jaghdani, T.J. Sustainable provider evaluation and selection with a novel two-stage DEA model in the presence of uncontrollable inputs and undesirable outputs: A plastic case study. Int. J. Adv. Manuf. Technol. 2018, 97, 2933–2945. [Google Scholar] [CrossRef]
  20. Azadi, M.; Saen, R.F. A new chance-constrained data envelopment analysis for selecting third-party reverse logistics providers in the existence of dual-role factors. Expert. Syst. Appl. 2011, 38, 12231–12236. [Google Scholar] [CrossRef]
  21. Momeni, E.; Azadi, M.; Saen, R. Measuring the efficiency of third party reverse logistics provider in supply chain by multi objective additive network DEA model. Int. J. Ship. Transp. Log. 2015, 7, 21–41. [Google Scholar] [CrossRef]
  22. Kannan, G.; Pokharel, S.; Sasi Kumar, P. A hybrid approach using ISM and fuzzy TOPSIS for the selection of reverse logistics provider. Resour. Conserv. Recycl. 2009, 54, 28–36. [Google Scholar] [CrossRef]
  23. Govindan, K.; Palaniappan, M.; Zhu, Q.; Kannan, D. Analysis of third party reverse logistics provider using interpretive structural modeling. Int. J. Prod. Econ. 2012, 140, 204–211. [Google Scholar] [CrossRef]
  24. Gardas, B.B.; Raut, R.D.; Narkhede, B.E. Analysing the 3PL service provider’s evaluation criteria through a sustainable approach. Int. J. Prod. Perf. Manag. 2019, 68, 958–980. [Google Scholar] [CrossRef]
  25. Liu, A.; Ji, X.; Lu, H.; Liu, H. The selection of 3PRLs on self-service mobile recycling machine: Interval-valued pythagorean hesitant fuzzy best-worst multi-criteria group decision-making. J. Clean. Prod. 2019, 230, 734–750. [Google Scholar] [CrossRef]
  26. Garg, C.P.; Sharma, A. Sustainable outsourcing partner selection and evaluation using an integrated BWM–VIKOR framework. Environ. Dev. Sustain. 2018, 22, 1529–1557. [Google Scholar] [CrossRef]
  27. Govindan, K.; Jha, P.C.; Agarwal, V.; Darbari, J.D. Environmental management partner selection for reverse supply chain collaboration: A sustainable approach. J. Environ. Manag. 2019, 236, 784–797. [Google Scholar] [CrossRef]
  28. Kumar, A.; Dixit, G. A novel hybrid MCDM framework for WEEE recycling partner evaluation on the basis of green competencies. J. Clean. Prod. 2019, 241, 118017. [Google Scholar] [CrossRef]
  29. Liu, A.; Xiao, Y.; Lu, H.; Tsai, S.-B.; Song, W. A fuzzy three-stage multi-attribute decision-making approach based on customer needs for sustainable provider selection. J. Clean. Prod. 2019, 239, 118043. [Google Scholar] [CrossRef]
  30. Zhou, F.; Wang, X.; Lim, M.K.; He, Y.; Li, L. Sustainable recycling partner selection using fuzzy DEMATEL-AEW-FVIKOR: A case study in small-and-medium enterprises (SMEs). J. Clean. Prod. 2018, 196, 489–504. [Google Scholar] [CrossRef]
  31. Luthra, S.; Govindan, K.; Kannan, D.; Mangla, S.K.; Garg, C.P. An integrated framework for sustainable provider selection and evaluation in supply chains. J. Clean. Prod. 2017, 140, 1686–1698. [Google Scholar] [CrossRef]
  32. Sasikumar, P.; Haq, A.N. Integration of closed loop distribution supply chain network and 3PRLP selection for the case of battery recycling. IJPR 2011, 49, 3363–3385. [Google Scholar] [CrossRef]
  33. Prakash, C.; Barua, M.K. A combined MCDM approach for evaluation and selection of third-party reverse logistics partner for Indian electronics industry. Sustain. Prod. Consump. 2016, 7, 66–78. [Google Scholar] [CrossRef]
  34. Dos Santos, B.M.; Godoy, L.P.; Campos, L.M.S. Performance evaluation of green providers using entropy-TOPSIS-F. J. Clean. Prod. 2019, 207, 498–509. [Google Scholar] [CrossRef]
  35. Senthil, S.; Srirangacharyulu, B.; Ramesh, A. A robust hybrid multi-criteria decision making methodology for contractor evaluation and selection in third-party reverse logistics. Expert Syst. Appl. 2014, 41, 50–58. [Google Scholar] [CrossRef]
  36. Prakash, C.; Barua, M.K. An analysis of integrated robust hybrid model for third-party reverse logistics partner selection under fuzzy environment. Resour. Conserv. Recycl. 2016, 108, 63–81. [Google Scholar] [CrossRef]
  37. Govindan, K.; Agarwal, V.; Darbari, J.D.; Jha, P.C. An integrated decision making model for the selection of sustainable forward and reverse logistic providers. AnOR 2017, 273, 607–650. [Google Scholar] [CrossRef]
  38. Chen, K.; Yu, X.; Yang, L. GI-TOPSIS Based on Combinational Weight Determination and its Application to Selection of Reverse Logistics Service Providers. J. Grey Syst. 2013, 25, 16–33. [Google Scholar]
  39. Zarbakhshnia, N.; Wu, Y.; Govindan, K.; Soleimani, H. A novel hybrid multiple attribute decision-making approach for outsourcing sustainable reverse logistics. J. Clean. Prod. 2020, 242, 118461. [Google Scholar] [CrossRef]
  40. Mavi, R.K.; Goh, M.; Zarbakhshnia, N. Sustainable third-party reverse logistic provider selection with fuzzy SWARA and fuzzy MOORA in plastic industry. Int. J. Adv. Manuf. Technol. 2017, 91, 2401–2418. [Google Scholar] [CrossRef]
  41. Luo, Z.; Li, Z. A MAGDM Method Based on Possibility Distribution Hesitant Fuzzy Linguistic Term Set and Its Application. Mathematics 2019, 7, 1063. [Google Scholar] [CrossRef] [Green Version]
  42. Li, Y.-L.; Ying, C.-S.; Chin, K.-S.; Yang, H.-T.; Xu, J. Third-party reverse logistics provider selection approach based on hybrid-information MCDM and cumulative prospect theory. J. Clean. Prod. 2018, 195, 573–584. [Google Scholar] [CrossRef]
  43. Zarbakhshnia, N.; Soleimani, H.; Ghaderi, H. Sustainable third-party reverse logistics provider evaluation and selection using fuzzy SWARA and developed fuzzy COPRAS in the presence of risk criteria. Appl. Soft. Comput. 2018, 65, 307–319. [Google Scholar] [CrossRef]
  44. Sen, D.K.; Datta, S.; Mahapatra, S.S. Decision Support Framework for Selection of 3PL Service Providers: Dominance-Based Approach in Combination with Grey Set Theory. Int. J. Inf. Technol. Decis. 2017, 16, 25–57. [Google Scholar] [CrossRef]
  45. Tian, G.; Zhang, H.; Feng, Y.; Jia, H.; Zhang, C.; Jiang, Z.; Li, Z.; Li, P. Operation patterns analysis of automotive components remanufacturing industry development in China. J. Clean. Prod. 2017, 164, 1363–1375. [Google Scholar] [CrossRef]
  46. Pourjavad, E.; Mayorga, R.V. A fuzzy rule-based approach to prioritize third-party reverse logistics based on sustainable development pillars. J. Intell. Fuzzy Syst. 2018, 35, 3125–3138. [Google Scholar] [CrossRef]
  47. Guarnieri, P.; Sobreiro, V.A.; Nagano, M.S.; Marques Serrano, A.L. The challenge of selecting and evaluating third-party reverse logistics providers in a multicriteria perspective: A Brazilian case. J. Clean. Prod. 2015, 96, 209–219. [Google Scholar] [CrossRef]
  48. Silva, A.L.E.; Moraes, J.A.R.; Machado, Ê.L. Proposta de produção mais limpa voltada às práticas de ecodesign e logística reversa. Eng. Sanit. Ambient. 2015, 20, 29–37. [Google Scholar] [CrossRef] [Green Version]
  49. Singhal, D.; Tripathy, S.; Jena, S.K. Remanufacturing for the circular economy: Study and evaluation of critical factors. Resour. Conserv. Recycl. 2020, 156, 104681. [Google Scholar] [CrossRef]
  50. Meherishi, L.; Narayana, S.A.; Ranjani, K.S. Sustainable packaging for supply chain management in the circular economy: A review. J. Clean. Prod. 2019, 237, 117582. [Google Scholar] [CrossRef]
  51. Saaty, T.L. The Analytic Hierarchy Process; McGraw-Hill Book, Co.: New York, NY, USA, 1980. [Google Scholar]
  52. Hwang, C.L.; Yoon, K. Multiple Attributes Decision Making Methods and Applications; Springer: Berlin, Germany, 1981. [Google Scholar]
  53. Li, S. Research on remanufacturing technology and Development Trend of Automobile engine. Intern. Combust. Engine Parts 2019, 18, 203–204. [Google Scholar]
  54. Jiang, Z.; Ding, Z.; Zhang, H.; Cai, W.; Liu, Y. Data-driven ecological performance evaluation for remanufacturing process. Energ. Convers. Manag. 2019, 198, 111844. [Google Scholar] [CrossRef]
  55. Chu, J.W. Automotive Recycling Engineering, 2nd ed.; People Transportation Press: Beijing, China, 2012. [Google Scholar]
  56. Jiang, Z.; Jiang, Y.; Wang, Y.; Zhang, H.; Cao, H.; Tian, G. A hybrid approach of rough set and case-based reasoning to remanufacturing process planning. J. Intell. Manuf. 2016, 30, 19–32. [Google Scholar] [CrossRef]
  57. Ding, Z.; Jiang, Z.; Zhang, H.; Cai, W.; Liu, Y. An integrated decision-making method for selecting machine tool guideways considering remanufacturability. Int. J. Comput. Integr. Manuf. 2020, 33, 686–700. [Google Scholar] [CrossRef]
Figure 1. Proposed research framework for RL provider selection.
Figure 1. Proposed research framework for RL provider selection.
Processes 09 00631 g001
Figure 2. Structure of RL provider selection.
Figure 2. Structure of RL provider selection.
Processes 09 00631 g002
Figure 3. Process of analytic hierarchy process (AHP).
Figure 3. Process of analytic hierarchy process (AHP).
Processes 09 00631 g003
Figure 4. Process of technique for order of preference by similarity to ideal solution (TOPSIS).
Figure 4. Process of technique for order of preference by similarity to ideal solution (TOPSIS).
Processes 09 00631 g004
Table 1. Summary of factors for reverse logistics (RL) provider selection.
Table 1. Summary of factors for reverse logistics (RL) provider selection.
FactorsRelevant Characteristics in the LiteratureReferences
SustainabilityThe evaluation criteria are determined based on the three dimensions of economy, environment and society[20,21,22,23,24,25,27,29,37,38]
GSCMGreen recycling, green purchasing, green transportation, resource and environmental management capabilities, social responsibility benefits, green core competencies[14,26,33]
CEAir pollution, environmental standards, eco-friendly raw materials, eco-design, eco-friendly packaging, eco-friendly transportation, clean technology[13]
Performance dimensionsReverse logistics services, reverse logistics functions, organizational role, user satisfaction, RL activities, organizational performance criteria, IT application[15,16,32,34]
Table 2. Scale of preference.
Table 2. Scale of preference.
ScaleExplanation
1Two factors are of equal importance compared to each other
3One factor is slightly more important than the other
5One factor is obviously more important than the other
7One factor is strongly more important than the other
9One factor is extremely more important than the other
2, 4, 6, 8Intermediate values
ReciprocalsReciprocals for inverse comparison
Table 3. Random index (RI).
Table 3. Random index (RI).
N1234567891011
RI000.580.901.121.241.321.411.451.491.51
Table 4. The weights and ranking of the four dimensions.
Table 4. The weights and ranking of the four dimensions.
DimensionsPairwise ComparisonsWeightsRank
A1A2A3A4
Economy (A1)10.50.20.250.07494
Society (A2)210.250.80.14193
Technology (A3)54150.59501
Circularity (A4)41.250.210.18832
Table 5. The weights and ranking of criteria.
Table 5. The weights and ranking of criteria.
DimensionsCriteriaRelative WeightsRelative RankGlobal WeightsGlobal Rank
EconomyE10.125020.009413
E20.875010.06555
SocietyE30.285720.04058
E40.571410.08113
E50.142930.020310
TechnologyE60.500010.29751
E70.100030.05956
E80.400020.23802
CircularityE90.203430.03839
E100.050850.009612
E110.406810.07664
E120.067840.012811
E130.271220.05117
Table 6. Original decision matrix of alternatives.
Table 6. Original decision matrix of alternatives.
E1E2E3E4E5E6E7E8E9E10E11E12E13
MT2.87526.256.3756.12577.56.375777.1257.57.375
TPT3.7546.3756.6256.56.6256.3756.8756.8756.565.6256.375
RT3.1253.3756.1256.256.55.2556.256.1255.6255.55.1255.25
Table 7. Normalized decision matrix.
Table 7. Normalized decision matrix.
E1E2E3E4E5E6E7E8E9E10E11E12E13
MT0.50750.35700.57730.57340.55450.63780.67930.56580.60520.63140.65870.70200.6661
TPT0.66190.71390.58880.59590.58840.60360.57740.61010.59440.58630.55470.52650.5758
RT0.55160.60240.56570.56220.58840.47840.45290.55470.52960.50740.50840.47970.4742
Table 8. Weighted normalized matrix.
Table 8. Weighted normalized matrix.
E1E2E3E4E5E6E7E8E9E10E11E12E13
MT0.00480.02340.02340.04650.01130.18970.04040.13470.02320.00610.05050.00900.0340
TPT0.00620.04680.02380.04830.01190.17960.03440.14520.02280.00560.04250.00670.0294
RT0.00520.03950.02290.04560.01190.14230.02690.13200.02030.00490.03890.00610.0242
V + = (0.0062, 0.0468, 0.0238, 0.0483, 0.0119, 0.1897, 0.0404, 0.1452, 0.0232, 0.0061, 0.0505, 0.0090, 0.0340)
V = (0.0048, 0.0234, 0.0229, 0.0456, 0.0113, 0.1423, 0.0269, 0.1320, 0.0203, 0.0049, 0.0389, 0.0061, 0.0242)
Table 9. Distances to ideal solution and closeness coefficient.
Table 9. Distances to ideal solution and closeness coefficient.
Alternatives D + D CCiRank
MT0.02580.05180.66792nd
TPT0.01520.04710.75651st
RT0.05400.01610.22963rd
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Zhang, X.; Li, Z.; Wang, Y.; Yan, W. An Integrated Multicriteria Decision-Making Approach for Collection Modes Selection in Remanufacturing Reverse Logistics. Processes 2021, 9, 631. https://doi.org/10.3390/pr9040631

AMA Style

Zhang X, Li Z, Wang Y, Yan W. An Integrated Multicriteria Decision-Making Approach for Collection Modes Selection in Remanufacturing Reverse Logistics. Processes. 2021; 9(4):631. https://doi.org/10.3390/pr9040631

Chicago/Turabian Style

Zhang, Xumei, Zhizhao Li, Yan Wang, and Wei Yan. 2021. "An Integrated Multicriteria Decision-Making Approach for Collection Modes Selection in Remanufacturing Reverse Logistics" Processes 9, no. 4: 631. https://doi.org/10.3390/pr9040631

APA Style

Zhang, X., Li, Z., Wang, Y., & Yan, W. (2021). An Integrated Multicriteria Decision-Making Approach for Collection Modes Selection in Remanufacturing Reverse Logistics. Processes, 9(4), 631. https://doi.org/10.3390/pr9040631

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop