Performance Analysis of the Capability Assessment Tool for Sustainable Manufacturing

Abstract

This paper explores the performance of a novel capability assessment tool, developed to identify capability gaps and associated training and development requirements across the supply chain for environmentally-sustainable manufacturing. The tool was developed to assess 170 capabilities that have been clustered with respect to key areas of concern such as managing energy, water, material resources, carbon emissions and waste as well as environmental management practices for sustainability. Two independent expert teams used the tool to assess a sample group of five first and second tier sports apparel and footwear suppliers within the supply chain of a global sporting goods manufacturer in Asia. The paper addresses the reliability and robustness of the developed assessment method by formulating the expected links between the assessment results. The management practices of the participating suppliers were shown to be closely connected to their performance in managing their resources and emissions. The companies’ initiatives in implementing energy efficiency measures were found to be generally related to their performance in carbon emissions management. The suppliers were also asked to undertake a self-assessment by using a short questionnaire. The large gap between the comprehensive assessment and these in-house self-assessments revealed the suppliers’ misconceptions about their capabilities.

1. Introduction

Industrial economics are increasingly dominated by major concerns about environmental factors, most notably climate change, the shortage of non-renewable energy sources, water and raw material supplies, and commodity price increases [1]. In response to these concerns, national and international authorities have established standards, protocols, regulations, and schemes, such as emissions trading and binding international targets for pollution control and resource efficiency, e.g., the United Nations Framework Convention on Climate Change (UNFCCC) [2]. Manufacturers that do not adapt to these environmental, social and economic changes will be affected in the emerging competitive market for sustainability [3]. Furthermore, the increasing public awareness of the social, environmental, and economic benefits associated with sustainable manufacturing, and of the risks, liabilities, and costs associated with unsustainable practices, represent additional drivers for a more fundamental shift towards sustainability within the manufacturing sector [2,3,4,5,6,7]. It is important to note that by adopting the sustainable manufacturing concept, companies have the potential to create new market opportunities and improve their corporate reputation [8,9]. This in essence supports the development of globally competitive manufacturing businesses [10,11] and positively impacts on current financial and market performance of such businesses [12].

Manufacturing involves continual management of a variety of inter-related disciplines and activities, including product design, raw material purchasing, production planning and processes, and environmental compliance [4]. To manage such operations effectively, complex decisions must be made, involving often conflicting requirements. Contrary to traditional manufacturing, which is typically dominated by decisions based on economics only, sustainable manufacturing management is based on a more comprehensive, multi-criteria, decision-making process, which seeks to balance social, e.g., [13], environmental and economic elements of manufacturing [2,14,15,16]. This is essential in order to minimize the likelihood that the solutions adopted today will not become problems in the future.

Criteria which are typically considered pertinent to environmental sustainability assessment include the following examples [15]:

• Made from renewable materials;
• Produced while making the most efficient use of resources such as water and energy;
• Produced with minimal pollution to the environment and waste;
• Capable of being washed in low temperatures using environmentally friendly agents (where applicable);
• Capable of being returned safely to the environment at the end of their useful life.

This paper is scoped around the environmental aspects of sustainable manufacturing. Implementation of environmentally sustainable practices in manufacturing of sports apparel and sports footwear must be based on clearly-defined environmental targets for reduction in emissions, water consumption and waste generation. The materials and processes used in manufacturing are of key concern as they carry with them many potential environmental risks. For example, while many manufacturers state that they aim to increase the percentages of recycled materials in their products, instead, virgin materials are being adopted as the main raw materials for sports apparel. Also, particular manufacturing operations in sports apparel and footwear manufacturing—e.g., injection moulding, used in manufacturing of outsoles and heels—are energy intensive, which means that they account for high levels of emissions [17]. Furthermore, the demand for textile fibre worldwide is increasing along with the demand for high performance sports apparel. Performance apparel items impact on the environment at every stage of their life cycle as shown in Table 1. Sports apparel and footwear in general have a heavy carbon footprint, which results primarily from pollution, consumption of non-renewable resources and waste associated with manufacturing. Manufacturers are required to reduce or alleviate these impact areas across the entire supply chain. Whether they are able to reduce energy consumption and emissions, water consumption and waste largely depends on whether staff and management are capable of identifying and implementing the required improvements within their particular environments.

Table 1. Environmental impacts across the life cycle of sport apparel.

Lifecycle stage	Environmental impact
Raw materials (growth, acquisition and processing)	Resource consumption, greenhouse gas emissions, air/water pollution and toxicity, soil degradation/contamination, biodiversity/land use, solid and hazardous waste
Fibre production (natural and synthetic)	Greenhouse gas emissions, air/water pollution and toxicity, soil degradation/contamination, biodiversity/land use
Clothing production and garment assembly	Greenhouse gas emissions, air/water pollution and toxicity, soil degradation/contamination, biodiversity/land use
Packaging	Solid and hazardous waste
Distribution	Greenhouse gas emissions
Retail	Solid and hazardous waste
Use	Resource consumption, solid and hazardous waste
End of life management	Greenhouse gas emissions, solid and hazardous waste

Table 1. Environmental impacts across the life cycle of sport apparel.

Lifecycle stage	Environmental impact
Raw materials (growth, acquisition and processing)	Resource consumption, greenhouse gas emissions, air/water pollution and toxicity, soil degradation/contamination, biodiversity/land use, solid and hazardous waste
Fibre production (natural and synthetic)	Greenhouse gas emissions, air/water pollution and toxicity, soil degradation/contamination, biodiversity/land use
Clothing production and garment assembly	Greenhouse gas emissions, air/water pollution and toxicity, soil degradation/contamination, biodiversity/land use
Packaging	Solid and hazardous waste
Distribution	Greenhouse gas emissions
Retail	Solid and hazardous waste
Use	Resource consumption, solid and hazardous waste
End of life management	Greenhouse gas emissions, solid and hazardous waste

Due to the relative newness of environmental sustainability in the manufacturing sector (often called green manufacturing) as a discipline [9], the industry needs to acquire new capabilities (knowledge, skills, and values) in order to convert environmentally-sustainable manufacturing aspirations into practice. Many global manufacturers have already set strategic environmental targets for themselves, e.g., [18]. Whilst setting and monitoring tangible environmental targets are essential steps towards practicing sustainability, the targets will not be achieved if a manufacturer does not possess the required capabilities that need to be identified strategically by using a comprehensive sustainability framework [3]. Such a framework needs to go beyond the environmental management systems traditionally practiced (which mainly encourage environmental awareness) [11], by including particular assessment approaches to identify the capability gaps that need to be addressed in order to achieve the set environmental sustainability targets and encourage suppliers to proactively take environmental initiatives [19]. Sustainability frameworks developed to date are typically focused on strategy and decision making processes, rather than capability assessment. De Bakker and Neijhof developed a framework for social aspects of sustainability [20]; however, no specific capabilities were proposed. Kinderytė reviewed a number of sustainability assessment frameworks [21]. One of these frameworks, the Sustainability Assessment for Enterprises (SAFE), developed by the Wuppertal Institute, is designed to assess a company’s performance on economic, ecological, social and communication metrics. One aspect of the SAFE framework is the identification of the qualification needs of employees [21]. The SAFE system, however, does not extend into the supply chain. Likewise, the Sustainability Competency and Opportunity Rating and Evaluation (SCORE) system [22] includes aspects of assessing capability across sustainability practices, but also does not extend this assessment to the supply chain. A common aspect of these frameworks is the need and importance of involving stakeholders in framework development [20,23], including the company, its customers and suppliers, and government agencies There are sustainable manufacturing frameworks and/or environmental assessment tools introduced by other researchers which are solely based upon some particular solutions only such as using cleaner energies, using materials with low environmental impacts, and using improved technologies to reduce environmental footprints—e.g., GHGs, and hazardous waste—associated with manufacturing processes [6,24]. This is while there are also examples of some frameworks and assessment tools (e.g., for environmental management performance) which have limited but broad objective-based capabilities—e.g., pollution reduction capability—along with some solution-based capabilities such as employing advanced technologies used as the basis to quantify (assess) the environmental management performance of manufacturing firms [25]. In sum, after reviewing existing frameworks, it was identified that there was a need to develop a new framework to assess particular capabilities across the supply chain that can be applied to a broad range of industries. Such a framework needs to be objective based, focusing on environmental footprints and the associated capabilities (at the manufacturers’ level) in order to provide a real value add to different types of manufacturers.

In an earlier article published by the authors [26], the development and implementation of a new sustainable manufacturing capability framework for an existing supply chain within the sports apparel and footwear industries was described in detail. This framework, which is primarily concerned with formulating critical sustainable manufacturing competencies, was used to establish a metrics assessment tool to evaluate capabilities relating to environmental sustainability at manufacturing level in order to identify capability gaps, improvement strategies, and training needs to achieve a set of environmental targets, i.e., reduction of energy and water usage, as well as the reduction of waste and emissions generation. Identifying such gaps is an important capability by itself that allows a manufacturer to realize its sustainability-related threats and opportunities and create new (dynamic) capabilities accordingly to keep pace with outside changes [27,28]—e.g., new environmental regulations and targets. The earlier paper published by the authors [26] mainly focused on using the framework and the associated assessment tool to identify such gaps and suggest training needs to fill the gaps. The present paper is dedicated to evaluating the performance, robustness, and reliability of this sustainable manufacturing framework and the metrics assessment tool built upon it. Although the framework developed as part of this research is focused on environmental sustainability at the manufacturing level, the approach used in this case can be extended beyond manufacturing to achieve a comprehensive system approach for sustainable manufacturing (i.e., life cycle approach) [8,16,24]. It is noteworthy that the present research was conducted in collaboration with a global sporting goods manufacturer, involving five of its first and second tier sports footwear and apparel suppliers in Asia, aimed at assisting the manufacturer to meet its strategic environmental targets across the supply chain.

2. Sustainable Manufacturing Capability Assessment Tool (CMAT)

The first step towards assessing the suppliers’ capabilities relating to sustainable manufacturing was to develop a sustainable manufacturing framework (SMF), (Figure 1). This framework was developed in consultation with the manufacturer and the participating suppliers, as well as by reviewing the sustainability targets and associated documents of the manufacturer and similar global manufacturers—e.g., [29] and other external resources from Australia’s National Training Information Service [30], the Australian Department of Education, Employment and Workplace Relations [31], and ISO standards [32,33]. The framework comprises of three levels termed clusters, applied outcomes, and capabilities.

Figure 1. Sustainable manufacturing framework (SMF) (Note: there are 170 capabilities under these eight clusters and 18 applied outcomes which have not been shown in this figure).

Figure 1. Sustainable manufacturing framework (SMF) (Note: there are 170 capabilities under these eight clusters and 18 applied outcomes which have not been shown in this figure).

The eight clusters in this framework reflect the environmental indicators and initiatives set by the manufacturer, similar to those also provided in corporate reports [34,35]. Environmental capabilities and the associated indicators [10,32,33,36,37] are broadly grouped within three management categories, including resources (energy, water, and material), emissions (waste and greenhouse gas emissions), and environment (e.g., ISO 14001 standards). These are the key elements that to some extent have also been considered by similar studies in these areas, e.g., Material; Energy; and Waste, MEW, used by Smith and Ball [38]. These eight clusters were grouped into three broader areas of sustainable manufacturing; resource management, emissions management, and improved management practices (in order to introduce environmental sustainability). The 18 applied outcomes relate to industry practices within the relevant environmental indicators and initiatives.

Based on the sustainable manufacturing framework (SMF), we have developed a comprehensive capability metrics assessment tool (CMAT), which was used to assess the suppliers’ capabilities and complete a needs analysis. The capabilities were assessed on a Likert Scale, ranging from one to five (unsatisfactory to excellent) based on the following examples of attainment:

• Unsatisfactory (1): There is no capability at all.
• Satisfactory (2): There are just enough, minimal or marginal capabilities.
• Good (3): Developing capability above minimal or marginal, can do basic things independently.
• Very good (4): Developed capability, can plan regular actions independently.
• Excellent (5): Can lead changes, plan improvements, and grasp new techniques.

In order to ensure a fair, robust, and reliable assessment, the capabilities were assessed by two independent expert teams (with up to four personnel in each team), one team from RMIT University and one team from the manufacturer. Furthermore, four assessment methods were employed in the assessment process. Each core capability (specified based on the industry being assessed) was assessed using two of the four methods, while the remaining capabilities were assessed using at least one method. The four methods used in this assessment were:

• Walk-through assessment, based on observing capability in work activities.
• Written answers to one or more questions based on simulated workplace activities.
• Interviews through responses to verbal questions.
• Work samples (e.g., a project report) indicating prior demonstration of capability.

Different questions were used to assess each capability using each of the four methods described above. For example, the “reduce energy use” applied outcome under the “energy efficiency” cluster (see Figure 1) includes the following capabilities to be investigated:

• Measure, record, and report energy use in factory operations.
• Compare energy use records for operations to previous records and industry best practice.
• Relate energy used in operation to total consumption for the factory.
• Use rankings and identify energy losses to determine process which can be improved.
• Document a flowchart showing energy flows for an operation in the factory.
• Apply continuous improvement.

Each assessment method for a particular capability includes a few questions/items to be asked/checked against pre-defined evidence of attainment as a reference to avoid registering any possible expert teams’ perceptions about the participating suppliers. For example, under the “reduce energy use” applied outcome, the following open questions were used in an interview with the focus group:

• Explain how you measure energy use for the factory (evidence: electricity bills, oil/diesel/coal bills, energy balances).
• Explain how you measure or calculate energy use for a machine or a production line (evidence: direct measurement using tongue tester; estimation based on machine energy rating and time of use, energy balances).
• Explain how you report energy use (evidence: the manufacturer’s Environmental Metrics Reporting Tool, own system; can be for all operations or machine-specific. Need to provide examples of internal or external reports).

The overall score of a cluster was calculated by following the procedure illustrated in Figure 2 and using the following equation:

(1)

Where “C” is the overall average score of the cluster (a number between 1 and 5) calculated by an assessment group i, “A” is the number of assessors; “N” is the number of applied outcomes under the cluster being assessed (1, 2, or 3 according to Figure 1); “M_n” is the number of capabilities under the applied outcome of “n” (n varies from 1 to N), which is a number between 5 and 16 depending on how many capabilities are listed under that applied outcome; and c is the score given to a capability by an assessor based on any method used for assessing that capability.

Figure 2. The overall structure of the assessment methodology used for a comprehensive capability metrics assessment tool (CMAT).

Figure 2. The overall structure of the assessment methodology used for a comprehensive capability metrics assessment tool (CMAT).

Notes:

• N, number of applied outcomes under each cluster, is either 2 or 3, depending on the cluster (see Figure 1);
• M, used to show the number of capabilities under an applied outcome, is an integer between 5 and 16, depending on the applied outcome;
• For a capability being assessed by an assessor, the average of the scores obtained from different methods will form the overall score of that capability;
• The average scores of different capabilities under an applied outcome will form the overall score of that applied outcome;
• The average scores obtained for different applied outcomes under a cluster will form the overall score of that cluster;
• Each capability is assessed by four assessors, so the same procedure is repeated for each assessor and the averages of all are used;
• The nature of capability determines what method is used for its assessment.

3. Capability Assessment and Gap Analysis Results

The manufacturer that collaborated with RMIT in this study is an original equipment manufacturer (OEM) and is at the top of a tiered supply chain. Tier one companies supply products directly to the OEM, while tier two companies supply products to tier one companies. Tier three companies supply tier two companies and so on. This study was limited to the assessment of selected tier one and tier two companies. CMAT was used to assess the capabilities relating to sustainable manufacturing within a sample group of tier 1 and 2 apparel and footwear suppliers that participated in this study (Table 2).

Table 2. Apparel and footwear tier 1 and 2 suppliers that participated in the study.

Supplier Name	Industry Type	Short Description
A	Footwear, Tier 1	Making sport shoes. The major manufacturing operations include cutting, stitching, gluing, finishing, and packaging.
B	Footwear, Tier 1	Making sport shoes for major global sporting manufacturers. The main manufacturing operations include cutting, stitching, gluing, finishing, and packaging.
C	Footwear, Tier 2	Making high-technology plastic shoe components. The major manufacturing processes include design, blending of raw materials, injection molding, and packaging.
D	Apparel, Tier 1	Making sport clothing products with five major divisions including: fabric warehouse; cutting/patterns; sewing; finishing; and sampling.
E	Apparel, Tier 2	Making textile products. The major manufacturing processes include desizing, scouring, washing, mercerizing, bleaching, printing and finishing.

Table 2. Apparel and footwear tier 1 and 2 suppliers that participated in the study.

Supplier Name	Industry Type	Short Description
A	Footwear, Tier 1	Making sport shoes. The major manufacturing operations include cutting, stitching, gluing, finishing, and packaging.
B	Footwear, Tier 1	Making sport shoes for major global sporting manufacturers. The main manufacturing operations include cutting, stitching, gluing, finishing, and packaging.
C	Footwear, Tier 2	Making high-technology plastic shoe components. The major manufacturing processes include design, blending of raw materials, injection molding, and packaging.
D	Apparel, Tier 1	Making sport clothing products with five major divisions including: fabric warehouse; cutting/patterns; sewing; finishing; and sampling.
E	Apparel, Tier 2	Making textile products. The major manufacturing processes include desizing, scouring, washing, mercerizing, bleaching, printing and finishing.

The results of the assessment are compared with the manufacturer’s minimum expectations for different applied outcomes and clusters in order to identify capability gaps and complete a training needs analysis for the sample group of suppliers. Figure 3 shows a summary of the overall results of the capability assessment (applied outcome-based) obtained for the five participating suppliers using CMAT. The details of the gap analysis and required training program have been published by Subic et al. [26].

According to Figure 3, there were some common gaps in capabilities relating to the following applied outcomes:

• 1.1 Reduce energy use
• 1.2 Maximize alternative energy resources
• 2.1 Reduce water use
• 2.2 Maximize alternative water supply (excluding supplier B)
• 4.1 Implement and apply waste management hierarchy
• 4.2 Handle, store, treat, and dispose waste appropriately
• 5.1 Account for carbon emissions
• 5.2 Reduce carbon emissions
• 6.1 Enhance auditing and environmental monitoring performance
• 6.2 Comply with environmental systems
• 7.1 Implement industrial clustering and resource pooling
• 7.3 Identify, develop, and implement business cases for sustainability improvement

The results of this analysis were then used to devise a training program for the suppliers to help them attain the required knowledge and skills in order to meet the environmental targets set by the manufacturer.

Figure 3. The overall results of capability assessment (applied outcome-based), done for five first and second tier footwear and apparel suppliers of a global sporting goods manufacturer, using CMAT (see Figure 1 for details on the applied outcomes 1.1–8.2).

Figure 3. The overall results of capability assessment (applied outcome-based), done for five first and second tier footwear and apparel suppliers of a global sporting goods manufacturer, using CMAT (see Figure 1 for details on the applied outcomes 1.1–8.2).

Notes: Considering the nature of the participating suppliers, some applied outcomes may not be applicable for some suppliers (e.g., 7.1 for supplier B). The manufacturer’s minimum expectations; Applied outcome-based assessment results.

4. Sensitivity Analysis and Assessment Tool Robustness

4.1. Sensitivity Analysis

The following analysis will demonstrate how cluster-based overall assessment results determined by one assessor (A = 1) can be different to that given by another assessor. Following Equation (1), the overall score of a cluster given by an assessor can be obtained using the following equation:

(2)

Hence, the difference between the overall scores of two assessors can be presented as follows:

(3)

and are numbers between 0 and 4. Considering each assessment team as an assessor, the assessment completed by two assessment teams in this study can be assumed as an assessment conducted by two independent representative “assessors”. By reviewing the scores obtained by these two assessors (or two assessment teams) for different capabilities, the maximum differences were mainly about 1 score (excluding some exceptional cases with score differences of about 2 or so). Hence, the maximum effect (when for both methods follow the same signs) of such a difference (1 score) for scores given for a capability by two assessors can appear as a difference in the overall score of the cluster as shown in Table 3. As suggested by this equation, the difference in the overall score depends on the number of applied outcomes (N) and the number of capabilities (M_n) under the applied outcome with different given scores:

(4)

Sensitivity estimations given in Table 3 underline the fact that overestimating or underestimating a capability by one score can lead to a negligible effect on the overall results. That is why the overall scores of different clusters given by two independent assessors show an overall difference of above 1 score in only 7% of cases (grey cells in this table). A further detailed review of the results at applied outcome level also shows that the two assessment teams had agreement on almost the same scores for 60% of the clusters assessed for these suppliers (Figure 4).

Table 3. Sensitivity of the cluster-based results in one score difference for one capability and the cluster-based score differences for five participating suppliers assessed by two independent assessment teams.

Cluster		1	2	3	4	5	6	7	8
N		2	2	2	3	2	2	3	2
M1		6	6	16	11	8	12	5	7
M2		5	13	12	9	4	14	11	13
M3		NA	NA	NA	5	NA	NA	12	NA
∆C1 (for the 1st applied outcome)		0.08	0.08	0.03	0.03	0.06	0.04	0.07	0.07
∆C2 (for the 2nd applied outcome)		0.1	0.04	0.04	0.04	0.125	0.036	0.03	0.04
∆C3 (for the 3rd applied outcome)		NA	NA	NA	0.07	NA	NA	0.03	NA
Actual ∆C calculated through assessment	Supplier A	0.41	0.22	1.1	0.08	0	0.11	0.01	0.09
	Supplier B	0.4	1.28	1.06	0.73	0.34	0.66	0	0.04
	Supplier C	0.05	0.52	0.02	0.12	0.38	0.41	0.57	0.29
	Supplier D	0.36	0.53	0.01	0.45	0.13	0.25	0.1	0.08
	Supplier E	0	0.16	0.05	0.09	0	0.13	0.15	0.35

Note: The overall scores of different clusters given by two independent assessors show an overall difference of above one score in only 7% of cases. These cells are shown in grey color.

Table 3. Sensitivity of the cluster-based results in one score difference for one capability and the cluster-based score differences for five participating suppliers assessed by two independent assessment teams.

Cluster		1	2	3	4	5	6	7	8
N		2	2	2	3	2	2	3	2
M1		6	6	16	11	8	12	5	7
M2		5	13	12	9	4	14	11	13
M3		NA	NA	NA	5	NA	NA	12	NA
∆C1 (for the 1st applied outcome)		0.08	0.08	0.03	0.03	0.06	0.04	0.07	0.07
∆C2 (for the 2nd applied outcome)		0.1	0.04	0.04	0.04	0.125	0.036	0.03	0.04
∆C3 (for the 3rd applied outcome)		NA	NA	NA	0.07	NA	NA	0.03	NA
Actual ∆C calculated through assessment	Supplier A	0.41	0.22	1.1	0.08	0	0.11	0.01	0.09
	Supplier B	0.4	1.28	1.06	0.73	0.34	0.66	0	0.04
	Supplier C	0.05	0.52	0.02	0.12	0.38	0.41	0.57	0.29
	Supplier D	0.36	0.53	0.01	0.45	0.13	0.25	0.1	0.08
	Supplier E	0	0.16	0.05	0.09	0	0.13	0.15	0.35

Note: The overall scores of different clusters given by two independent assessors show an overall difference of above one score in only 7% of cases. These cells are shown in grey color.

Figure 4. Comparison of assessment results obtained by two expert teams from the RMIT University and the manufacturer.

Figure 4. Comparison of assessment results obtained by two expert teams from the RMIT University and the manufacturer.

4.2. Supplier Self-Assessment

The self-assessment by participating suppliers was a separate assessment procedure (and not an integral part of the assessment method described in this paper), followed in order to understand the extent to which the suppliers perceive their knowledge against the reference capabilities. These assessments were conducted by the companies’ senior and middle managers, mainly responsible for or involved in maintaining the environmental standards of the participating companies (e.g., ISO 14000 family of standards). A lack of awareness about existing capability gaps can limit companies from improving their practices and moving toward a more sustainable business and meeting the environmental targets set by the manufacturer.

The suppliers were provided with a simple self-assessment questionnaire, including 18 multiple choice questions designed to address the applied outcomes specified in the SMF (Figure 1). The core capabilities and the associated evidence of attainment were indirectly used to design-in the multiple choice answers. The five choices for each question were linked to the rating scores of one to five in order. The suppliers were given access to this assessment questionnaire through a SurveyMonkey online link [39]. Below is an example of self-assessment questions relating to “reduce energy use” applied outcome under the “energy efficiency” cluster:

Question: Which of the following describes best your company’s capability to measure, record, and report energy use and develop options to reduce it?

Answers:

• We do not measure, record, and report the energy use for the factory (linked to score 1).
• We measure, record, and report the total energy use for the factory (linked to score 2).
• We measure, record, and report the total energy use for the factory as well as the energy use of operations/machines (linked to score 3).
• We measure, record, and report the total energy use of the factory as well as the energy use of operations/machines. The energy use data are used to relate the use in operations to the total use in the factory and identify high use operations (linked to score 4).
• We measure, record, and report the total energy use for the factory as well as the energy use of operations/machines. These data are used to relate use in operations to the total use in the factory and to identify high use operations. There is at least one demonstrable example of energy efficiency improvement within the factory implemented through following such an approach (linked to score 5).

The self-assessment results are presented in Figure 5. As expected, these results did not generally show sufficient reliability in identifying the suppliers’ capability gaps. “Carbon emissions” was the only cluster showing relatively-close agreement between the suppliers’ self-assessment and the assessment done based on CMAT by the expert teams from RMIT and the manufacturer (except Supplier B). Capabilities in other clusters were generally overestimated according to the suppliers’ self-assessment results. Such discrepancies between assessment results were not unexpected as an existing lack of suppliers’ knowledge in a particular cluster may not place well that supplier in critically assessing their capabilities across that cluster. Note that almost the same ranking orders for the suppliers in terms of their capabilities are suggested by the RMIT/manufacturer assessment teams, and by the suppliers’ self-assessment results (Figure 5).

Figure 5. The same order in supplier rankings based on their sustainability performance is suggested by the RMIT/manufacturer assessment teams using CMAT, and by the suppliers’ self-assessment using a simple short questionnaire. (a) RMIT/Manufacturer assessment results; (b) Supplier self-assessment results; (c) The average of clusters’ scores assessed by RMIT/Manufacturer team and self-assessed by the A–E participating suppliers.

Figure 5. The same order in supplier rankings based on their sustainability performance is suggested by the RMIT/manufacturer assessment teams using CMAT, and by the suppliers’ self-assessment using a simple short questionnaire. (a) RMIT/Manufacturer assessment results; (b) Supplier self-assessment results; (c) The average of clusters’ scores assessed by RMIT/Manufacturer team and self-assessed by the A–E participating suppliers.

4.3. Results Discussion

The inherent interconnections between the assessment results of different clusters can be used to evaluate the validity of the results and comment on the reliability of the assessment tool. Some of the main links between the clusters are discussed below.

4.3.1. Connection between Three Groups of Clusters

It is expected that a lack of capability in “improving environmental management practices” (clusters 6, 7 and 8 in Figure 1) is reflected in the lack of capability in using energy, water, and material resources efficiently (clusters 1, 2 and 3) and in achieving emissions reduction. One should be skeptical about the reliability of the assessment tool if the results show that a supplier with poor performance in capabilities relating to “improved management practices” has obtained a high score in that relating to resource efficiency and emission reduction. This also agrees with the generally-accepted definition of manufacturing environmental management as management practices to minimize the negative environmental impacts of the manufacturing activities [25]. In searching for such a connection, the average scores for clusters under resource efficiency, emissions reduction, and improved management practices (three main cluster categories as shown in Figure 1) were calculated for all five suppliers that participated in this study (Figure 6). Although different capabilities were assessed under each of these three categories of management, the average scores of the categories follow almost the same ranking order for the five participating suppliers. In other words, the supplier with the higher average score in the “improved management practices” capability cluster also obtained higher average scores in the other two cluster categories. Interestingly, for each individual supplier, the average scores for these three groups of clusters are all approximately the same with a maximum difference of 0.6 only (for Supplier A) and almost the same scores for Supplier C (2.42, 2.42, and 2.46). A similar link was also identified by Wong et al. [4] in studying 122 manufacturing firms investigating the role of environmental management capabilities in achieving green operation (GO) at the manufacturing level; Lo et al. [17] also studied the role for proper environmental management systems (EMSs) in controlling the environmental performance of manufacturing firms, particularly in developing countries where the implementation of such EMSs faces challenges such as lack of appropriate infrastructure, efficient policies and effective environmental regulations, and financial resources [40].

Figure 6. Average scores in resource efficiency, emissions reduction, and improved management practices cluster categories for the five participating suppliers.

Figure 6. Average scores in resource efficiency, emissions reduction, and improved management practices cluster categories for the five participating suppliers.

4.3.2. Integration of Energy Efficiency and Carbon Emission Clusters

Suggested energy management hierarchies for sustainability purposes have usually integrated carbon and energy-related capability, e.g., [41] also used by Smith and Ball [38], as energy can often serve as a proxy for carbon emissions, particularly for fossil-based energy systems. In fact, although some capabilities relating to “carbon emissions” and “energy efficiency” are connected, good capabilities in one cluster do not necessarily mean good capabilities in other clusters. In some instances, carbon emissions can be reduced without any reduction in energy consumption: for example, switching to photovoltaic electricity does not reduce energy consumption but can reduce carbon emissions. Furthermore, some manufacturers established targets based on separate energy and carbon emission indicators, e.g., [18]. For this reason, the integration of capabilities relating to carbon emissions and energy is considered not appropriate.

As shown in Table 4, the majority of capabilities in the carbon emissions cluster relate to those required to undertake different levels of carbon accounting which are different to those outlined under the energy efficiency cluster. However, as discussed, the existing links are reflected in the close scores for these two clusters as shown in Figure 7 for the five participating suppliers. Following Figure 7, the biggest gap between the two clusters (0.4) was for supplier B; interestingly, this supplier utilized onsite electricity generation using diesel generators. Hence, they could potentially reduce their greenhouse gas emissions while keeping their electricity consumption constant.

Table 4. Capabilities included under the “energy efficiency” and “carbon emissions” clusters.

Cluster	Energy efficiency		Carbon Emission
Capabilities	Capabilities Under Reduce Energy Use Applied Outcome	Measure, record and report energy used in operations for factory	Capabilities Under Account for Carbon Emissions Applied Outcome	State the three main greenhouse gases for energy processes
		Compare energy use records for operations to previous records and industry best practice		Recognize other greenhouse gas emissions, sources (e.g., some refrigerant gases) and characterisation factors
		Relate energy used in operations to total energy consumption for factory		List or name global warming factors (CO2-equivalent factors) for main greenhouse gases
		Use ranking and identified energy losses to determine processes which can be improved		or produce a document which lists emission factors for different energy types (e.g., fuel oil, electricity)
		Document a flow chart showing energy flows for an operation in the factory (energy balance)		Calculate greenhouse gas (GHG) emissions from energy use and emission factors
		Apply continuous improvement		Report on GHG emissions for the factory
	Capabilities Under Maximize Alternative Energy Resources Applied Outcome	Determine alternative energy sources (e.g., heat from gas vs. heat from electricity, solar)		Compare GHG emissions to previous records
		Implement alternative energy sources		Differentiate between Scope 1, 2 and 3 emissions
		Operate alternative energy resource technology (s)	Capabilities Under Reduce Carbon Emissions Applied Outcome	Use GHG emissions and energy data to find operations contributing to GHG impacts
		Maintain alternative energy resource technology (s)		Determine options to reduce GHG emissions
				Recommend changes to reduce GHG emissions
		Apply continuous improvement		Implement changes to reduce GHG emissions

Table 4. Capabilities included under the “energy efficiency” and “carbon emissions” clusters.

Cluster	Energy efficiency		Carbon Emission
Capabilities	Capabilities Under Reduce Energy Use Applied Outcome	Measure, record and report energy used in operations for factory	Capabilities Under Account for Carbon Emissions Applied Outcome	State the three main greenhouse gases for energy processes
		Compare energy use records for operations to previous records and industry best practice		Recognize other greenhouse gas emissions, sources (e.g., some refrigerant gases) and characterisation factors
		Relate energy used in operations to total energy consumption for factory		List or name global warming factors (CO2-equivalent factors) for main greenhouse gases
		Use ranking and identified energy losses to determine processes which can be improved		or produce a document which lists emission factors for different energy types (e.g., fuel oil, electricity)
		Document a flow chart showing energy flows for an operation in the factory (energy balance)		Calculate greenhouse gas (GHG) emissions from energy use and emission factors
		Apply continuous improvement		Report on GHG emissions for the factory
	Capabilities Under Maximize Alternative Energy Resources Applied Outcome	Determine alternative energy sources (e.g., heat from gas vs. heat from electricity, solar)		Compare GHG emissions to previous records
		Implement alternative energy sources		Differentiate between Scope 1, 2 and 3 emissions
		Operate alternative energy resource technology (s)	Capabilities Under Reduce Carbon Emissions Applied Outcome	Use GHG emissions and energy data to find operations contributing to GHG impacts
		Maintain alternative energy resource technology (s)		Determine options to reduce GHG emissions
				Recommend changes to reduce GHG emissions
		Apply continuous improvement		Implement changes to reduce GHG emissions

Figure 7. Close scores for energy efficiency and carbon emissions clusters obtained by all of the five suppliers that participated in the study.

Figure 7. Close scores for energy efficiency and carbon emissions clusters obtained by all of the five suppliers that participated in the study.

5. Conclusion

The paper presented a comprehensive Sustainable Manufacturing Framework (SMF) that was developed in collaboration with a global sporting goods manufacturer. This tool was used to identify the key requirements for building and/or enhancing capabilities across the manufacturing supply chain in Asia in order to promote and secure the uptake of environmentally sustainable manufacturing practices. The framework was designed to particularly target the key elements of resource management, emissions management and environmental management to help the manufacturer reduce their environmental footprint and meet their strategic measurable environmental targets. This framework was used as a platform to develop the Capability Metrics Assessment Tool (CMAT). The CMAT was used by authors to assess a select group of five tier one and tier two sports apparel and footwear suppliers. The assessment results obtained in this research identified distinctive capability gaps, characterized by the lack of capability to:

• Identify the resource/emissions intensive processes (or to conduct a “Hot-Spot” Analysis of their manufacturing operations);
• Identify the reasons for this intensity (or to conduct a “Root-Cause” Analysis);
• Evaluate the costs and benefits associated with environmental improvement options (or to conduct a “Full-Cost” analysis);
• Decide on a sustainable option using multi-criteria assessment (or to conduct a “Multi-Criteria Decision-Making” process).

Five suppliers were assessed by two expert assessment teams from the university and the manufacturer using four independent assessment methods; the differences between the obtained scores were mainly less than 1. Such small differences between the capability assessment scores led to the conclusion that 93% cluster-based score agreements between the two assessment teams were achieved if the agreement is defined as a score difference of 1 or less. In only three cases (highlighted in grey in Table 3) the overall scores of the clusters showed differences of slightly above 1. By reviewing the cluster-based scores obtained by the two assessment teams, it is apparent that there is an agreement on almost the same scores in 60% of the clusters assessed for the same group of suppliers (Figure 4). There are also some important connections between the scores of different clusters which confirmed the robustness and the reliability of the assessment tool and the methodology used. The assessment results for all the participating companies illustrated that those with poor performance in environmental management practices could not use their energy, water, and material resources efficiently in order to reduce their environmental footprint (e.g., waste, chemicals, and carbon emissions) effectively (Figure 6). Furthermore, the results clearly demonstrated links between the “Carbon Emissions” and “Energy Efficiency” capability clusters. Companies with low scores in the latter received low scores in capabilities relating to their ability to reduce their carbon emissions (Figure 7). Finally, the suppliers completed a self-assessment against the same set of capabilities. Whilst the self-assessment results did not agree with those determined by the expert teams from the university and the manufacturer both set of assessments resulted in the same ranking order of the suppliers (Figure 5).

In conclusion, the key focus in this paper was on the analysis of the obtained assessment results in order to determine the robustness of the developed CMAT tool. This was important in order to demonstrate how reliably the tool can be applied to assess a greater number of suppliers across the region and globally.