Using a DEA–AutoML Approach to Track SDG Achievements

Abstract

Each country needs to monitor progress on their Sustainable Development Goals (SDGs) to develop strategies that meet the expectations of the United Nations. Data envelope analysis (DEA) can help identify best practices for SDGs by setting goals to compete against. Automated machine learning (AutoML) simplifies machine learning for researchers who need less time and manpower to predict future situations. This work introduces an integrative method that integrates DEA and AutoML to assess and predict performance in SDGs. There are two experiments with different data properties in their interval and correlation to demonstrate the approach. Three prediction targets are set to measure performance in the regression, classification, and multi-target regression algorithms. The back-propagation neural network (BPNN) is used to validate the outputs of the AutoML. As a result, AutoML can outperform BPNN for regression and classification prediction problems. Low standard deviation (SD) data result in poor prediction performance for the BPNN, but does not have a significant impact on AutoML. Highly correlated data result in a higher accuracy, but does not significantly affect the R-squared values between the actual and predicted values. This integrative approach can accurately predict the projected outputs, which can be used as national goals to transform an inefficient country into an efficient country.

1. Introduction

Sustainable Development Goals (SDGs) are a global plan for developing sustainability in order to eradicate poverty while preserving the environment and quality of life of all living things around the world without leaving anyone behind. According to the 2019 Sustainable Development Report, none of the countries is currently pursuing a path to achieve the 17 goals [1]. Therefore, each country needs to monitor its progress in order to develop and promote strategies that meet the expectations of the SDGs [2]. Benchmarking, by setting goals to compete with, can help identify best practices and then apply them reliably to improve a country’s performance towards reaching their SDGs. Data envelope analysis (DEA) is a benchmarking technique in which the optimal combinations of effort (input) and performance (output) result from a linear optimization problem. DEA uses the data from each decision-making unit (DMU) involved in the efficiency assessment to establish an efficient frontier and then determines the positions of each DMU with respect to the frontier. Every country must set future goals in order to achieve their SDGs; however, DEA lacks predictive capability [3,4,5,6]. Some studies have tried to integrate DEA with other methods, such as machine learning, to predict future situations.

Machine learning is a field of artificial intelligence that applies statistical techniques to give computers the ability to learn from data without being explicitly programmed. It is easy to use and understand compared to statistical methods; however, a data scientist needs to understand the perspective of a problem, address the data situation, preprocess the data accordingly, and arrive at a model. The prediction capability limitation of DEA can be fulfilled by the machine learning capability [7,8]; however, designing a good machine learning framework is a crucial step and requires an expert [9,10,11,12]. Automated machine learning (AutoML), one of the most helpful automation techniques, simplifies machine learning by automating various machine learning processes. The aim is to provide machine learning problem solving opportunities for users who are looking for a way to reduce their time and manpower requirements. It helps data scientists make better use of machine learning in their work [13]. AutoML is an emerging research field; however, more studies need to be done to ensure the more widespread application of this technique, which is possible with the increasing availability of open data [14]. We therefore urgently need to use this tool in order to gain insights from this rapidly growing data base.

No individual technique can solve all problems. DEA is used in many studies as a method of evaluating performance, but it is unable to forecast the future efficiency frontier. AutoML is a powerful forecasting tool and can be helpful in choosing the right algorithm; however, it lacks the ability to optimize. Many studies have combined DEA with machine learning to improve the performance of both these decision-support tools. Although this integrative approach can evade the limitations of its individual methods and provide more detailed performance, there are still some gaps to expand the approach. None of the previous studies combined DEA and AutoML approaches to measure efficiency in the SDGs area. As a final refinement in 2020, at the 51st session of the United Nations Statistical Commission, there are now 169 goals and 231 unique indicators for the 17 SDGs, which are expected to be achieved by 2030 [15]. Although concrete goals have not yet been formulated or decided, the principles, on the basis of which the thematic priorities of the sustainable goals have been determined, have already been agreed to [16].

This work shows a method for assessing and predicting the performance of SDGs using an integrative method between DEA and AutoML. A stratification DEA approach was applied in this study to benchmark and evaluate the performance of each country. Then, an AutoML method was processed to predict its efficiency score (ES), efficiency tier (ET), and projected output (PO) in the future based on its historical performance. The experiment regarding the ES is to test the machine learning regression algorithm; for the ET, to test the classification algorithm; and for the PO, to test the regression problem with multiple outputs. There are two empirical studies to demonstrate the integration between DEA and AutoML, with various data properties. The first study consists of long-term and highly correlated data used to measure the annual goals for the sustainable prosperity of a global development strategy called the Belt and Road Initiative (BRI) project. The second study includes short-term and less correlated data, measuring the weekly performance in fighting the COVID-19 pandemic around the world. This integrative approach can efficiently perform both analysis and prediction in these studies, with a faster build and more flexible customization, as well as being user-friendly to everyone.

2. Materials and Methods

In this study, an integrative approach is presented in which DEA and AutoML are integrated to predict ES, ET, and PO, and to support the achievement of the SDGs. The integrative approach consists of four main steps, as shown in Figure 1:

• implementing a stratification DEA to evaluate the ES and ET, and to examine the PO that will turn an inefficient DMU into an efficient DMU;
• applying AutoML to predict the three outcomes of the DEA using classification algorithms to determine the ET and regression algorithms to predict the ES and PO;
• using a back-propagation neural network (BPNN) to produce a series of the same results to validate the AutoML outputs by comparing the rates of precision and accuracy along with the number of DMUs within an acceptable percentage error (PRED);
• employing DEA by using the predicted PO from the AutoML and BPNN to evaluate the ES and compare them to validate the integrative approach.

2.1. Stratification DEA

DEA is an efficiency analysis technique that is widely used in industry to compare the efficiency of organizational units. Charnes et al. [17] developed a basic DEA model in 1978, called the CCR model after the initials of its developers, which accepts the assumption of constant returns to scale (CRS). Banker et al. [18] introduced the BCC model in 1984, which accepts the assumption of variable returns to scale (VRS). Although DEA users can consider multiple inputs and outputs, only similar DMUs are considered in a benchmark, and all DMUs must have the same inputs and outputs to ensure that applying the DEA produces a significant result. These factors are often not comparable; therefore, the inputs and outputs are multiplied by meaningful weights. A feature of the DEA, compared to other efficiency analysis techniques, is that the weights of the inputs and outputs are determined in the model without user intervention. This work used the BCC output-oriented model to measure efficiency because the DMUs of different sizes can be processed.

Barr et al. [19] proposed the idea of peeling an onion to layer and classify DMUs using a stratification DEA. Stratification is an arrangement in superimposed layers and classes. It is a process of dividing the general population into homogeneous subgroups prior to operation, and separating data from different sources to analyze patterns. Stratification can bring advantages in statistics if the population can be divided into meaningful groups called strata [20,21]. Meaningful means that the layer is relatively homogeneous with regard to one or more properties, which also influences the expression of the property that is ultimately interesting and that differs from one another as clearly as possible. When the subpopulation in a data analysis varies in a general population, it may be advantageous to select data within each subpopulation or stratum. The stratum must be mutually exclusive, i.e., each element of the population is assigned to a unique layer; furthermore, none of the general population elements can be left out. The strata of the stratified dataset are expediently selected such that they are essentially homogeneous in terms of the characteristics relevant for the selection of the strata of the population and differ from one another in terms of the expected values in order to differentiate these characteristics as much as possible. This stratification technique was integrated into DEA as a classification tool in many works [22,23,24,25,26,27]. This work therefore used this technique in combination with DEA to improve the analysis of the results.

2.2. Automated Machine Learning

Machine learning requires a programmed mechanism full of algorithms, each of which can be adjusted more precisely with so-called hyperparameters. An important task of a data scientist is to find the right algorithm for the respective problem and to set it correctly. In fact, the area of responsibility is considerably larger. In a typical machine learning application, users must use the appropriate methods for data preprocessing, feature extraction, and feature selection in order to make datasets usable for machine learning. Following these preprocessing steps, users then need to perform algorithm selection and hyperparameter optimization in order to maximize the predictive performance of the machine learning model. Since many of these steps require time-consuming and human skills on the part of the user, AutoML was designed as an artificial intelligence-based solution to meet this challenge. AutoML simplifies and accelerates the machine learning workflow and enables data scientists with limited specific programming skills to create machine learning systems. As a result, it is no longer just the expertise of data scientists and programmers to create their own machine learning systems. Even with little expertise, a high-quality machine learning model can be trained and deployed using powerful AutoML tools. Not only will the field of activity of data scientists no longer exist, but their focus will shift to more specialized or sophisticated analysis techniques. In a complex machine learning process with difficult problem-solving tasks; however, not all steps can be automated [12,28,29]. Tasks like a basic understanding of the problem or complex data engineering tasks often still need to be performed by people.

As with traditional manual machine learning, AutoML can be used in many areas to create, train, optimize, and deploy machine learning models to solve problems. Typical tasks that are performed with the models implemented with AutoML are classification tasks. For example, it was used in the health sector [28,30,31], the corporate sector [32], the environmental sector [33,34], the energy sector [35,36], and others [37,38,39]. There are many AutoML tools and solutions available today to help data scientists. While some of them are intended for local use, others allow use over cloud-based platforms. Commercial software solutions such as Google Cloud AutoML and Azure Automated ML, which focus on business applications, enable the use of various AutoML services through a cloud-based platform.

There are also open source tools that use standard machine learning algorithms, such as Auto-WEKA [40], auto-sklearn [41], and TPOT [42], or tools that apply deep learning algorithms like Mcfly [43], which focuses on multivariate times series classification, or auto-Keras [44], which focuses on image classification. Olson et al. [45] introduced a tree-based pipeline optimization tool called Tree-based Pipeline Optimization Tool (TPOT) to automate pipeline design. Optimization models from TPOT use genetic programming, an optimization technique inspired by the mechanism of biological evolution in a population that works by selecting the individuals best suited to a constraint, by crossing over and adding random variations. The individuals of the population in this case are models defined by trees that contain only the most efficient pipelines in a development. During development, the population of trees is evolved by changing existing nodes, adding new nodes, or deleting them. This tool is one of the most popular open source AutoML systems because it can automatically build machine learning pipelines with satisfactory accuracy [13]. Balaji and Allen [46] compared three open source AutoML solutions and found that auto-sklearn is best for classification but worst for regression, while TPOT is best for regression and the second for classification problems. In contrast to auto-sklearn, TPOT offers a white box operation because the Python code of the best model is generated in the form of a scikit-learn pipeline. TPOT was chosen as the AutoML solution in this work because it can handle both regression and classification problems well and the structure of the pipeline generated in TPOT is described by a general tree-based pipeline.

2.3. Integration Between DEA and Machine Learning

Zhou et al. [47] showed that the trend towards combining DEA with other analytical methods, such as a neural network (NN), is increasing significantly and that future research should focus more on such combinations in order to improve the accuracy of the analyses and the explanation of the results. The integration between DEA and NN was proposed in a bank branch performance evaluation study to overcome the drawbacks and promote the benefits of both methods [48]. The approach was used not only for performance evaluation and prediction, but also for optimization purposes. As a frontier method, DEA also calculates the PO, which turns an inefficient DMU into an efficient DMU. An NN predicts efficient frontiers instead of an ES, and then the PO is predicted. The efficient frontier indicates the number of efficient DMUs, which accordingly have an ES of 1 or 100%. We adapted this concept by integrating DEA and AutoML instead of NN to perform these tasks and fill the previously mentioned gap. The previous integrative frameworks can be summarized in six categories:

• first evaluate the ES with DEA and then predict the ES with an NN [49,50,51];
• first forecast the inputs and outputs with the NN and then evaluate the future ES with DEA [4,6];
• first evaluate the ES using DEA, then manually group the DMUs based on the ES ranges, and finally predict the groups using an NN [52];
• first evaluate the ES and PO with DEA and then predict the ES and PO with an NN [3,53,54,55];
• first evaluate the ET and PO using a stratification DEA and then predict the ET and PO with an NN [5];
• first evaluate the ET using a stratification DEA to preprocess the NN learning datasets and then predict groups using each stratified learning datasets with an NN [56].

This work combined the fourth and fifth framework to assess and predict the ES, ET and PO. DEA was used as a first step in evaluating the ES to understand the current state of the DMUs and identifying the PO to establish the goal of improving the performance of an inefficient DMU. Original inputs and outputs were used in this step. The second step was to follow the concept proposed by Barr et al. [19] and the stratification technique proposed by Seiford and Zhu [20] to determine the ET of each country based on their performance. ET1 was accredited when the assessed countries achieved a 100% efficiency in the first round. These countries were then removed from the analysis and the DEA model re-executed. This process was repeated and stopped when the DEA’s rule of thumb was met. The rule of thumb was established to ensure that the DEA model can produce a result with a good discrimination between efficient and inefficient DMUs. The number of DMUs should be more than twice the total number of input and output variables to improve the discriminatory power [57,58]. For example, if the total number of input and output variables is 20, the number of DMUs in the final round evaluation should be more than 40. The remaining inefficient DMUs are automatically assigned to tier n + 1 if n is the last lap number. We then employed BPNN and AutoML to predict three targets, including the ES, ET and PO, which were the main results of the DEA experiment. Original inputs and outputs were used, along with the ES and ET resulting from DEA, to predict the ES and ET. To predict the PO, which is the efficient frontier projection, only the original inputs and PO of the DEA experiment were used as the learning dataset. We used the root mean square error (RMSE), PRED, and the coefficient of determination (R-squared) to assess the performance of the regression algorithm, and the precision and accuracy rates to evaluate the performance of the classification algorithm. Finally, DEA was employed to assess the ES using the original inputs and PO predicted from the BPNN and AutoML. Normally, the ES calculated from the projection frontier should be 100%, since the projection frontier is the optimal value for inputs and outputs. In this step, the multi-target regression algorithm was validated. We also applied the distance formula using Pythagorean theorem to find the distance between the efficiency points of a DMU and the maximum efficiency, which is 100% in each SDG. This allowed us to compare the performance of all DMUs in terms of the overall perspective.

2.4. Data Description of the Sustainability Study in the BRI Region

The Belt and Road Initiative (BRI), originally called One Belt One Road, was presented by the Chinese government in fall 2013 [59]. The entire project now affects more than 60% of the world’s population, around 35% of the global economy, and deals with around half of the global energy consumption and carbon emissions [60]. It is therefore important to focus on the progress of sustainability development in this region. An empirical study of this work aims at the area of prosperity, which consists of SDGs 7–11 [61,62]; however, only SDG7, SDG8, and SDG9 focus on the area of prosperity and relate to the energy, economic, and infrastructure developments [63] that correlate with the strategy of the BRI [64]. Lim et al. [65] also stated that SDG7, SDG8, and SDG9 are important to support overall growth, especially in economic terms.

The promotion of sustainable growth in the green economy and the creation of decent jobs while respecting human rights and global borders are important for all developing, emerging, and industrialized countries. SDG7 is a guarantee for affordable, reliable, sustainable, and modern access to energy. SDG8 includes sub-goals for economic growth, productivity growth, and decent job creation. It therefore calls for a global improvement in resource efficiency in consumption and production and strives to decouple economic growth from environmental destruction. Investing in scientific and technological research on sustainable infrastructure can stimulate economic growth, create jobs, and promote prosperity. SDG9 is about building resilient infrastructure and promoting sustainable industrialization and innovation. Research on energy efficiency and economic growth, as well as investments in energy infrastructure and clean energy technologies, should be encouraged.

This empirical study includes a performance assessment of the BRI countries in SDG7, SDG8, and SDG9 to find the best practices for improving performance development in the fields of energy, economics, and infrastructure. Based on the data available from 2011 to 2017, 80 countries in the BRI region were included in this sustainability study. The inputs and outputs of each SDG were extracted from the denominators and numerators of the indicators. Lamy and Millet [66] expressed energy efficiency as ratios depending on the measured aspect, and the authors also extracted the inputs and outputs from the denominators and numerators of the ratios. All indicators are defined by the United Nations and

Table 1. Description of the indicators in SDG7, SDG8, and SDG9.

Indicator	Extracted Denominator	Extracted Numerator
(7.1.1 *) Proportion of population with access to electricity	Population	Electricity access
(7.1.2 *) Proportion of population with primary reliance on clean fuels and technology	Population	Clean fuel access
(7.2.1 *) Renewable energy share in the total final energy consumption	Total energy consumption	Renewable energy
(7.3.1 *) Energy intensity measured in terms of primary energy and GDP	GDP	Total energy consumption
(7.a.1) International financial flows to developing countries in support of clean energy research and development and renewable energy production, including in hybrid systems	N/A	Clean energy finance flows
(7.b.1) Installed renewable energy-generating capacity in developing countries (in watts per capita)	Population	Electricity generated by renewable resources
(8.1.1 *) Annual growth rate of real GDP per capita	Population	GDP
(8.2.1 *) Annual growth rate of real GDP per employed person	Employment	GDP
(8.3.1) Proportion of informal employment in total employment, by sector and sex	Employment	Informal employment
(8.4.1) Material footprint, material footprint per capita, and material footprint per GDP	GDP	Material footprint
(8.4.2 *) Domestic material consumption, domestic material consumption per capita, and domestic material consumption per GDP	Population	Domestic material consumption
(8.5.1) Average hourly earnings of female and male employees, by occupation, age and persons with disabilities	Total working hours	Total income
(8.5.2 *) Unemployment rate, by sex, age and persons with disabilities	Labor force	Unemployment
(8.6.1) Proportion of youth (aged 15–24 years) not in education, employment or training	Youth population	Uneducated or unemployed youth
(8.7.1) Proportion and number of children aged 5–17 years engaged in child labor, by sex and age	Child population	Child employment
(8.8.1) Frequency rates of fatal and non-fatal occupational injuries, by sex and migrant status	Employment	Non-fatal occupational injuries and illnesses
(8.8.2) Level of national compliance with labor rights (freedom of association and collective bargaining) based on International Labour Organization (ILO) textual sources and national legislation, by sex and migrant status	N/A	National compliance with labor rights
(8.9.1) Tourism direct GDP as a proportion of total GDP and in growth rate	GDP	Tourism value added
(8.10.1 *) (a) Number of commercial bank branches per 100,000 adults and (b) number of automated teller machines (ATMs) per 100,000 adults	Adult population	Number of commercial bank branches
(8.10.2) Proportion of adults (15 years and older) with an account at a bank or other financial institution or with a mobile-money-service provider	Adult population	Adults with an account at a financial institution
(8.a.1) Aid for Trade commitments and disbursements	N/A	Commitments for aid for trade
(8.b.1) Existence of a developed and operationalized national strategy for youth employment, as a distinct strategy or as part of a national employment strategy	N/A	Number of national strategies for youth employment
(9.1.1) Proportion of the rural population who live within 2 km of an all-season road	Total rural population	Rural population who live within 2 km of an all-season road
(9.1.2 *) Passenger and freight volumes, by mode of transport	Population	Air, road or railways passengers or goods transferred
(9.2.1 *) Manufacturing value added as a proportion of GDP and per capita	Population	Manufacturing value added
(9.2.2 *) Manufacturing employment as a proportion of total employment	Employment	Manufacturing employment
(9.3.1) Proportion of small-scale industries in total industry value added	Industry value added	Small-scale industries value added
(9.3.2) Proportion of small-scale industries with a loan or line of credit	Number of small-scale industries	Small-scale industries with loan or line of credit
(9.4.1 *) CO2 emission per unit of value added	Gross value added	CO2 emissions
(9.5.1) Research and development expenditure as a proportion of GDP	GDP	Research and development expenditure
(9.5.2) Researchers (in full-time equivalent) per million inhabitants	Population	Researchers
(9.a.1) Total official international support (official development assistance plus other official flows) to infrastructure	N/A	Total official flows for infrastructure
(9.b.1) Proportion of medium and high-tech industry value added in total value added	Gross value added	Medium and high-tech industry value added
(9.c.1 *) Proportion of population covered by a mobile network, by technology	Population	Mobile phone subscriptions or Internet users

* Measured indicators based on the data available.

shows the description and variables extracted from the indicators in SDG7, SDG8, and SDG9. Based on the available data, we were able to measure some indicators in Targets 7.1–7.3, 8.1–8.2, 8.4–8.5, 8.10, 9.1–9.2, 9.4, and 9.c using the variables shown in

Table 2. Descriptive statistics of the variables in the sustainability study of the Belt and Road Initiative (BRI).

	Variables	Sources	Units	SDGs	Types	Mean	SD	Max	Min
1	Energy consumption	U.S. EIA	ktoe	7	Input	56,703	217,288	1,995,057	399
2	Population	World Bank	people	7, 8, 9	Input	48,895,906	155,696,380	1,386,395,000	416,268
3	Labor force	World Bank	people	8	Input	23,336,083	88,036,169	787,399,317	180,756
4	Adult population	World Bank	people	8	Input	36,366,959	126,249,267	1,137,712,323	354,703
5	Total employment	World Bank	people	8, 9	Input	22,120,921	84,072,312	752,753,746	169,227
6	Gross value added	UN	constant 2015 US$	9	Input	300,768,954,726	1,180,630,638,278	12,600,000,000,000	2,910,456,947
7	Access to clean fuels	World Bank	people	7	Output	27,966,632	90,083,247	816,996,879	278,423
8	Access to electricity	World Bank	people	7	Output	43,190,262	154,469,265	1,386,395,000	416,268
9	Renewable energy Consumption	U.S. EIA	ktoe	7	Output	6469	18,439	127,544	1
10	GDP	UN	constant 2015 US$	7, 8	Output	313,461,796,866	1,187,748,354,777	12,600,000,000,000	3,299,234,529
11	Domestic material consumption	World Bank	KG	8	Output	659,176,464,099	3,493,224,391,622	34,800,000,000,000	6,756,860,686
12	Unemployment	World Bank	people	8	Output (U)	1,215,027	4,032,204	36,101,083	10,040
13	Commercial bank branches	World Bank	branches	8	Output	4426	11,902	99,937	20
14	Air freight	World Bank	million ton-km	9	Output	904	3000	23,324	0.0001
15	Air passengers	World Bank	people	9	Output	14,766,277	48,542,713	551,234,509	899
16	Manufacturing employment	World Bank	people	9	Output	5,350,647	24,121,994	219,519,439	28,306
17	Manufacturing value added	UN	constant 2015 US$	9	Output	67,113,438,034	345,141,666,422	3,660,000,000,000	189,954,006
18	CO2 emissions	U.S. EIA	ktCO2	9	Output (U)	215,510	1,015,721	9,259,259	1,354
19	Internet users	World Bank	people	9	Output	19,199,110	73,370,618	752,812,485	113,488
20	Mobile phone subscriptions	World Bank	people	9	Output	49,323,520	146,141,851	1,469,882,500	478,860

. Energy efficiency is one of the main targets of SDG7, which is to double the global improvement in energy efficiency, ensure access to energy and affordability, and increase the share of renewable energies. The energy intensity of Indicator 7.3.1 is used to measure energy inefficiency; however, the aim of Target 7.3 is to improve energy efficiency. Energy intensity is defined as the energy supplied to the economy per unit of value of economic output. A lower ratio indicates that less energy is used to create an output unit. We therefore reversed it to calculate it as GDP per energy consumption for the evaluation of energy efficiency. The unemployment rate of Indicator 8.5.2 and the carbon emissions of Indicator 9.4.1 are undesirable outputs, and there are four methods to deal with undesirable outputs in DEA [67]. We used the multiplicative inverse transform to process these. In summary, there were two inputs and four outputs for SDG7, four inputs and outputs for SDG8, and three inputs and seven outputs for SDG9. The correlation matrix in

Table 3. Correlation matrix of the variables in the sustainability study of the BRI.

⁺⁺ Correlation is significant at the 0.01 level (2-tailed).

, which measured the R-squared, Spearman’s rank correlation coefficient (ρ), and statistical significance of all variables, shows that all variables were significantly correlated. In addition, access to electricity, the adult population, and manufacturing employment had the highest mean R-squared, and air freight, renewable energy consumption, and commercial bank branches had the lowest mean R-squared. Access to electricity, domestic material consumption, and manufacturing employment had the highest mean ρ, and the consumption of renewable energy, air freight, and passengers had the lowest mean ρ.

2.5. Data Description of the COVID-19 Pandemic Study

A UN report highlighted the need to learn from the current global health crisis called COVID-19 and to implement the SDGs more consistently and faster than before [68]. COVID-19 is currently the subject of intense research; however, there is limited literature using DEA in studies. Shirouyehzad et al. [69] evaluated the effectiveness of COVID-19 management of the outbreak response using DEA. The authors used population and International Health Regulations Core Capacity Scores as inputs and confirmed cases as undesirable outputs to measure the effectiveness of contagion control, and used confirmed cases, recovered cases, and deaths as the inputs, outputs, and undesirable outputs, respectively, to evaluate the effectiveness of the medical treatment. Ghasemi et al. [70] applied DEA to measure government performance during the COVID-19 outbreak. In the first step, the authors rated the effectiveness of preventing the spread of the coronavirus using population and population density as inputs and confirmed cases as the undesirable output. In this study, an index called the stringency index was used to analyze the prevention of spread. The authors rated the effectiveness of preventing coronavirus deaths in the second step, using the first step score, population, and population aged 65 and over as the inputs, and deaths as the undesirable output. The stringency index is one of the policy indices in the Oxford COVID-19 Government Response Tracker (OxCGRT) analysis, which shows how the infection situation is developing in different countries in relation to mitigation and containment measures [71]. In total, the researchers examined 17 indicators and aggregated the indicators under four parameters, including the overall government response index, the stringency index, the containment and health index, and the economic support index.

Good health is the goal and result of SDG3, which comprises a total of 28 indicators. In addition, nine indicators measure mortality rates and five indicators from Target 3.3 measure new and active cases caused by diseases. Half of the world’s population does not yet have access to basic health services [1]; therefore, government support in the health sector is included in SDG3. Since this empirical study is an assessment of management performance during the COVID-19 pandemic, we have adjusted all variables to focus only on the COVID-19 pandemic. In summary, there were three inputs including active cases, which were the weekly base load work, the containment and health index, which represented the base severity of the lockdown restrictions and closings in each country, and the newly confirmed cases, with one desirable output being active cases in the next three weeks, and two undesirable outputs being the newly recovered cases and new deaths in the next three weeks. The results of this part illustrate the effectiveness of the medical treatment for COVID-19.

SDG11 also targets security and sustainable communities. Target 11.5 points to the number of people affected by disasters; additionally, Target 11.b also focuses on policies and plans to manage and reduce disaster risk. These targets highlight issues related to public and potential disasters. The COVID-19 pandemic is a new type of infectious disease hitting the 21st century, affecting a wide range of people. To control the spread of COVID-19, many governments are applying social distancing and local lockdown restrictions. These restrictions affect the behavior of people in activities and travel [72,73]. Hadjidemetriou et al. [74] stressed that the reduction in human mobility had a significant impact on the confirmed COVID-19 deaths. Badr et al. [75] also found that the decline in mobility patterns correlated strongly with the slowdown in the growth rates of COVID-19 cases. Zhou et al. [76] mentioned that the extent of the impairment of mobility could help delay the peak number of transmission cases. Community mobility is therefore a key strategy for governments to deal with COVID-19. The Google COVID-19 Community Mobility Report shows a change in the mobility of people to categorized locations compared to base days. In this study, three places were categorized, namely, workplaces, residential places, and public places. The daily visits to public places were combined by people visiting four categorized locations, namely, retail and recreation, grocery and pharmacy, transit stations, and parks. Not only is the Google Mobility Report useful, the Google trend also shows during COVID-19. Pier et al. [77] evaluated Google searches for otolaryngology-related terms in the United States and found a significant change during COVID-19. Kurian et al. [78] also measured a correlation between COVID-19 cases and Google Trends in the United States and concluded that they were highly correlated. In addition, Google Trends could be used to monitor a new outbreak area. Husnayain et al. [79] analyzed search terms related to COVID-19, hand-washing, and face masks in Taiwan on Google and indicated that Google Trends could be used to monitor public nervousness about COVID-19. Effenberger et al. [80] reported that the public interest represented by Google Trends correlated with the number of new COVID-19 cases. In summary, there were four inputs, namely, days from the first confirmed case, new tests, trending inquiries about “covid” reflecting the awareness of people in each country, and the OxCGRT’s overall government response index, which served as the basis for the severity of government policies in each country, with three desirable outputs reflecting the results of the restrictions—daily visits to workplaces, residential places, and public places—and one undesirable output, which was newly confirmed cases representing the unwanted outcome of all responses from governments. This part shows a performance evaluation of the preparation for coping with the current infectious disease COVID-19.

The incubation period of COVID-19, which is the time between infection and the appearance of the first symptoms, averages five to six days; however, it can take up to two weeks for a symptom to appear [81]. The newly confirmed case dataset was therefore created with a delay of one week after the week evaluated. After an incubation period, fever, muscle pain, and a dry cough may occur. The disease often manifests itself in a general, severe feeling of illness and back pain [82]. The WHO–China Joint Mission on COVID-19 reported that the median time from first symptoms to recovery from mild cases is approximately two weeks [83]. This information was used to create a lag time for the newly recovered case dataset; therefore, the newly recovered case dataset was two weeks after the newly confirmed case dataset. Wang et al. [84] developed an algorithm to predict the days from hospitalization to death, and the result was 13 days. This result was used to create a delay time for the new death dataset; therefore, the new death dataset was also two weeks after the newly confirmed case dataset. In summary, the newly confirmed case dataset was collected from week t + 1, and the newly recovered case and newly death datasets were pooled from week t + 3 if the week evaluated was week t.

Based on data availability, 61 countries were assessed from 12 April to 23 May or from Weeks 16 to 21, 2020, using all of the variables listed in

Table 4. Descriptive statistics of the variables in the COVID-19 pandemic study.

	Variables	Sources	Units	SDGs	Types	Mean	SD	Max	Min
1	Active cases	Johns Hopkins University	cases	3	Input	26,611	122,361	1,171,805	23
2	Containment and health index	University of Oxford	score	3	Input	75	13	100	25
3	Days from the first confirmed case	Johns Hopkins University	days	11	Input	73	22	131	28
4	New COVID-19 tests	University of Oxford	cases	11	Input	103,594	284,720	2,651,203	839
5	Trending inquiries about “covid”	Google	score	11	Input	52	18	94	7
6	Government response index	University of Oxford	score	11	Input	73	12	96	21
7	Newly recovered cases at week t + 3	Johns Hopkins University	cases	3	Output	4907	11,706	92,863	2
8	New deaths at week t + 3	Johns Hopkins University	people	3	Output (U)	311	1143	12,452	0
9	Active cases at week t + 3	Johns Hopkins University	cases	3	Output (U)	34,848	156,540	1,402,525	5
10	Daily visits to workplaces	Google	score	11	Output	41	16	80	1
11	Daily visits to residential	Google	score	11	Output	18	8	41	2
12	Daily visits to public places	Google	score	11	Output	53	22	100	1
13	Newly confirmed COVID-19 cases at week t + 1	Johns Hopkins University	cases	3	Input	7140	23,867	210,512	0
				11	Output (U)

. Only Uganda was death-free during the study period. Although Taiwan was the fourth county to experience COVID-19 in this study, they had the lowest active cases at Week 24. Taiwan also had the lowest cumulative number of recovered cases, newly confirmed cases, and deaths during the investigation period when the country without deaths was excluded. Belarus had the lowest scores on the overall government response index and the containment and health index, followed by Taiwan. The United States had the highest active cases at Week 24. They also had the highest cumulative number of recovered cases, newly confirmed cases, and deaths during the period studied. In contrast, Costa Rica had the lowest cumulative COVID-19 tests in this study. Although El Salvador had the highest score on the overall government response index, they had the lowest trending inquiries about “covid”, which was a factor that reflected people’s interest. Restricted mobility to public places changed the most in Bolivia, while Panama was the country where mobility to workplaces changed the most, and second place after Singapore in mobility to residential places for all weeks. The correlation matrix in

Table 5. Correlation matrix of the variables in the COVID-19 pandemic study.

⁺⁺ Correlation is significant at the 0.01 level (2-tailed), ⁺ Correlation is significant at the 0.05 level (2-tailed), ⁻ Correlation is not significant.

shows that the variables in the COVID-19 pandemic study were less correlated than those in the BRI sustainability study. Only the newly confirmed case variable correlated significantly in rank with all variables. In addition, the newly confirmed cases, as well as active cases in week t + 3 and week t had the highest mean in both R-squared and ρ. Trend inquiries about “covid”, days from the first confirmed case, and daily visits to workplaces had the lowest mean value in both R-squared and ρ.

3. The BRI Sustainability Study

This section presents a study that compares and assesses sustainability progress in relation to SDG7, SDG8, and SDG9 in order to support policymakers in developing strategies to achieve the targets. The SDG7 performance assessment considered two inputs, namely energy consumption and the national population, and four outputs, namely access to clean fuels, access to electricity, renewable energy consumption, and GDP. For SDG8, we defined total workforce, adult population, and total employment as the inputs, GDP and domestic material consumption as the desirable outputs, and the number of unemployed as the undesirable output. National population, total employment, and gross value added were inputs; the number of air freight and passengers, employment, value added in manufacturing, internet users, and mobile phone subscriptions were the desirable outputs; and carbon emissions were an undesirable output in the performance assessment of SDG9. Annual national population was the variable that served as input for all SDGs, as people are the key factor for sustainability. Employment also played an important role in SDG8 and SDG9, showing that employment is essential for economic development. SDG7 and SDG8 also shared an important output, GDP, as these SDGs mainly focus on the area of prosperity.

3.1. Result of the DEA Experiment

We measured the effectiveness of SDG7, SDG8, and SDG9 from 2011 to 2017 in 80 BRI countries, and the results are shown in

Table A1. DEA experiment results in the sustainability study of the BRI.

	2011				2012				2013				2014				2015				2016				2017
	SDG7	SDG8	SDG9	Distance	SDG7	SDG8	SDG9	Distance	SDG7	SDG8	SDG9	Distance	SDG7	SDG8	SDG9	Distance	SDG7	SDG8	SDG9	Distance	SDG7	SDG8	SDG9	Distance	SDG7	SDG8	SDG9	Distance
AE	1.000	0.944	1.000	0.056	1.000	0.892	1.000	0.108	1.000	0.933	1.000	0.067	1.000	0.961	1.000	0.039	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
AO	1.000	0.271	0.386	0.953	1.000	0.272	0.381	0.956	1.000	0.301	0.364	0.945	1.000	0.297	0.375	0.941	1.000	0.302	0.336	0.964	1.000	0.303	0.332	0.965	1.000	0.316	0.330	0.957
AT *	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
AZ	1.000	0.281	0.953	0.720	1.000	0.287	0.905	0.719	1.000	0.299	1.000	0.701	1.000	0.306	1.000	0.694	1.000	0.304	1.000	0.696	1.000	0.306	1.000	0.694	1.000	0.314	1.000	0.686
BA	0.997	0.735	1.000	0.265	1.000	0.697	1.000	0.303	1.000	0.729	1.000	0.271	1.000	0.778	1.000	0.222	1.000	0.741	1.000	0.259	1.000	0.678	1.000	0.322	1.000	0.656	0.997	0.344
BD	1.000	0.216	0.949	0.785	1.000	0.223	0.963	0.778	1.000	0.224	0.986	0.776	1.000	0.234	1.000	0.766	1.000	0.266	1.000	0.734	1.000	0.296	1.000	0.704	1.000	0.307	1.000	0.693
BG	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	0.979	0.021	1.000	1.000	0.964	0.036	1.000	0.974	0.956	0.051	1.000	1.000	0.950	0.050
BJ	0.659	0.181	1.000	0.887	0.691	0.175	1.000	0.881	0.699	0.180	1.000	0.873	0.710	0.182	0.953	0.869	0.682	0.183	0.991	0.877	0.637	0.181	1.000	0.896	0.643	0.183	1.000	0.891
BO	0.925	0.493	0.896	0.523	0.921	0.542	0.901	0.475	0.904	0.580	0.842	0.459	0.913	0.646	0.901	0.378	0.915	0.669	0.946	0.345	0.918	0.763	0.818	0.310	0.918	0.825	0.818	0.265
BY	1.000	0.419	1.000	0.581	1.000	0.433	1.000	0.567	1.000	0.396	1.000	0.604	1.000	0.412	1.000	0.588	1.000	0.416	1.000	0.584	1.000	0.424	1.000	0.576	1.000	0.436	1.000	0.564
CG	0.929	0.128	0.921	0.879	0.955	0.128	0.941	0.875	0.915	0.124	0.893	0.886	0.934	0.130	0.866	0.883	0.932	0.131	0.893	0.878	1.000	0.131	0.912	0.873	1.000	0.132	0.900	0.874
CI	1.000	0.130	0.979	0.870	0.980	0.138	0.903	0.867	0.967	0.143	0.914	0.862	0.978	0.151	0.930	0.852	0.986	0.158	1.000	0.842	0.994	0.163	1.000	0.837	0.996	0.170	1.000	0.830
CL	1.000	1.000	0.900	0.100	1.000	1.000	0.887	0.113	1.000	1.000	0.878	0.122	1.000	1.000	0.877	0.124	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
CN *	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
CR	1.000	0.410	0.659	0.681	1.000	0.417	0.738	0.639	1.000	0.419	0.914	0.587	1.000	0.419	0.914	0.588	1.000	0.423	0.975	0.578	1.000	0.424	1.000	0.576	1.000	0.424	1.000	0.576
CY	1.000	0.792	0.749	0.326	1.000	0.733	0.793	0.338	1.000	0.556	0.873	0.462	1.000	0.543	0.875	0.473	1.000	0.529	0.867	0.490	1.000	0.521	0.915	0.486	1.000	0.790	0.949	0.216
CZ	1.000	0.686	1.000	0.314	1.000	0.668	1.000	0.332	1.000	0.671	1.000	0.329	1.000	0.695	1.000	0.305	1.000	0.704	1.000	0.296	1.000	0.697	1.000	0.303	1.000	0.706	1.000	0.294
DZ	0.992	0.391	1.000	0.610	0.990	0.414	1.000	0.587	0.996	0.411	1.000	0.589	1.000	0.437	1.000	0.563	0.999	0.448	1.000	0.552	1.000	0.454	1.000	0.546	1.000	0.468	1.000	0.532
EC	0.975	0.337	0.762	0.705	0.975	0.341	0.774	0.698	0.980	0.337	0.769	0.702	0.990	0.339	0.760	0.703	0.988	0.339	0.763	0.702	0.987	0.336	0.776	0.701	1.000	0.325	0.797	0.705
EE	1.000	0.851	1.000	0.149	1.000	0.871	1.000	0.129	1.000	0.944	1.000	0.056	1.000	0.958	1.000	0.042	1.000	0.925	1.000	0.075	1.000	0.952	1.000	0.048	1.000	0.986	1.000	0.014
EG	1.000	0.489	1.000	0.511	1.000	0.496	1.000	0.504	1.000	0.484	1.000	0.516	1.000	0.475	1.000	0.525	1.000	0.497	0.997	0.503	1.000	0.501	1.000	0.499	1.000	0.507	1.000	0.493
GE	0.996	0.358	0.889	0.651	1.000	0.437	0.937	0.567	0.999	0.511	0.962	0.490	1.000	0.505	0.980	0.496	1.000	0.504	1.000	0.496	1.000	0.552	1.000	0.448	1.000	0.566	1.000	0.434
GH	0.926	0.223	0.814	0.802	0.899	0.226	0.851	0.795	0.926	0.224	0.860	0.792	1.000	0.236	0.876	0.774	0.955	0.256	0.973	0.745	0.985	0.263	1.000	0.737	0.937	0.298	0.983	0.705
GQ	0.661	0.920	0.810	0.397	0.658	0.860	0.727	0.460	0.660	0.766	0.775	0.470	0.663	0.761	0.762	0.477	0.666	0.764	0.767	0.470	0.669	0.758	0.862	0.433	0.672	0.739	0.962	0.421
GR	1.000	0.835	0.878	0.205	1.000	0.906	0.959	0.102	1.000	0.842	1.000	0.158	1.000	0.791	1.000	0.209	1.000	0.751	0.935	0.257	1.000	0.713	0.941	0.293	1.000	0.692	0.933	0.315
HR	1.000	0.608	0.981	0.392	1.000	0.620	1.000	0.380	1.000	0.631	1.000	0.369	1.000	0.620	1.000	0.380	1.000	0.617	0.960	0.385	1.000	0.611	0.964	0.391	1.000	0.620	0.873	0.401
ID	1.000	0.555	1.000	0.445	1.000	0.623	1.000	0.377	1.000	0.657	1.000	0.343	1.000	0.697	1.000	0.303	1.000	0.770	1.000	0.230	1.000	0.817	1.000	0.183	1.000	0.824	1.000	0.176
IL	1.000	0.926	0.986	0.076	1.000	0.946	0.979	0.058	1.000	0.964	0.945	0.066	1.000	0.979	0.970	0.036	1.000	0.985	0.947	0.055	1.000	0.990	0.955	0.046	1.000	0.990	0.951	0.050
IQ	0.986	0.379	0.870	0.634	0.995	0.407	0.866	0.608	0.994	0.399	0.911	0.608	1.000	0.385	0.873	0.628	1.000	0.400	1.000	0.600	1.000	0.424	0.829	0.601	1.000	0.406	0.906	0.602
IR	0.992	1.000	1.000	0.008	0.995	1.000	1.000	0.005	0.997	1.000	1.000	0.003	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
IT *	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
JO	1.000	0.574	1.000	0.426	1.000	0.543	1.000	0.457	1.000	0.523	1.000	0.477	1.000	0.492	1.000	0.508	1.000	0.486	1.000	0.514	1.000	0.480	1.000	0.520	1.000	0.475	1.000	0.525
KE	0.929	0.173	0.752	0.866	0.974	0.177	0.775	0.853	0.978	0.174	0.783	0.855	0.952	0.178	0.736	0.865	0.915	0.185	0.815	0.840	0.994	0.186	0.816	0.834	0.996	0.189	0.826	0.830
KG	1.000	0.232	1.000	0.768	1.000	0.241	1.000	0.759	0.999	0.249	1.000	0.751	1.000	0.246	1.000	0.754	1.000	0.244	1.000	0.756	1.000	0.244	1.000	0.756	1.000	0.244	1.000	0.756
KH	0.857	0.180	1.000	0.833	0.859	0.186	1.000	0.826	0.866	0.183	1.000	0.828	0.860	0.198	1.000	0.814	0.846	0.211	1.000	0.804	0.879	0.233	1.000	0.776	0.946	0.250	1.000	0.752
KR	1.000	0.985	1.000	0.015	1.000	0.942	1.000	0.058	1.000	0.994	1.000	0.006	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
KZ	0.998	0.760	1.000	0.240	0.999	0.720	1.000	0.280	0.999	0.756	1.000	0.244	1.000	0.763	1.000	0.237	1.000	0.765	0.976	0.237	1.000	0.780	0.967	0.223	1.000	0.805	0.977	0.197
LA	0.942	0.519	0.794	0.527	0.954	0.523	0.689	0.571	0.936	0.519	0.708	0.566	0.933	0.508	0.671	0.596	0.980	0.500	0.551	0.673	0.971	0.479	0.508	0.717	0.962	0.500	0.466	0.732
LK	1.000	0.325	0.933	0.678	1.000	0.344	0.928	0.660	1.000	0.363	0.927	0.642	1.000	0.376	0.915	0.629	1.000	0.384	0.944	0.618	1.000	0.402	0.984	0.598	1.000	0.420	1.000	0.580
LT	1.000	0.448	1.000	0.552	1.000	0.456	1.000	0.544	1.000	0.455	1.000	0.545	1.000	0.458	0.997	0.542	1.000	0.450	0.982	0.550	1.000	0.448	0.979	0.552	1.000	0.465	1.000	0.535
LU *	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
LV	1.000	0.550	1.000	0.450	1.000	0.488	1.000	0.512	1.000	0.451	1.000	0.549	1.000	0.454	0.995	0.546	1.000	0.447	0.990	0.553	1.000	0.451	0.995	0.549	1.000	0.461	0.981	0.539
MA	1.000	0.465	1.000	0.535	1.000	0.506	1.000	0.494	1.000	0.547	1.000	0.453	1.000	0.574	1.000	0.426	1.000	0.603	1.000	0.397	1.000	0.639	1.000	0.361	1.000	0.689	1.000	0.311
MD	1.000	0.790	1.000	0.210	1.000	0.864	1.000	0.136	1.000	0.888	1.000	0.112	1.000	0.929	1.000	0.071	1.000	0.551	1.000	0.449	1.000	0.573	1.000	0.427	1.000	0.583	1.000	0.417
MM	1.000	0.166	1.000	0.834	1.000	0.154	1.000	0.846	0.987	0.161	1.000	0.839	0.946	0.173	1.000	0.829	0.944	0.174	1.000	0.828	0.938	0.172	1.000	0.831	0.860	0.190	1.000	0.822
MN	0.721	1.000	0.867	0.309	0.795	1.000	0.920	0.220	0.812	1.000	0.757	0.308	0.812	1.000	0.745	0.317	0.827	1.000	0.745	0.308	0.843	1.000	0.780	0.270	0.859	1.000	0.857	0.201
MT *	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
MY	0.996	0.653	1.000	0.347	0.998	0.586	1.000	0.414	0.999	0.597	1.000	0.403	1.000	0.599	1.000	0.401	1.000	0.602	1.000	0.398	1.000	0.608	1.000	0.392	1.000	0.620	1.000	0.380
MZ	0.819	0.133	0.777	0.913	0.882	0.123	0.745	0.921	0.905	0.125	0.887	0.887	0.857	0.134	1.000	0.877	0.832	0.138	1.000	0.878	0.694	0.143	0.959	0.911	0.641	0.147	0.650	0.989
NA	0.475	0.489	0.939	0.736	0.489	0.532	0.839	0.711	0.490	0.483	0.900	0.733	0.495	0.460	0.824	0.760	0.503	0.483	0.747	0.761	0.514	0.483	0.835	0.728	0.528	0.479	0.856	0.717
NG	1.000	0.265	0.728	0.784	1.000	0.297	0.790	0.734	1.000	0.292	0.842	0.725	1.000	0.305	0.877	0.706	1.000	0.294	1.000	0.706	1.000	0.283	0.886	0.726	1.000	0.274	1.000	0.726
NP	1.000	0.185	0.936	0.817	1.000	0.207	0.983	0.793	1.000	0.214	1.000	0.786	1.000	0.226	1.000	0.774	1.000	0.239	1.000	0.761	1.000	0.261	1.000	0.739	1.000	0.306	1.000	0.694
NZ	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	0.980	0.020	1.000	1.000	0.972	0.028
PA	0.920	0.391	1.000	0.615	0.922	0.412	1.000	0.593	0.928	0.439	1.000	0.566	0.941	0.445	1.000	0.558	0.952	0.453	1.000	0.550	0.970	0.458	1.000	0.543	1.000	0.464	0.866	0.552
PE	0.973	0.521	0.795	0.522	0.982	0.519	0.735	0.549	0.967	0.518	0.748	0.545	0.991	0.510	0.732	0.558	0.963	0.509	0.756	0.549	0.958	0.503	0.770	0.549	0.977	0.500	0.788	0.543
PH	1.000	0.273	0.991	0.727	1.000	0.266	1.000	0.734	1.000	0.257	1.000	0.743	1.000	0.266	1.000	0.734	1.000	0.276	1.000	0.724	1.000	0.286	1.000	0.714	1.000	0.294	1.000	0.706
PK	1.000	0.362	0.991	0.638	1.000	0.359	1.000	0.641	1.000	0.356	1.000	0.644	1.000	0.366	1.000	0.634	1.000	0.375	1.000	0.625	1.000	0.392	1.000	0.608	1.000	0.401	1.000	0.599
PL	1.000	0.899	1.000	0.101	1.000	0.892	1.000	0.108	1.000	0.868	1.000	0.132	1.000	0.896	1.000	0.104	1.000	0.896	1.000	0.104	1.000	0.926	1.000	0.074	1.000	0.933	1.000	0.067
PT	1.000	1.000	0.855	0.145	1.000	1.000	0.881	0.119	1.000	1.000	0.892	0.108	1.000	0.959	0.890	0.117	1.000	0.701	0.885	0.320	1.000	0.851	0.877	0.194	1.000	0.819	0.885	0.215
RO	1.000	0.656	0.984	0.344	1.000	0.603	0.912	0.407	1.000	0.667	0.920	0.342	1.000	0.671	0.923	0.338	1.000	0.651	0.923	0.358	1.000	0.679	0.948	0.325	1.000	0.681	0.929	0.327
RS	1.000	0.823	1.000	0.177	1.000	0.762	1.000	0.238	1.000	0.729	0.997	0.271	0.997	0.634	1.000	0.366	1.000	0.648	1.000	0.352	1.000	0.626	1.000	0.374	1.000	0.612	0.984	0.389
RU *	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
SA *	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000	1.000	1.000	1.000	0.000
SD	0.983	0.191	0.864	0.820	1.000	0.230	0.874	0.781	1.000	0.253	0.789	0.776	1.000	0.265	0.735	0.781	1.000	0.281	0.722	0.771	1.000	0.296	0.718	0.758	1.000	0.313	0.704	0.748
SI	1.000	0.666	0.939	0.340	1.000	0.651	0.944	0.354	1.000	0.604	1.000	0.396	1.000	0.588	0.985	0.412	1.000	0.584	1.000	0.416	1.000	0.580	1.000	0.420	1.000	0.582	1.000	0.418
SN	1.000	0.170	1.000	0.830	1.000	0.181	1.000	0.819	1.000	0.174	1.000	0.826	1.000	0.176	1.000	0.824	1.000	0.182	1.000	0.818	1.000	0.185	1.000	0.815	1.000	0.196	1.000	0.804
SR	0.922	0.910	1.000	0.119	0.931	0.926	1.000	0.102	0.937	0.912	1.000	0.108	0.943	0.938	1.000	0.084	0.951	1.000	1.000	0.049	0.960	1.000	1.000	0.040	0.968	1.000	1.000	0.032
SV	1.000	0.220	1.000	0.780	1.000	0.224	1.000	0.776	1.000	0.239	1.000	0.761	1.000	0.231	1.000	0.769	1.000	0.231	1.000	0.769	1.000	0.251	1.000	0.749	1.000	0.279	0.990	0.721
TG	0.925	0.230	1.000	0.774	1.000	0.227	1.000	0.773	0.992	0.233	1.000	0.767	1.000	0.226	1.000	0.774	0.986	0.227	1.000	0.773	0.991	0.216	1.000	0.784	0.959	0.215	1.000	0.786
TH	1.000	0.530	1.000	0.470	1.000	0.545	1.000	0.455	1.000	0.541	1.000	0.459	1.000	0.540	1.000	0.460	1.000	0.550	1.000	0.450	1.000	0.558	1.000	0.442	1.000	0.560	1.000	0.440
TN	1.000	0.487	1.000	0.513	1.000	0.495	1.000	0.505	1.000	0.497	1.000	0.503	1.000	0.507	1.000	0.493	1.000	0.526	1.000	0.474	1.000	0.537	1.000	0.463	1.000	0.575	1.000	0.425
TT	1.000	0.582	0.988	0.418	1.000	0.571	0.988	0.430	1.000	0.793	0.981	0.208	1.000	0.813	0.957	0.192	1.000	0.792	1.000	0.208	1.000	0.611	1.000	0.389	1.000	0.623	1.000	0.377
TZ	1.000	0.154	0.781	0.874	0.995	0.152	0.722	0.893	1.000	0.154	0.737	0.886	1.000	0.160	0.793	0.865	1.000	0.160	0.948	0.841	1.000	0.159	0.854	0.853	1.000	0.159	0.832	0.857
UA	1.000	0.486	1.000	0.514	0.999	0.428	1.000	0.572	1.000	0.454	1.000	0.546	1.000	0.455	1.000	0.545	1.000	0.450	1.000	0.550	1.000	0.453	1.000	0.547	1.000	0.458	1.000	0.542
UG	1.000	0.163	1.000	0.837	1.000	0.157	1.000	0.843	1.000	0.162	1.000	0.838	1.000	0.161	1.000	0.839	1.000	0.160	1.000	0.840	0.992	0.158	1.000	0.842	0.984	0.154	1.000	0.846
VE	0.989	0.475	0.795	0.564	0.992	0.523	0.778	0.526	0.994	0.521	0.807	0.517	0.996	0.500	0.816	0.533	0.998	0.471	0.889	0.540	1.000	0.429	0.806	0.603	1.000	0.435	0.938	0.568
VN	1.000	0.542	1.000	0.458	1.000	0.383	1.000	0.617	1.000	0.370	1.000	0.630	1.000	0.387	1.000	0.613	1.000	0.400	1.000	0.600	1.000	0.408	1.000	0.592	1.000	0.414	1.000	0.586
ZA	0.863	0.705	1.000	0.325	0.863	0.697	1.000	0.333	0.863	0.680	1.000	0.348	0.887	0.696	1.000	0.324	0.897	0.661	1.000	0.355	0.917	0.655	1.000	0.355	0.905	0.648	1.000	0.364
ZM	0.980	0.329	0.961	0.672	0.957	0.308	0.805	0.720	0.969	0.306	0.764	0.733	0.992	0.296	0.704	0.764	0.968	0.299	0.726	0.753	0.971	0.302	0.729	0.750	0.938	0.305	0.671	0.771
ZW	1.000	0.181	0.798	0.844	1.000	0.267	0.815	0.757	1.000	0.274	0.824	0.747	1.000	0.296	0.716	0.759	1.000	0.179	0.771	0.853	1.000	0.178	0.748	0.859	1.000	0.178	0.771	0.853

* Efficient DMUs in all studied periods.

in Appendix A. There were seven countries that performed efficiently in all periods studied, including Austria, China, Italy, Luxembourg, Malta, Russia, and Saudi Arabia. These countries were in the 30 countries with the highest GDP per capita in 2017. Six of these countries were also in 30 countries with the highest value added per employment in manufacturing and the highest energy consumption per capita. These results are consistent with the selection of inputs and outputs as the national population, employment, and GDP were the main drivers for SDG7, SDG8, and SDG9.

In the first step of the DEA experiment, we measured the ES of the first evaluation round. When compared over the years, the mean ES was relatively stable across all the SDGs, as shown in

Table 6. DEA experiment summary in the sustainability study of the BRI: (a) yearly efficiency score; (b) number of decision-making units (DMUs) in each efficient tier.

(a)									(b)
		2011	2012	2013	2014	2015	2016	2017			2011	2012	2013	2014	2015	2016	2017
SDG7	Mean	0.967	0.971	0.971	0.972	0.971	0.972	0.971	SDG7	ET1	52	53	52	57	56	58	62
	SD	0.088	0.082	0.080	0.080	0.080	0.085	0.086		ET2	24	22	23	18	18	16	13
	Max	1	1	1	1	1	1	1		ET3	4	5	5	5	6	6	5
	Min	0.475	0.489	0.490	0.495	0.503	0.514	0.528	SDG8	ET1	13	13	13	13	15	14	15
SDG8	Mean	0.555	0.555	0.559	0.562	0.556	0.560	0.570		ET2	15	12	16	16	16	13	15
	SD	0.300	0.295	0.295	0.295	0.289	0.289	0.288		ET3	14	18	13	18	16	21	17
	Max	1	1	1	1	1	1	1		ET4	19	17	18	14	13	12	13
	Min	0.128	0.123	0.124	0.130	0.131	0.131	0.132		ET5	10	13	13	14	14	14	13
SDG9	Mean	0.936	0.933	0.939	0.936	0.946	0.943	0.941		ET6	9	7	7	5	6	6	7
	SD	0.106	0.109	0.106	0.111	0.114	0.115	0.119	SDG9	ET1	43	46	49	46	49	50	47
	Max	1	1	1	1	1	1	1		ET2	33	31	28	32	29	28	29
	Min	0.386	0.381	0.364	0.375	0.336	0.332	0.330		ET3	4	3	3	2	2	2	4

a. Only SDG8 had the lowest mean and lowest minimum, but always the highest standard deviation (SD) in ES. The mean ES was around 55% and the minimum ES less than 15% each year. SDG8 mainly focuses on economic growth and labor productivity to alleviate poverty; therefore, the progress in economic development of each country in the BRI region seems to be quite small and very different. In addition, Luxembourg was one of the three largest GDP per capita countries in the world, while Mozambique was among the five lowest GDP per capita countries in the world, according to the United Nations in 2017. SDG7 had the highest mean and the highest minimum, but the lowest SD in ES. The mean ES was 97% and the minimum ES was around 50%. This means that most of the countries performed pretty well at providing people with clean and affordable energy and reducing their energy intensity. Although the mean ES in SDG9 was lower than in SDG7, it was still relatively high and satisfactory. The mean ES was 94% and the minimum ES was 36%. This shows that the BRI region considered sustainable industrialization, innovation, and infrastructure equally. In summary, the trends in the mean ES and the minimum ES in SDG7 and SDG8 showed increasing trends, while SD showed a decreasing trend, meaning that countries in the BRI improved performance to get closer. Although SDG9 showed an uptrend from the mean, the minimum showed a downtrend and SD showed a negative uptrend. These showed that the inequality in SDG9 was increasing.

We analyzed the sustainability performance of each country and, in a second step, grouped them based on their performance. In SDG7, SDG8, and SDG9, the total number of input and output variables were six, eight, and ten, so the number of DMUs to analyze in the final round should be greater than 12, 16, and 20, respectively, according to the DEA’s rule of thumb. As shown in Table 6b, all periods studied produced the same number of ET. The results of this ET assessment were comparable to those of the ES assessment because they shared the same characteristics. The number of DMUs in each ET was similar from year to year. SDG8 had the largest number of ET corresponding to six tiers; furthermore, the number of DMUs in each ET from ET 1–5 was quite comparable. Only ET5 contained half of the units compared to the other tiers. The trends in ET1, ET3, and ET5 increased, but ET4 and ET6 had downtrends while ET2 was stable. These showed the same characteristics in relation to ES, leading to the conclusion that SDG8 had the highest variance in ES and the trend of ES increased. SDG7 and SDG9 had the same number of ET, namely, three tiers. ET1 in SDG7 made up almost 70% of the total DMUs, ET2 had about a quarter, while ET3 covered less than 10%. Although the trends in ET1 and ET3 increased, ET2 showed a downward trend. The large number of DMUs in ET1 made the mean ES high, and the narrow distribution in ET2 and ET3 resulted in a low SD in SDG7. Around 60% of the DMUs in SDG9 were in ET1, less than 40% in ET2, and almost 5% in ET3. The proportion of DMUs in ET1 and ET2 was certainly not as different as in SDG7, and the proportion in ET3 was very low. This clearly showed that there were two classified groups in SDG9. While ET1 was in the uptrend, ET2 and ET3 built downtrends. The number of DMUs in ET1 of SDG9 was lower than in SDG7, so the mean ES of SDG9 was also lower than that of SDG7. The higher number of DMUs in ET2 and ET3 of SDG9 resulted in a higher SD than that of SDG7.

3.2. Result of the AutoML Experiment

This section shows an experiment predicting the ES, ET, and PO in 2017. We used the collected data and the results of the DEA from 2011 to 2016 to create the learning dataset for training and testing in the AutoML approach. We also employed BPNN to predict those targets using the same learning dataset and compare its performance against AutoML. Clementine software was used to perform traditional BPNN because it is widely recognized and used in many academic studies [85,86,87,88,89]. The ES and ET in 2017 were not that different from previous years, so the DEA’s results could be a capable learning dataset for BPNN and AutoML to get accurate results.

Table 7. Performance comparison of prediction methods in the sustainability study of the BRI: (a) regression evaluation metrics in the efficiency score prediction; (b) precision and accuracy in the efficient tier classification; (c) regression evaluation metrics in the frontier projection prediction.

(a)							(b)
	SDG7		SDG8		SDG9			SDG7		SDG8		SDG9
	BPNN	AutoML	BPNN	AutoML	BPNN	AutoML		BPNN	AutoML	BPNN	AutoML	BPNN	AutoML
Mean	0.977	0.973	0.508	0.543	0.940	0.935	ET1	62/80	58/59	8/12	14/14	38/61	42/46
SD	0.000	0.081	0.246	0.273	0.016	0.099	ET2		11/16	5/9	11/13	9/19	25/33
Max	0.978	1.000	1.000	1.000	0.998	1.000	ET3		4/5	13/49	15/21		1/1
Min	0.974	0.514	0.130	0.142	0.926	0.381	ET4			1/10	11/12
RMSE	0.086	0.026	0.163	0.047	0.116	0.052	ET5				13/14
PRED(0.20)	0.950	0.988	0.500	0.975	0.925	0.975	ET6				6/6
PRED(0.10)	0.925	0.975	0.300	0.863	0.863	0.875	Accuracy	0.775	0.913	0.338	0.875	0.588	0.850
PRED(0.05)	0.900	0.963	0.213	0.550	0.325	0.775
R2	0.008	0.909	0.725	0.983	0.058	0.821
(c)
		BPNN						AutoML
	Projected Outputs	RMSE		PRED (0.20)	PRED (0.10)	PRED (0.05)	R2	RMSE		PRED (0.20)	PRED (0.10)	PRED (0.05)	R2
SDG7	Clean fuels	29,508,671		0.088	0.025	0.000	0.903	4,571,957		0.900	0.850	0.613	1.000
	Electricity	40,248,413		0.175	0.125	0.050	0.937	2,589,071		0.988	0.938	0.813	1.000
	GDP	1,468,794,073,162		0.000	0.000	0.000	0.001	283,599,211,370		0.588	0.350	0.138	0.989
	Renewables	6792		0.125	0.100	0.050	0.872	1076		0.675	0.400	0.225	0.998
SDG8	GDP	1,535,253,602,486		0.000	0.000	0.000	0.010	256,121,857,571		0.825	0.563	0.263	0.997
	DMC	4,006,196,741,083		0.000	0.000	0.000	0.013	412,449,897,927		0.888	0.663	0.375	1.000
	Unemployment	8.96 × 10−6		0.300	0.150	0.075	0.866	5.50 × 10−6		0.763	0.575	0.413	0.948
	Bank branches	5516		0.300	0.188	0.138	0.854	1218		0.850	0.650	0.275	0.995
SDG9	Air freight	1609		0.063	0.025	0.013	0.878	925		0.363	0.138	0.050	0.979
	Air passengers	23,297,874		0.088	0.025	0.025	0.967	17,770,131		0.388	0.150	0.013	0.994
	Manufacturing employment	2,731,618		0.250	0.138	0.038	0.991	379,469		0.800	0.588	0.413	1.000
	Manufacturing value added	94,784,937,949		0.150	0.063	0.025	0.985	345,878,375,449		0.025	0.000	0.000	0.997
	CO2 emissions	0.0001		0.325	0.150	0.088	0.851	4.26 × 10−5		0.588	0.375	0.300	0.922
	Internet users	19,810,610		0.150	0.088	0.038	0.976	17,691,665		0.563	0.175	0.088	0.999
	Mobile phone	36,174,109		0.175	0.075	0.050	0.988	32,220,596		0.875	0.575	0.388	0.996

a shows the comparison of the regression evaluation in the ES prediction. The mean ES predicted by BPNN and AutoML were quite similar; however, the SD in the AutoML results were higher than in the BPNN. The maximum ES is normally 100%, but the maximum ES of the BPNN in SDG7 and SDG9 were lower. These two SDGs characteristically had a low SD in ES, so this could affect the algorithms. The minimum ES of BPNN in SDG7 and SDG9 obviously differed from the actual values, but both BPNN and AutoML were able to predict close to the actual minimum ES in SDG8. The RMSE was measured to compare the accuracy of the BPNN and AutoML; moreover, AutoML outperformed BPNN in all predictions. The RMSE of BPNN was the lowest in SDG7 and lower in SDG9 than in SDG8. Although the AutoML’s RMSE was also lowest in SDG7, it was higher in SDG9 than in SDG8. Not only RMSE, but also the number of DMUs with a percentage error less than 20% (PRED(0.20)), 10% (PRED(0.10)), and 5% (PRED(0.05)) from AutoML were higher than BPNN in all SDGs. PRED in SDG7 and SDG9 of both methods were quite similar with the exception of PRED (0.05) in SDG9. PRED and R-squared of BPNN in SDG8, which had the highest SD in ES, were much lower than those of AutoML. AutoML’s predicted ES correlated better with the actual score than BPNN’s because all of the AutoML’s R-squared values were much higher than the BPNN’s in all SDGs. We can say that AutoML predicted the ES better than the popular BPNN technique in this BRI sustainability study.

We then predicted ET using the inputs, outputs, and ET of the DEA experiment as the learning dataset. The precision and accuracy of the outputs predicted from the classification AutoML algorithms were also better than the outputs from the BPNN, as shown in Table 7b. BPNN classified all DMUs in SDG7 with a very low SD as ET1, but AutoML was able to identify all ETs and classified the DMUs based on their performance. Although the BPNN could predict greater than 50% accuracy in SDG7, AutoML could predict greater than 90% accuracy. SDG8 contained six tiers and AutoML again recognized them all and achieved an accuracy of over 85%. It was unlikely that BPNN could only set four tiers, resulting in an accuracy of around 30%. BPNN classified more than half of the DMUs as ET3, but only 13 DMUs that matched. BPNN could detect only ET1 and ET2 in SDG9, but only 59% of the DMUs were correct. Although SDG9 had a higher SD than SDG7 in ES, AutoML still performed well and resulted in an accuracy of 85%. These show that AutoML could also beat BPNN in this classification problem.

We predicted the PO using BPNN and AutoML to study the efficient frontier projection that turned an inefficient DMU into an efficient DMU. Only the original inputs and PO of the DEA experiment were used as the learning dataset. Table 7c shows the evaluation metrics of both methods for comparing the prediction performance. All of AutoML’s RMSE and PRED values were better than the BPNN’s ones, with the exception of the manufacturing value added in SDG9. These results were consistent with the previous finding that AutoML’s regression algorithm could overwhelm the BPNN. Next, we used these predicted POs from the BPNN and AutoML along with the original inputs to measure performance in 2017 using the DEA application. This measurement would compare the performance in the multivariable prediction algorithm, since a machine learning model is basically good at supporting a single-target prediction and ignoring the multiple relationships between target variables. This study aims to optimize all targets together and not just a single target, as all variables are related to each other. Nevertheless, the multiple regression feature was missing from the selected TPOT library, so AutoML predicted the PO as a single target and ignored the relationship of the other outputs when fitting the predictive model. Figure 2 shows the comparison of the ES and the distance to the efficient frontier of all DMUs between BPNN and AutoML. For this experiment, the BPNN’s ES in diamond shape was mostly closer to 100% than that of AutoML in the line shape. The mean ES of all SDGs in BPNN was 0.998, while in AutoML it was 0.988. In addition, the mean minimum ES of BPNN was 0.991 and for AutoML it was 0.872. Although the BPNN could generate more efficient DMUs in each SDG, AutoML could generate more efficient DMUs in all SDGs. While all DMUs of SDG8 in BPNN achieved an ES of 100%, only 25 DMUs in all SDGs could perfectly reach the efficient frontier with zero distance. There were 32 DMUs from AutoML that could perfectly distance zero from the efficient frontier; however, the mean distance in AutoML was still higher than that in the BPNN. Since the original TPOT library was unable to predict regressions with multiple outputs, this may result in a more accurate prediction, but poorer mean ES and distance.

4. The COVID-19 Pandemic Study

This section provides a study on the integrative approach to addressing the COVID-19 pandemic in the context of the SDGs to understand the situation and develop guidelines for management and risk mitigation. To assess the effectiveness of medical treatment in SDG3, three inputs, one desirable output and two undesirable outputs, were applied to the DEA. Performance in preparing to deal with COVID-19, which was defined as a segment in SDG11, was assessed based on four inputs, three desirable outputs, and one undesirable output. The newly confirmed case variable was used in both SDGs, and only this variable correlated significantly in rank with all variables. Some variables were measured at weeks t + 1 and t + 3, when the week evaluated was week t, due to incubation, convalescence, and hospitalization to death.

4.1. Result of the DEA Experiment

Table A2. DEA experiment results in the COVID-19 pandemic study.

	16th Week			17th Week			18th Week			19th Week			20th Week			21st Week
	SDG3	SDG11	Distance	SDG3	SDG11	Distance	SDG3	SDG11	Distance	SDG3	SDG11	Distance	SDG3	SDG11	Distance	SDG3	SDG11	Distance
AR	0.181	0.855	0.832	0.462	0.921	0.544	0.533	0.935	0.471	0.317	0.887	0.693	0.136	0.857	0.875	0.423	0.897	0.587
AT	0.495	0.761	0.559	0.533	0.661	0.578	0.582	0.658	0.540	0.463	0.617	0.660	0.231	0.601	0.866	0.592	0.736	0.486
AU	0.693	0.701	0.428	0.670	0.676	0.463	0.309	0.654	0.773	0.502	0.618	0.627	1.000	0.528	0.472	1.000	0.627	0.373
BD	0.561	0.796	0.485	0.322	0.808	0.704	0.370	0.816	0.656	0.242	0.790	0.786	0.128	0.680	0.929	0.257	0.751	0.783
BE	0.105	0.871	0.904	0.137	0.791	0.888	0.117	0.799	0.906	0.052	0.780	0.973	0.063	0.732	0.975	0.088	0.812	0.931
BG	0.183	0.849	0.830	0.227	0.777	0.805	0.306	0.582	0.810	0.278	0.703	0.780	0.369	0.560	0.769	0.649	1.000	0.351
BH	1.000	0.447	0.553	0.587	0.529	0.626	0.885	0.581	0.434	0.417	0.560	0.731	0.495	0.538	0.685	0.995	0.572	0.428
BO	0.159	1.000	0.841	0.232	1.000	0.768	0.105	1.000	0.895	0.116	1.000	0.884	0.164	1.000	0.836	0.168	1.000	0.832
BY *	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000
CA	0.496	0.748	0.564	0.521	0.728	0.551	0.433	0.668	0.657	0.327	0.746	0.719	0.250	0.753	0.790	0.442	0.794	0.595
CH	0.656	0.685	0.466	0.382	0.588	0.743	0.456	0.668	0.637	0.317	0.609	0.787	0.378	0.496	0.801	0.371	0.751	0.677
CL	0.557	0.919	0.450	0.720	1.000	0.280	0.841	1.000	0.159	0.762	1.000	0.238	1.000	1.000	0.000	1.000	1.000	0.000
CO	0.266	0.896	0.742	0.303	0.914	0.702	0.267	0.894	0.741	0.284	0.863	0.729	0.417	0.853	0.601	0.502	0.873	0.514
CR	1.000	1.000	0.000	0.481	1.000	0.519	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	0.333	1.000	0.667
CZ	0.448	0.584	0.691	0.689	0.504	0.586	0.505	0.552	0.668	0.271	0.611	0.827	0.159	0.379	1.045	0.434	0.455	0.786
DE	0.850	0.654	0.377	0.812	0.507	0.527	0.782	0.532	0.516	0.459	0.466	0.761	0.324	0.421	0.891	0.511	0.736	0.556
DK	0.439	0.766	0.608	0.578	0.532	0.631	0.473	0.497	0.729	0.336	0.592	0.779	0.252	0.463	0.921	0.554	0.571	0.619
EE	1.000	0.806	0.194	0.617	0.839	0.416	1.000	0.943	0.057	0.307	0.908	0.699	0.336	1.000	0.664	1.000	1.000	0.000
GH	0.214	0.696	0.843	1.000	0.687	0.313	0.338	0.684	0.734	0.223	0.667	0.845	0.249	1.000	0.751	0.223	1.000	0.777
HR	0.440	0.774	0.604	0.638	0.671	0.489	0.395	0.714	0.669	0.207	0.566	0.904	0.509	0.473	0.720	1.000	0.848	0.152
HU	0.188	0.634	0.891	0.451	0.527	0.724	0.314	0.574	0.807	0.433	0.539	0.730	0.097	0.544	1.011	0.494	0.523	0.695
ID	0.224	0.678	0.840	0.308	0.787	0.724	0.315	0.833	0.705	0.230	0.921	0.774	0.208	0.867	0.803	0.520	0.827	0.510
IL	0.404	0.887	0.606	0.497	0.730	0.571	1.000	0.752	0.248	0.569	0.558	0.618	0.297	0.521	0.850	0.451	0.442	0.783
IN	0.700	0.755	0.387	1.000	0.791	0.209	1.000	0.781	0.219	1.000	0.722	0.278	0.496	0.655	0.611	1.000	0.704	0.296
IT	0.948	0.948	0.074	1.000	0.939	0.061	1.000	0.903	0.097	1.000	0.836	0.164	0.704	0.864	0.326	1.000	0.732	0.268
JP	0.710	0.499	0.579	1.000	0.654	0.346	1.000	0.675	0.325	0.331	1.000	0.669	0.497	0.711	0.580	1.000	0.748	0.252
KE	0.328	0.631	0.767	0.591	0.630	0.551	1.000	0.689	0.311	0.225	0.624	0.861	0.239	0.669	0.830	0.578	0.662	0.540
KR	1.000	0.290	0.710	0.965	0.368	0.633	0.684	0.395	0.683	0.298	0.384	0.933	0.265	0.171	1.108	0.364	0.258	0.977
KZ	0.451	0.771	0.595	0.635	0.790	0.421	0.752	0.752	0.350	0.428	0.776	0.614	0.323	0.603	0.785	0.478	0.510	0.716
LT	0.348	0.747	0.699	0.493	0.591	0.651	0.439	0.661	0.655	0.301	0.486	0.868	0.501	0.477	0.722	0.748	0.426	0.627
LU	0.439	0.966	0.562	0.604	0.856	0.421	0.300	0.956	0.702	0.502	0.920	0.505	1.000	0.876	0.124	1.000	0.983	0.017
LV	0.434	1.000	0.566	0.986	1.000	0.014	0.273	1.000	0.727	0.333	1.000	0.667	0.500	1.000	0.500	1.000	1.000	0.000
MA	0.458	1.000	0.542	0.565	1.000	0.435	0.583	1.000	0.417	0.314	1.000	0.686	0.621	1.000	0.379	0.457	1.000	0.543
MX	1.000	0.724	0.276	1.000	0.862	0.138	1.000	0.878	0.122	1.000	0.883	0.117	0.695	0.890	0.325	1.000	0.908	0.092
MY	0.450	0.947	0.552	0.587	0.949	0.416	0.481	0.958	0.520	1.000	0.915	0.085	0.343	0.798	0.687	0.756	0.737	0.359
NG	0.550	0.837	0.479	0.476	0.756	0.578	0.312	0.759	0.729	0.212	0.730	0.833	0.156	0.721	0.889	0.345	0.776	0.692
NP	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	0.248	0.835	0.770	0.686	0.945	0.319
PA	1.000	1.000	0.000	0.516	1.000	0.484	0.087	1.000	0.913	1.000	1.000	0.000	0.172	1.000	0.828	1.000	1.000	0.000
PE	0.757	0.927	0.254	0.575	0.950	0.428	1.000	0.947	0.053	0.634	0.935	0.371	0.255	0.980	0.745	0.746	0.992	0.254
PH	0.204	0.842	0.812	0.212	0.865	0.800	0.206	0.883	0.803	0.117	0.847	0.897	0.099	0.834	0.916	0.217	0.822	0.803
PK	0.372	0.567	0.763	0.366	0.573	0.764	0.491	0.643	0.622	0.457	0.594	0.678	0.208	0.461	0.958	0.786	0.474	0.568
PL	0.360	0.852	0.657	0.382	0.624	0.723	0.473	0.565	0.683	0.304	0.526	0.842	0.184	0.475	0.970	0.288	0.424	0.916
PT	0.129	0.892	0.878	0.271	0.861	0.742	0.825	0.834	0.241	1.000	0.770	0.230	0.252	0.769	0.783	0.364	0.747	0.685
PY	1.000	1.000	0.000	0.502	0.747	0.558	1.000	0.813	0.187	1.000	0.888	0.112	1.000	0.973	0.027	1.000	1.000	0.000
QA	0.521	0.640	0.599	0.376	0.711	0.688	0.595	0.768	0.466	1.000	0.780	0.220	0.535	0.753	0.526	0.671	0.700	0.445
RO	0.511	0.714	0.567	0.640	0.768	0.429	0.433	0.675	0.653	0.373	0.662	0.712	0.268	0.516	0.878	0.490	0.495	0.718
RS	0.314	0.980	0.686	0.507	0.988	0.493	0.608	0.893	0.407	0.275	0.666	0.799	1.000	0.522	0.478	0.479	0.999	0.521
RU	0.889	0.572	0.443	1.000	0.581	0.419	1.000	0.570	0.430	1.000	0.672	0.328	0.799	0.590	0.457	1.000	0.505	0.495
RW	1.000	0.930	0.070	1.000	0.905	0.095	1.000	1.000	0.000	0.501	1.000	0.499	0.500	1.000	0.500	1.000	1.000	0.000
SG	0.299	0.968	0.701	0.935	1.000	0.065	0.895	1.000	0.105	1.000	1.000	0.000	0.393	1.000	0.607	0.748	1.000	0.252
SK	0.407	0.664	0.682	1.000	0.647	0.353	1.000	0.723	0.277	1.000	0.624	0.376	1.000	0.555	0.445	1.000	0.475	0.525
SN	1.000	0.642	0.358	0.555	0.612	0.590	0.654	0.659	0.486	0.277	0.657	0.800	0.386	0.640	0.712	0.765	0.708	0.375
SV	1.000	1.000	0.000	0.704	1.000	0.296	0.343	1.000	0.657	0.678	1.000	0.322	0.157	1.000	0.843	0.506	1.000	0.494
TH	0.389	0.693	0.684	1.000	0.751	0.249	1.000	0.846	0.154	0.590	0.648	0.541	0.500	0.559	0.667	1.000	1.000	0.000
TR	1.000	0.728	0.272	1.000	0.800	0.200	0.640	0.778	0.423	0.490	0.751	0.568	0.384	0.645	0.712	1.000	0.836	0.164
TW *	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000
UA	0.248	0.629	0.839	0.371	0.695	0.699	0.412	0.594	0.714	0.291	0.604	0.812	0.301	0.535	0.840	0.417	0.406	0.833
UG *	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000	1.000	1.000	0.000
US	1.000	0.547	0.453	1.000	0.548	0.452	1.000	0.490	0.510	1.000	0.562	0.438	1.000	0.560	0.440	1.000	0.545	0.455
UY	0.561	1.000	0.439	0.674	1.000	0.326	0.600	1.000	0.400	1.000	1.000	0.000	0.501	1.000	0.499	1.000	1.000	0.000

* Efficient DMUs in all studied periods.

in Appendix A shows the results of the DEA in this study. The results are the ES in medical treatment associated with SDG3 in Target 3.3 and in preparedness for COVID-19 related to SDG11 in Targets 11.5 and 11.b, as well as the distance between the assessed DMU and the DMU with the best performance. Based on the data available, three countries including Belarus, Taiwan, and Uganda performed best overall in the fight against COVID-19 over all periods studied. Belarus, Taiwan, Uganda, and the United States showed the best effectiveness of medical treatments, while Bolivia, Belarus, Costa Rica, Latvia, Morocco, Panama, El Salvador, Taiwan, Uganda, and Uruguay were best prepared for COVID-19. When looking at the ratio of newly recovered cases to newly confirmed cases during the period studied, Taiwan had the highest number. It also had the lowest ratio of newly confirmed cases to new COVID-19 tests, which was a result in the area of prevention, and it also reduced the number of active cases well. Uganda, the only country with no fatalities, also performed well in terms of the ratio of newly confirmed cases to new COVID-19 tests. Belarus did well for the ratios of deaths to active cases and deaths to newly confirmed cases. Morocco and Uruguay had done well in reducing the number of active cases due to their good prevention practices.

Table 8. DEA experiment summary in the COVID-19 pandemic study: (a) weekly efficiency score; (b) number of DMUs in each efficient tier.

(a)								(b)
		wk 16	wk 17	wk 18	wk 19	wk 20	wk 21			wk 16	wk 17	wk 18	wk 19	wk 20	wk 21
SDG3	Mean	0.588	0.648	0.645	0.555	0.459	0.687	SDG3	ET1	16	15	19	18	11	23
	SD	0.301	0.262	0.298	0.319	0.296	0.288		ET2	18	17	13	14	13	12
	Max	1.000	1.000	1.000	1.000	1.000	1.000		ET3	14	14	15	12	13	10
	Min	0.105	0.137	0.088	0.052	0.063	0.088		ET4	9	11	7	12	12	8
									ET5	4	4	7	5	12	8
SDG11	Mean	0.800	0.784	0.790	0.772	0.731	0.778	SDG11	ET1	12	13	14	15	15	18
	SD	0.165	0.169	0.168	0.176	0.213	0.209		ET2	27	26	25	24	25	19
	Max	1.000	1.000	1.000	1.000	1.000	1.000		ET3	15	16	17	18	16	19
	Min	0.290	0.368	0.395	0.385	0.171	0.258		ET4	7	6	5	4	5	5

a shows that the mean ES in both SDG3 and SDG11 were unstable from year to year. SDG3 produced a smaller mean and a smaller minimum, but a higher SD in ES. SDG3 in this study focuses only on the medical management of COVID-19 to measure recovery, deaths, and remaining active cases. It ranged from the country with the highest confirmed and recovered cases and deaths in the world to countries with no deaths; therefore, the medical treatment situation was different in each country. The results in SDG3 show that many DMUs could handle this situation well, but more DMUs deteriorated at Week 20 due to the lowest mean ES over that period. The maximum of the minimum ES in SDG3 was below 15%, which was quite low compared to SDG11, at almost 40%. SDG11 aims to assess the effectiveness of the preparations that most countries had made regarding social distancing and local lockdown restrictions; therefore, the SD was lower than that of SDG3. The mean ES for all periods were nearly 80%, except for Week 20, which was less than 75%. This means that most countries were pretty well prepared for COVID-19. In summary, the mean ES and SD of SDG3 were fairly stable over the periods studied, while the minimum ES showed a small downward trend. This means that medical treatment performance did not change significantly over the period studied and the worst countries did not improve. The mean ES of SDG11 showed a slight downward trend, the minimum ES showed an upward trend, but the SD showed a downward trend. These describe that there were more countries that were poorly prepared for COVID-19, but the worst countries tried to improve and narrow the gap with more efficient countries.

Based on the DEA’s rule of thumb, plus six and eight of the total input and output variables in SDG3 and SDG11, the final round of DEA should have more than 12 and 16 DMUs for analysis, respectively. We used the stratification DEA to classify the ETs and the results varied over time, as shown in Table 8b. In addition, they were relatively identical to the results of the ES assessments. SDG3 had more ETs than SDG11 because the variance in SDG3 was higher than in SDG11. ET1 had the largest number of DMUs in Weeks 18, 19, and 21, while ET2 had the largest number in Weeks 16 and 17. Week 20 was completely different from the others as all ETs had similar numbers of DMUs; moreover, this week had the lowest mean ES. This lowest ES could come from the largest number of DMUs in ET5. Week 21 was also special as ET1 was more than twice the size of the other ET, resulting in the highest mean ES. In addition, ET1, ET4, and ET5 generated upward trends, while ET2 and ET3 generated downward trends. These express that most countries performed well in terms of medical treatment, but the gap between countries with high and low efficiency had widened. There were four ETs in SDG11; additionally, the number of DMUs in ET4 was lowest each week. Results varied slightly from Weeks 16 to 20, with ET2 being greatest and ET3 greater than ET1. In contrast, the number of DMUs in ET1, ET2, and ET3 at Week 21 were comparable. The trends in ET1 and ET3 were positive, while ET2 and ET4 were declining. These show that most countries performed well in preparing for COVID-19 and that performance improved.

4.2. Result of the AutoML Experiment

This part shows an experiment to predict the ES, ET, and PO at Week 21 using the collected data and DEA results at Weeks 16 to 20 as the learning dataset. When comparing the results of the regression score in ES prediction between the BPNN and AutoML, the final conclusion was the same as in the BRI sustainability study, as shown in

Table 9. Performance comparison of prediction methods in the COVID-19 pandemic study: (a) regression evaluation metrics in the efficiency score prediction; (b) precision and accuracy in the efficient tier classification; (c) regression evaluation metrics in the frontier projection prediction.

(a)						(b)
		SDG3		SDG11				SDG3		SDG11
		BPNN	AutoML	BPNN	AutoML			BPNN	AutoML	BPNN	AutoML
Mean		0.973	0.588	0.741	0.761	ET1		18/21	23/23	13/15	15/17
SD		0.033	0.267	0.165	0.190	ET2		5/13	6/12	14/23	15/24
Max		1.000	1.080	0.998	1.027	ET3		5/15	6/10	15/20	13/15
Min		0.899	0.109	0.343	0.363	ET4		2/8	4/8	2/3	5/5
RMSE		0.390	0.182	0.115	0.092	ET5		3/4	3/8
PRED(0.20)		0.393	0.541	0.836	0.934	Accuracy		0.541	0.689	0.721	0.787
PRED(0.10)		0.393	0.475	0.475	0.803
PRED(0.05)		0.393	0.410	0.279	0.525
R2		0.487	0.724	0.732	0.811
(c)
		BPNN					AutoML
	Projected Outputs	RMSE	PRED (0.20)	PRED (0.10)	PRED (0.05)	R2	RMSE	PRED (0.20)	PRED (0.10)	PRED (0.05)	R2
SDG3	Recovered at week t + 3	6801	0.115	0.049	0.033	0.919	4303	0.230	0.131	0.033	0.932
	Deaths at week t + 3	0.254	0.295	0.164	0.016	0.617	0.128	0.410	0.361	0.230	0.917
	Active at week t + 3	0.072	0.016	0.016	0.016	0.071	0.048	0.066	0.049	0.016	0.879
SDG11	Workplaces	10	0.426	0.148	0.098	0.863	7	0.787	0.312	0.033	0.961
	Residential	3	0.853	0.492	0.197	0.886	2	0.902	0.705	0.459	0.945
	Public	15	0.721	0.098	0.033	0.933	12	0.787	0.475	0.197	0.828
	Confirmed at week t + 1	0.136	0.016	0.016	0.000	0.392	0.112	0.033	0.016	0.016	0.551

a. The mean ES predicted by the BPNN was very different from the AutoML in SDG3, but was quite similar in SDG11. In contrast, the predicted ES of both methods in the BRI study was quite similar. Only the maximum ES predicted by the BPNN in SDG3 was 100%, but AutoML in the BRI study was able to achieve 100% in all SDGs. The minimum ES predicted by BPNN in SDG3 was completely different from the actual values, but the minimum ES predicted by BPNN and AutoML in SDG11 was close to the actual values. Not only were the RMSE of AutoML in SDG3 and SDG11 lower than that of BPNN, but all of AutoML’s PREDs were also better than those of the BPNN, which were the same as in the BRI study. Nonetheless, the AutoML’s PREDs were lower in this study than in the BRI study. PREDs and R-squared in SDG11 were better than in SDG3, which could be due to the lower SD. The ES predicted by AutoML correlated better with the actual score than that predicted by the BPNN, and the R-squared of the AutoML in this study was similar to the BRI study. We can summarize that AutoML performed better than BPNN in this COVID-19 prediction experiment. The ET of each SDG was predicted by the BPNN and AutoML using the inputs, outputs, and ET of the DEA experiment as a learning dataset to compare the performance in the classification algorithm, as shown in Table 9b. Both the BPNN and AutoML approaches were able to capture all ETs equally; however, AutoML was more accurate than BPNN. Therefore, AutoML could also perform better than BPNN on this COVID-19 classification problem. We then predicted the PO of SDG3 and SDG11 using only the original inputs and PO from the DEA experiment as the learning dataset. The evaluation metrics in Table 9c show the comparison of the BPNN and AutoML approaches. As with the results in the BRI study, all of the AutoML’s RMSE, PRED, and R-squared scores were better than the BPNN’s ones, with the exception of PRED (0.05) on daily visits to workplaces and R-squared on daily visits to public places in SDG11. Finally, we used this predicted PO along with the original inputs to measure Week 21′s performance using the DEA application and to compare performance on regression with multiple outputs. Figure 3 illustrates the comparison of the ESs and the distance to the efficient frontier of all DMUs between BPNN and AutoML. The ESs assessed by the BPNN were closer to 100% than those of AutoML in all SDGs. The mean ES of the BPNN was 0.973 in SDG3 and 0.983 in SDG11, while the mean ES of AutoML was 0.816 in SDG3 and 0.980 in SDG11. The minimum ES of BPNN was 0.899 in SDG3 and 0.912 in SDG11, while the minimum ES of AutoML was 0.272 in SDG3 and 0.778 in SDG11. These values were lower than the results in the BRI study. There were 31 efficient DMUs in SDG3 and 22 efficient DMUs in SDG11 from BPNN. In contrast, AutoML’s efficient DMUs were 19 DMUs in SDG3 and 32 DMUs in SDG11. There were 12 DMUs from BPNN and 11 DMUs from AutoML that could perfectly distance zero from the efficient frontier; however, the mean distance in BPNN was quite higher than that in AutoML. As previously described, the data in the BRI study were better correlated than the data in this COVID-19 study; therefore, the results of the BRI study showed a higher ES and a higher ratio of efficient DMUs to total DMUs. Due to the limitation of TPOT, BPNN performed slightly better than AutoML in this regression problem with multiple outputs. Moreover, the integrative approach can be used to measure current performance in preparing for and dealing with COVID-19. Then each country understands its own capabilities and can choose to follow best practices to fight the COVID-19 pandemic. The projected results can be used as a target in action.

5. Conclusions

Since the DEA does not have predictive capabilities, it is integrated with an AutoML application to predict the future performance and projected outputs, thus developing such a predictive ability. This article introduces the integrative method to assess and predict the performance of a country’s SDGs. There are two empirical studies with different data properties to demonstrate the integrative approach. Three prediction targets were used to measure the performance of AutoML in the regression, classification, and multi-target regression algorithms. A BPNN was used to validate the AutoML outputs. There were three main outcomes, namely, a performance measurement analysis, performance evaluation of the predictive methods, and a goal analysis.

Most countries in the BRI region performed reasonably well on SDG7, but progress on SDG8 was quite limited. Countries in the BRI region were considered equally in SDG9, but inequality in SDG9 increased as the countries were clearly divided into two groups. Most countries performed well in treating COVID-19 and preparing for COVID-19; additionally, performance did not change significantly over the period studied.

In particular, policymakers should improve access to clean fuels and electricity to improve performance in SDG7. Domestic material consumption and employment in manufacturing are the key outputs leading to high performance in SDG8 and SDG9. To improve the performance of medical treatment for COVID-19 in SDG3, recovered cases and deaths are the primary goals that everyone should focus on. Daily visits to workplaces and residential areas are two main criteria by which the performance of preparing to deal with COVID-19 in SDG11 should be measured. These factors should be used as national goals in support of the achievement of the SDGs.

A low SD results in poor regression and classification prediction performance for the BPNN, but does not significantly affect AutoML. AutoML can outperform a BPNN on regression and classification prediction problems; however, a BPNN can handle the problem of multi-target optimization better than AutoML due to the limitation of TPOT in the multi-output regression. Long-term data with a high correlation can provide a more accurate and precise prediction result than short-term data with a lower correlation; however, the R-squared values of both data properties were quite similar.

The integrative approach can accurately predict the projected outputs that can be used as national goals to transform an inefficient country into an efficient country to meet the expectations in the SDGs. The use of this method to reduce the population size and variance in the population analysis can be expanded in future work.