Effective Decision Making: Data Envelopment Analysis for Efficiency Evaluation in the Cloud Computing Marketplaces

Abstract

Assessing business performance is a critical issue for practicing managers, and business performance has always been of interest to managers and researchers. In recent years, the world has experienced a rapid growth in the cloud computing service sector thanks to its benefits to business organizations and economic development. Therefore, the performance efficiency of this sector has been concerned as one of the keys in today’s economic environment. This study aimed to assess the performance efficiency of cloud computing service providers in the United States of America, one of the biggest global markets in terms of cloud computing, by applying the data envelopment analysis models. The efficiency of cloud computing providers was evaluated based on the assumption of the non-cooperative game among cloud computing providers in which providers selfishly choose the best strategy to maximize their payoff with three stages. In the first stage, the performance of these providers over the past period was measured by a super slack-based measure. In the second stage, the performance in the future period was predicted by the new data envelopment analysis model: the past–present–future model based on resampling. In the last stage, the efficiency improvement was investigated by adopting the Malmquist productivity index. The findings of this study indicated that the percentage of inefficient providers would increase from 10% in the period from 2017 to 2020 to 20% for 2021 and 2024. Moreover, 30% of providers showed a regress in performance efficiency over the research period of 2017 to 2024. The results of this study provide an insight picture to the decision-makers, and this research will fill the gap in the literature as the first study that measures and predicts the performance efficiency of cloud computing service providers, which will provide a helpful reference for future studies.

1. Introduction

Although cloud computing has appeared since the early 2000s, many organizations were reluctant to implement it due to the technology’s lack of knowledge and trust [1]. However, the cloud computing market has experienced rapid growth in recent years thanks to the impressive development of technology and the apparent ease of use, scalability, and security from the users. Due to the advantages of cloud computing, more and more public and private sector organizations choose to test cloud workloads and even migrate everything to the cloud [2].

The COVID-19 pandemic was one of the reasons that boosted the increasing demand for cloud computing services. The coronavirus pandemic led to the sudden shutdown of almost every sector in many countries all over the world. Offices, business enterprises, and schools have been forced to shut down the offline channels and implement the online channel. People work from home; students participate in online classes; enterprises do business via the internet. That has resulted in greater demand for cloud solutions. In 2020, the market size of cloud computing was US$219.0 billion, with a growth of 13.7% compared to the average year-on-year growth during 2017–2019, and it is projected to reach $250 billion in 2021 and $791 billion in 2028 [3].

The advent of cutting-edge technologies such as artificial intelligence (AI) and machine learning (ML), as well as the increased investment by countries such as China, the United Kingdom, India, and the United States, are driving the growth of the cloud computing market. North America is the biggest market of cloud computing, with the market size reaching US $78.28 billion in 2020, which is an increase of US $8.69 billion, equal to 12.49% [3]. This market is forecasted to account for about 40% of the global cloud computing market [3].

As mentioned above, more and more organizations and firms have used cloud computing thanks to its benefits, such as decreasing operating costs and improving the flexibility of strategic decision making, thereby increasing the company’s lifespan and resilience [4]. In addition to benefiting firms, the cloud computing sector also has an outstanding contribution to national economic development. In 2012, cloud computing contributed around US $165 billion (1.04%) of added value to the US gross domestic product (GDP). It also supported nearly 1.7 million American jobs. This sector is expected to increase US GDP. In the next ten years, it will grow by approximately US $2 trillion [5]. This sector was found to contribute a total of €763 billion in added to value five European countries—Spain, France, Italy, Germany, and the United Kingdom—from 2010 to 2015 and created almost 2.4 million jobs during this period by promoting the expansion of an existing business, doing new business, reducing costs and indirect impact [5]. These benefits mainly come from developing existing businesses, launching new businesses, cost savings, and side effects. The study of Deloitte [6] in 2018 revealed that Google cloud services had a meaningful impact on the economic productivity of 14 selected countries, between $300 million and $1.2 billion in large countries and $100 million to $600 million in medium-sized countries. Moreover, cloud computing is also evidenced to reduce energy consumption [7].

In the new era of communications, the benefits of cloud technology for business and economic development are increasingly obvious. Hence, the growth of this sector is very important. Therefore, the efficiency performance of cloud computing providers captured great attention. Thus, this study will focus on the performance efficiency of the top 10 cloud computing providers in the United States market—one of the biggest markets in terms of cloud computing—by applying the two-stage data envelopment analysis (DEA) model, which included the super slack-based measure model (Super-SBM) to evaluate the performance during the past period from 2017 to 2020 and the resampling past–present–future model to predict the efficiency in the future period from 2021 to 2024.

The rest of the research is structured as follows: Section 2 contains a detailed review of the literature; Section 3 introduces the research methodology; Section 4 introduces the results analysis, and the last section contains the conclusions and discussion.

2. Literature Review

There are different methods for analyzing the performance, such as the ratio analysis, the performance pyramid, the analytic hierarchy process (AHP), the stochastic frontier analysis (SFA), variable factor productivity (VFP), and the data envelopment analysis (DEA). In the past, the ratio analysis was used as the favored instrument for measuring the business performance through standard ratios groups such as activities ratios, profitable ratios, liquid ratios, etc. The ratio method is attractive because of its simple methodological perspective, and the results of ratio analysis are easy to interpret. However, this method has several notable limitations. The first notable disadvantage of ratio analysis is that it requires or assumes that all the decision-making units (DMUs) operate under conditions of similar return of scale [8]. Another limitation is that each ratio group focuses on the different activities of the business organization, not the whole operation performance [9].

Another popular method is the performance pyramid, which can avoid the limitation of the ratio method. A performance pyramid is an integrated performance method that can capture multiple perspectives [10]. Each side of the pyramid represents a hierarchical perspective of success factors, controls, and process drivers. This model pays attention to the importance of influencing internal and external factors of the implementation and allows the construction of a performance monitoring system in the pyramid floors, starting with vision organization, using both financial and non-financial indicators [11]. However, it takes a long time, and the results may not be good for the organization in any way. The model also restricts the leadership from seeing what it highlights and leaves no room for other decisions [12].

The limitation of the ratio analysis method and the performance pyramid led to the application of more sophisticated methodologies [8]. There are two basic approaches, the parametric and nonparametric ones, to measure efficiency parametric frontier models and nonparametric methods of efficiency analysis, which have become dominant in the field of efficiency analysis [13]. Parametric and nonparametric approaches have been widely applied in measuring efficiency and achieved highly correlated results in most cases [14]. In comparison, the nonparametric method is more straightforward than the parametric approach. Thus, its application has increased in recent decades, which is applied in measuring the efficiency of diversified management fields thanks to its various models that suit many different requirements of the researcher [15].

One of the most common nonparametric methods is data envelopment analysis (DEA). DEA is a mathematical technique that uses linear programming strategies to convert inputs into outputs to estimate the overall performance of an organization or similar product [13]. Since Charnes, Cooper, and Rhodes [16] first introduced the DEA-CCR model in 1978, many different DEA models have been established to deal with the requirements of researchers, such as the Window model that can deal with the small set of DMUs developed by Charnes et al. [17] in 1984 and the slack-based model (SBM) that can deal with undesirable output introduced by Tone [18] in 2003. DEA was quickly regarded as a cutting-edge performance measurement instrument. Since then, a vast and substantial number of articles have appeared, including key theoretical discoveries and an enormous number of works on DEA applications, both in the public and private sectors, to evaluate the efficiency and productivity of their activities [19]. For example, Zhang et al. [20] used the slacks-based measure DEA model to examine the environmental efficiency of the 16 listed cement enterprises in China from 2008 to 2013. Xia et al. [21] evaluated the environmental efficiency of China’s mining sector from 2007 to 2016 by the meta-frontier slacks-based measure method. The environmental performance of the top 20 industrial countries was estimated by using the data envelopment analysis model by Iqbal et al. [22]. In another research, Behera et al. [23] calculated the productivity change of the coal-based power plants in India by the Malmquist productivity index. Park et al. [24] applied the DEA window and SBM DEA model to estimate the operational efficiency of the coastal ferry sector in South Korea. The application of DEA appeared to be popular in the high-tech sector. For example, Zhang et al. [25] evaluated the efficiency of the hi-tech industry in China by applying a multi-activity network DEA. Wang et al. [26] applied two-stage DEA in the efficiency evaluation of China’s high-tech industry. Bai et al. [27] used the dynamic network SBM model in DEA to assess the efficiency of the high-tech zone in China in the post-financial crisis era. Li et al. [28] measured the innovation efficiency of the semiconductor industry in China with the application of the new DEA model. Healthcare represents the main application area for DEA. The study of Kohl et al. [29] reported that there were about 262 papers of DEA application in healthcare focusing on hospitals from 2005 to 2016. The literature proves that the DEA is a powerful method in measuring the efficiency of DMU performance in every sector.

In terms of forecasting methodology, the Grey system theory is considered to be one of the most widely used methods due to its computational efficiency [30]. The Grey model has been applied widely to forecast the outputs or productions of different fields. Numerous previous studies that applied the Grey model can be found in the literature [31,32,33,34]. Thanks to the advantages of the DEA and the Grey model, these two methods are usually combined to measure and forecast performance efficiency. The study of Wang et al. [35] presented an integrated approach by combining the super slack-based model and the Grey model GM(1,1) to evaluate and predict the energy consumption efficiency of 25 different countries in the world. Chen and Chen [36] applied the Malmquist productivity index (MPI) and the Grey model GM(1,1) to investigate and forecast the operation performance of the Taiwanese wafer fabrication firms. Carboni and Russu [37] used a combination of the DEA with the Grey model to measure and predict the local economy and environmental efficiency in Italy. Wang et al. [38] assessed the performance of major Asian airline companies by combining the Grey model GM(1,1) and the DEA Window model. However, this integrated method required sophisticated steps and a large database, which is necessary to make a base for forecasting by the Grey method.

In 2016, Tone [39] proposed the model in DEA based on resampling that enables the researchers to evaluate and forecast the efficiency at the same time. In previous research, Wang et al. [40] applied this DEA resampling model to estimate the macroeconomic performance of 17 economies (12 Asian developing countries and five developed countries) from 2013 to 2020. The research shows that DEA resampling is an effective model for predicting and measuring the performance of multiple decision-makers. Chiu et al. [41] introduced the combination of the merger potential gains model and the resample past–present–future model to evaluate the performance of the financial industry in Taiwan. Wang et al. [42] applied the newly developed DEA resampling model to evaluate the profitability of Vietnamese commercial banks. The results of these studies supported the feasible application of the new DEA method proposed by Tone [39] not only in measuring but also in forecasting the efficiency of decision making without the need of combining two separate methodologies, such as DEA and the Grey model.

In recent years, measuring the efficiency of cloud computing service providers has newly captured researchers’ attention due to the significant development of technology and the consequence of the COVID-19 pandemic. For example, Azaidi et al. [43] in 2019 proposed the network DEA method as a tool to evaluate the efficiency of the 18 cloud service providers. The results of this research indicated that DEA could effectively measure the efficiency of this industry. Another study by Azadi et al. [44] in 2021 also assessed the sustainability of 30 cloud computing service providers with the application of DEA. One considerable research on this topic is the work conducted by Subirats and Guitart [45]. They evaluated and forecasted the energy efficiency of cloud computing platforms. However, most of the related studies found in the literature focused on analyzing the benefits and the challenges of adopting cloud computing in organizations, the efficiency of cloud computing [46,47,48,49], and many other studies concentrating on the advantages, difficulties, and opportunities related to cloud computing can be found in the literature. On the contrary, there are very few studies on the utility of DEA in cloud computing.

An increasing quantity of businesses and organizations around the world use cloud computing offerings to enhance their performance inside the aggressive marketplace. Measuring the performance helps to understand these providers’ behavior and provides a guideline to improve performance. However, it is challenging to evaluate service industry performance such as cloud computing because of the unbalancing service quality and resource usage, which is closely related to game theory. Game theory is known as the practice of simulating strategic interaction between two or more players [50]. Game theory is used in many different fields to analyze players’ strategic decisions in two cases: cooperation and competition [50]. In the technology field, Teng and Magoulès [51] in 2010 tackled the resource allocation issue in cloud computing by game theory. Riahi and Riahi [52] in 2018, Ghosh [53] in 2020 evaluated the energy efficiency in wireless systems by applying game theory. Azadi et al. [43], in 2019, measured the efficiency of cloud service providers using DEA with a game theory base.

3. Research Methodology

3.1. Research Flow

The DEA past–present–future based on the resampling model is applied to measure and predict the performance efficiency of cloud computing service providers in the United States. The historical data in the period of 2017 to 2020 are used to measure the actual performance of these biggest ten providers. Then, the authors apply the resampling of the past–present–future model to analyze future results from 2021 to 2024. The process of carrying out this research consists of four main stages:

Stage 1	The background of the research: This section firstly proposes research questions. The authors learn about the reality of the cloud computing sector in the United States and identify limitations and motivations to promote development. Some theories and business data related to the selected topic are investigated in this part.
Stage 2	Data description: In order to achieve the goals of the research, it is necessary to collect data. This stage includes data collection and selection of input and output. In the first step, the authors collect the sample cloud computing service providers in the US market. The target providers have a meaningful influence on this research cloud computing sector, and their financial statements were published at least from 2017 to 2020. After that, input/output factors are selected before applying and implementing DEA models in the next stage. The choice of analytical factors is extremely important because it affects the evaluation results. Therefore, the appropriate inputs and outputs are carefully considered in the second step based on the literature review.
Stage 3	Research analysis: The authors apply an integrated DEA model to measure and assess the forecasting efficiency of US cloud computing providers. Selecting the suitable performance evaluation model provides accurate, in-depth analysis results, which help answer the question of this study. Among the set of DEA models, the super slack-based measure (Super-SBM) model and the Malmquist productivity index (MPI) are chosen based on their advantages over other DEA methods [54,55]. First, the Super-SBM model is applied to analyze the performance of 10 cloud computing corporations in the United States from 2017 to 2020. Afterward, the performance efficiency over the future period of 2021 to 2024 is predicted by applying the resampling method. In the last step, the MPI is employed to estimate the productivity changes from 2017 to 2024.
Stage 4	Conclusions and discussion: The results of the Super-SBM model, resampling, and Malmquist model provide valuable insights for the US cloud computing sector. The authors will also make some suggestions in this section.

3.2. Super-SBM Model

Tone [55] developed the Super-SBM model from DEA in 2002, which was modified from the slacks-based measure model. The research assumes that there are $n$ DMUs and each ${\mathrm{DMU}}_j$($j=1,\dots ,n)$ has $m$ input factors and $g$ output factors. The input and output matrices can be defined as $X=\left(x_{ij}\right) \in R^{m\times n}$ and $Y=\left(y_{ij}\right) \in R^{g\times n}$, respectively. While λ is a non-negative vector in $R^n$, the vectors $s^{-}\in R^m$ and $s^{+}\in R^g$ represent the excessive input and insufficient output of the expression, which is referred to as slacks. In fractional form, the SBM-DEA model to evaluate the efficiency of ${\mathrm{DMU}}_k$ is as follows:

$$\min \rho =\frac{1-\frac{1}{m}{{\displaystyle \sum }}_{i=1}^m{s}_i^{-}/x_{ik}}{1+\frac{1}{g}{{\displaystyle \sum }}_{i=1}^g{s}_i^{+}/y_{ik}}$$

(1)

subject to:

$$\begin{array}{c}{x}_k=X \lambda +s^{-}\\ y_k=Y \lambda -s^{+}\\ \lambda \ge 0,s^{-}\ge 0 and s^{+}\ge 0.\end{array}$$

(2)

In this study, the authors consider the super-efficiency problem under the assumption of ${\rho }^{* }=1$, $S^{-*}$ = 0 and $S^{+*}$ = 0; the inputs are assured, and the output is constant in the optimal solution; the DMU $\left(x_k,y_k\right)$ is defined as the SBM efficient model. The efficiency with which the SBM is estimated ranges from 0 to 1.

The Super-SBM model is introduced by Tone [55] in 2002 for separating and ranking these efficient DMUs. If ${\rho }^{*}=1$, then the DMU $\left(x_k,y_k\right)$ is efficient, and the Super-SBM model can be described as follows:

$$\min \delta =\frac{\frac{1}{m}{{\displaystyle \sum }}_{i=1}^m{\overline{x}}_i/x_{ik}}{\frac{1}{g}{{\displaystyle \sum }}_{i=1}^g{\overline{y}}_i/y_{ik}}$$

(3)

subject to:

$$\begin{array}{c}\overline{x}\ge {\displaystyle {\displaystyle \sum }_{j=1, j\ne k}^n}{\lambda }_j{x}_j\\ \overline{y}\le {\displaystyle {\displaystyle \sum }_{j=1, j\ne k}^n}{\lambda }_j{y}_j\\ \overline{x}\ge x_k, 0\le \overline{y}\le y_k,\lambda \ge 0.\end{array}$$

(4)

Suppose $y_{rk}$ ≤ 0. $\overline{y_r^{+}}$, and $\overline{y_{-r}^{+}}$ will be defined by:

$${\overline{y}}_r^{+}=\underset{j=1,\dots ,n}{\max }\left\{y_{rj}|y_{rj}>0\right\},$$

(5)

$${\overline{y}}_{-r}^{+}=\underset{j=1,\dots ,n}{\min }\left\{y_{rj}|y_{rj}>0\right\}.$$

(6)

If the output $r$ has no positive factors, then it is denoted as ${\overline{y}}_r^{+}$ = ${\overline{y}}_{-r}^{+}$= 1. The elements in the objective function are replaced by $s_r^{+}/y_{rk }$ as follows, whereas the value $y_{rk}$ never changes.

$$\mathrm{If}{\overline{y}}_r^{+}>{\overline{y}}_{-r}^{+},\mathrm{subjected}\mathrm{to}:s_r^{+}/\frac{y_{-r}^{+}\left({\overline{y}}_r^{+}-y_{-r}^{+}\right)}{{\overline{y}}_r^{+}-y_{rk}}$$

(7)

$$\mathrm{If}{\overline{y}}_r^{+}={\overline{y}}_{-r}^{+},\mathrm{subjected}\mathrm{to}:s_r^{+}/\frac{{\left({\overline{y}}_{-r}^{+}\right)}^2}{B\left({\overline{y}}_r^{+}-y_{rk}\right)}$$

(8)

DEA-solver has a B-Score of 100. In any situation, the suggestions are constructive and exceedingly tight $y_{-r}^{+}$. It is also related to the size of the unsupported output and is inversely proportional to the distance ${\overline{y}}_r^{+}-y_{rk}$. The resulting score is constant concerning the unit of measurement; it does not depend on the unit of measure used.

3.3. Resampling Model

When analyzing the performance of any DMU, Chang et al. [56] stated that it is critical to consider the history, present record, and future potential. However, if the history, current record, and future production are all taken into consideration, it is necessary to integrate diverse methodologies, such as integrating the data envelopment analysis and Grey model. In order to cope with this issue, Tone [39] submitted a model in DEA called resampling past–present and resampling past–present–future based on the resampling method. Based on the super slack-based measure model, the past–present model measures the confidence interval of the DEA score during the past and present time structure, and the past–present–future model is expanded to this past–present model.

3.3.1. Past–Present Model

Let us suppose ($X^t, Y^t$) (t = 1, …, T) is a historical input and output matrix where t = 1 is the first recognized period, and t = T is the last recognized period with the input vector $X^t=\left(x_1^t, \dots ,x_n^t\right)$ ($x_j^t \in R^m)$ and the output vector $Y^t=\left(y_1^t, \dots , y_n^t\right)$ ($y_j^t \in R^g$), and the number of DMU is n.

To achieve efficiency in the first stage, the authors employ the Super-SBM model. This model was used in this study because it is not constrained by unity ($\le$1) and so makes studying efficient DMUs more challenging [39].

In the next step, the weight is determined based on the decision-maker’s perspective and knowledge of how the past should affect the present. The weight chosen by Tone [39] was applied for this paper: the $W_t$ for a time period t, under the assumption that more recent periods provide information that is more important to estimating current efficiency scores. Thus, the following Lucas number series $\left(l_1,\dots ,l_t\right)$, is a candidate where $l_{t+2}=l_t+l_{t+1}\left(t=1,\dots ,T,T-2;l_1=1, l_2=2\right)$. Let L stand for the total of the series: L = ${\sum }_{t=1}^T{l}_t$. Then, the weight $W_t$ is as follows:

$$W_t=\raisebox{1ex}{$l_t$}\!\left/ \!\raisebox{-1ex}{$L$}\right.\left(t=1,\dots ,T\right).$$

(9)

Finally, the confidence interval is calculated using a bootstrapped replica of historical data. Since replicas are typical of the dataset, a preliminary examination of the data should be performed to determine dataset characteristics [39]. The replicas should be rejected and resampled if they are not representative of the dataset. The hypothesis test or confidence interval based on Fisher’s z transformation can be used to compare patterns in the past and present data for non-correlated and homoscedastic datasets. For the current time period data, the correlation between all pairs of inputs, outputs, and the input–output of all DMUs can be determined. Then, using Fisher’s z transformation, compute their 95% confidence intervals [57]. Based on the corresponding correlation, the resampled data will be deleted or allowed. If the relevant correlation is within this interval’s range, the resampled data is accepted; if the corresponding correlation is outside the interval’s range, the resampled data is discarded. As a result, the last period’s incorrect samples are removed from the sampling. This 95% confidence interval is optional; nevertheless, the narrower the interval, the closer the sample is to the data from the previous period [57].

3.3.2. Past–Present–Future Model

The forecast for the future, namely $\left(X^{t+1}, Y^{t+1}\right)$, is obtained by taking the past–present data $\left(X^t, Y^t\right)$ with (t = 1, …, T) and measuring the DMU efficiency value in the future period alongside their confidence intervals [58]. The past–present time-based frame is expanded to the past–present–future time-based frame in this resampling. To do so, take the following observed data from the past–present with an exact input i (i = 1, …, m) and output r (r = 1, …, $g$) of a DMU:

$$h^t\left(t=1,\dots ,T\right).$$

(10)

The $h^{T+1}$ will be predicted from $h^t\left(t=1,\dots ,T\right)$. The trend analysis, Lucas weight analysis, and the hybrid model are three available prediction tools that are introduced to produce forecasts [39]. The estimation of the super-efficiency index for the future period is implemented after getting the forecast.

3.4. Malmquist Model in DEA

The Malmquist productivity index was launched by Caves et al. [59] in 1982 to determine the change in the efficiency of each DMU in two periods. In the research, the authors assessed the dynamic productivity trend of cloud computing providers by using the original and expanded MPI by Färe et al. [60].

The change in total factor productivity from period t to period t + 1 is calculated as the following equation [59]:

$${\mathrm{MPI}}_t^{t+1}=\sqrt{\frac{{\rho }_{0 }^t\left(x^{t+1},y^{t+1}\right)}{{\rho }_{0 }^t\left(x^t,y^t\right)}\times \frac{{\rho }_{0 }^{t+1}\left(x^{t+1},y^{t+1}\right)}{{\rho }_{0 }^{t+1}\left(x^t,y^t\right)}}.$$

(11)

${\mathrm{MPI}}_t^{t+1}$ > 1 indicates positive DMU performance growth in the period between t and t + 1, whereas ${\mathrm{MPI}}_t^{t+1}$ = 1 and ${\mathrm{MPI}}_t^{t+1}$ < 1 indicate that the performance has no change and negative growth.

The Malmquist productivity index is broken down into two different components as the efficiency change (EC) and the technical changes (TC) by Färe et al. [60]. The efficiency change is also named the catch-up effect, reflecting the change in DMU’s efficiency, while the technical change is also named the frontier shift effect, demonstrating the fluctuation in the efficient frontier. The MPI index can be rewritten as follows:

$$\begin{gathered} \mathrm{MPI}\left(x^{t+1},y^{t+1},x^t,y^t\right)=\frac{{\rho }_{0 }^{t+1}\left(x^{t+1},y^{t+1}\right)}{{\rho }_{0 }^t\left(x^t,y^t\right)} \sqrt{\frac{{\rho }_{0 }^t\left(x^{t+1},y^{t+1}\right)}{{\rho }_{0 }^{t+1}\left(x^t,y^t\right)}\times \frac{{\rho }_{0 }^t\left(x^t,y^t\right)}{{\rho }_{0 }^{t+1}\left(x^t,y^t\right)}} \\ =\mathrm{Efficiency}\mathrm{change}\left(\mathrm{EC}\right)\times \mathrm{Technical}\mathrm{change}\left(\mathrm{TC}\right). \end{gathered}$$

(12)

If MPI > 1, this indicates productivity increases from period t to period t + 1. On the other hand, MPI = 1 means there is no change in productivity indexes, and MPI < 1 illustrates a productivity reduction. Since the total productivity changes are a multiplicative composite of efficiency change and technical change, the primary cause of productivity improvement is impacted by the change in efficiency and technology.

3.5. Data Collection

This research uses a database of listed companies in the cloud computing industry in the United States. After the authors searched and reviewed the completeness of data for companies in this industry, the ten largest cloud computing companies in terms of revenue were chosen to consider and investigate the performance. The list of all companies is shown in

Table 1. Characteristics of 10 listed US cloud computing companies.

DMUs	Company Name
C1	Amazon
C2	Alphabet
C3	Microsoft
C4	Salesforce
C5	ServiceNow
C6	Splunk
C7	Dropbox
C8	Shopify
C9	Atlassian
C10	Twilio

.

The selection of inputs and outputs plays a significant role in applying the DEA model. By referring to relevant literature reviews on DEA in

Table 2. List of previous studies using the data envelopment analysis method.

No.	Author (s)	Input Factors	Output Factors	Research Areas
1	Yang et al. (2009) [62]	Operating expenses, Operating costs, Assets	Operating revenues, Mobile phone calls, Mobile phone subscribers	Evaluating the efficiencies of the three leading telecommunication firms in Taiwan
2	Lu et al. (2011) [63]	Operating expenses, Liability, Equity, Employee	Net income, Net sales, Intangible value, Market value	Evaluating the operating performance of 30 global e-retailing companies
3	He-Boong Kwon (2014) [64]	Cost, Asset	Revenue, Operating income	Investigating the comparative performance of major mobile phone providers
4	Yang et al. (2016) [65]	Operating expense, Total assets, The number of employees	Revenue, Market share	Examining the operational efficiency of e-commerce firms
5	Ko et al. (2017) [66]	Store size, Number of items, Number of employees, Rental cost	Sales revenue, Number of customers	Examining the relative efficiency of retail chain stores in Korea
6	Shah et al. (2019) [67]	Assets, Equity, Employees, Expense	Revenue, Profit	Measuring the performance and productivity of the sustainable and non-sustainable banks
7	Wang et al. (2020) [68]	Asset, Liabilities, Equity	Revenue, Gross profit	Assessing the efficiency score of the ten largest e-commerce marketplaces in the US
8	Wang et al. (2021) [69]	Assets, Deposit, Operating expense, Liabilities	Loan, Net income	Measuring the performance of the commercial banking system in Vietnam

and examining the suitable correlation between the inputs and outputs [61], the researchers decided to choose two inputs (cost of revenue, operating expense) and two outputs (gross profit, total revenue). Figure 1 shows the operational process of cloud computing companies. These variables are defined as follows.

Inputs

(1)

Cost of revenue (I₁): Cost of revenue is the total cost of manufacturing and providing products or services to consumers.

(2)

Operating expense (I₂): Operating expense is expenses acquired by the company in its daily business activities. Operating costs include rent, equipment, inventory costs, marketing, wages, insurance, processing costs, and research and development funds.

Outputs

(1)

Gross profit (O₁): Gross profit appears as revenue minus the cost of goods sold. Note that gross profit is not deducted from other fixed and variable expenses, i.e., rent, utilities, payroll.

(2)

Total revenue (O₂): Total revenue is the total sales of products and services determined by multiplying the overall sales of goods and services by the prices of goods and services.

The data on inputs and outputs over the period of 2017 to 2020 were acquired from the National Association of Securities Dealers Automatic Quotation System (Nasdaq) [70], which published the financial statements of US public joint-stock corporations. The unit of measurement is million US dollars. The authors show the statistical data from 2017 to 2020 in

Table 3. Descriptive statistics of the inputs and outputs from 2017 to 2020.

Year	Statistics	Input Factors		Output Factors
		Cost of Revenue	Operating Expense	Gross Profit	Total Revenue
2017	Max	137,183	173,760	65,272	177,866
	Min	119.161	282.199	216.125	399.02
	Average	22,152.07	25,560.81	17,363.32	39,515.39
	SD	41,439.04	51,124.42	24,628.07	60,430.86
2018	Max	173,183	220,466	77,270	232,887
	Min	172.69	464.461	349.226	650.067
	Average	27,684.85	31,972.57	22,490.02	50,174.88
	SD	52,304.68	64,768.63	31,252.04	77,556.87
2019	Max	205,768	265,981	89,961	280,522
	Min	210.285	978.702	608.917	1134.468
	Average	32,789.48	38,160.54	26,882.86	59,672.34
	SD	62,144.21	78,020.21	36,758.81	92,432
2020	Max	291,824	363,165	97,795	386,064
	Min	268.807	1331.28	915.661	1614.173
	Average	43,252.4	49,043.8	31,530.23	74,782.62
	SD	87,057.44	106,410.1	42,629.72	121,402.4

.

4. Results Analysis

4.1. Efficiency Evaluation during the Past Period from 2017 to 2020

In this section, the Super-SBM model is applied to compute the efficiency scores and rank the efficiency of 10 DMUs.

Table 4. Results of the efficiency from 2017 to 2020.

DMUs.	2017	2018	2019	2020	DMUs	2017	2018	2019	2020
Efficiency Scores					Ranking
C1	1.00	1.00	1.00	1.00	C1	7	7	8	8
C2	1.56	1.40	1.42	1.35	C2	2	3	3	3
C3	1.08	1.30	1.39	1.55	C3	6	4	4	2
C4	1.60	1.57	1.66	1.61	C4	1	1	1	1
C5	0.96	0.99	1.00	1.05	C5	8	8	9	6
C6	1.26	1.21	1.22	0.74	C6	5	6	5	9
C7	0.85	0.76	0.90	1.01	C7	10	10	10	7
C8	0.93	0.89	1.06	1.11	C8	9	9	6	5
C9	1.44	1.48	1.60	1.29	C9	3	2	2	4
C10	1.41	1.28	1.03	0.68	C10	4	5	7	10

presents the obtained scores and ranking of these DMUs over the period of 2017 to 2020, along with the illustration in Figure 2.

As observed, the efficiency scores of each DMU experienced different trends. Some DMUs remained unchanged and remained stable, such as C1-Amazon and C4-Salesforce, while the rest of the DMUs showed a notable fluctuation in efficiency scores over the period of 2017 to 2020. The results divided these ten DMUs into three groups: the stable group, the increasing group, and the decreasing group.

The efficiency score of C1-Amazon remained unchanged over four years with a score of 1. The efficiency score of C4-Salesforce slightly changed over the period. However, the change was insignificant, approximately 1% over a 4-year period and a lower than 6% year-to-year change. It was also noted that C4-Salesforce had the highest score among the ten DMUs and ranked at the first position for all four years from 2017 to 2020. From the results, it can be seen that these two DMUs controlled well the inputs and outputs over the observed period.

While the scores of C1-Amazon and C4-Salesforce were stable, the efficiency scores of the remaining eight DMUs significantly fluctuated. C2-Alphabet had a fluctuation in score with a decreasing trend from 1.56 in 2017 to 1.35 in 2020, approximately a 13.5% decrease over the period, which led to a drop in ranking from the second position among ten to the third position. The efficiency scores of C3-Microsoft, on the other hand, significantly increased with the drastic change of 43.5% from 1.08 in 2017 and ranked at the 6th position to the score of 1.55 and become the second most efficient DMU in 2020. The scores of C5-ServiceNow experienced an increasing trend of 9.4% over the period of 2017 to 2020, with the score changing from 0.96 in 2017 to 1.05 in 2020. It should be noted that C5-ServiceNow was inefficient in 2017 and 2018; then, it turned to be efficient in 2019 with a score of 1.0 and increased to 1.05 in 2020, which thanked both the increase in gross profit and the decrease in the cost of goods sold and operating expenses. C7-Dropbox and C8-Shopify experienced the same trend with C3-Microsoft and C5-ServiceNow, with the scores increasing over a 4-year period. Scores of C7-Dropbox increased from 0.85 in 2017 to 1.01 in 2020, approximately 18.8%, and the score of C8-Shopify improved from 0.93 to 1.11, about 19.4%. These two DMUs also turned from inefficient to efficient.

On the other hand, C6-Splunk, C9-Atlassian, and C10-Twilio went through the decreasing trend of 41.3%, 10.4%, and 51.8%, respectively, due to the higher cost of revenue and operating cost and also the lower gross profit and total revenue. Among these three, C6-Splunk and C10-Twilio turned from efficient to inefficient.

4.2. The Choice of Replicas Illustration

This stage demonstrates the efficiency score of ten DMUs in 2020 and then compares the actual scores of DMUs in 2020 obtained by 5000 replicas with those obtained by 500 replicas. The variation of scores by using different replicas within a 95% confidence interval was obtained by applying the past–present model, and the results are presented in

Table 5. Comparisons of 5000 and 500 replicas (Fisher 95%) for the year 2020.

DMUs	5000 Replicas			500 Replicas			Difference
	97.50%	DEA	2.50%	97.50%	DEA	2.50%	97.50%	2.50%
C1	1.00	1.00	1.00	1.00	1	1.00	0	0
C2	5.68	1.35	0.62	5.68	0.89	0.63	0	−0.01
C3	7.71	1.55	1.06	7.62	1.65	1.02	0.09	0.04
C4	2.72	1.61	0.59	2.72	1.03	0.56	0	0.03
C5	1.82	1.05	0.55	1.83	1.04	0.57	−0.01	−0.02
C6	1.82	0.74	0.52	1.73	1.51	0.50	0.09	0.02
C7	1.35	1.01	0.53	1.36	1.39	0.52	−0.01	0.01
C8	2.42	1.11	0.37	2.27	0.41	0.36	0.15	0.01
C9	2.54	1.29	1.08	2.52	1.57	1.09	0.02	−0.01
C10	2.87	0.68	0.40	2.56	1.18	0.39	0.31	0.01

. In the research, most of the results of 5000 replicas and 500 replicas were statistically negligibly small except for the scores of C10-Twilio with the difference of 0.31. Thus, 5000 replicas can be applied in the next stage.

Before conducting further analysis, the correlation analysis is assessed to ensure the appropriate inputs and outputs. The value of the correlation coefficient is always between (−1) and (+1), and if the value is near (±1), it means that there is a stronger linear relationship between factors. The results of correlation analysis reported in

Table 6. Correlation matrix of inputs and outputs in 2020.

	Cost of Revenue	Operating Expense	Gross Profit	Total Revenue
Cost of revenue	1.00	0.99	0.72	0.97
Operating expense	0.99	1.00	0.64	0.93
Gross profit	0.72	0.64	1.00	0.87
Total revenue	0.97	0.93	0.87	1.00

confirm that the selected inputs and outputs in this study are suitable, which is proved by the high correlation between inputs and outputs.

4.3. Illustration of Selecting the Prediction Model

This section illustrates the forecasted efficiency of ten DMUs through three different prediction models in the past–present–future framework in DEA. The most appropriate prediction model is selected based on the comparison of forecasted results obtained by each prediction model with the actual efficiency scores (for the year 2020 only).

The Trend, Lucas weight, and the hybrid are three considered prediction models in this stage. The results of efficiency scores obtained by these models and the actual efficiency score of all DMUs in 2020 are compared. The comparison indicates that the results of these models were consistent with all actual efficiency of all ten DMUs were within the 95% confident interval of all three prediction models. However, the average difference between forecasted and actual scores is 24.7% with the trend prediction, 10.7% with the Lucas weight prediction, and 12.4% with the hybrid model. Therefore, the Lucas weight model is the most appropriate prediction model in this study, and it would be used to get the efficiency scores of ten DMUs in the future period from 2021 to 2024.

Table 7. Forecast scores by the Lucas weight model, actual scores, and confidence interval in 2020.

DMUs	97.50%	Forecasted Score	Actual Score	2.50%
C1	1.00	1.00	1.00	1.00
C2	5.15	1.38	1.35	0.87
C3	7.03	1.43	1.55	1.11
C4	2.45	1.62	1.61	0.70
C5	1.65	1.02	1.05	0.67
C6	1.64	0.86	0.74	0.59
C7	1.29	0.89	1.01	0.62
C8	2.21	1.07	1.11	0.46
C9	2.36	1.53	1.29	1.14
C10	2.46	1.04	0.68	0.46

exposes the difference between the actual and predicted results in 2020 (predicted by the Lucas weight model).

4.4. Future Forecast Efficiency Evaluation over the Future Period of 2021 to 2024

The efficiency scores of ten DMUs in the future period of 2021 to 2024 are predicted by using the past–present–future model with the Lucas weight prediction. The obtained scores are displayed in

Table 8. Efficiency score from 2021 to 2024.

DMUs	2021	2022	2023	2024	DMUs	2021	2022	2023	2024
Efficiency Scores					Ranking
C1	1.00	1.00	1.00	1.00	C1	7	7	7	8
C2	1.38	1.38	1.38	1.38	C2	1	2	3	4
C3	1.45	1.46	1.46	1.45	C3	2	3	4	3
C4	1.62	1.62	1.62	1.62	C4	3	1	1	1
C5	1.02	1.03	1.03	1.02	C5	5	6	6	6
C6	0.86	0.85	0.84	0.85	C6	9	10	10	10
C7	0.90	0.90	0.90	0.90	C7	10	9	9	9
C8	1.08	1.08	1.08	1.08	C8	6	5	5	5
C9	1.50	1.47	1.47	1.48	C9	4	4	2	2
C10	1.03	1.02	1.02	1.03	C10	8	8	8	7

, along with the bar chart illustration in Figure 3.

The obtained scores in 4 years tend to be a stable trend corresponding with the insignificant changes in efficiency scores of all DMUs from 2021 to 2024. From 2021 to 2024, ten DMUs can be categorized into two different groups: inefficient and efficient groups. The inefficient group includes C6-Splunk and C7-Dropbox, with the corresponding efficiency score smaller than 1. On the other hand, eight DMUs are included in the efficient group as the obtained efficiency scores of those are equal and higher than 1.

It is also notable that the average efficiency score from 2021 to 2024 is slightly higher than that from 2017 to 2020. However, the average efficiency score of C2-Alphabet and C6-Splunk from 2021 to 2023 decreased compared to those during 2017 to 2020, as can be seen in Figure 4.

4.5. Efficiency Improvement over the Research Period of 2017 to 2024

In this section, the Malmquist productivity index is applied to investigate the efficiency changes over the period of 2017 to 2024. The change in efficiency is categorized into three groups: the catch-up index measuring the efficiency changes of each DMU, frontier-shift measuring the movement of the efficiency frontier over two periods, and the MPI, which is the production of catch-up and frontier-shift.

Table 9. Efficiency improvement from 2017 to 2024.

DMUs	Catch-Up	Frontier-Shift	MPI
C1	1.00	1.04	1.04
C2	1.00	1.01	1.01
C3	1.05	1.01	1.06
C4	0.99	1.03	1.01
C5	1.01	1.01	1.02
C6	0.97	1.03	1.00
C7	1.01	1.01	1.02
C8	1.02	0.94	0.96
C9	1.01	0.96	0.97
C10	0.98	0.89	0.89
Average	1.01	0.99	1.00

summarizes the obtained results.

According to the catch-up scores in Table 9, three DMUs (C4-Salesforce, C6-Splunk, and C10-Twilio) showed a weakening in efficiency, with a catch-up score lower than 1. On the contrary, five DMUs (C3-Microsoft, C5-ServiceNow, C7-Dropbox, C8-Shopify, and C9-Atlassian) demonstrated progress from 2017 to 2024 with a catch-up score above 1. The rest two DMUs (C1-Amazon and C2-Alphabet) displayed no changes in the degree of their efforts attained for improving their efficiency. The results of the frontier-shift indicated the technological regress of C8-Shopify, C9-Atlassian, and C10-Twilio. The rest of the seven DMUs experienced technological progress with the corresponding frontier-shift score higher than 1. MPI is the production of catch-up and frontier-shift represents the total improvement over two time periods. The results in Table 9 indicated the efficiency improvement of seven DMUs from C1 to C7 with the MPI higher than 1. On the other hand, three DMUs (C8-Shopify, C9-Atlassian, and C10-Twilio) showed the regress with the MPI lower than 1.

5. Conclusions and Discussion

The rapid growth in the cloud computing sector, along with its importance, measuring and forecasting the performance of cloud computing providers was considered to be very necessary. Therefore, this study applied the Super-SBM and the past–present–future model in DEA based on the resampling technique to evaluate and predict the performance efficiency of ten cloud computing providers in the United States of America, which is one of the most developing markets in terms of cloud computing services.

After forming the research framework, the first step was taken is to select the appropriate indicators. Since there was no related study measuring the efficiency of cloud computing providers found in the literature, the inputs and outputs of this study were selected based on reviewing the previous studies that measured the performance of service providers.

The second step was to measure the performance efficiency of ten selected providers over the past and present period (2017–2020) by applying the Super-SBM. This step found that 30% of cloud computing service providers were inefficient due to the problem in controlling the cost of goods sold, operating cost, and the capacity to generate revenues of these providers. However, when observing the whole ten providers as one, the results indicated that the cloud computing sector was efficient with an average efficiency score of 1.19 over the period of 2017 to 2020. During this period, it was also notable that most of the providers fluctuated in terms of efficiency with the corresponding unstable efficiency scores.

After assessing the performance over the past and present periods, the efficiency in the future was predicted by applying the past–present–future model in DEA that was based on the resampling method. Since accuracy was the most severe concern in forecasting, choosing an appropriate number of replicas and a correct prediction model was considered first in this step. In order to ensure the right choice of the number of resampling, 5000 replicas and 500 replicas were conducted. The results of these two were similar in most cases. However, the results of 5000 replicas and 500 replicas in the case of C10 showed a significant difference. Thus, the results of 5000 replicas were chosen. The next step was to identify the most accurate prediction model among the three available models: trend, Lucas weight, and the hybrid model. The accuracy of these three models was checked by comparing the efficiency scores calculated from each prediction model with the actual scores obtained by the Super-SBM model. The one with the smallest forecast-actual ratio would be selected. After the comparison, the Lucas weight model was the selected prediction model with the lowest difference between the forecast and actual score (10.7%). The final step was to forecast the efficiency of these ten providers by using the Lucas weight prediction model. The obtained results showed stable efficiency over the future period of 2021 to 2024. From 2021 to 2024, all providers were forecasted to perform more stably than their performance from 2017 to 2020, with the corresponding efficiency scores slightly changed. It was also noted that the average efficiency score of ten providers in the future would be slightly lower than that over the past period from 2017 to 2020. This result might be explained by the significantly increasing demand for cloud computing services in 2019 and 2020 due to the COVID-19 pandemic.

In the next step, the DEA Malmquist productivity index was used to investigate the efficiency improvement during the research period of 2017 to 2024. The results of MPI revealed that during this research period, 70% of the selected providers showed efficiency progress with an MPI value higher than 1.

The results of this study provided valuable information and practical implications to the decision-makers. The measure of the past and present performance helped better understand how efficient the firm was and how it compared to other providers in the same field. The result of forecasted efficiency helps a firm to have an insight picture of how it would perform in the future. That can help the firm make the appropriate policies and strategies to be more efficient and gain competitive advantages. Additionally, this study will fill the gap in the literature as the first study that measures and predicts the performance efficiency of cloud computing service providers, which will provide a helpful reference for future studies. However, there are still many things that need to be done in this research area. For example, this study only focused on measuring the performance in general without in detail analyzing the performance of each provider. This study applied the new approach of DEA to measure and forecast efficiency at the same time. Thus, further research can apply another forecasting technique to compare the accuracy of the forecast model. Finally, economic efficiency is not the only criterion that needs to be considered regarding the performance of the decision-making unit. In fact, there are two major issues in the cloud computing industry related to how to develop services with high levels of quality of service and maximize the benefit for the service providers. To answer these two questions, the general evaluation of the economic efficiency of cloud computing providers is critical, but there is not enough. In this study, the performance efficiency was evaluated based on the assumption of the non-cooperative game among cloud computing providers in which providers selfishly choose the best strategy to maximize their payoff. However, it will be more practical if the study can analyze the cooperative behavior and conflict between providers based on the theoretical tool-game theory. Further research should focus on the resource allocation problems which arise in the cloud computing industry by applying non-cooperative and cooperative game theory.