Integral-Valued Pythagorean Fuzzy-Set-Based Dyna Q+ Framework for Task Scheduling in Cloud Computing

Abstract

Task scheduling is a critical challenge in cloud computing systems, greatly impacting their performance. Task scheduling is a nondeterministic polynomial time hard (NP-Hard) problem that complicates the search for nearly optimal solutions. Five major uncertainty parameters, i.e., security, traffic, workload, availability, and price, influence task scheduling decisions. The primary rationale for selecting these uncertainty parameters lies in the challenge of accurately measuring their values, as empirical estimations often diverge from the actual values. The integral-valued Pythagorean fuzzy set (IVPFS) is a promising mathematical framework to deal with parametric uncertainties. The Dyna Q+ algorithm is the updated form of the Dyna Q agent designed specifically for dynamic computing environments by providing bonus rewards to non-exploited states. In this paper, the Dyna Q+ agent is enriched with the IVPFS mathematical framework to make intelligent task scheduling decisions. The performance of the proposed IVPFS Dyna Q+ task scheduler is tested using the CloudSim 3.3 simulator. The execution time is reduced by 90%, the makespan time is also reduced by 90%, the operation cost is below 50%, and the resource utilization rate is improved by 95%, all of these parameters meeting the desired standards or expectations. The results are also further validated using an expected value analysis methodology that confirms the good performance of the task scheduler. A better balance between exploration and exploitation through rigorous action-based learning is achieved by the Dyna Q+ agent.

1. Introduction

Today’s business environment is very complex and cannot easily be supported by traditional IT solutions due to the explosive growth in application sizes, large volume of e-content generation, exponential growth in the computing capability of devices, introduction of newer architectures, etc. Cloud computing is one of the extremely important on-demand computing platforms used to perform tasks instead of these being performed using self-configurable computing resources. Several distinct properties of cloud computing include multitenancy, elasticity, pay per use features, resiliency, workload movement ease, and so on. Several advantages offered by high-end computation include scalability, high performance, intense computational power, better availability, reduced cost of operation, and many more [1,2,3].

Despite all the advantages, cloud computing is subject to a lot of challenges pertaining to security, cost of operation, resource management, multi-cloud management, performance, segmented adaption, application migration, interoperability, reliability, and availability. Out of all the challenges, performance-related challenges (task scheduling, load balancing, and resource management) are of paramount importance because good performance is vital for the overall success of cloud computing. Poor performance leads to the dissatisfaction of users and, in turn, a decrease in revenue generation. Also, it introduces hindrances to the seamless successful execution of high-end applications [4,5,6].

Task scheduling, being a paramount performance concern, has garnered significant attention from researchers over the past few decades. Precise scheduling of tasks is considered a nondeterministic polynomial time hard (NP-Hard) problem because it is difficult to find near-optimal solutions within the stipulated time limits under conditions of uncertainty using classical algorithms. Machine learning algorithms have been found to be very promising in tackling task scheduling problems [7]. However, these algorithms also suffer from lower convergence rates, and higher tendencies to converge toward a local optimal solution. Hence, there is a need to develop intelligent, uncertainty-proof algorithms by properly balancing exploration and exploitation activities and to achieve enhanced results in very few iterations of training [8]. The existing task scheduling algorithms have limitations in managing uncertainty, leading to higher task failure rates. They are often reactive, stochastic, and fuzzy, lacking adaptability and dynamic computing capabilities, and tend to converge on suboptimal solutions.

Uncertainty is one of the main issues that affect the computing efficiency of cloud computing. Five major uncertainty parameters in cloud computing are security, traffic, availability, price, and workload. Vital sources of uncertainty are data (variety, value), virtualization, job arrival rates, job migration rates, energy consumption, fault tolerance, scalability, dynamic pricing, resource availability, elasticity, consolidation, communication, replication, elastic provisioning, etc. The performance metrics affected due to uncertainty are throughput, scalability, cost, adaptability, accuracy, transparency, and response time. Hence, there is a necessity to efficiently handle the parameters causing uncertainty and then make intelligent task scheduling decisions [9,10].

The integral-valued Pythagorean fuzzy set (IVPFS) is a modification of the fuzzy set. The IVPFS is based on intuitionistic fuzzy sets. The uncertainty is represented as the membership and non-membership degree in terms of the integral value, which is in the range of [0, 1]. With respect to the IVPFS, the sum of the square of membership and non-membership values is less than 1. The IVPFS, composed of operators like concentration, dilation, and normalization, helps in handling imprecise, incomplete, and inadequate data to express an opinion in precise numerical values [11,12].

The Dyna Q+ algorithm is a modified form of the Dyna Q algorithm [13]. The Dyna Q+ learning agent receives bonus rewards for actions that have not been carried out for a longer time. It updates the agent’s rewards based on time. If the Q-learning agent has visited a state long back, the rewards obtained will be increased, which will empower the agent to visit that particular state again. The Dyna Q+ agent is suitable for dynamically changing environments and provides exploration bonus rewards that encourage exploration activity [14]. Some potential applications of the proposed work include resource allocation, cost management, load balancing, production planning, inventory management, and maintenance scheduling.

In this paper, the Dyna Q+ algorithm is made uncertainty-proof with the application of the IVPFS mathematical framework. The IVPFS mathematical model exhibits excellent ability to handle imprecise and vague parameters of tasks and virtual machines. The Dyna Q+ learning agent is designed to adapt to the changing dynamics of cloud systems. Scheduling policies are formulated through vigorous action-based learning.

The main objectives of the paper are as follows:

• A mathematical representation of the cloud computing system model is constructed for task scheduling, and definitions of the performance metrics are given;
• Mathematical definitions of the performance metrics are set to evaluate the efficiency of the proposed framework;
• A novel IVPFS-Dyna Q+ task scheduler is designed with the supporting algorithms as a component of the framework;
• The IVPFS-Dyna Q+ task scheduler is simulated using the CloudSim 3.3 simulator by considering three different types of workloads: a random dataset, GOCJ dataset, and synthetic dataset;
• The results are validated through expected value analysis of the proposed IVPFS-Dyna Q+ task scheduler.

The remaining part of the paper is organized as follows: Section 2 discusses the existing works. Section 3 presents the cloud system model considered for operation along with the definitions of the performance metrics. Section 4 presents the proposed IVPFS-enabled task scheduler with two subcomponents: the interval-valued Pythagorean fuzzy set resource pool (IVPFS_RP), and the interval-valued Pythagorean fuzzy set client workflow (IVPFS_CWF). Section 5 presents an expected value analysis of the proposed work. Section 6 presents the results and discussion, and finally, Section 7 presents the conclusion.

2. Related Work

Tong et al. [15] present a novel task scheduling scheme based on Q-learning embedded with a heterogeneous earliest finish time policy. The scheme works in two phases. The first phase consists of sorting the available list of tasks in the optimal order using Q-learning. The second phase involves allocating the processor for the tasks using the earliest finish time policy. The static scheduling problem is solved using the proposed scheduling scheme. By providing immediate rewards, the Q-learning agent is made to go through a better learning experience. The immediate reward for each action is provided using the upward rank. After every action, the Q-table is updated through a self-learning process. The performance is tested against several benchmarks, and the results obtained reveal a significant reduction in makespan time and response time. However, the scheme leads to an overestimation of policies and may be too optimistic in policy formation.

Kruekaew et al. [16] discuss a hybrid bee colony algorithm with a reinforcement learning technique embedded to balance the load of virtual machines in the cloud. The main goal of load balancing is to ensure that the load is balanced across all virtual machines. It must be ensured that none of the virtual machines are overloaded or underloaded. By applying reinforcement for every action by the agent, the speed of the bee colony algorithm is enhanced. Task scheduling decisions are made by making predictions using an appropriate scheduling table. A mathematical model is formulated to include the following performance metrics: cost of operation, resource utilization, and makespan time. The algorithm is tested on the CloudSim simulator by considering three random datasets, and the performance is good with respect to resource utilization and throughput. However, the performance of the scheduler is not optimized on every dataset considered for evaluation. There might be a chance that it generates poor-quality solutions and that it converges toward suboptimal solutions.

Hou et al. [17] present a specialized review of the energy efficient task scheduling algorithm based on deep reinforcement learning for cloud computing. Energy consumption is of primary concern in cloud data centers. The potential of deep reinforcement learning (DRL) is to make energy efficient task scheduling decisions. First, a classification of energy models in the cloud data centers is carried out. An energy consumption model is developed by considering the energy consumed by the data centers. However, measuring the power consumed in every partition is challenging in practical scenarios. The existing DRL methods are analyzed by considering several benchmarks with respect to type, space, state, action, and reward metrics. A brief guideline is provided for the formulation of a reward function and the objective to be considered while scheduling tasks. From the survey, it was found that there is a lack of performance comparisons between DRL scheduling algorithms. Even the effectiveness of the algorithms is not determined with respect to policy formulation and value computation.

Neelakantan et al. [18] discuss an optimized machine learning strategy to effectively schedule the jobs in the cloud. Job scheduling for cloud environments is considered a problematic job as these environments have heterogeneous operating systems and necessitate user requirement validation by a virtual machine before scheduling. A novel hybrid framework composed of a convolutional neural network and whale optimization strategy (CNN-W) was proposed for task scheduling. The cloud framework is composed of a fixed number of virtual machines for task execution. The deadline of the tasks was considered a metric for performing scheduling using CNN-W. In order to reduce resource consumption and the task execution time, the deadline was given higher priority. The framework first allocated the tasks, followed by deadline prediction and priority setting. Priorities were assigned based on the duration of tasks. Short-duration tasks were given higher priority, and the remaining jobs were given lower priority. The performance of the framework was tested on the Python platform by considering several benchmark datasets. The prediction accuracy was enhanced, but the fault tolerance was not considered. As a result, the virtual machines became more vulnerable to damage. The robustness score was lower, and because of this, the task execution performance was low.

Attiya et al. [19] discussed a hybrid algorithm combining the Manta ray foraging optimization (MRFO) and salp warm optimization (SSO) algorithms for the scheduling of internet of things (IoT) tasks in the cloud. The MRFO metaheuristic algorithm uses three types of foraging operators, i.e., chain, cyclone, and somersault, for solving the optimization problem. The SSO is another metaheuristic algorithm inspired by the swarming behavior of salps in the ocean. The random selection of reference points in MRFO leads to weakened search capability for promising solutions. However, the search ability of MRFO is improved by incorporating SSO. The performance of MRFO-SSA was tested by considering different real-world datasets, which resulted in higher throughput and an improved convergence rate. However, the MRFO-SSA was not able to balance between exploration and exploitation operations.

The drawbacks observed in the existing works are as follows:

• Uncertainties of the task and resource parameters are poorly or insufficiently modeled;
• They show an inability to search within the large search spaces of cloud systems, resulting in a low probability of arriving at a global optimal solution;
• They come with a high probability of task failure due to the improper mapping of tasks to resources;
• Fundamental approaches available in the literature are reactive, stochastic, and fuzzy, and these approaches lack adaptability, dynamic computing ability, and the ability to converge towards suboptimal solutions;
• The robustness scores achieved are lower as cloud resources are more vulnerable to becoming damaged;
• There is an improper balance between exploration and exploitation, which results in poor task scheduling policies;
• Existing task scheduling policies are static as they do not deal with highly dynamic cloud scenarios;
• Some of the task schedulers are inflexible in handling multi-cloud environments as they are trained for specific types of cloud environment;
• Scheduling policies are found to violate SLA due to ineffective scheduling.

3. System Model

Consider a typical cloud computing environment involving m and the collection of resource pools, RP = $<{RP}_1,{RP}_2,{RP}_3,\dots ,{RP}_m>$, where m represents the total count of virtual machines. Each of the resources in ${RP}_i$ has resources like RAM, CPU, and bandwidth. Similarly, n and the independent collection of client workflows are available for execution, ${WF}_i=<{WF}_1,{WF}_2,{WF}_3,\dots ,{WF}_n>$, where n represents the total count of tasks. The uncertainty in the resource and client workflow are handled by applying the IVPFS, i.e., $IVPFS\_RP=IVPFS\_RP<IVPFS\_{RP}_1,\dots ,{IVPFS\_RP}_m>$, and $IVPFS\_WF=<IVPFS\_{WF}_1,\dots ,{IVPFS\_WF}_n>$.

The system model determines the stability of an environment for task scheduling by considering sensitive performance objectives like workflow execution time, $\mathrm{W}\mathrm{F}\mathrm{E}\mathrm{T}(\mathrm{D}\mathrm{y}\mathrm{n}\mathrm{a} \mathrm{Q}+)$; makespan time, $MST\left(\mathrm{D}\mathrm{y}\mathrm{n}\mathrm{a} \mathrm{Q}+\right)$; operation cost, OC$(\mathrm{D}\mathrm{y}\mathrm{n}\mathrm{a} \mathrm{Q}+)$; and resource utilization rate, $\mathrm{R}\mathrm{U}(\mathrm{D}\mathrm{y}\mathrm{n}\mathrm{a} \mathrm{Q}+)$. An optimal solution is designed by computing the fitness of the scheduling solution. The main performance objectives of the proposed framework are defined below:

PO1: Workflow Execution Time $\left(\mathbf{W}\mathbf{F}\mathbf{E}\mathbf{T}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$: This is the time taken by the cloud system to complete the last workflow.

$$WFET(Dyna Q+({RP}_i))=\sum _{T_i\in {vm}_i}{\displaystyle \frac{\sum length\left({WF}_i\right)}{CPU\left({RP}_i\right)}}$$

(1)

where the $length\left({WF}_i\right)$ is defined as the summation of the number of instructions taken by each of the workflows in the workflow, set ${WF}_i\subseteq WF$, to be executed, and $CPU\left({RP}_i\right)$ is the CPU rate of ${RP}_i$ for processing.

The fitness of $WFET(Dyna Q+({RP}_i\left)\right)$ is determined as the ratio of the minimum workflow execution time to the actual workflow execution time.

$$F(WFET(Dyna Q+({RP}_i)))={\displaystyle \frac{Min\left(WFET\right(Dyna Q+\left({vm}_i\right))}{WFET(Dyna Q+({vm}_i\left)\right)}}$$

(2)

PO2: Makespan Time $\left(\mathbf{M}\mathbf{S}\mathbf{T}\right(\mathit{D}\mathit{y}\mathit{n}\mathit{a} \mathit{Q}+\left)\right)$: This is defined as the maximum workflow execution time of all the resource pools in the cloud system.

$$MST\left(Dyna Q+\left({RP}_i\right)\right)=Max\left(WFET\right(Dyna Q+\left({RP}_i\right)\left)\right)$$

(3)

The fitness of $MST(Dyna Q+({RP}_i\left)\right)$ is determined as the ratio of the minimum makespan time to the actual makespan time.

$$F(MST(Dyna Q+({RP}_i)))={\displaystyle \frac{Min\left(MST\right(Dyna Q+\left({RP}_i\right))}{MST(Dyna Q+({RP}_i\left)\right)}}$$

(4)

PO3: Operation Cost $\left(\mathbf{O}\mathbf{C}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$: This is defined as the cost incurred by the resource pool in processing the requests.

$$OC\left(Dyna Q+\right)=\sum _{i=1}^m(C_1 \ast TET(Dyna Q+\left({RP}_i\right)\left)\right)+(C_2 \ast TET(Dyna Q+\left({RP}_i\right)\left)\right)+(C_3 \ast TET(Dyna Q+\left({RP}_i\right)\left)\right))$$

(5)

where $C_1$ represents the CPU usage cost, $C_2$ represents the memory usage cost, and $C_3$ represents the bandwidth usage cost.

The fitness of F(OC $(\mathrm{D}\mathrm{y}\mathrm{n}\mathrm{a} \mathrm{Q}+))$ is determined as the ratio of the minimum operation cost to the operation cost.

$$F(OC(Dyna Q+({RP}_i)))={\displaystyle \frac{Min\left(OC\right(Dyna Q+\left({RP}_i\right))}{OC(Dyna Q+({RP}_i\left)\right)}}$$

(6)

PO4: Resource utilization rate $\left(\mathbf{R}\mathbf{U}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$: This is defined as the summation of the memory load on the resource pool, $LM\left({vm}_i\right)$, and the CPU load on the resource pool, $LC\left({RP}_i\right)$.

$$RU\left(Dyna Q+\right)=\sum _{i=1}^{i=m}(LM\left({RP}_i\right)+LC\left({RP}_i\right))$$

(7)

where $LM\left({RP}_i\right)$ is computed by considering the memory used before the execution of the task set, $BM\left(T_i\right)$, the memory occupied by the workflow, ${WF}_{i}OM\left({WF}_i\right)$, and the total memory available in the resource pool, $TM\left({RP}_i\right)$.

$$LM\left({RP}_i\right)=BM\left({WF}_i\right)+{\displaystyle \frac{OM\left({WF}_i\right)}{TM(\left({RP}_i\right)}}$$

(8)

Similarly $LC\left({RP}_i\right)$ is computed by considering the CPU used before the execution of the workflow, $BC\left({WF}_i\right)$, the CPU occupied by the workflow, $OM\left({WF}_i\right)$, and the total CPU available in the resource pool, $TM(\left({RP}_i\right)$.

$$LC\left({RP}_i\right)=BC\left({WF}_i\right)+{\displaystyle \frac{OC\left({WF}_i\right)}{TC(\left({RP}_i\right)}},$$

(9)

The fitness of F(RU$(\mathrm{D}\mathrm{y}\mathrm{n}\mathrm{a} \mathrm{Q}+))$ is determined as the ratio of the weighted proportion of $LM\left({RP}_i\right)$ and $LM\left({RP}_i\right)$.

$$F(RU(Dyna Q+))={\displaystyle \frac{w_1}{1-LM\left({RP}_i\right)}} \ast {\displaystyle \frac{w_2}{1-LC\left({RP}_i\right)}}$$

(10)

where $w_1$ and $w_2$ are the weights assigned to the memory and CPU, such that $w_1+w_2=1$.

The overall fitness function is computed as follows:

$$\begin{array}{ll}F\left(Dyna Q+\right)=& {\gamma }_1\ast F\left(TET\left(Dyna Q+\left({RP}_i\right)\right)\right)+{\gamma }_2\ast F\left(MST\left(Dyna Q+\left({RP}_i\right)\right)\right)+{\gamma }_3\\ & \ast F\left(OC\left(Dyna Q+\left({RP}_i\right)\right)\right)+{\gamma }_4\ast F\left(RU\left(Dyna Q+\left({RP}_i\right)\right)\right)\end{array}$$

(11)

where, ${\gamma }_1$, ${\gamma }_2$, and ${\gamma }_3$ are the balance coefficients required to determine optimal solution. A higher fitness function value leads to an optimal solution.

4. Proposed Work

The proposed work is mainly composed of three subcomponents, which are the interval-valued Pythagorean fuzzy set resource pool (IVPFS_RP), interval-valued Pythagorean fuzzy set client workflow (IVPFS_WF), and Dyna Q+ task scheduler. The IVPFS_RP is responsible for removing the parametric uncertainties in the virtual machine resource pool. Similarly, the IVPFS_WF is responsible for reducing the parametric uncertainties in the client workflow. The Dyna Q+ task scheduler generates the scheduling policies over the reduced set of resources pool and client workflows. The Dyna Q+ agent mainly executes Dyna Q logic and provides additional bonus rewards for actions that are left pending for a longer duration through the exploration activity.

4.1. IVPFS_RP

The virtual machine resources in cloud computing systems are associated with several forms of uncertainties, which include network congestion, improper placement of resources, loss of data, inadequate processor cores, compatibility problems, frequent repartitioning, more downtime, resource contention between the collocated virtual machines, overloading of resources, random variation in the processing capability, and so on. The uncertainties in the resources are reduced with the application of the IVPFS.

The IVPFS form of the resource pool, ${RP}_i$, is defined as follows.

$IVPFS\left({RP}_i\right)=\{<{RP}_i,\left[{\mu }^l\left({RP}_i\right),{\mu }^u\left({RP}_i\right)\right],\left[V^l\left({RP}_i\right),V^u\left({RP}_i\right)\right]>:{RP}_i\in RP\}$, where ${\mu }^l\left({RP}_i\right),{\mu }^u\left({RP}_i\right)$ represent the lower and upper membership degrees of ${RP}_i$, and $V^l\left({RP}_i\right),V^u\left({RP}_i\right)$ represent the lower and upper non-membership degrees of ${WF}_i$. These satisfy the conditions $1\ge {\mu }^u\left({RP}_i\right)\ge {\mu }^l\left({RP}_i\right)\ge 0$ and $0\le V^u\left({RP}_i\right)\le V^l\left({RP}_i\right)\le 1$. An additional constraint is $0\le {{\mu }^l\left({RP}_i\right)}^2+{V^l\left({RP}_i\right)}^2\le 1$, and $0\le {{\mu }^u\left({RP}_i\right)}^2+{V^u\left({RP}_i\right)}^2\le 1$. The approximate degree of $IVPFS\left({RP}_i\right)$ is computed as follows:

$$\prod {RP}_i=[{\pi }^l\left({RP}_i\right), {\pi }^u\left({RP}_i\right)]$$

(12)

$$\prod {RP}_i=[\sqrt{1-{{\mu }^l\left({RP}_i\right)}^2-{V^l\left({RP}_i\right)}^2} , \sqrt{1-{{\mu }^u\left({RP}_i\right)}^2-{V^u\left({RP}_i\right)}^2}]$$

(13)

The workflow of the IVPFS_RP is provided in Algorithm 1.

Algorithm 1: Working of IVPFS_RP

1: Begin 2: Input: $RP=<{RP}_1,{RP}_2,{RP}_3,\dots ,{RP}_n>$ 3: Output:$IVPFS\_RP=<IVFFN\_{RP}_1,\dots ,{IVFFN\_RP}_m>$ 4: Training phase of IVPFS_RPUR 5: for each incoming training client workflow ${RP}_i\in RP$ do 6:    Compute lower and upper membership degree of ${RP}_i$    ${\mu }^l\left({RP}_i\right), {\mu }^u\left({RP}_i\right)$ 7:    Compute lower and upper non membership degree of ${WF}_i$                 $V^l\left({RP}_i\right), V^u\left({RP}_i\right)$ 8:  end for 9: Testing: IVPFS_RPUR 10: for every incoming test client workflow ${RP}_i\in RP$ do 11:    Compute lower and upper membership degree of ${RP}_i$    ${\mu }^l\left({RP}_i\right), {\mu }^u\left({RP}_i\right)$ 12:    Compute lower and upper non membership degree of ${RP}_i$                 $V^l\left({RP}_i\right), V^u\left({RP}_i\right)$ 13:  end for 14:   Enumerate $IVPFS\_RP$       $IVPFS\_RP\colon\colon =IVPFS\_{RP}_i\cup ({\mu }^l\left({RP}_i\right), {\mu }^u\left({RP}_i\right),V^l\left({RP}_i\right), V^u\left({RP}_i\right))$ 15: Output IVFTFS_RP resource pool           $IVPFS\_RP=<IVPFS\_{RP}_1,\dots ,{IVPFS\_RP}_n>$ 16: End

Client workflows in cloud computing systems are associated with several forms of uncertainties, which include variations in the task arrival rate, poor data representation, fluctuations in the data volume, frequent pre-emption of tasks, unrealistic task deadlines, improper task deployment, task parallelization, failure of task execution, high energy consumption, and so on.

The IVPFS form of the client workflow, ${WF}_i$, is defined as follows.

$IVPFS\left({WF}_i\right)=\{<{WF}_i,\left[{\mu }^l\left({WF}_i\right),{\mu }^u\left({WF}_i\right)\right],\left[V^l\left({WF}_i\right),V^u\left({WF}_i\right)\right]>:{WF}_i\in WF\}$, where ${\mu }^l\left({WF}_i\right),{\mu }^u\left({WF}_i\right)$ represent the lower and upper membership degrees of ${WF}_i$, and $V^l\left({WF}_i\right),{ V}^u\left({WF}_i\right)$ represent the lower and upper non-membership degrees of ${WF}_i$. These satisfy the condition $1\ge {\mu }^u\left({WF}_i\right)\ge {\mu }^l\left({WF}_i\right)\ge 0$ and $0\le V^u\left({WF}_i\right)\le V^l\left({WF}_i\right)\le 1$. Additional constraints include $0\le {{\mu }^l\left({WF}_i\right)}^2+{V^l\left({WF}_i\right)}^2\le 1$ and $0\le {{\mu }^u\left({WF}_i\right)}^2+{V^u\left({WF}_i\right)}^2\le 1$. The approximate degree of $IVPFS\left({WF}_i\right)$ is computed as follows:

$$\prod {WF}_i=[{\pi }^l\left({WF}_i\right), {\pi }^u\left({WF}_i\right)]$$

(14)

$$\prod {WF}_i=[\sqrt{1-{{\mu }^l\left({WF}_i\right)}^2-{V^l\left({WF}_i\right)}^2} , \sqrt{1-{{\mu }^u\left({WF}_i\right)}^2-{V^u\left({WF}_i\right)}^2}]$$

(15)

The protocol of the IVPFS_WF is provided in Algorithm 2.

Algorithm 2: Working of IVPFS_WFUR

1: Begin 2: Input: $WF=<{WF}_1,{WF}_2,{WF}_3,\dots ,{WF}_m>$ 3: Output:$IVPFS\_WF=<IVPFS\_{WF}_1,\dots ,{IVPFS\_WF}_M>$ 4: Training phase of IVPFS_WF 5: for each incoming training client workflow ${WF}_i\in WF$ do 6:    Compute lower and upper membership degree of ${WF}_i$                 ${\mu }^l\left({WF}_i\right), {\mu }^u\left({WF}_i\right)$ 7:    Compute lower and upper non membership degree of ${WF}_i$                 $V^l\left({WF}_i\right), V^u\left({WF}_i\right)$ 8:   end for 9: Testing: IVPFS_WF 10: for every incoming test client workflow ${WF}_i\in WF$ do 11:    Compute lower and upper membership degree of ${WF}_i$                 ${\mu }^l\left({WF}_i\right), {\mu }^u\left({WF}_i\right)$ 12:    Compute lower and upper non membership degree of ${WF}_i$                 $V^l\left({WF}_i\right), V^u\left({WF}_i\right)$ 13:  end for 14:   Enumerate $IVPFS\_WF$       $IVPFS\_WF\colon\colon =IVPFS\_{WF}_i\cup ({\mu }^l\left({WF}_i\right), {\mu }^u\left({WF}_i\right),V^l\left({WF}_i\right), V^u\left({WF}_i\right))$ 15: Output IVPFS_WF client workflows           $IVPFS\_WF=<IVPFS\_{WF}_1,\dots ,{IVPFS\_WF}_m>$ 16: End

4.2. Proposed IVPFS Based Dyna Q+ Task Scheduler

The high-level architecture of the proposed IVPFS Dyna Q+ task scheduler is shown in Figure 1. It is composed of three main components: the client pool, Dyna Q+ framework, and resource pool. In the client pool component, the incoming clients submit their workflows, and each client is composed of varying degrees of workflows. The uncertainties involved in the client workflow are reduced by the functional module IVPFS_WF. Similarly, the resource pool component is composed of virtual machines with varying degree of resources. The uncertainties involved in the virtual machine resource pool are reduced using the IVPFS_RP. Finally, the Dyna Q+ framework is responsible for formulating the task scheduling policies. The Dyna Q+ framework mainly executes Dyna Q logic and provides an additional bonus reward for the actions that are left pending for a longer duration through the exploration activity. The workflow of the proposed IVPFS-based Dyna Q+ task scheduler is provided in Algorithm 3.

Algorithm 3: Working of IVPFS-Dyna Q+ task scheduler

1: Begin 2: Input:           $IVPFS\_WF=<IVPFS\_{WF}_1,\dots ,{IVPFS\_WF}_m>$            $IVPFS\_RP=<IVPFS\_{RP}_1,\dots ,{IVPFS\_RP}_n>$ 3: Output:           $\prod \left(IVPFS\_WF\to IVPFS\_RP\right)=<{\pi }_i,\dots .,{\pi }_k>$ 4: Training Phase: IVPFS-Dyna Q+ task scheduler 5: Initialize $Q(S,A)=\oslash$ and $Model(S,A)=\oslash$ for all $s\in Sanda\in A$ 6: for every training $IVPFS\_{WF}_i$ and $IVPFS\_{RP}_j$do 7:  Begin the Dyna Q+ model learning phase 8:    Initialise the Q+ learning agent state $St\leftarrow$ current non-terminal state 9:    Perform the action Aa $\leftarrow ϵ\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{e}\mathrm{d}\mathrm{y} \mathrm{p}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{y}$ (St, Aa) 10:     Take an action Aa, in which ${Aa}_i\in Aa$, change the state from $S to S^1$ 11:     Calculate the Q value        $Q\left(St,Aa\right)=Q\left(St,Aa\right)+\alpha [Rt+\gamma {max}_{a}Q\left({St}^1,Aa\right)-Q\left(St,Aa\right)]$ 12:   End the Dyna Q+ model learning phase 13:   Begin the Dyna Q+ real interaction learning phase 14:       Get a robot to some random state $St\leftarrow$ current non-terminal state 15:       Generate an action Aa $\leftarrow$ Robot Experience (St, Aa) 16:         Execute the action Aa in the environment 17:         Update the model, save the action ${Rt}_i\in Rt$, and move to next state ${St}^1$ 18:           Update the Q value: Send to the Dyna Q+ model learning phase            $UQ\left(St,Aa\right)=UQ\left(St,Aa\right)+\alpha [Rt+\gamma {max}_{a}Q\left({St}^1,Aa\right)-Q\left(St,Aa\right)]$ 19:   End the Dyna Q+ real interaction learning phase 20: End for 21: Formulate policies $\Pi \colon\colon =\Pi \bigcup {\Pi }_i$ 22: Testing Phase: IVPFS-Dyna Q+ task scheduler 23: Initialize $Q(S,A)=\oslash$ and $Model(S,A)=\oslash$ for all $s\in Sanda\in A$ 24: every testing $T2SS\_{CT}_i$ and ${T2SS\_GR}_i$ do 25: Begin the Dyna Q+ model testing phase 26:      Take an action Aa, where $Aa\in A$, update the state $St\to {St}^1$ 27:        Compute the Q value       $Q\left(St,Aa\right)=Q\left(St,Aa\right)+\alpha [Rt+\gamma {max}_{a}Q\left({St}^1,Aa\right)-Q\left(St,Aa\right)]$ 28: End the Dyna Q+ model testing phase 29: Begin the Dyna Q+ real interaction testing phase 30:     Update the testing model $Model\left(S,A\right)\leftarrow R$ and $Model\left(S,A\right)\leftarrow S^1$ 31:     Update the Q value: Send to the Dyna Q+ model testing phase       $UQ\left(St,Aa\right)=UQ\left(St,Aa\right)+\alpha [Rt+\gamma {max}_{a}Q\left({St}^1,Aa\right)-Q\left(St,Aa\right)]$ 32: End the Dyna Q+ real interaction testing phase 33: Output the $\prod \left(IVPFS\_WF\to IVPFS\_RP\right)=$ $<{\Pi }_1,{\Pi }_2,\dots ,{\Pi }_{\mathrm{p}}>$ 34: End

5. Expected Value Analysis

The expected value analysis of the proposed Dyna Q+ task scheduler is performed by considering three of the recent existing works (E1, E2, and E3). The four performance metrics (POs) considered for analysis are the workflow execution time, makespan time, operation cost, and resource utilization rate.

PO1: Workflow Execution Time $\left(\mathbf{W}\mathbf{F}\mathbf{E}\mathbf{T}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$: The expected task execution time $℮\left(\mathrm{W}\mathrm{F}\mathrm{E}\mathrm{T}\right(\mathrm{D}\mathrm{y}\mathrm{n}\mathrm{a} \mathrm{Q}+)$) is influenced by two factors, the expected length of the workflow, $℮\left(length\left({WF}_i\right)\right)$, and the expected CPU utilization rate of the resource pool, $℮\left(CPU\left({RP}_i\right)\right)$.

$$℮\left({\displaystyle \frac{WFET\left(Dyna Q+\right)}{℮\left(length\left({WF}_i\right)\right),℮\left(CPU\left({RP}_i\right)\right) }}, T\right)={\int }_{\alpha }^{\infty }{\displaystyle \frac{\sum _{\alpha ϵ\varphi }℮\left(\mathrm{W}\mathrm{F}\mathrm{E}\mathrm{T}\right(Dyna Q+\left)\right)}{|℮\left(length\left({WF}_i\right)\right),℮(CPU\left({RP}_i\right)\left)\right|}}$$

$$℮\left({\displaystyle \frac{WFET\left(Dyna Q+\right)}{℮\left(length\left({WF}_i\right)\right),℮\left(CPU\left({RP}_i\right)\right) }}, T\right)={\sum }_{c\in C}c{\int }_{\alpha }^{\infty }{\displaystyle \frac{\sum _{\alpha ϵ\varphi }℮\left(\mathrm{W}\mathrm{F}\mathrm{E}\mathrm{T}\left(Dyna Q+\right)\right)\left(\mathbf{a}\right)}{|℮\left(length\left({WF}_i\right)\right),℮(CPU\left({RP}_i\right)\left)\right|}}$$

$$℮\left({\displaystyle \frac{WFET\left(Dyna Q+\right)}{℮\left(length\left({WF}_i\right)\right),℮\left(CPU\left({RP}_i\right)\right) }}, T\right)=℮\left({\displaystyle \frac{1\ast ℮\left(length\left({WF}_i\right)\right),℮\left(CPU\left({RP}_i\right)\right)\ast ℮\left(TET\left(Dyna Q+\right)\right)}{\mathcal{P}(℮\left(length\left({WF}_i\right)\right)+℮(CPU\left({RP}_i\right)\left)\right)}}, T\right)$$

$$\mathrm{Dyna} \mathrm{Q}+: ℮\left({\displaystyle \frac{WFET\left(Dyna Q+\right)}{\mathrm{A}\Pi }}, T\right)\cong low$$

$$\mathrm{E}1=℮\left({\displaystyle \frac{WFET\left(E1\right)}{\mathrm{A}\Pi }}, T\right)\cong High$$

$$\mathrm{E}2=℮\left({\displaystyle \frac{WFET\left(E2\right)}{\mathrm{A}\Pi }}, T\right)\cong Medium$$

$$\mathrm{E}3=\mathrm{℮}\left({\displaystyle \frac{WFET\left(E3\right)}{\mathrm{A}\Pi }},T\right)\cong High$$

PO2: Makespan Time $\left(\mathit{M}\mathit{S}\mathit{T}\right(\mathit{D}\mathit{y}\mathit{n}\mathit{a} \mathit{Q}+\left)\right)$: The expected makespan time is influenced by the expected value of the maximum execution time of the resource pool.

$$℮\left(MST\left(Dyna Q+\right)\right)=℮\left(Max\right(TET(Dyna Q+({RP}_i\left)\right))$$

$$℮\left({\displaystyle \frac{MST\left(Dyna Q+\right)}{℮\left(Max\right(TET(Dyna Q+({RP}_i\left)\right)) }}, T\right)={\sum }_{c\in C}c{\int }_{\alpha }^{\infty }{\displaystyle \frac{\sum _{\alpha ϵ\varphi }℮\left(\mathrm{T}\mathrm{E}\mathrm{T}\left(Dyna Q+\right)\right)\left(\mathbf{a}\right)}{|℮(Max\left(TET\right(Dyna Q+\left({RP}_i\right)\left)\right|}}$$

$$℮\left({\displaystyle \frac{MST\left(Dyna Q+\right)}{℮\left(\right(Max\left(TET\right(Dyna Q+\left({RP}_i\right)\left)\right) }}, T\right)=℮\left({\displaystyle \frac{1\ast ℮\left(Max\right(TET(Dyna Q+({RP}_i\left)\right))}{\mathcal{P}(℮(\left(Max\right(TET(Dyna Q+({RP}_i\left)\right))}}, T\right)$$

$$\mathrm{Dyna} \mathrm{Q}+: ℮\left({\displaystyle \frac{MST\left(Dyna Q+\right)}{\mathrm{A}\Pi }}, T\right)\cong low$$

$$\mathrm{E}1=℮\left({\displaystyle \frac{MST\left(E1\right)}{\mathrm{A}\Pi }}, T\right)\cong Medium$$

$$\mathrm{E}2=℮\left({\displaystyle \frac{MST\left(E2\right)}{\mathrm{A}\Pi }}, T\right)\cong High$$

$$\mathrm{E}3=℮\left({\displaystyle \frac{MST\left(E3\right)}{\mathrm{A}\Pi }}, T\right)\cong Medium$$

PO3: Operation Cost $\left(\mathbf{O}\mathbf{C}\right(\mathit{D}\mathit{y}\mathit{n}\mathit{a} \mathit{Q}+\left)\right)$: The expected value of the operation cost is influenced by the expected value of the cost incurred to process the requests by the resource pool.

$$℮\left({\displaystyle \frac{OC\left(Dyna Q+\right)}{℮\left({\sum }_{k=1}^3\sum _{i=1}^m\left({C_k\ast \mathcal{D}}_i\right)\right) }}, T\right)={\int }_{\alpha }^{\infty }{\displaystyle \frac{\sum _{\alpha ϵ\varphi }℮\left(\mathrm{O}\mathrm{C}\right(Dyna Q+\left)\right)}{|℮\left(length\left({WF}_i\right)\right),℮(CPU\left({RP}_i\right)\left)\right|}}$$

where, ${\mathcal{D}}_i=TET(Dyna Q+({RP}_i\left)\right)$.

$$℮\left({\displaystyle \frac{OC\left(Dyna Q+\right)}{℮\left({\sum }_{k=1}^3\sum _{i=1}^m\left({C_k\ast \mathcal{D}}_i\right)\right) }}, T\right)={\sum }_{c\in C}c{\int }_{\alpha }^{\infty }{\displaystyle \frac{\sum _{\alpha ϵ\varphi }℮\left(OC\left(Dyna Q+\right)\right)\left(\mathbf{a}\right)}{|℮(Max\left(OC\left(Dyna Q+\right)\right)|}}$$

$$℮\left({\displaystyle \frac{OC\left(Dyna Q+\right)}{℮\left({\sum }_{k=1}^3\sum _{i=1}^m\left({C_k\ast \mathcal{D}}_i\right)\right) }}, T\right)=℮\left({\displaystyle \frac{1\ast ℮\left(Max\right(OC(Dyna Q+({RP}_i\left)\right))}{\mathcal{P}(℮\left(Max\left({\sum }_{k=1}^3\sum _{i=1}^m\left({C_k\ast \mathcal{D}}_i\right)\right)\right)}}, T\right)$$

$$\mathrm{Dyna} \mathrm{Q}+: ℮\left({\displaystyle \frac{OC\left(Dyna Q+\right)}{\mathrm{A}\Pi }}, T\right)\cong low$$

$$\mathrm{E}1=℮\left({\displaystyle \frac{OC\left(E1\right)}{\mathrm{A}\Pi }}, T\right)\cong High$$

$$\mathrm{E}2=℮\left({\displaystyle \frac{OC\left(E2\right)}{\mathrm{A}\Pi }}, T\right)\cong High$$

$$\mathrm{E}2=℮\left({\displaystyle \frac{OC\left(E3\right)}{\mathrm{A}\Pi }}, T\right)\cong Medium$$

PO4: Resource utilization rate $\left(\mathit{R}\mathit{U}\right(\mathit{D}\mathit{y}\mathit{n}\mathit{a} \mathit{Q}+\left)\right)$: The expected value of the resource utilization rate is influenced by the expected value of the memory load on the resource pool, $e\left(LM\left({RP}_i\right)\right)$, and the CPU load on the resource pool, $e\left(LC\left({RP}_i\right)\right)$.

$$℮\left({\displaystyle \frac{RU\left(Dyna Q+\right)}{\sum _{i=1}^m℮\left(LM\left({RP}_i\right)\right),℮\left(LC\left({RP}_i\right)\right) }}, T\right)={\int }_{\alpha }^{\infty }{\displaystyle \frac{{\sum }_{\alpha ϵ\varphi }℮\left(RU\right(Dyna Q+\left)\right)}{\left|\sum _{i=1}^m℮\left(LM\left({RP}_i\right)\right)+℮\left(LC\left({RP}_i\right)\right)\right|}}$$

$$℮\left({\displaystyle \frac{RU\left(Dyna Q+\right)}{\sum _{i=1}^m℮\left(LM\left({RP}_i\right)\right),℮\left(LC\left({RP}_i\right)\right)}}, T\right)={\sum }_{c\in C}c {\int }_{\alpha }^{\infty }{\displaystyle \frac{\sum _{\alpha ϵ\varphi }℮\left(\mathrm{R}\mathrm{U}\left(Dyna Q+\right)\right)\left(\mathbf{a}\right)}{\left|\sum _{i=1}^m℮\left(LM\left({RP}_i\right)\right)\_℮\left(LC\left({RP}_i\right)\right)\right|}}$$

$$℮\left({\displaystyle \frac{RU\left(Dyna Q+\right)}{\sum _{i=1}^m℮\left(LM\left({RP}_i\right)\right),℮\left(LC\left({RP}_i\right)\right) }}, T\right)=℮\left({\displaystyle \frac{1\ast \sum _{i=1}^m℮\left(LM\left({RP}_i\right)\right)+℮\left(LC\left({RP}_i\right)\right)\ast ℮\left(RU\left(Dyna Q+\right)\right)}{\mathcal{P}(\sum _{i=1}^m℮\left(LM\left(RP\right)\right)+℮\left(LC\left({RP}_i\right)\right) )}}, T\right)$$

$$\begin{gathered} \mathrm{Dyna} \mathrm{Q}+: ℮\left({\displaystyle \frac{RU\left(Dyna Q+\right)}{\mathrm{A}\Pi }}, T\right)\cong High \\ \mathrm{E}1=℮\left({\displaystyle \frac{RU\left(E1\right)}{\mathrm{A}\Pi }}, T\right)\cong High \end{gathered}$$

$$\mathrm{E}2=℮\left({\displaystyle \frac{RU\left(E2\right)}{\mathrm{A}\Pi }}, T\right)\cong Low$$

$$\mathrm{E}3=℮\left({\displaystyle \frac{RU\left(E3\right)}{\mathrm{A}\Pi }}, T\right)\cong Medium$$

6. Results and Discussion

For the simulation of the proposed Dyna Q+ task scheduler, the CloudSim 3.3 simulator was used, which is one of the most widely used simulation tools for cloud computing environments [20]. CloudSim extends its support to simulate a wide range of virtual resources and allows for the experimentation of virtualized cloud data. Details on the simulation parameter setup are as follows: host (number of host = 30, MIPS = 188,770, bandwidth = 20 GB/s, storage = 3 TB, RAM = 16 GB, VM monitor = Xen), data center (number of data centers = 1, virtual machine scheduler = time shared, memory cost = 0.1–1.0, storage cost = 0.1–1.0, virtual machine monitor = Xen), client workflow (length of the workflow = 1 K–900 K, number of workflow = 300–1000), and virtual machine (number of virtual machine = 10–100, virtual machine speed = 4500–100,000 MIPS, memory = 1–4 GB, bandwidth = 2000–10,000, memory cost = 0.1–1.0, storage cost = 0.1 to 1.0, cloudlet scheduler = time shared, virtual machine monitor = Xen).

The efficiency of the proposed Dyna Q+ task scheduler is tested over three benchmark datasets: the random dataset, GOCJ dataset, and synthetic dataset. Comparisons of the proposed Dyna Q+ task scheduler against three of the existing task schedulers, E1 [15], E2 [18], and E3 [19], are conducted using performance metrics like task execution time, makespan time, operation cost, and resource utilization rate. The random dataset is composed of 1000 varieties of randomly generated workflows. This offers entirely random data for testing purposes. The dataset is generated using a built-in function of Python. It is composed of two columns: the index and value. The index represents the row ID and the value represents a randomly generated value. The GOCJ dataset is a realistic dataset generated using the bootstrapped Monte Carlo method. It comprises several files, and each file is composed of tasks in terms of millions of instructions. The tasks are derived from the workload behavior exhibited by Google cluster traces. The synthetic dataset is composed of random numbers that are generated using the Monte Carlo method for simulation. Typically, the client repeatedly requests the same kind of file, and the size of the file keeps varying in every test. This dataset allows the server to perform at its highest capacity since the requested file is stored in the server’s main memory.

6.1. Experiment 1: Random Dataset

The random dataset composed of four different workflow sizes, which are small (3 K–10 K), medium (20 K–40 K MI), large (50 K to 60 K MI), and extra-large (70 K to 79 K MI), is considered for evaluation purposes.

6.1.1. Workflow Execution Time $\left(\mathbf{W}\mathbf{F}\mathbf{E}\mathbf{T}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$

A graph of different types of client workflows (small, medium, large, and extra-large) versus the WFET (ms) is shown in Figure 2. It is observed from the graph that the WFET of Dyna Q+ is consistently shorter for the entire variety of client workflows as the Dyna Q+ agent continuously keeps updating the rewards with respect to the time taken to formulate policies with the maximum number of rewards. On the other hand, the WFET of E1 becomes very long as the size of the Q-table grows exponentially in a large state space. Even the WFET of E2 and E3 is moderate due to the low precision of computation and the random selection of reference points for mimicking the foraging behavior of swarms.

6.1.2. Makespan Time $\left(\mathbf{M}\mathbf{S}\mathbf{T}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$

A graph of different types of client workflows versus makespan time (ms) is shown in Figure 3. The MST of Dyna Q+ is very short for the entire variety of client workflows as the Q+ learning agent effectively remembers all the visited states through the exploration bonus. On the other hand, the MST of E1 and E2 is very long due to the improper balance between exploration and exploitation and the slow convergence speed. The MST of E3 is above a moderate length due to the opposition-based learning policy and poor tuning of the optimization parameters.

6.1.3. Operation Cost $\left(\mathbf{O}\mathbf{C}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathrm{Q}+\left)\right)$

A graph of different types of client workflows versus operation cost (ms) is shown in Figure 4. It is observed from the graph that the OC of dyna Q+ is lower for the entire variety of client workflows as it steadily increases the action value through repeated visits to unvisited areas. On the other hand, the OC of E1 is more than moderate as a large number of computational resources need to be stored and updated in the Q-table. The OCs of E2 and E3 are very high as they are highly sensitive to the choice of hyperparameters, and the agent’s action cannot be predicted from the swarm function.

6.1.4. Resource Utilization Rate $\left(\mathbf{R}\mathbf{U}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathrm{Q}+\left)\right)$

A graph of different types of client workflows versus resource utilization rate is shown in Figure 5. The maximum amount of resources are utilized by the Dyna Q+ scheduler for the entire variety of client workflows, as obtaining a bonus for exploration leads to the Q+ agent having a faster learning rate. On the other hand, the RU rates of E1 and E2 are moderate due to their poor accuracy and the fact that it is easy for them to get trapped in moderate solutions. The RU of E3 is poor as it converges to a suboptimal solution even after training for many iterations.

6.2. Experiment 2: GOCJ Dataset

The GOCJ consists of a dataset that ranges from 20 K to 1000 K millions of instructions. It is composed of five different kinds of workflows, which are small-size workflows (15 K to 55 K millions of instructions), medium-size workflows (59 K to 99 K million instructions), large-size workflows (101 K to 135 K million instructions), extra-large-size workflows (150 K to 337 K million instructions), and huge-size workflows (525 K to 900 K MI).

6.2.1. Workflow Execution Time $\left(\mathbf{W}\mathbf{F}\mathbf{E}\mathbf{T}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathrm{Q}+\left)\right)$

A graph of different types of client workflows (small, medium, large, extra-large, and huge) versus WFET (ms) is shown in Figure 6. The WFET of Dyna Q+ is very short for the entire variety of workflows, as it easily balances between exploration and exploitation over the large state space. The WFETs of E1 and E3 are moderate due to the self-updating of the Q-table and due to them mimicking chain foraging behavior. The WFET of E2 is very high for the entire variety of client workflows due to its restricted global search capability.

6.2.2. Makespan Time $\left(\mathbf{M}\mathbf{S}\mathbf{T}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathrm{Q}+\left)\right)$

A graph of different types of client workflows versus makespan time (ms) is shown in Figure 7. The MST of Dyna Q+ is consistently shorter for the entire variety of client workflows, as it has an adaptable architecture that makes it suitable for dynamic environments. The MSTs of E1 and E3 are high due to the fine-tuning towards the optimal solution not being good. On the other hand, the MST of E2 is very long due to its poor global search capability.

6.2.3. Operation Cost $\left(\mathbf{O}\mathbf{C}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$

A graph of different types of client workflows versus operation cost (USD) is shown in Figure 8. The OC of Dyna Q+ is lower as it successfully operates in dynamic environments through action-based learning. The OC of E1 is moderate as the heterogeneous workflows are handled properly. The OCs of E2 and E3 are very high due to their poor local and global search capability to find a promising solution.

6.2.4. Resource Utilization Rate $\left(\mathbf{R}\mathbf{U}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$

A graph of different types of client workflows versus resource utilization rate is shown in Figure 9. It can be seen that the RU rate of Dyna Q+ is very high for the entire variety of client workflows because of the proper balance between the exploration and exploitation processes of the Q+ agent. The RU rates of E1 and E2 are lower as they easily become trapped in local optima. The RU rate of E3 is moderate as it takes the maximum amount of time for the search process to become saturated.

6.3. Experiment 3: Synthetic Dataset

The synthetic dataset is composed of five different varieties of workflows, which include tiny-size workflows (1 K to 250 K MI), small-size workflows (800 to 1200 MI), medium-size workflows (1800 to 2500 MI), large-size workflows (7 K to 10 K MI), and extra-large-size workflows (30 K to 45 K MI).

6.3.1. Workflow Execution Time $\left(\mathbf{T}\mathbf{E}\mathbf{T}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathrm{Q}+\left)\right)$

A graph of different types of client workflows versus workflow execution time is shown in Figure 10. The WFET of Dyna Q+ is very short for the entire variety of workflows due to its convergence toward a promising solution through reward exploration. The WFETs of E2 and E3 are moderate due to their poor exploitation of the search space and their less adaptive control parameter strategies. On the other hand, the WFET of E1 is very high as chain foraging leads to the local optimum solution.

6.3.2. Makespan Time $\left(\mathbf{M}\mathbf{S}\mathbf{T}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$

A graph of different types of workflows versus makespan time is shown in Figure 11. It can be observed from the graph that the MST of Dyna Q+ is short as it extends more support for the exploration of large state spaces by gathering more cumulative rewards. On the other hand, the MST of E2 is moderate due to its inconsistent convergence speed when dealing with complex optimization problems. The MSTs of E1 and E3 are very long as the accuracy of the exploitation of large state spaces in later iterations of training is very low.

6.3.3. Operation Cost $\left(\mathbf{O}\mathbf{C}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathrm{Q}+\left)\right)$

The graph for different types of client workflows versus operation cost is shown in Figure 12. The OC of Dyna Q+ is low due to the exploration bonus. On the other hand, the OC of E1 is very high due to the pseudo-intermediate rewards present during task mapping. The OCs of E2 and E3 are moderate.

6.3.4. Resource Utilization Rate $\left(\mathbf{R}\mathbf{U}\right(\mathbf{D}\mathbf{y}\mathbf{n}\mathbf{a} \mathbf{Q}+\left)\right)$

A graph of the different types of client workflows versus resource utilization rate is shown in Figure 13. The RU rate of Dyna Q+ is very high due to its uniform random sampling of search spaces. The RU rate of E1 is low. The RU rates of E2 and E3 are very high due to them becoming trapped in local optima and their slower exploration of state spaces, owing to the diversity in the optimal solution.

7. Conclusions

This paper proposes a novel IVPFS-based Dyna Q+ task scheduler for cloud computing systems. The parameter uncertainty among the workflows and resource pools are handled via the application of the IVPFS mathematical framework. The proposed Dyna Q+ task scheduler is made uncertainty-proof and exhibits a high-adaptability feature for the changing dynamics of cloud systems by gathering exploration bonus rewards. The performance of the task scheduler is found to be good in terms of the following parameters: workflow execution time, makespan time, accuracy, and resource utilization rate. Its performance is further validated using expected value analysis, and the results are found to be satisfactory. The limitation of the proposed work is that it uses heterogeneous real-time dynamic cloud scenarios for testing. Our future work will concentrate on comparative analytical modeling of the scheduler by considering dynamic cloud scenarios.