A Change Management Approach with the Support of the Balanced Scorecard and the Utilization of Artificial Neural Networks

Abstract

Artificial Intelligence (AI) has revolutionized the way organizations face decision-making issues. One of these crucial elements is the implementation of organizational changes. There has been a wide-spread adoption of AI techniques in the private sector, whereas in the public sector their use has been recently extended. One of the greatest challenges that European governments have to face is the implementation of a wide variety of European Union (EU) funding programs which have evolved in the context of the EU long-term budget. In the current study, the Balanced Scorecard (BSC) and Artificial Neural Networks (ANNs) are intertwined with forecasting the outcomes of a co-financed EU program by means of its impact on the non-financial measures of the government body that materialized it. The predictive accuracy of the present model advanced in this research study takes into account all the complexities of the business environment, within which the provided dataset is produced. The outcomes of the study showed that the measures taken to enhance customer satisfaction allows for further improvement. The utilization of the proposed model could facilitate the decision-making process and initiate changes to the administrational issues of the available funding programs.

1. Introduction

All kinds of organizations operate in a continuously changing environment in which they have to adapt on a constant basis. For this reason, upper management must make critical decisions for immediate and effective actions to promote conformity to new environments. Organizational changes aim at an improved utilization of the available resources, fortification of the value chain and the creation of favorable results for the stakeholders (Alolabi et al. 2021). The embodiment of AI techniques in the process of arriving at such decisions is considered to be an important step in reaching utmost effectiveness.

AI is drastically changing the way that organizations face decision-making issues. In the aftermath of the fundamental shifts that Industry 4.0 has brought about, humans and AI create synergies so as to cope with the multidimensional problems of our times. Having the ability to self-learn and enhance decisions, AI nowadays is undertaking a significant part of the decision-making process in a hybrid collaboration with the human intellect (Vincent 2021).

As far as the private sector is concerned, specifically in large organizations, the utilization of AI techniques has been established, whereas in the public sector, their use has been recently applied to an increasing extent. In the majority of cases, the public sector lags behind the private sector in service quality and efficiency (Kotková Stríteská and Sein 2021); therefore, the advancement of this concept in public administration relates to the provision of innovative services and quality issues. The main aim of the pioneering technologies associated with the AI concept is to minimize the time spent during the decision-making process (Di Vaio et al. 2022).

The generalized application of AI in the field of public administration can provide governments with a wide variety of opportunities. European countries are undergoing a digital transformation in order to develop and provide relative services to their citizens so as to heighten public satisfaction, transparency, and entrust governments with public administration (Kuziemski and Misuraca 2020). Taking into consideration the COVID-19 crisis has brought the importance of offering digital services to the surface, given the restrictions imposed in order to tackle the pandemic (Sobrino-García 2021).

The association of a data driven approach with sophisticated AI systems could be the building block of smart citizen-centric governance. Smart government is regarded as one of the main elements to improve the decision-making process. The way that public service is delivered is totally altered to achieve better citizen engagement, accountability, and interconnectivity among public institutions (Kankanhalli et al. 2019).

One of the most challenging tasks that European governments have to undertake is the implementation of the various EU funding programs that have been developed in the context of the EU long-term budget. In addition to the implementation of these programs, equally important is measuring their efficiency. The goal is to measure how funds are used by the program’s target groups and what impact their utilization has on management capacities (McMaster et al. 2019). A prevalent method used to estimate the impacts of an EU program is counterfactual analysis. This method compares a situation where a policy/program has been applied and a relevant situation where the same policy/program has not been implemented (Castaño et al. 2019). The comparisons must be conducted properly in order to unveil the cause and effect relationships.

In recent years, governments look thoroughly into the efficiency of bodies which are responsible for implementing these kinds of programs. Efficiency refers to the ratio between the invested funds and the accomplished results. Research has indicated that the funding allocated to a specific sector (i.e., healthcare or education) has an immediate impact on the improvement of services which relate to this field (Kotková Stríteská and Sein 2021). In addition, the available funding programs are effective when there are existing mechanisms which allow the efficient appropriation of the available funds. The aforementioned advances make clear which activities should be financed for the common benefit (Valle-Cruz et al. 2021).

In cases where the available data does not facilitate counterfactual analysis, other approaches can be applied to evaluate a program. For instance, other qualitative methods compare key performance indicators during and after a program with those prior to the program (Michalek 2012). This method was applied by Psarras et al. (2020) who combined the BSC and predictive analytics to evaluate the performance of the co-financed EU financial program “Competitive reinforcement of the Greek Small and Medium-sized Enterprises (SMEs)”. A Greek government body was assigned with the implementation and the supervision of the whole funding process. This included the evaluation of the investment plans submitted by the companies. The financial perspective of the BSC was used to assess how the program’s actions influenced the financial indicators of the companies which received the funding.

Based on the research effort of Psarras et al. (2020), this paper utilizes the data for the remaining perspectives of the government body’s BSC which materialized the aforementioned financial program. More specifically, data extracted from the customers, the internal process, as well as the learning and innovation perspectives, are analyzed with the use of an ANN, which is the main contribution of this paper. The proposed approach can be used to establish a pilot decision support system, which would improve the decision-making process in terms of efficiency and effectiveness to these funding programs. It should be noted that this paper examines the program’s outcomes from the viewpoint of the government body that materialized it. The long-term objectives that the specific government body has set for this program have common elements with the EU’s objectives, but they cannot be perceived as identical.

The rest of the paper is organized as follows: Section 2 provides the relevant literature review and is structured in two parts. In the first part, the impact of AI in decision-making is discussed. In the second part, the concept of BSC is briefly reviewed. Section 3 describes the research methodology that was used for the study, and in Section 4, the research results are presented. Finally, the paper closes with a discussion of the results in Section 5 and a summary of the conclusions in Section 6.

2. Literature Review

2.1. AI in Decision-Making

The concepts of AI and AI systems were put forward to the scientific community during the 1950’s. Recent technological advancements such as the boosting of computing power and data processing speed contributed to the rapid development of AI systems (Duan et al. 2019); therefore, with all this technological progress, the field of Big Data emerged. Big Data are so intricate that conventional statistical methods cannot be applied in an effort to analyze them (Di Vaio et al. 2022). On the contrary, AI techniques could be exploited for Big Data analysis in order to obtain the maximum advantage of the available information.

AI is commonly defined as “the ability of a digital computer or computer-controlled robots to perform tasks commonly associated with intelligent beings such as the ability to reason, discover meaning, generalize, or learn from past experience” (Copeland 2021). The element which differentiates AI systems from the rest of computer systems is to self-learn from the input data. Consequently, these systems do not merely process data, but they update their decisions whenever new inputs are received (Vincent 2021); therefore, the term AI emerged because these applications resemble the function of the human brain.

The utilization of AI in the decision-making process is one of the major breakthroughs in Computer and Management Science. Bader et al. (1988) categorized six roles in AI’s contribution during this process: assistant, critic, second opinion, expert consultant, tutor, and automaton. Decision-making is reinforced by AI, whereas with the use of predictive analytics, one can timely identify the upcoming changes and receive the appropriate corrective action. Additionally, the time required for the analysis of big datasets is diminished and the possibility of human error is reduced (Valle-Cruz et al. 2021); however, in order to make user results tangible, considering the application of AI systems in the decision-making process, upper management should incorporate them into their organizational strategy (Di Vaio et al. 2022).

It should be noted that AI systems cannot proceed to decision-making unless predefined instructions have been provided by the user (Di Vaio et al. 2022). Unlike human effort, the decisions of AI systems should not be influenced by prejudgments or emotions. Nevertheless, an AI system can extract decisions by merely taking into account the information which has been entered into it. Hence, there is a collateral bias in this process because AI cannot take into account information that exists outside its system (Vincent 2021). Moreover, the ambiguity of the way in which AI produces outputs could result in a lack of trust and transparency (Valle-Cruz et al. 2021). Humans tend to be reluctant when they do not fully comprehend the way in which an algorithm comes to a decision and are not truly involved in understanding how AI systems operate before they completely integrate them into organizational processes (Jarrahi 2018). AI is highly efficient when there is enough available data from the past to foresee future actions (Tambe et al. 2019); however, in certain cases, there is a lack of historical data, and their potential to arrive at precise decisions is still under examination (Vincent 2021).

Despite the aforementioned concerns regarding the use of AI in decision-making, several research efforts have shown the beneficial results of the collaboration between humans and AI. These works have highlighted the numerous applications related to the utilization of AI techniques, as well as the particular issues concerning their contribution in assisting or replacing humans in several processes. For instance, a recent experiment at Yale University indicated that smart bots, in the context of an online game, assisted human teams to enhance their performance by minimizing the medium time needed to solve a problem by 55.6% (Shirado and Christakis 2017). Other researchers have looked into the application of AI in making weather forecasting predictions and the possibility of replacing traditional weather models (Schultz et al. 2020). The current evolution brought about by this technology assists decision-makers in comparing the possible effects and risks related to decision alternatives (Vincent 2021). Moreover, in the medical field, numerous sophisticated applications of AI have been recently developed. Wang et al. (2016) presented an algorithm which had the ability to detect metastatic cancer. Their algorithm had 92.5% success rate, whereas an unassisted pathologist performing the same task had a success rate of 96.6%. When algorithm predictions were combined with human diagnoses, the pathologist’s success rate improved to 99.5%, thus the human error rate was decreased by 85%.

Organizations that have integrated AI into their organizational strategy have seen significant benefits. Manzoor (2016) highlighted that advanced analytics techniques are utilized by top-class organizations five times more regularly than low performing ones. With the accelerating progress of these technologies, AI could be integrated in increasingly more complicated functions which require intellectual capacity. For instance, in the field of financial advising and wealth management, companies have recently been using AI and have remodeled their operations by providing investors with cutting-edge technology named robo-advisors (Méndez-Suárez et al. 2019). These algorithms utilize AI to mechanically allocate, manage, and improve clients’ assets for short- or long-term investments (D’Acunto et al. 2019).

The application of AI in government practices could create value in multiple functional areas such as e-government, healthcare, transportation, energy management, and defense. The European Parliament included AI as one of the leading technologies which will facilitate the strategic objective of digitization during this century (Sobrino-García 2021). A big challenge that policy makers have to address is how society can take advantage of emerging technologies without affecting individual freedom and privacy (Anagnostopoulos et al. 2021). To this end, the European Commission made the first worldwide attempt to regulate AI with the “Artificial Intelligence Act”. This draft regulation attempts to set out horizontal rules for the utilization of AI-driven products, services, and systems across the EU market (Kop 2021).

A wide variety of studies have proven the positive impact of AI utilization in public administration. Al-Mushayt (2019) suggested an AI solution which improves e-government services, whereas Wirtz et al. (2019) proposed a comprehensive framework which represents the crucial aspects of AI in public administration. Furthermore, the analysis of large datasets can assist authorities in detecting tax frauds (Baghdasaryan et al. 2022). In their research, Al Nuaimi et al. (2015) analyzed the key applications that can be implemented in the development of smart cities. Anagnostopoulos et al. (2017) utilized smart devices in waste management and Moustris et al. (2020) developed a forecasting model based on ANNs to predict the energy demand of Tilos Island in Greece, respectively. Additionally, the thorough research of Valle-Cruz et al. (2021) proposed an approach based on an ANN which assists public spending allocation to increase Gross Domestic Product, decrease inflation, and reduce income inequality.

Apparently, in many fields of management, prediction problems emerge. The solution lies in determining the relationship between two or more variables. The conventional methods used to solve these problems rely on regression techniques; however, if the relationship among the variables is too complex, traditional techniques are bound to fail. It has been proven that problems with many complicated attributes can be solved more efficiently by ANNs (Kosa 2013).

In their previous research effort, Psarras et al. (2020) utilized predictive analytics to assess the extent to which the financial measures of companies that participated in a funding program were influenced. As shown in Figure 1, this process affected the financial perspective of the government body which implemented the specific program. In the present study, the effects of non-financial measures are examined by utilizing an ANN. In sum, this research effort aims at predicting how the funding program being studied influenced the non-financial measures of the government body that implemented it.

2.2. The Association of the Balanced Scorecard with Performance and Change Management

The utilization of performance measurement data is facilitated by recent technological advancements. In business environments, performance data are used to make the most important decisions, such as the formulation of a strategy, the design of a service, or the implementation of change. More specifically, in the public sector, measuring performance forms an obligation to publicly report program results. Performance management emerged as a concept in the context of New Public Management (NPM), which is a managerial approach that revolutionized public administration in the early 1990s (McDavid et al. 2019). Its foundations lie in the importance of clarifying program objectives, measuring and reporting results, as well as holding all the involved parties responsible for achieving the expected outcomes (Hood 1991). NPM emphasizes associating financing with targeted outcomes. Psarras et al. (2020) investigated how the actions of a funding program affected the financial indicators of the companies which received financing. In the current study, the scope of the previous research effort will be further extended. Additional data are utilized and investigated, by means of ANN as well as through the impact of financing on the rest of the perspectives of the BSC.

In some instances, performance measures can effectively be integrated into evaluations. Program evaluation has been transformed by the aforementioned concept which relates to public administration. It is a process which exploits information systems to minimize the uncertainty levels for decision makers (McDavid et al. 2019). The evaluation outcomes and performance management systems play a crucial role as to how managers deal with their programs (Hunter and Nielsen 2013). Performance measurement and reporting may sometimes lead to numerous implications concerning program alterations or the revision of strategic objectives. By means of this process, the expected outcomes of a specific program are measured and its progress is monitored over time. In cases where deviations from the initial objectives are detected, certain corrective actions can be considered, such as the reallocation of funding.

The initial performance measurement models placed more emphasis on monitoring the financial indicators of an organization. Given the fact that business environments have been changing rapidly, and have become increasingly complex, a more balanced and integrated approach was needed to evaluate performance more holistically (Van Looy and Shafagatova 2016); therefore, organizations were compelled to go beyond conventional performance measures and develop operational measures whose performance cannot be assessed through financial indices (Papalexandris et al. 2005). Kaplan and Norton (1992, 1996) conceptualized a management system that would integrate both traditional quantitative and qualitative performance measures, which aided the development of the BSC.

The BSC constitutes one of the most significant business tools developed over the last few years and is widespread in various fields (Grigoroudis et al. 2012). It investigates the organization by means of four different perspectives: (1) financial, (2) customer, (3) internal business process, and (4) learning and innovation. The BSC helps translate strategy into operational performance measures and connects the organizational targets with those of its relevant departments. It is a tool that supports strategic planning and change management by diffusing the strategy and vision of the organization to each and every employee (Kaplan and Norton 1996). Additionally, the BSC facilitates change management as the goals of this process are linked to specific actions and timetables (Salmon et al. 2019). Finally, the implementation of the action plan is measured through the indices which have been assigned to each and every aforementioned perspective.

Many research efforts have utilized the BSC to measure the performance of various schemes. Greatbanks and Tapp (2007) investigated the impact of applying the BSC in a pubic organization. Their findings suggest that the employees who have a better understanding of their role, prompt the enhanced fulfillment of the organizational strategy. Northcott and Taulapapa (2012) indicated that the BSC is perceived by public organizations as a performance measurement tool, although its performance management role remains underutilized. Additionally, Rompho (2020) applied the BSC to measure school performance and discovered that there is a cause-and-effect relationship between the three viewpoints (customers/students, internal processes, learning and innovation). In any case, they did not unveil a relationship between the three previously mentioned perspectives and the financial perspective. Elbanna et al. (2015) applied 33 indicators in order to measure hotel performance by means of the BSC. Additionally, Gambelli et al. (2021) studied the performance of small ruminant farms in seven European countries. Their findings indicate that not much emphasis is placed on innovation issues, which may give an explanation for the low performance and longstanding downturn of this sector.

This study utilizes the BSC to evaluate how a funding program influenced the non-financial measures of the government body that implemented it. More specifically, measures are extracted from the customers (customer satisfaction index), the internal process (number of applications per program), as well as the learning and innovation perspectives (number of Greek SMEs with an e-shop). Given the aforementioned points, Figure 1 exhibits the BSC of the government body which materialized the financial program studied in the present paper.

3. Research Methodology

3.1. Area of Study

Experiments were applied to a dataset, which was structured with a certain format, and the sample is presented in

Table A1. Dataset sample.

Company ID	Y1P1	Y1P2	Y1P3	Y2P1	Y2P2	Y2P3	Y3P1	Y3P2	Y3P3	Y4P1	Y4P2	Y4P3	Y5P1	Y5P2	Y5P3	Y5C1	Y5C2	Y5C3
1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0
2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1
3	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0	1	0
4	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0
5	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0
6	1	1	0	1	1	0	1	1	0	1	1	0	1	1	0	1	1	1
7	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0	1
8	1	1	0	1	1	0	1	1	0	1	1	0	1	1	0	0	0	0
9	1	0	0	1	0	0	1	0	0	1	0	0	1	0	0	1	1	0
10	1	1	0	1	1	0	1	1	0	1	1	0	1	1	0	0	0	0
11	1	1	0	1	1	0	1	1	0	1	1	0	1	1	0	1	1	0
12	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0	0
13	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1
14	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1
15	1	0	0	1	0	0	1	0	0	1	0	0	1	0	0	0	1	0
16	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1
17	1	0	0	1	0	0	1	0	0	1	0	0	1	0	0	0	1	1
18	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0	0
19	1	0	0	1	0	0	1	0	0	1	0	0	1	0	0	1	1	0
20	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0

(Appendix A). The provided dataset consisted of 4.071 companies, which submitted their investment plans to participate in the co-financed EU financial program, “Competitive reinforcement of the Greek Small and Medium-sized Enterprises”. These companies went through three evaluation phases, but only a few proceeded to the subsequent phases (Psarras et al. 2020).

During the first phase, 4.071 companies were evaluated on the basis of covering the basic prerequisites for participating in the program. In the second phase, the business plans of the companies that were successful in the previous phase were evaluated. Finally, in the third phase, funding was disbursed to the companies that passed the second phase; however, it was granted only to those whose business plans completely materialized. There were also 575 participants whose business plans were approved in the second phase, but they did not materialize completely, so they did not receive the total approved disbursement (Psarras et al. 2020). As shown in Figure 1, these phases refer to the actions taken by the government body to materialize the program.

3.2. Experimental Setup Dataset Structure

The provided data were retrieved from the database of the Greek government body which was responsible for implementing the program. For each one of the three phases, the evaluations were included in the given dataset. As a result, the predictive attributes of this research were utilized in relation to the three discrete evaluation phases, namely: (1) close examination of the basic prerequisites for participation in the program; (2) approval for receiving funding; and (3) total disbursement of funding. Each predictive attribute is binary as they each take two discrete values (i.e., {0,1} based on the results of their evaluation, which is either rejected or accepted). As a consequence, each year is composed by using three predictive attributes (P1, P2, P3). The available data were selected from five consecutive years (Y1, Y2, Y3, Y4, Y5) of experimental study. Let us define predictive attributes in the form of $YnPm$, where $n=\left[1, \dots , 5\right]$ and $m=\left[1,\dots ,3\right]$; therefore, the predictive attributes are defined as follows: (1) Y1P1, (2) Y1P2, (3) Y1P3, (4) Y2P1, (5) Y2P2, (6) Y2P3, (7) Y3P1, (8) Y3P2, (9) Y3P3, (10) Y4P1, (11) Y4P2, (12) Y4P3, (13) Y5P1, (14) Y5P2, and (15) Y5P3.

In addition, at the end of the fifth year, three class attributes were observed, namely: (1) customer satisfaction; (2) submitting a proposal at least to one program; and (3) e-shop. Each class attribute is binary as they each take two discrete values (i.e., {0,1}); therefore, class attributes are defined as follows: (1) C1, (2) C2, and (3) C3. As shown in Figure 1, each one of the three class attributes represents a strategic objective of the government body which materialized the program. More specifically, the government body targeted a satisfaction rate of over 80%. As follows, C1 takes a value of 0 when the satisfaction rate of each company was below 80%, and a value of 1 when the same rate was over 80%. C2 takes a value of 0 when companies did not submit at least one proposal to another program that the government body materialized, and a value of 1 when they submitted at least one. C3 takes a value of 0 when companies did not have an e-shop, and a value of 1 when they had one.

The dataset incorporated using the given methodology supports a certain area of study which is presented in the current research effort. In general, optimally, researchers must use a given dataset to solve a specific problem; however, combinations between predictive and class attributes should not be performed according to random selections. More specifically, in the present research, predictive attributes rationally depict the characteristic of added value to the prediction process. In addition, class attributes are rationally correlated with the given predictive attributes to provide a solution to the studied problem. For the given problem, the notation of attributes to certain predictive and class quantities is based on the domain of specific knowledge and is inherent in the adopted model.

The initial dataset had 4.087 total instances; however, 16 missing values were observed, which were less than 5%. Instances with missing values were removed from the initial dataset, so the final experimental dataset has a total of 4.071 valid instances. The dataset structure is presented in

Table 1. Adopted Dataset Structure.

Attribute	Type	Value
Y1P1	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y1P2	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y1P3	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y2P1	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y2P2	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y2P3	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y3P1	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y3P2	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y3P3	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y4P1	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y4P2	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y4P3	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y5P1	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y5P2	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
Y5P3	Predictive	$\mathrm{Binary}\in \left\{0,1\right\}$
C1	Class	$\mathrm{Binary}\in \left\{0,1\right\}$
C2	Class	$\mathrm{Binary}\in \left\{0,1\right\}$
C3	Class	$\mathrm{Binary}\in \left\{0,1\right\}$

.

3.3. Model

3.3.1. Model Selection

To predict each one of the three class attributes, a Multi-Layer Perceptron ANN was utilized, as provided by the open-source machine learning tool Weka (Frank et al. 2016). ANN is a classifier which consists of a large number of interconnected neurons. Each and every connection is assigned to different weights. These weights are adjusted according to back propagation, which is a training technique of the Multi-Layer Perceptron used to classify given predictive attributes and predict class attributes. At first, data are entered into the first layer and then distributed to the next layer, which is the hidden layer. There is no rule that defines whether the hidden layer should have one or more sub-layers. The most common method to configure this is by making trials (Méndez-Suárez et al. 2019). In the hidden layer, mathematical operations are performed which aim at obtaining the outputs.

An ANN model was selected for the experiment, since the data source is based on business and real time data, which have a complex relationship among their attributes. The ANNs are considered one of the most suitable classification models when the observed data are highly complicated, which is the case in this research effort.

3.3.2. Experimental Parameters

The adopted ANN model is finely tuned with certain parameters for each class attribute application. More specifically, the input layer was composed by 15 nodes, such as the number of the predictive attributes. Three hidden layers were used: (1) the first is composed of 15 nodes; (2) the second is composed of 30 nodes; and (3) the third is composed of 15 nodes. The output layer for each class is composed of two nodes according to the number of binary values (i.e., {0,1}). All the adopted nodes in the ANN are sigmoid. The number of epochs used to train the ANN is set to 500. The learning rate for weight updates is set to 0.3. The momentum applied to the weight updates is set to 0.2. Regarding the model incorporated, the number of nodes by the three layers was converged into the resulted values by experimenting with a given interval of values provided by the Weka machine learning software. This was used to observe optimal results. Accordingly, the number of epochs was experimentally defined by the provided values of the incorporated Weka software. Subsequently, the learning rate and the momentum applied to the weight updates were experimentally provided by the adopted Weka machine learning software. This is because machine learning is an empirical science and it is initially not known which values should be used when training a neural network classification model. For this reason, several parameters are experimented with different values to converge to the model, which achieves optimal efficiency. The number of hidden layers and their nodes, as well as the rest of the model parameters, was observed experimentally as presented in

Table 2. Experimental Neural Network parameters.

Parameter	Value
Input layer nodes	15
Number of hidden layers	3
First hidden layer nodes	15
Second hidden layer nodes	30
Third hidden layer nodes	15
Output layer nodes	2
Type of nodes	Sigmoid
Number of Epochs	500
Learning rate for weight updates	0.3
Momentum applied to weight updates	0.2

.

The graphical representation of the adopted ANN is presented in Figure 2.

3.4. Evaluation

The evaluation of the proposed ANN is performed with a certain evaluation method and metric systems. More specifically, the method was tested with a 10-fold cross validation, whereas the metric system was assessed by calculating the predictive accuracy of the adopted ANN classification model.

3.4.1. 10-Fold Validation Evaluation Method

The adopted ANN classification model was evaluated with a 10-fold cross validation, which divides the initial dataset to 10 equal sized parts. Then, in certain loops, it incorporates the first nine parts to train the classifier, and the remaining part to test the classifier. This process is repeated until each and every part has been used for training and testing.

3.4.2. Prediction Accuracy Evaluation Metric

The effectiveness of the adopted classifiers was assessed by incorporating the prediction accuracy evaluation metric, $a\in \left[0,1\right]$, which is defined in Equation (1) as follows:

$$a=\frac{t_p+t_n}{t_p+f_p+t_n+f_n}$$

(1)

where, $t_p$, indicates the instances correctly classified as positives, and $t_n$, are the instances correctly classified as negatives. In addition, $f_p$, signifies the instances which are falsely classified as positives, and $f_n$, indicates the instances which are falsely classified as negatives. A low value of $a$ implies that the classifier is weak, whereas a high value of $a$ indicates that the classifier is efficient.

4. Research Results

Experiments were performed on the adopted dataset of 4.071 valid instances with the proposed Multi-Layer Perceptron ANN classification model. The ANN was finely tuned based on certain experimental parameters, which were evaluated with a 10-fold cross validation evaluation method. The results of the application of the ANN classification process for all the three class attributes were assessed with certain values of predictive accuracy. More specifically, the predictive values of accuracy are as follows: (1) C1 prediction accuracy a = 0.731761; (2) C2 prediction accuracy a = 0.900516; and (3) C3 prediction accuracy a = 0.803488. The results of predictive accuracy, $a$, are presented in Figure 3.

5. Discussion

Every EU program undergoes extensive evaluation to assess its effectiveness and efficiency. Effectiveness could be related with the degree to which the predefined objectives of the program are met. Efficiency could be linked with the extent to which overall program outcomes relate to its costs, respectively (Michalek 2012). Program evaluations highlight useful lessons learned, which can be essential for the preparation of new initiatives; however, the existence of an ineffective monitoring system and the lack of modern evaluation techniques could make it difficult for the evaluators to conduct their assessments.

This paper utilizes an advanced AI technique related to ANNs in order to analyze the available data of a co-financed EU program. The main research objective is to predict the program’s outcomes by means of its impact on the non-financial measures of the government body that materialized it. These measures are set within the context of the BSC and are related to the customers, the internal process, and the learning and innovation perspectives. Each and every perspective of the BSC operates with the following sequence: strategic objectives -> measures -> targets -> actions, as shown in Figure 1. For instance, the strategic objective that the government body set concerning the learning and innovation perspective was the digital transformation of the Greek SMEs. The action that contributed to the implementation of this strategic objective was the materialization of funding programs, such as those previously mentioned in the current study. The measure used to monitor the effectiveness of the aforementioned action was the number of Greek SMEs owing an e-shop.

Bearing in mind that the perspectives of the BSC are interconnected with each other (Kaplan and Norton 1992), it is presupposed that the results of one perspective have an impact on the others. As previously mentioned, the companies that participated in the program, and which are under study, underwent three evaluation phases: (1) close examination of the basic prerequisites for participation in the program; (2) approval for receiving funding; and (3) total disbursement of funding. As shown in Figure 1, these are also the actions that contributed to the realization of the financial strategic objectives that the government body had set. Consequently, data retrieved from the financial perspective are related to the materialization of the funding program, and have been utilized to predict the results of the remaining perspectives.

The research question is whether someone, knowing the results of the actions taken by the government body for the realization of the financial strategic objectives, could predict the level of achievement of the remaining three perspectives. The answer to this research question could improve the decision-making process and facilitate change management, by altering the funding criteria or the financing areas if needed.

AI which is utilized in the present research effort is an empirical science. This implies that the number of hidden layers and the number of nodes in each hidden layer are related to the predictive accuracy and the total performance of the model. More specifically, the greater the number of hidden layers and nodes within each layer, the greater the predictive accuracy, which increases until a certain convergence threshold. As previously mentioned, this is defined by trial and error according to the adopted experimental parameters; however, after a certain level, there is a decrease in predictive accuracy, no matter how many hidden layers and nodes continue to be incorporated. In addition, predictive accuracy not only depends upon the number of hidden layers and nodes of the ANN, but also upon the quality of the provided dataset. Moreover, a classification model is effective for certain data if it achieves high predictive accuracy concerning the fine tuning of the experimental parameters of the ANN model. Finally, this process depends not only on quality, but it is also connected to the quantity of the training data. Such data are preprocessed accordingly in order to remove outliers, missing values, and redundant data, which could affect the effectiveness of the adopted model.

The predictive accuracy of the presented model was 73% for customer satisfaction, 90% for submitting proposals to other programs, and 80% for owing an E-shop. These predictive accuracies are considered efficient, bearing in mind that the provided data were analyzed and produced by multimodal human activity in complicated business environments. Such environments are characterized by a high degree of volatility; thus, the final results are important and may have a significant impact upon the decision-making process. This study indicated that the government body which materialized the program could apply changes to the funding process and improve the predicted results of the customer perspective. The results for the remaining perspectives seem to satisfy the targets that were set in the initial BSC.

The limitations of the present study should also be noted. As mentioned in the introduction, this paper examines the program’s outcomes and efficiency from the viewpoint of the government body that materialized it. Additionally, the strategic objectives of the BSC should be revised on a yearly basis. In the current study, it has been assumed, for experimental purposes, that the strategic objectives have remained the same over the years. Moreover, the available data for the experiment is concerned with a single measure for each and every strategic objective; however, the intricate relationships among the four perspectives could be analyzed by exploiting all the measures for all perspectives put forward. Finally, the provided dataset consists only of binary variables, which is also a limitation of the current study.

6. Conclusions

The evaluation of EU programs is essential because useful insights are provided on whether the programs met their initial objectives. This process also helps policy makers to redesign programs and make them more efficient by optimizing the absorption of the invested funds. The existence of an effective monitoring system and a modern evaluation technique should help the evaluators to conduct their assessments. This study integrated the BSC and AI to predict the outcomes of a co-financed EU program by means of its impact on the non-financial measures of the government body that materialized it. The predictive accuracy of the model developed in this research effort is considered efficient, taking into account the complexity of the business environment in which the provided data were produced. The results indicated that corrective actions could be addressed by the government body which implemented the funding program in order to improve the outcomes of customer satisfaction. The utilization of the proposed model could improve the decision-making process and initiate changes to administrational issues in the available funding programs. Future research will be centered upon predicting program results more holistically by incorporating more variables related to the BSC.