Preventive Risk Management of Resource Allocation in Romanian Higher Education by Assessing Relative Performance of Study Programs with DEA Method

Abstract

Risk management is a key activity in every organization. The identification and evaluation of all risks in higher education institutions lead to the continuous monitoring of investments in people, high technology, and innovation. This paper evaluates the relative efficiency of study programs in Romanian higher education using the DEA method. This study is based on 38 study programs from a public university in Romania, using a traditional DEA approach: CRS-DEA and VRS-DEA models, with an output orientation for three academic years (2016–2019). To avoid distortions in the efficiency scores, we decided to implement the bootstrap method to correct DEA efficiencies. The results show that only four study programs were efficient during this period under the CRS-DEA approach, and eight study programs were efficient under the VRS-DEA model. According to scale efficiency and the bootstrap method, the results also showed that four study programs were efficient during the period analyzed. Finally, we observed that the inefficiency of study programs is relatively persistent (89%), compared with efficient DMUs (11%). Based on these findings, higher education institutions should consider the possibility of increasing the quality of study programs correlated with the degree of attractiveness of various programs in the current socio-economic environment.

1. Introduction

The educational system has gone through numerous transformations over time, trying to adapt to economic changes, depending on the material and social conditions of society. There is a connection between the development of a nation and education. Adam Smith, the “father of economics” said in the eighteenth century that “a country is rich if it has individuals,” which means for us that “a country is rich if it has educated individuals”. Progress increasingly depends on research capacity, innovation, and the ability to adapt to new generations [1]. However, there is still an open issue on how to balance performance with efficiency and risk. Achieving higher performance may need more resources, but if they are used in an ineffective or inefficient way, this goal is not achieved. Thus, it is tremendously important to introduce tools in the management practices of higher education institutions for measuring the risks and efficiencies related to investments and resource allocation for their study programs.

Higher education institutions (HEIs) have a mission to prepare specialists that can face the various situations and requirements of society and the economy. Moreover, students’ demands are becoming more and more complex, and HEIs must ensure that the students receive high-quality service [2].

The value of a university is how it serves the community that pays for education to deliver higher quality outputs in a more productive way as well as improve the performance of research activities and educational programs. Leaders in higher education institutions are dedicated to providing educational programs with a high level of performance and quality, and they face severe budgetary pressures and increased threats to their reputation that can affect their capacity to recruit and retain students. Better recruitment and retention support higher competitiveness, better quantification of the contributions of the academic staff, and critical public funding [3].

In Romania, there remains a gap between what programs universities offer and what skills the socio-economic system requires. The cooperation between the academic environment and economic actors has been a topic of debate for these educational programs. In principle, universities must play a transformative role and change their business models toward ones of higher value from deep and intense cooperation with the Romanian industry and society [4]. Moreover, constructing a modern educational system, with sustainable and efficient study programs, is an important issue in our society because well-trained employees play a key role in improving the competitiveness of their organizations [5].

For Romanian higher education institutions, it is relevant to measure the efficiency of resource allocation as long as they are financed from public funds (i.e., the Ministry of National Education). There are many ways to measure the efficiency of an organizational process, one of these being the normalized ratio between results and inputs. Nevertheless, a given process can have various types of efficiency. This depends on the specificity of inputs and outputs, meaning the targets with which we measure efficiency. The preventive risk management of resource allocation in higher education could play an important role in adequate financial support.

Risk management is a key activity in every organization that consists of standards, decisions, regulations, and compliance. The identification and evaluation of all risks show the constant focus of higher education on their investments in people, technology, multilayer innovation, and the capabilities required to survive sustainably in practice to deliver higher quality outputs in a more productive way [6]. Risk management is a recent topic in public higher education institutions, especially after the crisis of 2008, when the budgets allocated for education were not being fairly used according to their mission. There is a challenge for the kind of approaches to be considered when measuring the absolute or relative efficiency of allocated resources in higher education. If the absolute measurement is an issue of debate, relative assessment could be a more practical approach. Among the various methods for measuring the relative efficiency of a set of comparable units, the data envelopment analysis (DEA) method proved to be very practical because it is based on quantitative data that managers use as metrics (key performance indicators (KPIs)) in performance analysis. It has the quality of operating, both with inputs (resources allocated), and outputs (KPIs), which are well-fitted to the culture of top management.

This paper evaluates the relative efficiency of study programs in Romanian higher education using the data envelopment analysis (DEA) method. The research is based on 38 study programs from a public university in Romania, using a traditional DEA approach, namely CRS-DEA and VRS-DEA models, with an output orientation. Two inputs and two outputs were selected for the three academic years (2016–2019), and the efficiency scores were calculated for every year (pooled data). To avoid distortions in the efficiency scores, we decided to implement the bootstrap method to correct DEA efficiencies [7].

The main practical result of this research is that universities may consider the DEA for reducing the risk in budget allocation for various study programs and could analyze, by comparison, the lower-performing study programs with the top ones. The paper also provides a new perspective on the practice of risk management, which can be generalized to various fields of application.

The research is structured as follows: Section 2 gives some points of view about the funding of the higher education system in Romania, and Section 3 highlights the basic references of DEA application in higher education institutions from different countries. Section 4 describes some important aspects of DEA. Section 5 highlights the research undertaken to analyze 38 representative study programs of a public university in Romania using the DEA method. The main results are further discussed in Section 6, and Section 7 includes some conclusions and proposals for future research.

2. Funding of the Public Higher Education System in Romania

The proper allocation of funds in higher education is necessary for a new smart transformation of the higher education system. The public university in Romania is full of conflicts, such as the recruitment and retention of students versus employability, cuts in public funds for higher education versus costly and redundant infrastructure and professors, very few sources for unrestricted budgets versus the increasing costs of higher education, and decreased levels of public support [3]. Conflicts can be identified and innovatively approached in a systematic and structured way to remove them. Conflicts represent a very relevant measure of a university’s competitiveness and capacity to deliver value for the money it receives from public funds [4]. The financing of education expenditures differs, depending on the structure of the budgetary system; the expenses for education are mainly distributed to the ministry that organizes and conducts the education. Higher education institutions are allocated 10–20% of the total public financial resources for education. The allocation of gross domestic product (GDP) for education in Romania is lower than in other countries. A study published by Eurostat shows that Romania ranks 26th out of 27 (no data for Croatia), with a percentage of 2.69% of GDP given to education in 2017 [8].

The financing system of Romanian universities has two components: basic financing for the current functioning system and complementary financing.

• To determine the level of basic financing, the main calculation reference is the indicator known as “budgetary benefit on equivalent students.” Determining the number of equivalent students in the university depends on the number of active students and on the educational forms;
• Complementary financing is an indicator that is established to provide the funds for social expenses for students, namely accommodation and meals, local transport, and scholarships. The subsidies for accommodation and meals are calculated according to the number of actual students and the actual expenses planned for each period of the calendar year in which the students are accommodated in hostels [9].

The number of conventional hours represents an essential synthetic indicator for improving the financial management of any structure in Romanian universities [10]. Based on the average cost of the conventional hour unit, comprehensive analyses can be made regarding the cost of study programs, which can then be compared with the results obtained in the different study programs of the university or with the same program developed in different higher education institutions in Romania.

According to the methodology for the allocation of budgetary funds for the basic and supplementary financing of higher education and public institutions in Romania, the amounts established for basic and supplementary funding, allocated from the budget of the National Ministry of Education, are specifically mentioned within the institutional contracts of higher education institutions. The contract specifies the number of students (unit equivalents) financed from the state budget, the cycles of studies, and the number of state-funded doctoral programs.

The revenues used for basic and supplementary financing follow the rules of university autonomy to achieve its existing objectives in the framework of state policies regarding education and academic scientific research.

The basic financing concerns personnel costs (PC) (salaries for the teaching staff, auxiliary teaching and non-teaching staff involved in conducting programs, bonuses, social insurance contributions (SICs), other legal contributions, domestic and international travels); material costs (MC) (maintenance and household expenses, expenses for materials and services, related research programs, inventory items, repairs, publications, improving staff safety, protocol, labor protection, etc.); expenditures on supporting educational projects and human resource development; overhead expenditures of higher education institutions to conduct their programs of study.

To encourage excellence in higher education institutions, a national financing fund for additional expenditures in universities has been set up. This expenditure represents a minimum of 30% of the amount allocated to national universities as basic funding.

The funds attributed to each university as basic financing for the students currently enrolled are actually allocated based on the figures of tuition received by the University from the Ministry of Education. These figures are distributed in accordance with the numbers of students enrolled in undergraduate and graduate degree programs but proportionally to the number of equivalent students. The number of equivalent students at a university needs to be calculated by weighting the number of its students with the physical coefficients of equivalence and cost.

According to the Law on National Education, study programs constitute the basis for personal establishments. The personal establishment is a document that includes the posts, functions, payrolls, and hiring limits of the teaching staff in a university.

Personal establishments are developed annually. The number of posts is established considering the following indicators: educational programs; study groups; university rules and workloads; the financial sustainability of the faculty and the financial sustainability for research posts through contracts.

The total number of conventional hours in an academic year can be determined by multiplying the number of conventional hours included as weekly averages in the positions of the function states, with 28 weeks related to a university year.

As a result, universities plan their expenditures and staff costs starting from the approved and allocated tuition figures, within the limits of the approved budget and in accordance with the institutional contract. Students learn the subjects throughout their educational programs in accordance with a certain specialization.

3. Literature Survey of DEA Application in Higher Education Institutions

Some references that address the efficiency of study programs in higher education are reported in the scientific literature. Some representative results are further highlighted in this section. In this respect, Thanassoulis et al. (2011) analyzed the performance of higher education institutions in England using the DEA method. In the empirical analysis, they adopted an input orientation and measured the productivity and efficiency of 121 HEIs in England. For input, they used the total operating costs adjusted for inflation, from which six outputs were established: Three of them showed the full-time equivalent (FTE) undergraduate students enrolled by subject area; FTE postgraduate students (PG) in all disciplines; and the value of research grants and third-leg activities (income from other services rendered in constant prices). The results showed that DEA appears to offer better unit cost estimates, and for a significant subset of HEIs, productivity worsened during the analyzed period [11].

Recently, Eiduks, Paura, and Arhipova (2017) analyzed the quality and efficiency of the study programs in the Latvian Higher Education Institutions using two study period performance indicators of the Latvian University of Agriculture engineering science faculties. The knowledge, skills, and performance of graduate students were defined as outputs; graduation rates and the proportion of students who graduated in due time were defined as outcomes. In this analysis, they used the number of enrolled students as input. Quality, efficiency, and effectiveness were selected as performance indicators. The efficiency in the case of the study programs was equal to the number of graduates divided by the number of students at the time of enrollment. Effectiveness was calculated by the number of graduates with marks equal to or higher than 9 (outcome 1) and/or 8 divided by the number of graduates (outcome 2). The conclusion was that effectiveness increased by 3.5% for the first outcome and by 9.5% for outcome 2. The efficiency of all faculties in the last study year was 11.1% less than that in the first year. As a result, the study program quality evaluation decreased during the study period [12].

Kovalenko et al. (2019) analyzed efficiency in the higher education system in Bosnia and Herzegovina. In this research, for measuring efficiency, the superefficiency model and input-oriented CRS were used. For decision-making units (DMUs), the total number of the main fields of the education system in Bosnia and Herzegovina was considered, namely agriculture, health, social sciences, manufacturing, business and law, natural sciences, humanities and arts, construction, engineering, and mathematics and informatics. The inputs used in this research were the budget, co- and self-financing, and the enrolled students. The output was the number of graduates in each educational domain. The results showed that agriculture is the most efficient domain while manufacturing technologies, construction, and engineering had the worst performance in terms of budget expenditures. They concluded by emphasizing the need to find proper solutions for reducing the number of graduate students in the fields of mathematics and natural sciences, engineering, manufacturing technologies, and construction [13].

To find and effectively use resources toward more performance-oriented specialization, Youn and Park (2009) [14] suggested four models of universities, namely knowledge-specific (a university in which intensive research is conducted through specialized education); knowledge-expanding (a university in which collaborative research is conducted between interdisciplinary programs to enhance its capacity for knowledge expansion), knowledge-learning (a university focused on educational knowledge for helping students to integrate into society), and knowledge-replenishing (a university oriented more towards delivering practical knowledge). An analysis of the efficiency of Korean universities using the DEA method was performed for 60 universities (15 universities for each model). The inputs used in this paper were the number of undergraduates, the number of graduates, and the number of full-time faculty staff. For outputs, they used the number of BSc graduates, the employment rate of undergraduates, and the ratio of the undergraduates enrolled. This study also categorized their results relative to the size of the student body. Thus, they were divided into small (with a student body of fewer than 5000), medium (with a student body of between 5000 and 15,000), and large (with a student body of more than 15,000).

In this research, the CCR model was used for efficiency analysis under input orientation. The results showed that the most efficient model was knowledge-replenishing, followed by knowledge-expanding, knowledge-learning, and knowledge-specific models. Notably, the knowledge-replenishing and -expanding models were relatively easy to distinguish [14].

In another study [15], DEA was applied to Czech higher education institutions to measure their efficiency. The data from 2013 were considered on the following variables: academic personnel and other costs (inputs), and BSc, MSc, and PhD graduates and students (outputs). The analysis divided HEIs into three groups with similar cost coefficients to eliminate high differences in the inputs and outputs and to obtain better information about the efficiency of HEIs.

No specific application of DEA for the study programs in the Romanian higher education institutions has been published in the scientific literature. It is, therefore, the goal of this research to use DEA in this case. A brief description of the DEA method is addressed in the next section.

4. Data Envelopment Analysis Methodology

4.1. Data Envelopment Analysis

The data envelopment analysis (DEA) is a method for assessing the relative efficiency of decision-making units (DMUs) using the efficiency frontier it empirically forms. DEA was elaborated by Charnes, Cooper, and Rhodes (1978) and has been used in many fields, such as health, education, banks, and the public administration sector [16]. The main idea behind DEA was to compare a set of DMUs to identify the most efficient units that could form an efficient frontier [17]. There are numerous variants of DEA models with various combinations of outputs and inputs that could provide different results. The choice of method is important in the analysis of efficiency [18].

DEA is a method for measuring the efficiency of a model with multiple inputs and outputs. The efficiency scores indicate which DMUs are more efficient relative to their peers. DEA assigns a score of one to efficient units and a score of less than one to inefficient units. This shows that a linear combination of other units from the tested sample could generate the same set of outputs, using a smaller set of inputs. The score underlines the radial distance from the estimated production frontier to the analyzed DMU [19].

Since 1978, impressive progress has been made in the development of DEA models in terms of both theoretical development and practical applications, some of which include the following models:

• Various DEA models: the constant return to scale (CRS) model [16], the variable return to scale (VRS) model [20], the additive model or Pareto–Koopmans (PK) model [21], the Russell measure [22], the free disposal hull (FDH) model [23], and cross-evaluation [24];
• Multilevel models: network DEA [25], supply chains [7], multicomponent/parallel models [26], and hierarchical/nested models [27].

For the assessed DMU, the efficiency indicator in DEA compares the ratio between its outputs and inputs with the value of the same ratios for the other DMUs or in other formulations.

The models used in DEA in this paper are CRS, VRS, and scale efficiency. The efficiency scales are linked to each type of surface envelope, which establishes the efficiency frontier [16]. Those units that are part of this surface are treated as being efficient and those that are away from this surface are inefficient. For each inefficient DMU, the DEA method highlights the sources and level of inefficiency relative to each input and output. This level is established relative to a reference DMU. It must be located on the efficient frontier and must utilize the same inputs and produce the same types of outputs [28] (p. 6). There are three variants for measuring efficiency, namely reducing the inputs while keeping the outputs constant (input-oriented); increasing the outputs while keeping the inputs constant (output-oriented); increasing the outputs and simultaneously reducing the inputs (the dual version). Input orientation is equal to output orientation only in the case of CRS [22].

4.2. Bootstrapping in DEA

The distance of observation with respect to the efficiency boundary indicates both inefficiency and noise. Bootstrapping in DEA is used to correct it due to the fact that the observed input–output data are prone to error in many cases.

The first step in the bootstrap algorithm is the calculation of DEA efficiencies, named here as θ_k, k = 1, …, n, for all n units, followed by the construction of a set D2n of 2n reflected efficiencies. The reflected efficiencies are 2-θ_k k = 1, …, n. Using only the θ_k ≠1 from D2n, a smoothing constant h is calculated. This constant is used for adjustments by means of a discrete distribution of efficiencies. After drawing with replacement, a random sample size n out of the set D2n of reflected efficiencies is created. The next step adjusts the sample of efficiencies using the bandwidth h and a kernel function. This is used to generate a bootstrap sample of the input–output levels in the case of each DMU. For DMU k, the input levels are obtained by projecting them to the efficient level and after that to an inefficient level in line with the sampled adjusted efficiency for that DMU. Its bootstrap outputs are the same as the original ones. Using the set of n bootstrap input–output levels for each DMU the bootstrap efficiency λ^b_k of DMU k is calculated. After repeating the algorithm B times, the result is b = 1, …, B bootstrap efficiencies λ^b_k for each DMU k. In the end, we sort the differences λ^b_k − θ_k for each unit. Next, we identify the value dl of the difference below which α/2 % of the differences λ^b_k − θ_k lie, as well as the value du above which α/2 % of the differences λ^b_k − θ_k lie. The (100-α)% confidence interval on the true efficiency θ^t_k is [L_k = θ_k − du, U_K = θ_k − dl].

Bias corrections to the DEA efficiency θ_k:

• Mean

$${\mathsf{\theta }}_{\mathrm{k} \mathrm{corr}}=2{\mathsf{\theta }}_{\mathrm{k}}-\frac{{\sum }_{\mathrm{b}=1}^{\mathrm{B}} {\lambda }_{\mathrm{k}}^{\mathrm{b}}}{\mathrm{B}}$$

(1)

• Median

$${\mathsf{\theta }}_{\mathrm{k} \mathrm{corr}}=2{\mathsf{\theta }}_{\mathrm{k}-\mathrm{median}}({\lambda }_{\mathrm{k}}^{\mathrm{b}},\mathrm{b}=1\dots \dots \mathrm{B})$$

(2)

where B is the number of bootstrap iterations, and ${\lambda }_{\mathrm{k}}^{\mathrm{b}}$ are the bootstrap efficiencies.

As a result, the initial efficiency is an overestimate of the true efficiency and, in a few cases, reaches the upper bound on the confidence interval, which is 100% or higher [29].

5. Research Design

The majority of study programs in Romanian universities are very similar regarding the same subjects, irrespective of the institution offering them (be it an old or new institution, a comprehensive or specialized one, and a public or private institution). Moreover, they hardly meet the various needs of the labor market. Consequently, educational programs should become more flexible and better tailored to the demands of the labor market.

The completion of the study programs, especially those for undergraduate programs, should lead to gaining competences, acquiring specialization knowledge, and developing other educational skills. It should also encourage students to experience different situations, choose from various options and alternative methods, and take responsibility for their choices. In addition, educational programs should stimulate and increase the degree of inter- and multidisciplinarity, as well as the amount of acquired information, and raise students’ awareness of sustainability.

Our aim in the current research study was a comparative analysis of the undergraduate programs by using the CRS and VRS models with an output orientation. Using input and output data in the DEA analysis, a comparison was made between the analyzed study programs with regard to their efficiency in order to rank them by the level of efficiency. Consequently, suggestions were made regarding the values to be reached by inefficient study programs in order for them to have their efficiency level raised so that they can become efficient. In addition, we implemented the bootstrap method for calculating efficiency that was developed by Simar and Wilson (2008) [30].

This comparative analysis of study programs could help the top management of Romanian universities to make the right decisions about their efficient management system. Furthermore, the assessment of the performance and quality of the various specializations needs to be linked to the academic and financial performance of universities.

For this analysis, 38 representative study programs of a public university in Romania were selected to estimate the relative efficiency based on DEA under various approaches (

Table 1. Study programs from UTCN.

DMU	Description	DMU	Description
S1	Hydraulic engineering and construction	S20	Economic Engineering in Civil Engineering
S2	Architecture	S21	Economic Engineering in the Field of Electrical, Electronic, and Energy (ETTI)
S3	Automation and Applied Informatics (in English)	S22	Economic Engineering in the Field of Electrical, Electronic, and Energy (IE)
S4	Automation and Applied Informatics	S23	Industrial Economic Engineering
S5	Vehicles	S24	Medical Engineering
S6	Railways, Roads, and Bridges	S25	Urban Engineering and Regional Development
S7	Computers (in English)	S26	Building Services
S8	Information Technology	S27	Instrumentation and Data Acquisitions
S9	Computers	S28	Energy Management
S10	Civil, Industrial, and Agricultural Engineering (in English)	S29	Agriculture and Food Industry Machinery and Equipment
S11	Civil, Industrial, and Agricultural Engineering	S30	Land Measurements and Cadaster
S12	Industrial Design	S31	Fine Mechanics and Nanotechnologies
S13	Electromechanics	S32	Mechatronics
S14	Applied Electronics	S33	Robotics
S15	Applied Electronics (in English)	S34	Thermal Systems and Equipment
S16	Power Electronics and Electrical Actuators	S35	Materials Science
S17	Electrotechnics	S36	Machine Construction Technology
S18	Engineering and Environmental Protection in Industry	S37	Telecommunications Technologies and Systems
S19	Transport and Traffic Engineering	S38	Telecommunications Technologies and Systems (in English)

).

All the study programs of the university that we chose were homogeneous because, in DEA, the homogeneity of DMUs must satisfy three rules: They must have similar activities and the same objectives, they must utilize similar inputs to produce the same outputs, and they should operate within similar environments [31]. We used the CRS-DEA model (output-oriented) and the VRS-DEA model (output-oriented). The output efficiency is the proportion of the observed output levels to the maximum possible output levels for given input levels.

After the CRS-DEA and VRS-DEA analysis, we calculated the value of scale efficiency (SE) for each study program. Scale efficiency is the ratio between technical efficiency and pure technical efficiency and indicates deviations from the most productive scale size [32]. Finally, we implemented the bootstrap method for calculating the efficiency and selected to apply the software tool PIM-DEA-V3 [33]. The analysis was performed for three academic years between 2016 and 2019 from one university.

The input and output variables were based on cost-efficiency criteria for inputs and quality and performance in higher education for outputs (

Table 2. Definition of input–output variables and descriptive statistics.

Variables	Definition	Mean	SD	Minimum	Maximum
Input variables
The number of conventional hours (CH)	An indicator for improving the financial management of any structure in the university from Romania. Based on the average cost of the conventional hour unit, comprehensive analyses can be made regarding the cost of study programs, which can then be compared with the results obtained in different study programs of the university or with the same program developed in different higher education institutions.	3897.96	3100.21	1004.08	15,834 1
		3947.22	3145.52	915.32	16,030 2
		3972.95	3134.53	1077.44	15,288 3
The space allocated in square feet (SP)	The space allocated to each faculty in which students have courses and the staff teach.	3722.32	1026.58	1277.75	5039.99 1
		3722.32	1026.58	1277.75	5039.99 2
		3722.32	1026.58	1277.75	5039.99 3
Output variables
The total number of graduates (GS)	The number of students in each study program who completed their studies.	42.05	31.94	9	116 1
		39.74	27.63	9	116 2
		44.82	30.86	5	137 3
The average marks (scores) of undergraduate students (AMS)	The average scores of students when they finished the study programs of the university.	7.53	0.4	6.9	8.44 1
		7.53	0.38	6.68	8.21 2
		7.56	0.4	6.8	8.31 3

¹ The data on the academic year 2016/2017; ² the data on the academic year 2017/2018; ³ the data on the academic year 2018/2019.

). The selection of input and output variables was based on the criteria internally used by the top management of the analyzed university to allocate the public funds for educational purposes in relation to its study programs. If in other universities, some other criteria are considered, these variables might undergo modifications.

The total number of conventional hours in an academic year can be determined by multiplying the number of conventional hours included as weekly averages in the positions of the payrolls, by 28 weeks. In terms of the analysis of university efficiency, some researchers refer to other indicators such as the number of undergraduate and graduate degrees granted, the student credit hours generated, and the FTE produced [34]. A credit hour is equivalent to “one hour of classroom or direct faculty instruction (defined as a nominal 50 min classroom hour) and a minimum of two hours of out-of-class student work each week for approximately fifteen weeks for one semester hour of credit or the equivalent amount of work over a different amount of time” [34,35,36,37]. The number of student credit hours results from multiplying the number of course credits taught by a specific department by the number of enrolled students in those courses. This input is used in some papers such as those by Chiang Kao and Hsi-Tai Hung [35], Lopes and Lanzer [36], Moreno and Tadepalli [37], and Salah R. Agha et al. [38]. The produced FTE is a value used to measure student and faculty activity at the undergraduate and graduate levels. The produced FTE is calculated by first multiplying the student credit hours obtained from undergraduate coursework in a semester by 15, multiplying the student credit hours obtained from master’s coursework in a semester by 12, and multiplying the student credit hours obtained from doctoral coursework in a semester by 9, and then adding these three numbers [34]. The produced full-time equivalent (FTE) is used in various papers such as those by Lopes and Lanzer [36], Moreno and Tadepalli [37], and Thanassoulis et al. [11].

The second input, the space allocated in square feet (SP) was used in some other studies. For example, in a study conducted by Lopes and Lanzer [36], the authors used the building space allocated to each academic unit to assess the efficiency of faculty departments. Additionally, Moreno and Tadepalli [37] included this input in their research, and Kao and Hung [35] used the floor space as an input to measure the relative efficiency of faculty departments at National Cheng Kung University.

The first output used in this analysis was the total number of graduates (GS), which indicates the quality and quantity of teaching, and it was also used by Youn and Park [14], Worthington and Lee [39], and Katharaki and Katharakis [40]. The average marks (scores) of undergraduate students (AMS) upon the completion of their study programs was the second output used in the DEA analysis in this paper. Some researchers used other indicators such as the number of undergraduate, postgraduate, and PhD awards, which was introduced by Worthington and Lee [39] in their analysis of efficiency, technology, and productivity change in Australian universities; the number of graduates and the number of higher degrees awarded used in the study by Athanassopoulos and Shale [41]; the number of examinations and the number of finished supervised diploma thesis included in the DEA analysis for evaluating intellectual capital by Leiter [42].

This research was based on 38 study programs in a public university in Romania for three academic years (2016–2019). The data were obtained from the reports of the university rector, the websites of the faculties within the university, and other sources to improve data accuracy. Two inputs and two outputs were used to compare the efficiency of some representative study programs of a university in Romania [43].

6. Results and Discussions

The results in this study were based on the efficiency scores of study programs in a public university in Romania using the DEA analysis. The period under consideration was between 2016 and 2019, and this period included three academic years.

In the first model of analysis, 38 study programs from the university were compared under CRS-DEA output-oriented approach, and the results showed that only four study programs were efficient during the three academic years (S2, S9, S26, and S35). Specifically, there were eight DMUs (S2, S9, S21, S22, S24, S26, S35, and S38) efficient for the first academic year (period 1), seven DMUs (S2, S9, S13, S16, S26, S35, and S38) were efficient in the second academic year (period 2), and six DMUs (S2, S9, S15, S17, S26, and S35) were efficient in the last academic year (period 3). The most inefficient DMUs in all three academic years were S1, S6, S8, and S30. All these data are shown in

Table 3. Results from CRS-DEA model.

DMU	2016/2017	2017/2018	2018/2019	DMU	2016/2017	2017/2018	2018/2019
S1	62.79	63.91	66.16	S20	72.19	68.64	70.11
S2	100	100	100	S21	100	96.02	97.3
S3	81.25	79.57	80.81	S22	100	95.34	98.17
S4	66.64	92.82	79.14	S23	62.27	71.65	81.25
S5	72.16	62.41	87.46	S24	100	99.03	98.9
S6	59.42	57.54	59.02	S25	62.01	64.96	69.62
S7	69.69	81.67	73.63	S26	100	100	100
S8	59.69	67	57.19	S27	85.79	85.44	88
S9	100	100	100	S28	88.48	85.22	92.2
S10	68.16	65.33	66.94	S29	75.19	78.27	74.59
S11	59.88	55.59	80.72	S30	62.84	63.54	64.59
S12	71.41	65.73	84.74	S31	80.1	87.25	82.94
S13	89.93	100	93.02	S32	79.47	77.66	77.29
S14	86.72	82.68	90.05	S33	66.84	68.22	64.27
S15	97.56	93.84	100	S34	88.02	74.05	73.18
S16	86.32	100	87.97	S35	100	100	100
S17	89.38	90.62	100	S36	64.37	68.85	75.08
S18	64.67	65.78	67.74	S37	83.82	84.46	88.22
S19	72.23	78.69	77.5	S38	100	100	98.82

and Figure 1 below.

Under the VRS-DEA approach, 8 study programs appeared to be efficient during the three academic years (S2, S4, S9, S21, S24, S26, S35, and S38), but there were 10 DMUs efficient for the first academic year, 11 DMUs efficient for the second year, and 12 DMUs for the last year analyzed. The most inefficient DMUs under the VRS approach were S27 and S29.

Table 4. Results from VRS-DEA model.

DMU	2016/2017	2017/2018	2018/2019	DMU	2016/2017	2017/2018	2018/2019
S1	89.1	89.81	87.78	S20	100	100	98.8
S2	100	100	100	S21	100	100	100
S3	95.05	96.2	93.38	S22	100	95.96	98.24
S4	100	100	100	S23	85.7	94.65	96.87
S5	91.37	89.65	97.5	S24	100	100	100
S6	92.38	88.04	88.45	S25	86.85	90.9	91.2
S7	99.97	98.21	100	S26	100	100	100
S8	96.56	93.14	94.4	S27	88.3	87.2	88.39
S9	100	100	100	S28	89.49	91.26	92.28
S10	98.46	99.88	99.04	S29	86.18	88.33	86.13
S11	95.46	95.06	100	S30	96.07	97.09	95.79
S12	92.75	87.59	96.91	S31	91.75	97.67	94.85
S13	93.29	100	93.47	S32	93.31	92.12	91.85
S14	91.42	89.29	91.7	S33	90.98	96.57	91.22
S15	97.9	94.44	100	S34	98.77	83.58	84.49
S16	87.76	100	88.88	S35	100	100	100
S17	90.65	95.64	100	S36	87.63	92.58	89.16
S18	93.38	93.79	91	S37	91.6	92.53	92.63
S19	86.07	91.76	89.55	S38	100	100	100

and Figure 2 show the results of this model.

After the CRS-DEA and VRS-DEA analyses, we calculated the scale efficiency for each study program. The results from

Table 5. Results from scale efficiency (SE).

DMU	2016/2017	2017/2018	2018/2019	DMU	2016/2017	2017/2018	2018/2019
S1	70.47	71.16	75.37	S20	72.19	68.64	70.96
S2	100	100	100	S21	100	96.02	97.3
S3	85.49	82.71	86.54	S22	100	99.35	99.93
S4	66.64	92.82	79.14	S23	72.66	75.7	83.88
S5	78.97	69.61	89.7	S24	100	99.03	98.9
S6	64.32	65.36	66.73	S25	71.4	71.46	76.34
S7	69.72	83.16	73.63	S26	100	100	100
S8	61.81	71.94	60.58	S27	97.15	97.98	99.56
S9	100	100	100	S28	98.87	93.38	99.91
S10	69.22	65.41	67.6	S29	87.25	88.6	86.61
S11	62.73	58.48	80.72	S30	65.42	65.44	67.43
S12	76.99	75.04	87.44	S31	87.3	89.34	87.44
S13	96.41	100	99.52	S32	85.17	84.3	84.15
S14	94.86	92.6	98.2	S33	73.47	70.64	70.45
S15	99.66	99.36	100	S34	89.11	88.6	86.61
S16	98.36	100	98.98	S35	100	100	100
S17	98.59	94.75	100	S36	73.46	74.37	84.21
S18	69.26	70.13	74.44	S37	91.51	91.28	95.24
S19	83.92	85.75	86.55	S38	100	100	98.82

and Figure 3 show that four DMUs were efficient during the analyzed period (S2, S9, S26, and S35). Additionally, three study programs were the most inefficient DMUs during the three academic years (S6, S8, and S30).

Particularly, there were eight DMUs (S2, S9, S21, S22, S24, S26, S35, and S38) that were efficient in the first period (2016/2017), and the most inefficient DMUs were S6, S8, S11, and S30. For the second period (2017/2018), there were seven efficient DMUs (S2, S9, S13, S16, S26, S35, and S38), and the most inefficient DMUs were S6, S11, and S30. Finally, there were six efficient DMUs (S2, S9, S15, S17, S26, and S35) in the last period (2018/2019), and the most inefficient DMUs were S6, S8, and S30.

In the case of DMU S6 and DMU S30 (the most inefficient units), certain observations can be drawn. During the three academic years, their efficiency average was 65% and 66%, respectively, out of 100% efficiency and, compared with their efficient peers (S9 and S20), DMU S6 and S30 should better administrate outputs by increasing the number of graduate students (GS) on average by 68% and 62%, respectively, and the average marks of undergraduate students on average by 11% and 3%; they should also proportionally decrease the number of conventional hours in an academic year (CH) on average by 21% and 19%, respectively, and the space allocated in square feet (SP) by 19% and 25%.

To avoid distortions in the efficiency scores, we decided to implement the bootstrap method to correct DEA efficiencies (with 1000 iterations and a confidence interval of 95%) under output-orientated CRS and VRS approaches.

The results reported in

Table 6. The average efficiency DEA score estimates in the analyzed period.

DMU	CRS	CRS Bias Corrected Mean	VRS	VRS Bias Corrected Mean	DMU	CRS	CRS Bias Corrected Mean	VRS	VRS Bias Corrected Mean
S1	64.29	63.15	88.90	88.41	S20	70.31	69.02	99.60	99.43
S2	100	100	100	100	S21	97.77	96.12	100	100
S3	80.54	77.51	94.88	93.55	S22	97.84	96.86	98.07	97.12
S4	79.53	72.95	100	100	S23	71.72	68.08	92.41	91.80
S5	74.01	68.60	92.84	90.76	S24	99.31	98.73	100	100
S6	58.66	57.46	89.62	89.14	S25	65.53	64.23	89.65	89.12
S7	75.00	69.92	99.39	98.99	S26	100	100	100	100
S8	61.29	54.74	94.70	92.91	S27	86.41	83.28	87.96	86.39
S9	100	100	100	100	S28	88.63	85.29	91.01	89.14
S10	66.81	66.00	99.13	98.77	S29	76.02	74.31	86.88	86.26
S11	65.40	61.71	96.84	96.00	S30	63.66	61.56	96.32	95.59
S12	73.96	69.95	92.42	91.75	S31	83.43	80.71	94.76	93.90
S13	94.32	93.17	95.59	94.96	S32	78.14	76.61	92.43	91.64
S14	86.48	82.33	90.80	88.69	S33	66.44	65.02	92.92	92.38
S15	97.13	95.50	97.45	96.24	S34	78.42	75.43	88.95	88.32
S16	91.43	90.62	92.21	91.39	S35	100	100	100	100
S17	93.33	92.37	95.43	94.67	S36	69.43	66.26	89.79	89.23
S18	66.06	60.24	92.72	91.75	S37	85.50	80.79	92.25	90.24
S19	76.14	75.03	89.13	88.52	S38	99.61	99.25	100	100

and Figure 4 indicate that only four DMUs were efficient in all three academic years. In comparison with the classical CRS-DEA and VRS-DEA approaches, it was observed that DMU 2, DMU9, DMU26, and DMU 35 remained efficient during the analyzed period. Moreover, the results show that DMU6 remained the most inefficient DMU.

In this analysis, it was observed that the inefficiency of study programs was relatively persistent (89%), compared with the efficient DMUs (11%) for the period between 2016 and 2019.

In this context, one should carefully analyze the possibility of increasing the quality of study programs correlated with the degree of the attractiveness of various programs in the current socio-economic environment, by anticipating the reduction in the number of students. The university is a place for knowledge transfer and creativity but can also be a place of economic innovation and high-performance education by increasing the number of candidates for truly attractive programs.

7. Conclusions and Future Research

According to the Law of Education no 1/2011, higher education institutions in Romania function as institutions that grant academic degrees in various subjects based on the full-time attendance of on-campus study programs, in both undergraduate and postgraduate education, which is predominantly by public means through a national government. These institutions are financed through different funds: public funds, revenues from various projects, and other sources [44]. All these funds form the institution’s own revenues and are allocated in the context of university autonomy. To be more productive, a university could monitor its annual dynamics in terms of the number of enrolled students in different study programs, correlated with the attractiveness of similar study programs from other universities. Another factor in the institutional autonomy context is the dynamics of the drop-out rate [45].

This research paper introduces the DEA method as an effective tool to assist decision makers from universities and national bodies to assess the efficiency of resource allocation in their study programs. It is possible to assess similar study programs from various universities based on public data and to encourage the sharing of good practices among universities. The consideration of DEA for risk management encourages continuous innovation in delivering study programs, as long as the implemented measures are relative to the best in the given pool of universities. It also highlights the general public transparency in the use of public funds. The preventive risk management of resource allocation in higher education could be measured within the diversification of current education activities on teaching and enrollment; the funding of different sources of finance by introducing new attractive programs and updating the current ones; better quality and more efficient work from the staff; possibilities for employed and distant students to study under the collaborative network developed with other HEIs. This study provides an analysis of the study programs in Romanian higher education institutions according to the importance of their optimization. The goal of this research was to analyze the relative efficiency of study programs in the public university in Romania using the DEA method. This type of analysis enables us to understand how efficient study programs can be compared with their peers and can help them improve their performance. The results obtained in the three academic years analyzed were according to specific data.

For the three academic years of study (2016/2017, 2017/2018, and 2018/2019) used in this research, efficiency was measured at the level of the output variables and depended on the number of graduates with a bachelor’s degree and the arithmetic average of the marks obtained by these graduates in each study program. The observations of this study were made by directing the analysis toward maximizing the values of the output variables. The results showed that only four study programs were efficient during this period under the CRS-DEA approach and seven study programs were efficient under the VRS-DEA model. According to the scale efficiency, the results also showed that the four study programs were efficient during the analyzed period.

In the next step, we implemented the bootstrapping method to correct DEA efficiencies, and the results showed that only four study programs were efficient: DMU 2, DMU9, DMU26, and DMU 35.

In conclusion, the efficiency of higher education institutions is important for achieving a high level of performance by reallocating resources to optimize their educational programs. The university’s mission is to serve the community that pays for education in quality study programs. The investigation of the university model and the operational mechanisms designed by academic staff enables us to understand how efficient certain study programs can be in terms of using various resources, compared with other study programs of the university.

There are still some limitations in this research. The DEA method is not capable of showing us how well the best study program is running, as it deals only with relative comparisons. It is also not capable to make a clear link between cause and effect. For these limitations, other methods must be considered. Additionally, inputs and outputs can relativize the indicator of efficiency, but this issue is not a specific weakness for DEA, as the same is true for any method that operates with inputs and outputs (garbage-in-garbage-out).

The next step in our research is the development of a methodology for encouraging high performance and raising the quality of study programs in which current investments can bring good future achievements. With this point of view, it is useful to prioritize the study programs in a university considering their level of efficiency as a major criterion. By establishing sustainability criteria, a matrix of correlations between study programs and the sustainability criteria can be obtained. Thus, different levels of importance for each educational program can be determined. Moreover, the analysis of study programs in all public universities in Romania is necessary for future research.