A Systematic Review of the Application of Maturity Models in Universities

Abstract

A maturity model is a widely used tool in software engineering and has mostly been extended to domains such as education, health, energy, finance, government, and general use. It is valuable for evaluations and continuous improvement of business processes or certain aspects of organizations, as it represents a more organized and systematic way of doing business. In this paper, we only focus on college higher education. For this reason, we present a novel approach that allows detecting some gaps in the existing maturity models for universities, as they are not models that address the dimensions in their entirety. To identify these models and their validities, as well as a classification of models that were identified in universities, we carried out a systematic literature review on 27,289 articles retrieved with respect to maturity models and published in peer-reviewed journals between 2007 and 2020. We found 23 articles that find maturity models applied in universities, through exclusion and inclusion criteria. We then grouped these items into nine categories with specific purposes. We concluded that maturity models used in Universities move towards agility, which is supported by the semantic web.

1. Introduction

According to [1], based on the interactions of people and the differentiation of their roles, defined the university as an organization of the type “professional bureaucracies.” A bureaucracy for Max Weber is the efficient organization par excellence and that it defines even in the smallest details how things should be done [2]. That is to say, the universities mostly regulate their work, as is the case for the selection of their teachers, who assume significant control over their own work, in the teaching-learning process. Since universities are organizations, maturity models have proven to be valuable in evaluating their processes, and determine by levels the path for academic excellence. Maturity models had their origin in the field of quality proposed by Philip B. Crosby, and were consolidated in the field of software engineering [3,4]. The model “Quality Management Maturity Grid” is divided into five (5) progressive stages (uncertainty, awakening, enlightenment, wisdom, and certainty) and six (6) measurement categories (management’s attitude and understanding, organizational quality situation, problem management, cost of quality as a percentage of sales, actions for the improvement of quality and summary of the position of the company with respect to quality) that show the experience relationships that an organization has to go through. The form of evaluation proposed by Crosby is at the organizational, division or area level, and must be verified by three individuals: the quality manager of the operation, the general manager of the operation and a staff member who is not assigned to evaluation area, in order to cross opinions and establish an organization at a maturity level. This model recognizes the importance of human factors, such as leadership, attitude, and collaborative work [4,5].

Currently, the maturity models have been extended to different domains such as education, health, energy, finance, industrial sector, government and general use [6]. Moreover, a maturity model is a widely used tool in evaluating fundamental processes or certain aspects of organizations, since it represents an increasingly organized and systematic way to do business [7].

To identify eventual gaps and the level of development of the maturity models and their validities in university higher education, the maturity models selected were grouped into nine categories, according to their orientation. In this context, we carried out a systematic literature review on 27,289 articles retrieved with respect to maturity models and published in peer-reviewed journals between 2007 and 2020. We found 23 articles (presented in Appendix A) that find maturity models applied in universities, through exclusion and inclusion criteria. However, the efforts are oriented towards evaluating the maturity of specific areas in universities, however not its entire structure.

There are many relevant maturity model proposals in literature. In one study, a list of seven methods was found for the development of maturity models for software engineering and organizational management domains, based on ISO/IEC 15504 [8]. In other study, nine maturity models based on Capability Maturity Model Integration (CMMI) and Information Technology Infrastructure Library (ITIL) were found, oriented towards e-learning, computer courses, IT services and computer science courses for higher education institutions [9].

After following a systematic approach to our literature review, we have considered a selection of previous studies that are related to this research, focusing our analysis in order to identify the level of approach achieved in the maturity models in the universities.

This article seeks to contribute and expand in the field of continuous improvement with topics that answer relevant questions regarding the form of the implementation of maturity models in universities. In the field of education, generally, these models have a broad spectrum that encompasses all higher education (university, technological and pedagogical); however, this study only focuses on university higher education. For this reason, a novel approach is presented through three research questions and two bibliometric analyses that allow detecting some gaps in the existing maturity models for universities, as they are not models that address the dimensions in their entirety. The rest of the paper is organized as follows: Section 2 of this study describes maturity model foundations, and Section 3 describes the methodology addressed in our study. The results and discussions are presented in Section 4 and Section 5. Finally, Section 6 presents conclusions and future studies.

2. Maturity Model Foundations

We have presented theoretical and practical foundations of maturity models, their form of organization by levels and their purpose and field of application in universities. A maturity model is a tool that describes and analyzes the behaviors, practices and processes that enable an organization to reach reliable and sustainable results [10]. In this sense, a maturity model is a reference that allows a better understanding of reality, with the purpose of explaining a phenomenon and giving the ability to make predictions. For example, architectural models are not used to construct buildings, but rather to understand challenges represented by its construction.

There is a report that maturity models are an evolution of methodologies used to manage university quality [11], while also guiding organizations in implementing good practices [12], a maturity model with sequences of levels or stages that form a desired future [13], or logical planned routes, from an initial state to maturity [14,15,16]. In other words, a maturity model is a reference from which to understand the reality of states through which a university must pass towards excellence; that is, it defines routes to take and quality mechanisms at each level of maturity.

For [17], a maturity model can be descriptive, prescriptive or comparative. It is descriptive if it achieves a deeper understanding of the current situation of the domain; it fulfills its purpose if it establishes a level of quality at a given time. It is prescriptive if it allows a path for improvement, or if it determines mechanisms and practices needed to reach maturity levels. If a maturity model has comparison mechanisms, and is generalized with other domains, it is said to be comparative. In addition, these authors also highlight the evolution of a maturity model from a descriptive stage for the understanding of the current situation to become prescriptive, achieving substantial improvements. In other words, assessment is necessary to ensure continuous improvement and organizational maturity.

In addition, Reference [18] cataloged a mature company as an intelligent organization that manages knowledge and maintains it as a competitive advantage, along with the quality and improvement of its products and processes. In other words, the company is able to manage tactical and explicit knowledge for continuous learning.

3. Method

The methodology approach for this study is developing the following review protocol (see Figure 1).

We performed a Systematic Literature Review to develop our study, which refers to the method proposed by Kitchenham and Charters [19]. In order to meet the objective of the study, we formulated the following research questions: Under what categories are the maturity models oriented towards Universities grouped? What are the characteristics of maturity models in universities? How are maturity models in universities going to change? In addition, bibliometric questions were formulated: How has the frequency of publications on this topic evolved over time? In which academic journals are these studies found? Inclusion and exclusion criteria considered for the selection of the primary studies were the following:

3.1. Inclusion Criteria

Establishing inclusion criteria is a required practice for the design of high-quality standards in this study. Inclusion criteria are defined as the key characteristics of documents published in scientific journals. Next, in Figure 2, the criteria are presented:

3.2. Exclusion Criteria

The exclusion criteria allow us to see additional characteristics that could interfere with the success of the study. In addition, they include characteristics of the articles that meet the conditions (see Figure 3).

3.3. Temporality

We considered studies developed in the last 14 years. We evaluated the frequency of publications of maturity models within the university education domain.

Table 1. Terms and logical connectors used in the search.

Terms	Chain
Domain: University education
Maturity model	(“maturity model” OR “capability model” OR “maturity level”)
University education	(higher education) OR (university education) OR (university organization)

shows the chains used, linked with the AND and OR logical operators, to produce a single search string. The following digital libraries were considered for their scientific relevance: Scopus, Web of Science, ScienceDirect, IEEE, ERIC, EBSCO Discovery and Wiley.

3.4. Maturity Model Selection Procedure

The maturity model selection procedure (see Figure 4) was divided into four stages:

4. Results

Table 2. Results of the maturity model selection focused on University Education.

Database	Relevant	Pre Candidates	Candidates	Selected
Scopus	1464	1127	35	10
Web of Science	33	27	2	1
ScienceDirect	308	78	4	2
IEEE	564	307	7	3
ERIC	57	25	4	2
EBSCO Discovery	982	71	17	5
Wiley	23,881	13	0	0
Total	27,289	1648	69	23

lists 27,289 articles that were considered to be “relevant”. These articles were narrowed down by temporality (2007–2020), leaving 1648 “pre-candidate” articles. Pre-candidate articles were then exported to Excel according to Database characteristics, reducing the articles to 69 “candidate” articles. We proceeded to exclude articles whose titles had no relation to the object of study; we excluded duplicate articles and articles that did not refer to maturity levels from the seven Databases, to identify the final “selected” articles: 10 from Scopus, 1 from Web of Science, 2 from ScienceDirect, 3 from IEEE, 2 from ERIC, 5 from EBSCO Discovery and 0 from Wiley, for a total of 23 articles that directly focused on evaluating maturity in University Education domains.

Finally, the 23 selected articles were registered in the Mendeley tool for bibliographic reference management.

4.1. Research Questions

The following contains conclusions regarding the research questions presented above.

• Under what categories are the maturity models oriented towards University Education grouped?

No categories were found that properly grouped the maturity models oriented to university higher education through information extraction; however, upon analyzing their purpose, the articles were organized into nine categories, according to common criteria, as shown in

Table 3. Categories of selected Maturity Models.

Categories	Number of Articles
Maturity models oriented towards teaching.	8
Maturity models oriented towards Information and Communication Technology (ICT).	3
Maturity models oriented towards student monitoring.	2
Maturity models for intellectual capital.	1
Maturity models for E-Learning.	3
Maturity models aimed at evaluating university entrepreneurship.	1
Maturity model oriented to the employability of graduates.	1
Maturity model oriented to the strategic planning of universities.	1
Maturity model for IT governance in university institutions.	3
Total	23

.

• What are the characteristics of maturity models in university education?

The focus of this review, regarding maturity models in university education, does not include specific guidelines due to the diversity of approaches. In [9], they highlight that indicating maturity models oriented to university education give an idea of gaps, since they are focused on a specific area of the university, such as those oriented to students, curricula, online courses, government of TI among others. In this sense, none of these models present mature practices that focus on the different entities of university education. Typically, the models demonstrate “what to do”, but none of them offer “how to” use best practices for continuous improvement established by maturity levels. As in other sectors, CMM [20], together with its successor CMMI [21], are flagship models of maturity in the university environment.

• Which direction will maturity models in higher education take?

For [22], maturity models have a tendency to move according to the agility of their components or initiatives seen from software engineering projects, with greater resources to improve and optimize the path to maturity in a sustained and incremental fashion. Likewise, Reference [7] warns that maturity evaluation with traditional methods consumes many resources, and the lack of automation often renders reference points impossible to measure. In this sense, the tendency is to apply agile methods, to automate the methods of maturity evaluation through ontologies using the semantic web.

4.2. Bibliometric Analysis

• How has the frequency of publications on this topic evolved over time?

When analyzing the results obtained from the search chain and the maturity model selection shown in Table 2, an increase in the number of publications describing maturity models in different domains of university education can be seen in Figure 5 from 2013 onwards. Out of a total of 23 articles, 14 (61%) were published during the last seven years and nine (39%) were published between 2007 and 2013. This shows greater growth and interest in proposing models of maturity that allow for strengthening the university system.

• What are the publications in which studies related to the subject have been found?

Table 4. Scientific journals of selected articles.

Scientific Journals	Quantity
Advances in Intelligent Systems and Computing	1
IEEE Xplore	7
Lecture Notes in Electrical Engineering	1
Total Quality Management and Business Excellence	1
Journal of Intellectual Capital	2
Higher Education Research and Development	1
Direccion y Organizacion	1
SAGE Open	1
Procedia Computer Science	1
Procedia - Social and Behavioral Sciences	1
Electronic Journal of e-Learning	1
Turkish Online Journal of Educational Technology	1
Kaufman Hall	1
IJEDICT	1
Higher Education Research & Development	1
Governance	1
Total	23

presents 23 academic journals, where articles focused on university education have been found, with very specific and isolated domains.

5. Discussion

Our study generates new knowledge proposals in the field of university education maturity, considering studies and prospects from the software engineering domain.

Twenty three of the 27,289 relevant articles were selected, with titles oriented towards university education and maturity levels. The remaining 1648 articles incorporated concepts of maturity in their purposes, but were very generic and did not define a method for evaluating maturity between levels and are oriented towards software development. The challenge will be to establish a flexible maturity model that contemplates the particularities of purposes defined by Universities, in order to diminish the impact of isolated maturity initiatives.

The selected articles corresponded to nine categories: oriented towards teaching, ICT, student monitoring, intellectual capital, e-learning, university entrepreneurship, employability of graduates, strategic planning of universities and IT governance in university institutions because no criteria were found for grouping articles in the literature review. The final 23 articles were analyzed according to their purpose, results and conclusions. We established nine categories and provided an overview of the specific domains of universities.

Maturity models in university education have moved towards agility, automation through ontologies, and the semantic web. These characteristics are being incorporated in maturity models of the application scope of greater use, software engineering, with favorable results due to the use of ontologies, a term that explains the capture of knowledge regarding a specific domain of interest. Thus, in university education, maturity models must have the capacity to add value to society in the acquisition and transfer of knowledge, supported by technology, automation and the semantic web.

In the teaching-learning category, Reference [23] proposes a maturity model for curricular design of higher education institutions in Malaysia. Based on CMMI, it presents five levels of maturity with process areas for each level. Reference [24] presents a maturity model based on the capacity of academic processes, with a maturity scale divided into five levels (see Figure 6).

Reference [25] developed a maturity model with the goal of an engineering program accreditation based on CMMI and knowledge management. The maturity model compares software engineering processes with accreditation processes. It also defines how to advance from one level to another. The proposal of the authors is based on maturity models from the software engineering field; they use CMMI processes to align them with training processes of study programs or universities. Maturity model designs must be viewed from the needs and perspectives of University interest groups, to define assessment mechanisms for teaching and learning with flexible components that respond to institutional goals.

In the category of maturity models oriented towards ICT use, Reference [26] proposes an eight level maturity model, focusing on the use of generic hardware and software as the lowest level necessary for the infrastructure of student formation. Likewise, Reference [27] proposes a maturity model for IT Outsourcing in universities based on ISO 20,000, ISO 38,500, ITIL v3 standards, and Control Objectives for Information and Related Technologies (COBIT) methodologies as best practices in IT governance, with five maturity levels. Levels 1, 2 and 4 are based on ISO 20,000 & ITIL, level 3 is based on ISO 20,000 & author contributions and level 5 is strictly the author’s contribution. Assessing the maturity in the use of ICT for training with an outsourcing environment means transferring management to third parties, allowing updated technologies and services, and avoiding complications with technological waste and inventories. For [28], institutions of higher education require that ICT be part of the daily tools in educational processes. On the one hand, they must address the needs of students who expect, require and essentially demand digital technologies in their learning process. These proposals ensure maturity in the use of hardware, software and connectivity resources to reduce operating costs and improve educational services.

In the category of maturity models oriented towards student monitoring, Reference [29] frames the generational model to a model of capacity for maturity. The generational model implies an evaluation of the institutional capacity to initiate and plan student participation practices in the university. The capacity for maturity model includes five levels, a specific practice relationship, processes and categories. In addition, the model proposes a greater understanding and contextualization of student commitment. Reference [30] presents the SESR-MM as a maturity model to strengthen commitment, success and student retention in universities. The model structure has five maturity dimensions in five categories: learning, support, sense of belonging, integration and resources, as well as 18 processes and practices, and focuses on the improvement of training processes. Maturity models have components for assessing student performance throughout their education, with curricular and extracurricular activities included, and must ensure the achievement of the competencies established in the graduation profile.

In the category of maturity models focused on Intellectual Capital (IC), Reference [31] proposes an Intellectual Capital Maturity Model (ICMM) for universities within a framework divided in phases of actions for change, based on current maturity levels of significant environments. The ICMM contemplates six maturity levels: data collection, IC measurement and management relates to unstructured knowledge management; it is originated by the action, experience and participation of the university community in a specific context. For [32], the teaching process in universities represents intellectual capital for a variety of activities characterized by a mental and intellectual nature as well as a series of other traditional activities. In other words, it defines the ability to achieve the competencies that are defined in the graduation profile.

In the category of maturity models for E-learning, Reference [33] proposes a framework for the continuous improvement of university level e-learning (OCQMM). The OCQMM is based on CMMI and describes the characteristics and requirements of five maturity levels. Its structure includes processes that contemplate practices evaluated by dimensions. Reference [34] presents a pilot study using a method to compare e-learning in university institutions. The pilot was designed to evaluate the operational feasibility of a method based on the e-Learning Maturity Model (eMM), which was developed at the University of Wellington, New Zealand, which in turn was derived from the widely accepted Carnegie Mellon capacity maturity model. Reference [35] also applied a maturity model for e-learning for universities based on CMMI. It includes four process categories: learning, development, coordination and evaluation. In addition, it includes five dimensions: delivery, planning, definition, management and optimization. For [36], most universities tend to increase their use of Web 2.0 technologies to offer online courses, which implies using videos. E-learning is a field that has taken force in university education due to its ease of use and access to ICT, in addition to multiple human activities that facilitate online training. Currently, it is necessary to define criteria for the assessment of capacity and orientation in student formation.

In the category entrepreneurship maturity model category, Reference [37] proposes a business maturity model. The model describes 13 factors and 45 sub factors grouped into a business management framework that includes processes, resources and contexts. It contemplates three maturity scales and describes evaluation mechanisms. According to their objectives, some universities need criteria in order to evaluate the degree of student and graduate entrepreneurship, contributing to society in terms of productivity, skills and innovation, as a point of evaluation.

In the category of a support maturity model for university graduates for students, Pažur and Divjak [38] present a model that assesses the maturity of practices that contribute to the employability of graduates of the technology career. The model was developed using a five-step approach, from identification of purpose, to validation, following the paradigm of design science and exploiting four case studies across Europe. The final maturity model contains 65 practices within four key process areas: 13 within strategic planning, 26 within curriculum design and delivery, 16 within student support, and 10 within provision of extracurricular activities. Additionally, a description of the capacity assessment criteria is provided at the five maturity levels for each practice.

The model presented in the category of maturity model oriented to strategic planning, there is the model presented by Kaufman Hall [39]. This maturity model measures the performance of the strategy-oriented financial planning of universities. Likewise, it is based on four levels of maturity with its practices, which measure performance from the formulation of the annual budget to a budget with periodic reviews, which ensure establishing strategies that allow it to achieve its institutional purposes.

Finally, in the category of IT governance-oriented maturity models, in [40], they present a model that assesses maturity from the edge of service delivery processes, under the COBIT 4.1 framework. This has five maturity levels or indicators of process capacity: level 1-executed, level 2-managed, level 3-established, level 4-predictable and level 5-optimized of IT governance in university institutions under the model of COBIT 4.1 and contemplates the service delivery processes. Each level of maturity has continuous improvement practices in the context of the use of Information Technologies in universities. Additionally, Reference [41] proposes an integration of the BSC with COBIT 4.1 in universities for integration of IT with their organizational strategy. On the other hand, Reference [42] proposes a model that assesses digital maturity in Croatian universities, based on rubrics and qualifications. This model proposes five levels: basic, initial, e-Enabled, e-Confident and e-Mature, it also has an instrument that determines the level of maturity.

6. Conclusions

The models developed or adapted in the university environment are based on a very specific isolated domain, neglecting other key areas of university organizations.

Generally, university maturity models do not suggest areas of work and practices that allow evaluating the level achieved, in order to strengthen continuous improvement.

With this study, it contributes to the state of the art, providing a methodology for identifying maturity models for universities.

We presented the results of a systematic review of 23 academic articles found in digital libraries and indexed databases of great relevance in the scientific and academic field. Likewise, the classification of the studies by publication year is shown in the bibliometric analysis, where an increase in the number of relevant studies published between 2014 and 2020 can be noted.

The maturity models with greater applicability are those oriented towards university education, and those of less applicability towards university entrepreneurship, due to the diversity in the declaration of their objectives that direct their university work. For some universities, the declaration of their objectives is oriented towards teaching, while others focus on teaching with components of university entrepreneurship. These characteristics diverge in the graduation profile, curriculum and educational objectives of each study program and by university.

Universities must adopt descriptive and prescriptive or evaluative maturity models with agility and automation components in order to ensure quality and continuous improvement in education. These models will be developed with tools that support automation through ontologies and the semantic web in order to ensure knowledge management.

Maturity models in their distinct orientations (teaching, ICT and student monitoring) are focused on measuring the maturity of student training, face-to-face or in a virtual modality, with the establishment of criteria that measures the capacity of the formation processes.

We recommend, as future work, to focus on the maturity models of higher education—both technological and pedagogical—evaluating each reality in a concrete way.