Evaluation of Public Services Considering the Expectations of Users—A Systematic Literature Review

Abstract

Evaluating public services has become an important task in order to direct actions that may positively affect with the quality of the service provided by governments. To undertake an effective evaluation, it is necessary to analyze data and information on the impact of the services. A wide range of studies have been proposed to measure how an organization delivers its services according to the expectations of the stakeholders. This paper investigates approaches for evaluating public services from the perspective of users. The goal is to identify and describe evaluation-based models, as well as the instruments and tools employed in the service evaluation process. A systematic literature review was conducted to search and analyze studies published in the last 15 years. From the analysis of 31 studies, we identified four main dimensions regarding service evaluation: quality, success and acceptance of information systems, user satisfaction, and user experience. This work contributes to the identification of models, dimensions, instruments, and tools to evaluate public services from the perspective of users. The results of this work can be used as a guide to the Public Administration in the construction of effective models to evaluate their public services and to guarantee quality standards that meet the expectations of the users.

1. Introduction

The relationship between governments and users has changed in recent decades because of digital transformation. The use of technologies not only improves the performance of the service but also increases the service’s reach [1]. The need to assess the effectiveness of public services arises as governments increase the development of online systems to deliver public services [2]. Such evaluation efforts can enable government organizations to determine if they are capable of doing the task required and delivering services as expected. In this context, performance based models emerge to evaluate the effectiveness and efficiency of public services, in order to make them closer and more acceptable to users [3,4].

A great challenge to evaluate the performance of public services is to obtain valid data to guide the Public Administration with direct actions that may positively interfere in the quality of the service. The first models for service evaluation used to consider organizational contexts and dimensions that could impact the quality of a service, such as empathy and courtesy [5,6,7]. With technological evolution, the perception of the user about a service changed. The previous evaluation models also evolved and started to consider new dimensions, such as usability [8,9].

Evaluating public services from the perspective of the user became an opportunity to detect which factors hinder or facilitate their impact on society. However, there is no single unitary indicator in the literature to assess the effectiveness of public services [10]. For this reason, a mapping study of the main differences between evaluation-based models and their applications would be an excellent guide for the Public Administration to implement an effective evaluating model for its services.

The goal of this work is to investigate approaches related to the evaluation of public services from the perspective of the user. The proposal is to identify the most used models, tools and dimensions in the evaluation of public services. For this, a systematic literature review (SLR) was defined and conducted, following the guidelines proposed by [11,12]. After conducting the SLR research protocol, 31 papers related to the goal of this work were identified and selected. Their models are characterized by their approaches as well as their evaluation dimensions, instruments, and tools. A significant contribution of this work is the mapping of several works in a succinct meta-analysis, allowing the manager or service evaluator to easily identify the most widely used dimensions in the literature considering the expectation of the users.

This paper is organized as follows. In Section 2, the research methodology is described detailing the steps of the SLR process. The results of the SLR conducted are presented in Section 3, which explains the service evaluation approaches; in Section 4, the service evaluation dimensions are analyzed; and Section 5 presents the models, instruments, and tools of service evaluation employed by the works studied. A discussion on the relevant findings and their implications is presented in Section 6. Finally, Section 7 presents the final considerations.

2. Research Design

In this work, we conducted a systematic literature review (SLR) to identify, select, and critically appraise researches related to the evaluation of public services from the perspective of the user. The SLR technique aims to identify, evaluate, interpret, and aggregate research results to provide an objective summary of evidence related to a research question [13,14]. Our SLR followed the guidelines proposed by [11], who defined a research protocol characterized by three phases:

• Planning: objective and research questions are defined, as well as the search strategy, selection of the databases, and the criteria for inclusion and exclusion;
• Conduction: execution of the search string in the selected databases, followed by the application of the selection criteria and the data analysis;
• Report: report the execution of the research and the analysis of the results obtained.

2.1. Planning Phase

Research questions guide the execution of the SLR [14]. They consist of one main question (MQ) and derived secondary questions (SQ).

Table 1. Research questions.

ID	Research Question
MQ	What are the existing approaches to evaluate public services from the perspective of the user?
SQ1	What are the dimensions used to evaluate public services from the perspective of the user?
SQ2	What models, instruments, and tools have been adopted to assist in the evaluation of a service, considering the satisfaction of the user?

presents the research questions defined for this work.

The research protocol followed in this SLR is composed of the development of the search expression, the selection of scientific databases, the delimitation of the coverage period, and the definition of inclusion and exclusion criteria. To assist in the elaboration of the search expression, we used the PICOC approach [15]. The acronym PICOC stands for five elements: Population, Intervention, Comparison, Outcome, and Context.

Table 2. Research terms based on the PICOC approach.

	Term	Synonyms
Population	Service evaluation	Service quality, service assessment
Intervention	Citizen perspective	User perspective, user satisfaction
Comparison	-	-
Outcome	Model	Tools, dimensions, approach
Context	Government	Public sector, Public services

shows the search terms obtained from the PICOC analysis.

To select the relevant studies, we defined some inclusion (IC) and exclusion criteria (EC), which are presented in

Table 3. Inclusion criteria.

ID	Inclusion Criterion
IC01	Studies that present approaches for the evaluation of public services according to the perspective of the user
IC02	Studies that present models, techniques, and tools for the evaluation of public services according to the perspective of the user
IC03	Studies that suggest dimensions for the evaluation of public services according to the perspective of the user
IC04	Studies that present a history of the evaluation of public services according to the perspective of the user
IC05	Studies that present reports of evaluation of public services according to the perspective of the user

and

Table 4. Exclusion criteria.

ID	Exclusion Criterion
EC01	Studies related to the evaluation of public services from a point of view other than that of the citizen/user
EC02	Studies related to the evaluation of services in the context of industry
EC03	Studies related to digital government outside the context of service evaluation or quality of services
EC04	Studies written in a language other than English
EC05	Studies that do not fit into any inclusion criteria

. Besides the inclusion and exclusion criteria, three other filters were also defined to select the most relevant studies to compose the SLR analysis. Figure 1 shows these filters.

2.2. Conduction Phase

Based on the PICOC analysis, the keywords for the search expression was defined as: –TITLE-ABS-KEY((“service evaluation” OR “service quality” OR “service assessment”) AND ((“citizen perspective” OR “citizen satisfaction”) OR (“user perspective” OR “user satisfaction”)) AND (“model” OR “tool” OR “technique” OR “dimension” OR “approach”) AND (“government”)).

Three databases (Scopus, ACM Digital Library, and IEEE Xplore) were selected to run the search expression. They were chosen because of their advanced search engines, which allow the use of complex Boolean expressions, as well as several filters to refine the results retrieved. Scopus, for example, also indexes publications from different sources, including Springer Link [16]. The search expression was applied in each database, which in total returned 113 articles, as presented in

Table 5. Databases’ results.

Database	Results
SCOPUS	80
IEEE Xplore	13
ACM Digital Library	20
Total	113

.

Next, the coverage period was defined: at first, only papers published in the last 10 years would be considered in the research; however, as the bibliometry results showed only a few papers had been published before that timespan, we decided to include all papers found in the databases. Then, the type of works to be considered in the review was defined: Only conference and journal papers were included. If a paper was presented at a conference and later published in a journal, the latter was included.

Figure 2 shows the full process of paper selection. After applying Filter 1, we identified 14 duplicate articles (12.3%), which were excluded. There were also four records duplicated which were also discarded. After the application of Filter 1, 95 articles remained. Filter 2 consisted of reading the abstracts of the publications, considering the inclusion and exclusion criteria shown in Table 3 and Table 4. Of the 95 articles analyzed, 40 (42.1%) met the inclusion criteria. Filter 3 consisted of the complete reading of these 40 articles. Four of them were not available for download (10%) and were therefore excluded from the research. Of the 36 articles analyzed after full reading, 31 (86.1%) met the inclusion criteria and were selected.

Table 6. List of papers selected.

Code	Ref.	Code	Ref.	Code	Ref.	Code	Ref.
S01	[17]	S09	[18]	S17	[19]	S25	[20]
S02	[21]	S10	[22]	S18	[23]	S26	[24]
S03	[25]	S11	[26]	S19	[27]	S27	[28]
S04	[2]	S12	[29]	S20	[30]	S28	[31]
S05	[32]	S13	[33]	S21	[34]	S29	[35]
S06	[36]	S14	[37]	S22	[38]	S30	[39]
S07	[40]	S15	[41]	S23	[42]	S31	[43]
S08	[44]	S16	[45]	S24	[46]	-	-

presents the references of the 31 studies analyzed in this SLR. All inclusion criteria were met by at least 1 study. Most of the studies presented approaches (IC1), models, techniques, and tools (IC2), and/or dimensions to evaluate services from the perspective of the user. In the next sections, we discuss the qualitative aspects of the 31 articles selected.

3. Evaluation Approaches

Our study identified four evaluation approaches of public services from the perspective of users: quality of services, user satisfaction, user experience, and success and acceptance of Information Systems (IS).

Table 7. Evaluation approaches and their respective studies.

Approach	Studies
Service quality	S03, S07, S09, S11, S12, S16, S20, S29, S30
User satisfaction	S06, S08, S14, S23, S26
User experience	S18, S22, S25
Success and acceptance of IS	S02, S04, S05, S10, S13, S15, S19, S21, S27, S28, S31

summarizes each study and their corresponding approach. In the following, we discuss the characteristics of each approach.

3.1. Service Quality

Quality is a broad and complex topic. It covers everything from institutional practices to the application of specific statistical tools and techniques to improve and monitor quality levels [18]. According to [45], the quality of services is intangible and therefore its measurement is complex. In general, the evaluation models based on the quality of services seek to measure the perceptions of the users against the results of service delivery.

Most studies that take this approach use or are based on the SERVQUAL [6] model, which adopts a scale divided into five dimensions: tangibles, responsiveness, reliability, assurance, and empathy. This instrument assesses users’ expectations and perceptions of a service. Based on this, several studies focus on the perceptions of quality, which is often compared to the expected quality to obtain a final metric. Thus, the user’s expectation and perception of the quality of the service received are compared.

Furthermore, some of these studies state that user satisfaction is a direct result of the quality of services. It is frequently pointed out that, despite the focus on the perception of users, a complete evaluation of the quality must include the entire service delivery mechanism, analyzing, for example, its technical and functional aspects.

According to [18], the most important factors that influence public service quality are:

• Access to the service;
• Communication level;
• An understandable administrative system;
• The ability to provide a flexible and quick reply;
• Receptivity to service;
• Politeness and kindness of the administration staff;
• Credibility in the service supply;
• Reliability and responsibility;
• Security and quality of tangible aspects.

In [33], factors influencing the e-service quality of government portals were identified from an extensive review of research performed by academic scholars and practitioners. Seven constructs were identified from the analyses: citizen centricity, transaction transparency, technical adequacy, usability, complete information, privacy and security, and usefulness of information. These constructs can be used to assess the service quality of government portals.

3.2. User Satisfaction

User satisfaction is a psychological aspect related to the customer’s goals and needs. It serves as a benchmark tool for assessing user-oriented government strategies [37]. According to [24], satisfaction is a central field of interest for contemporary organizations. Other common dimensions to this approach are quality of services, customization or flexibility, utility, and user loyalty.

Evaluation based on user satisfaction generally comprises variables of service quality. Broadly speaking, service quality and use determine satisfaction. In [44], the authors argue that user satisfaction guarantees an immediate and significant response to the preferences and expectations of service users. However, the same authors also point out that many different models can be adopted to measure user satisfaction.

According to [36], governments ensure user satisfaction by utilizing the information and communication technology properly, specially the internet. By improving this channel of communication, the public sector ensures the accessibility and completeness of government information and service delivery in a more convenient way. However, the quality of websites is frequently overlooked during the design and implementation stages of online public services. Besides transparency, ease of navigation and comprehensive information require adequate monitoring resources, including targets for responses and reports. In fact, online users expects the organization to respond to their inquiries without delay. An immediate response will assist government website users to make decisions if it answers questions and resolves problems [29].

User satisfaction is a performance measurement that can be assessed with various techniques, for example Technology Acceptance Model (TAM), McLean and Delone Model, SERVQUAL, and EGOVSAT Model. It provides immediate, meaningful and objective feedback about user’s preference and expectations. On the other hand, the performance of public services will be evaluated in relation to a set of dimensions that indicate the strong and the weak factors affecting user satisfaction with public services [44].

3.3. User Experience

User experience encompasses all the tools, people, and processes involved in building a co-produced user experience [20]. It comprises all activities required of the user to obtain the service and/or fulfill their obligations to the government.

This approach correlates with the area of human–computer Interaction. As [38] suggest, it is closely linked to usability and can serve as an extension of it. Common dimensions to this approach are safety, flexibility or convenience, errors, and attractiveness.

According to [38], users have differing backgrounds, experiences and cultures. Users require effective, easy, and enjoyable interaction, which is key to successful use and acceptance of applications. In [23], a broad based research instrument was developed to capture perceptions of the user experience with the Nigeria Immigration Service website. They identified eight factors related to user experience: security and support, content and information, ease of use, benefits, barriers, convenience, trust, and website quality.

Usually, citizens do not have a clear understanding of the various functions of the service and how to navigate them, nor should they. In fact, [20] demonstrated that service usability can be improved by taking different perspectives of the service experience. However, the political will to make changes across government is missing. Public services will remain fragmented, reliant and difficult to consume until a conscious decision is made to evaluate the success of government based on the user experiences.

3.4. Success and Acceptance of IS

Understanding success in Information Systems (IS) is a complex challenge made more difficult when set in the public sector environment, while private sector studies may focus on profitability efficiency, quality, and reliability, public sector evaluation must combine these concerns with accountability, citizen trust, and the creation of public value [22].

To evaluate service delivery strategies, most of the studies identified in this SLR examine the success and acceptance of the IS based on the DeLone and McLean Model of Information Systems Success (D&M model) [21] and the Technology Acceptance Model (TAM) [47]. In [34], the authors emphasized that the DeLone and McLean IS success models are among few such theories and models that have helped researchers to establish some of the salient factors that influence the acceptance and use of public services by citizens before testing and post adoption. TAM [47] is a basic model used for technology acceptance studies. This model identifies the causal relationships between system design features, perceived usefulness, perceived ease of use, attitudes towards using, and actual usage behavior. It suggests that only ease of use and usefulness are insufficient to predict the behavior of users, as it is also necessary to include features such as system flexibility, system availability, information reliability, etc.

Studies that use acceptance of IS as an evaluation approach seek to measure service success by relating variables concerned with the overall quality, the constructs directly related to quality, and the behavioral and intentional variables of system use. The net benefit received by the user is also often analyzed. The variable of use—in its behavioral aspect—allows for determining the acceptance of the system. Then, the cyclical interrelationship between use, satisfaction and benefits makes it possible to point out the success of a system.

In conclusion, acceptance of the system and its subsequent use are vital points for the success and improvement of a user-oriented service.

4. Evaluation Dimensions

Similar evaluation dimensions are found in different models although these dimensions are constructed under a variety of perspectives and theories. In this sense, several models try to develop an evaluation based-approach that can be adapted to the situation. They may vary depending on the perspective used, the goal of the assessment and even the context of the object of study.

Table 8. Evaluation dimensions and their respective studies.

Dimension	Studies
Satisfaction	S01, S02, S03, S04, S06, S07, S11, S14, S15, S18, S21, S22, S24, S26, S27, S28
Service quality	S02, S03, S04, S06, S07, S12, S13, S15, S19, S23
Information quality	S02, S04, S07, S12, S15, S18, S19, S21, S22, S27, S28, S31

shows the most used dimensions among the models and evaluation approaches identified in our SLR. It should be noted that similar dimensions may appear with slightly different names in the literature.

Satisfaction is the most used dimension to assess the performance of public services. The popularity of this dimension is because the studies evaluate public services from the perspective of the user; therefore, in general, they assess user satisfaction. Sigwejo and Pather [46] argue that satisfaction can serve as an important indicator of the effectiveness of digital public services.

Other common dimension to evaluate the performance of public services is related to the service quality. In general, this dimension measures perceptions about the delivery and quality of the services, investigating the gap between the user’s expectations and experiences. Service quality is often a predictor of user satisfaction. Traditionally, user satisfaction has been measured indirectly through information quality, system quality, service quality, and other variables [21].

There are also dimensions related to information and its quality. Broadly speaking, this dimension evaluates if the information provided to users is easily understandable, complete, and relevant. In [2], the authors indicate that perceived usefulness is influenced directly by beliefs about system quality and information quality. Information quality exerts the stronger effect on user satisfaction, in comparison to service quality.

5. Models, Instruments, and Tools

Two main research tools were identified in the SLR: questionnaires and interviews. Of the 31 articles, 21 used questionnaires, 7 performed interviews, and 5 used both. The questionnaires usually comprised the analysis of existing literature and reviews or interviews with government officials. The interviews were often used in support of the questionnaires, to raise the aspects determining the quality of the services or to validate the researchers’ ideas.

Some studies are based on bibliographic reviews to build their models [31,38]. Others make use of practices from other areas, such as focus groups [46]. Other practices, such as service documentation analysis [46] and analysis of user complaint reports [20] also appear.

The D&M model has a broad procedural focus, establishing specific causal relationships. The TAM model also has a procedural approach but considers a more individual rather than general perspective, emphasizing the influence of individual variables such as subjective norms on the decision-making process regarding the use of a service. SERVQUAL, in turn, focuses on the performance of the service, considering its general characteristics. Future studies may propose integrated models for evaluating public services.

Most studies use linear regression models or structural equations based on exploratory and confirmatory factorial analyses, such as Principal Component Analysis (PCA) and Structural Equation Modeling (SEM), to evaluate the structure of the measuring instrument, as well as reliability analyses to evaluate the consistency and reliability of the sample. Other models are also used, such as Partial Least Squares (PLS) [25], Multicriteria Satisfaction Analysis (MUSA) [24], Grounded Theory [46], content analysis [29], Principal Axis Factoring [33], and support of the Parasuraman Gap model [6].

Table 9. Models and context of application.

Model	Country
D&M	China, Greece, India, Taiwan
SERVQUAL	China, India, Singapore
ECSF	Tanzania
TAM	India, Saudi Arabia
UTAUT	India
EGOVSAT	Malaysia
XE	Australia
ACSI	USA
ECSI	Europe
Context-specific	Afghanistan, Greece, Nigeria, Saudi Arabia, Spain, Sweden

presents an overview of the models used according to the country. In general, research in developing countries adopts models already established in the area, without well-defined mechanisms for their adaptation, while developed countries create context-specific models, as is the case with the American Customer Satisfaction Index (ACSI) and the European Customer Satisfaction Index (ECSI).

6. Discussion

In this section, we discuss results observed from our systematic mapping.

6.1. Performance Evaluation of Public Services

Performance is a term that defines the successful fulfillment of one or more activities performed by an organization. In this sense, evaluating the performance of public services is an utmost task to reveal the full value of the service projects. In the public sector, performance is difficult to identify because the processes are more difficult to quantify than in the private sector where the main purpose is profit. This evaluation is also not effective without considering the perspectives of all the key issues held by each stakeholder, especially the final users [48].

The success of governmental organizations depends heavily the quality of the services they provide to citizens. People place strong importance on the attention the administration pays to the complaints they make. So, a useful and suitable reply system to deal with these complaints is one of the necessary improvements for all the administrations. As a consequence, government service managers should be able to reduce risks (e.g., investing valuable resources in service quality characteristics that may not work effectively).

However, it is important to know what dimensions of quality of public services contribute to enhance users’ satisfaction. As governments increase the development of online systems, it is necessary to assess the effectiveness of their digital services, while information systems (IS) success models have received much academic attention, little research has been conducted to assess the success of digital public services [21]. In this sense, there is a need to investigate whether the success of traditional IS models can be extended to online systems. Such evaluation efforts can enable government agencies to ascertain whether they can do the task required and delivering services as expected.

Currently, the user satisfaction dimension is strongly related with the management, use, and appearance of the online services, especially in town councils with a large population [25]. However, new technologies constitute a basic service to any citizen, regardless of the size of the town, so public actions are necessary to promote the use of new technologies. This would have a positive effect on the relationship between citizens and the public administration.

Service quality is another dimension related with the performance evaluation of public services. The quality of a service can help the public administration to make decisions that increase citizen satisfaction. Prior to building a service quality management system, the problems of the public service should be identified and the measurement tool of public service should be defined [18].

Finally, a limitation of most studies observed by our SLR is that the practical investigations of public services are based on a mandatory information system in a particular country and region, with specific characteristics. This can affect the generalizability of the results. Thus, the interpretation of the results should be confined to the countries and regions with similar settings.

6.2. Practical Utilization of Service Evaluation

The use of performance indicators by public sector organizations has been on the public sector management agenda in many governments around the world. This SLR identified works carried out to explore the evaluation of several applications according to the user’s perspective.

In [17], the authors provide a set of clear and useful e-health evaluation criteria that help improve the use of services by citizens, to integrate the e-health evaluation framework, and to address areas that require further attention in the development of future e-health initiatives. Other works, such as [2,36], focus on the assessment of services related to tax payment. The results point to strong connections between the following constructs: system quality, service quality, information quality, user satisfaction, and perceived usefulness.

Many works use indicators to evaluate the performance of local governments [18,25,37]. Local governments are administrative services that are smaller than a state. The term is used to contrast services at the nation-state level, which mention the central government or (where appropriate) the federal government. It is important to highlight the importance and shortage of studies about public service delivery and quality by local governments. Local organizations have the advantage of knowing better what their citizens want. This is very significant because the greater quality attributed to the public services by citizens, the less they will oppose financing them. This results in a better cooperation between citizens, and there will be a better match between what the citizens want and what the administration provides.

Finally, the need to know whether a public organization has met its aims has led to various forms of assessment. In this sense, it is necessary to develop more understandable forms to approach users more effectively. Some people find it difficult to understand forms they need to fill in. The use of plain language by the public administration is highly appreciated. In this sense, diverse means of communications (besides fax, telephone, Internet, etc.) can become valuable sources for users to evaluate their experiences with the service received. E-mail, comment sections, chat rooms, search features, broadcasting of government events, and web site personalization should facilitate connections and interaction between government institutions and users.

7. Final Considerations

This study identified the approaches for evaluating services from the perspective of the user, specifically the models and dimensions of evaluation of public services mostly used in the last 15 years. It was verified that most of the studies use or adapt older models, some of them developed in the private sector. The three most used models were the DeLone and McLean Model of Information Systems Success (D&M), the Technology Acceptance Model (TAM), and SERVQUAL.

This SLR also identified four approaches of service evaluation considering the user perspective: quality of service, user satisfaction, user experience, and success and acceptance of IS. It also identified three evaluation dimensions: satisfaction, service quality, and information quality.

The lack of service evaluation models that present clear guidelines for adaptation in different contexts creates a loophole for a model to be applied in completely different ways. This SLR showed that the main affected by this are developing countries, which generally use consolidated general models. On the other hand, developed countries usually have instruments specific to their contexts, as is the case with ACSI and ECSI.

A limitation of this paper is inherent in the design and execution of the primary studies reported from the results of a set of articles, which were restricted to the selected databases (Scopus, ACM, and IEEE). Future research may expand the databases and validate the findings obtained in this work.

Public sector services should be designed to allow citizens to manage their lives most efficiently. Such designs should be based on an understanding of the experience and the services shaped to integrate into the user’s world. A good strategy will avoid (or mitigate) failure, especially of electronic government projects. Several factors pertinent to public administration have been investigated and analyzed by previous projects. Knowing these factors allows governments to build more effective plans and actions that contribute positively to the success of their public projects while also avoiding problems that may arise in the perceived effectiveness of public sector services considering the expectations of users.