Sustainable Assessment of Public Works through a Multi-Criteria Framework

Abstract

Enhancing the sustainability of public works has been a key agenda in recent years for many governmental organizations. Public works contribute significantly to a large portion of engineering works and have great potential to impact the sustainability of cities. Thus, evaluating the sustainability of these projects is highly relevant, mainly regarding their impacts on environmental, social, and economic aspects. There are currently assessment systems and methods with different scopes and approaches. Yet, there remains uncertainty when it comes to considering public works’ sustainability and how useful criteria can be incorporated into the proposed assessment tasks to ensure such a goal. This study contributes to filling this gap by developing, through an extensive and detailed bibliographic research, a flexible and comprehensive framework composed of 214 criteria distributed across nine categories that measure the degree of sustainability of public works, with emphasis on economic, social and environmental goals. The proposed framework can act as a practical tool, functioning as a checklist applicable to all types of public construction works, and at any stage of the lifecycle. Evaluation of the framework by professionals indicated its suitability when encompassing sustainability objectives, its viability, and its ease of use.

1. Introduction

The construction industry is known as a sector that consumes a lot of energy, with high rates of use of natural resources, which negatively impacts the environment [1,2,3]. Some of the reported undesirable effects of the industry include air and water pollution, solid waste, deforestation, toxic waste, greenhouse gas emissions, health risks, global warming and other negative consequences [4,5,6,7].

The impacts of construction on the environment are higher in developing countries [8,9,10,11], whose participation in the world economy is growing at a notable rate [12]. Bibliographic research has shown that there is a scarcity of studies that focus on how public works’ evaluation is carried out to ensure the sustainability of construction projects in developing countries.

Public work can be defined as any implementation or modification of a public good promoted by governments [13]. In general, public administration requires dealing with large-scale contractors of the construction industry to provide products and services [14,15]. In Brazil, public works account for 31.7% of the sector [16], with a significant impact on issues of sustainability. However, when compared to private enterprises, public enterprises are still lagging in terms of their sustainability assessment tools for their projects [17].

Public works have particularities that differentiate them from works in the private sector. In this, the financing comes from the private sector—there is no bidding process required by law for the contracting of services; there is no external and internal inspection for the product to be in conformity; they mainly aim at the return on investment through profit; they are mainly focused on the real estate market, and there is relative freedom for changes in the object to be built. In public works—financing is directly or indirectly derived from taxes; there is a bidding process required by law; there is an external inspection carried out by the courts of account and internally carried out by the public contracting body over its expenses, so as not to be legally penalized; they are mainly directed to meeting the needs of the population and, therefore, are more comprehensive, including urbanization and infrastructure works of the most varied sizes; they do not aim at the return on investment through profit, and, in modification of the object to be built (which has been bid), they follow stringent rules.

Although the sustainability criteria are similar for public and private works, public works serve a larger segment of society, and are more subject to demands for improvement in aspects of sustainability. Additionally, the difficulties imposed by the legislation for changes in the scope of the object to be executed makes it necessary to place greater emphasis on sustainability assessment in the preliminary stages of the contracting process.

Currently, in Brazil, there are no statistics that identify tools that are used to assess the sustainability of public works. Some have indicated that the lack of a tool for assessing public works can be handled by adopting private sector tools, such as the LEED tool [17]. The issue, however, is that public works can be different from private works, mainly due to the nature of the objectives to be achieved and by the contracting system and execution method of the construction works. Some notable differences between public and private works, when it comes to contracting, are as follows: (i) Procurement strategies in terms of the contract, price and segmentation strategy, and procurement procedure; (ii) Solicitation of tender offers; (iii) Evaluation of tender proposals; (iv) Awarding the contract; (v) Administration of the contracts and monitoring of compliance with requirements.

Sustainability assessment methods have been proposed with differing scopes and approaches, yet there is still no consensus as to which method is the most appropriate for any given case [18]. All the systems and methods have limitations that may prejudice their use and efficacy in the determination of sustainable performance [19]. Lately, governments’ public policies have been focusing on establishing sustainable practices not only for new works but also for existing works [8]. This has not been easy to implement, as the assessment of public projects for their sustainability can be vague [8,9,20]. Developing tools that can assess the sustainability of projects can provide insight into understanding the relationship between construction activities and associated social and environmental concerns [19]. As a result, if they can be put into practical use, assessment tools can be a solution to lifting some of the obstacles restraining the enhancement of sustainability policies aimed at public works [21,22].

Even when a sustainability tool is available, often, it can be complex to use, as they consider elements of culture and value, as well as multidisciplinary aspects that cover environmental, social, and economic values [23,24,25]. Some of the reported barriers of the implementation of traditional sustainability assessment tools include technical difficulties [25,26], the need for consultants and specialists to use the existing tools [26], the bureaucratic nature of the bidding process [22] and application costs (conference and certification charges) [27].

Table 1. Sustainable procurement: Barriers and benefits.

Barriers [20]	Benefits [28]
Lack of funding, restrictions on expenditure and reluctance to incur a higher capital cost when needed	Reducing the use of natural resources
Lack of awareness, understanding, information, commitment, and demand	Meeting existing and forthcoming legislation around the climate change agenda
Insufficient/inconsistent policies, regulations, incentives and commitment by leadership	Reducing harmful emissions and waste generation
Insufficient/confusing guidance, tools, demonstrations, and best practice	Improving working conditions and labor standards, health, and safety
The vagueness of definitions and diversity of interpretations	Assisting disadvantaged groups in society
The separation between capital budget and operational budget	Upskilling your workforce to meet the future needs of your organization
Lack of sufficient time to address sustainability issues	Saving the long term by considering the whole lifecycle cost
Lack of long-term perspective	Meeting international obligations (e.g., the Kyoto Protocol)
The general perception that is addressing sustainability always leads to incurring a higher capital cost	Improving the efficiency and transparency of procurement procedures
Resistance to change	Stimulate the market for green technologies
Insufficient integration and link-up in the industry
Insufficient research and development

presents the main barriers and benefits of the incorporation of sustainable contracts.

The need for such assessment tools on most public works’ projects have been firmly established as they offer guidance to the decision-making process when it comes to the development of sustainable policies [29]. Any sustainability assessment tool should ideally act as a reference standard to monitor and measure performance [30,31], besides serving as a source promoting knowledge and teaching of sustainability. Existing tools, however, are currently restrictive in terms of application due to their complexities and high time consumption [26].

The present study contributes to filling this gap by proposing a framework for sustainability assessment of public works, which incorporates and synthesizes logically related concepts, guidelines, and criteria proposed by researchers and institutions working within the field. The objective was to develop a tool that would offer a broader set of criteria by incorporating and making compatible the criteria considered important by researchers and those contained in tools: BREEAM, LEED, Green Star, CASBEE, and SBTool. In practice, this means that the evaluator will have a tool that includes a higher number of indicators for each sustainability concept to be evaluated, among which he will be able to select the most suitable ones to fit the characteristics of the enterprise to be evaluated. For public works, this higher coverage is essential, given its variety of typologies. A framework can be understood as a set of coherently associated stages to perform tasks that promote the achievement of a given objective [32]. To make sure that the tool developed in this work is widely accessible to practitioners and researchers in the field, an Excel spreadsheet is developed that operationalizes and facilitates the use of the framework. The developed framework will, therefore, be of great benefit to administrators and professionals in the area of public works. The emphasis in the developed framework has been on ensuring that it is a simple instrument for everyday use, which enables public administrators, even without being specialists in sustainability, to evaluate the sustainability of a public project from its conception. This way, the proposed method can act in a preventative way and promote the necessary alterations to carry out projects with higher levels of sustainability.

The proposed framework meets the following premises: (a) it covers any type of work from the simplest to the most complex; (b) it reduces the range and quantity of assessment criteria to only those with correspondence, having similar specificities and characteristics to work being assessed; (c) it enables the assessor to value or devalue a given criterion in relation to the others.

Considering that public works include all types of activities such as construction, renovation, manufacture, recovery or expansion [13], and all sorts of project types, including edification, transport infrastructure, public space or public service (water, sewage, and energy), the wide depth of coverage of the proposed framework is essential to assist public administrators in acting sustainably at the various stages of the construction lifecycle. As a result, the integration of the proposed framework with viability studies, planning, project, construction, operation/use, and demolition, is possible. In Section 2.7.1, we present the summaries of the phases of the construction lifecycle, what each phase contemplates, and the interconnections of the criteria used in the framework with these phases.

To validate the proposed sustainability assessment tool, it was evaluated by professionals that work on public projects, in order to certify its applicability and practicality. In the next section, an outline of the proposed framework is given. Later, some results will be demonstrated based on the evaluations of specialized personnel that have evaluated the tool.

2. Method

Figure 1 outlines the method that has been developed in this study as a framework to guide the sustainability assessment of public work projects. The first step involves conducting a comprehensive literature review to reveal criteria and tools for sustainability assessment that have been previously proposed. The second step involves the selection of a set of suitable criteria from the ones identified in the literature and the creation of a category structure. The third step builds the Consolidated Structure of Criteria (CSC) by incorporating the selected criteria into each of the categories created in the previous step. In the fourth stage, the criteria contained in the CSC are linked to the different characteristics of public works, resulting in the sustainability assessment spreadsheet (SAS). The fifth step is the development of a computerized instrument that facilitates the use of the framework. Finally, in the sixth step, the framework is evaluated by professionals who work with public works to test out the need for further improvement. The details of each of the steps outlined in the framework will be covered in each of the sections below.

2.1. Literature Review

Bibliographic research was conducted to identify sustainability assessment criteria and tools in public works. A wide and detailed bibliographic search was performed on databases such as Web of Science, Scopus, and SciELO.

The search was initially conducted through the following keywords: “sustainable construction”, “green construction”, “sustainable procurement” and “green procurement” and these were also associated with words like “public”, “government”, “criteria” “indicator”, “assessment” and “legislation”.

Bibliographic research initially explored publications from the last 10 years presenting criteria for sustainability assessment in public works. Subsequently, due to the scarcity of results, the research increased the period to 20 years, incorporating environmental studies not related to the public sphere and consulting references from the publications selected initially. Other studies made available by different sources were also chosen, providing a list of 2146 studies.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) recommendations were adopted to guide the bibliographic search. The aim of using PRISMA is to improve the reporting of systematic reviews and meta-analyses. A systematic review relies on selecting relevant research and analyzing data from what is included. When necessary, meta-analysis is used to incorporate statistical techniques into the analysis process. The four phases of the PRISMA flowchart were followed to summarize the results of the literature search by highlighting: (i) the number of articles identified; (ii) articles included and (iii) excluded articles, and (iv) the reason for the removal of the articles [33].

After selecting the publications based on keywords, the next step was to eliminate the 724 duplicated studies. The titles and abstracts of the remaining 1422 papers were then read, and only those presenting clear evidence of providing a relevant contribution to the theme were selected. Studies that were excluded include: those without a named author; those published in periodicals without the peer review system; those whose complete texts were not available; studies which had been developed in a language other than English, Spanish or Portuguese, and those whose title or abstract were not related to the field of concern.

In the remaining 326 articles, we did a quick reading in all sections to capture the general idea of the text and, thus, to be able to decide on its originality, validity, and clarity of the methodology, which resulted in the exclusion of 198 articles.

The remaining 128 publications were read in detail, 59 of which were effectively used. Figure 2 summarizes the bibliographic research using PRISMA.

2.2. Identification of the Criteria for the Assessment of Sustainability in Works

Among the 59 studies incorporated, 33 publications [1,8,21,22,27,28,29,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59] possessed a set of sustainability assessment criteria in their content. These were included in the analysis conducted for the framework, regardless of their focus, objective, or whether they were aimed exclusively at public works or not.

The other 19 publications used in this article, although not containing criteria for assessing sustainability in public works, contained theoretical references on different subjects related to the theme; such themes included sustainable development, sustainable construction, bidding processes, and sustainable contracting. The reviewed publications also collectively encompassed relevant economic and social data, and other information that justified the development of the theme and which give consistency to the content of the work.

Besides, among the 59 works listed, only 14 refer to public issues and these address topics such as: (i) Sustainable procurement [14,21,49,50]; (ii) Barriers, Opportunities, and Challenges to sustainable procurement [20,37,40,46] and (iii) Sustainable Construction [17,22,44,55,60,61].

2.3. Selection of Sustainability Criteria for the Assessment of Public Works

A total of 33 documents were selected from the bibliographic review, from which 873 preliminary criteria were extracted, with no type of filter, as per

Table 2. List of criteria.

Description of Preliminary Criteria	Documents
	1	2	...	32	33
Criteria 1 from document 1 (1.1)	X
Criteria 2 from document 1 (1.2)	X
...	...
Criteria N from document 1 (1.N)	X
Criteria 1 from document 2 (2.1)		X
Criteria 2 from document 2 (2.2)		X
...		...
Criteria N from document 2 (2.N)		X
Criteria 1 from document 33 (33.1)					X
Criteria 2 from document 33 (33.2)					X
...					...
Criteria N from document 33 (33.N)					X

. These indicators were sequentially listed, each associated only with a description and reference, without any kind of treatment.

The table created from the indicators, and which was associated with the structure of categories described in item 2.5 of Figure 1, served as a base for categorization, reordering, and unification of similar criteria that have the same objective in the assessment of sustainability.

2.4. Identification of Sustainability Assessment Tools in Construction Work

According to Wang et al. [61], the most widely used tools for assessing the sustainability of works are LEED, BREEAM, CASBEE, and Green Star. Analyzing the 59 studies incorporated, 34 publications were found to refer to the evaluation tools, according to

Table 3. List of the relevant assessment tools.

Relevant Assessment Tool	Reference Studies
LEED	[1,3,4,7,8,11,17,18,19,22,25,26,27,28,30,31,34,38,39,41,42,43,44,46,47,54,56,57,58,59,60,61]
BREEAM	[3,7,8,11,18,19,20,21,26,27,30,31,38,39,41,42,43,44,47,58,60,61]
CASBEE	[3,7,19,27,39,42,44,56,58,60,61]
GREEN STAR	[3,18,19,26,30,39,47,56,58,61]
SBTOOL	[3,7,19,22,38,39,43,47,60]
HKBEAM/BEAM	[18,19,26,38,39,58]
BEPAC	[19,30,39,42,44]
HQE	[18,22,39,44,61]
AQUA	[3,43,44]
GREEN GLOBES	[27,58,60]
GBC	[22,42,44]
GBI	[56,58]
VERDE	[18,22]
BEES	[27,57]
ECO-QUANTUM	[19,27]
ECOEFFECT	[18,27]
NABERS	[19,44]
GREEN MARK	[56,58]
ITACA	[22,39]
SBAT	[19,26]
GRIHA, SPeAR, NATHERS, DQI, EMGB, EPGB, GHEM, ABGR, ACCURATE, BASIX, CEPAS, CPA, LBC, ATHENA, ENVEST, C-2000, ECO-PROFILE, PRESCO, PROMISE, SBAT, GBL, DGLB, GSHIP, KGBCC, ASGB, 3 STAR, GSEED, GBAS	[3,18,19,26,27,30,39,56,58,61]

, and the most cited tools were also LEED, BREEAM, CASBEE, and Green Star. For the reasons mentioned above, they were selected to serve as a basis for forming the category structure of the proposed framework.

The SBTool tool, although not belonging to the group above, has unique characteristics that motivated its incorporation. It has international relevance, has a generic structure that covers various types of works, is flexible, can be adjusted to meet local conditions, and can be used freely by any organization.

A summary containing the most relevant features of the chosen tools is presented below:

• BREEAM (Building Research Establishment’s Environmental Assessment Method) was the second most cited tool among the studies evaluated: This was the first sustainable assessment method and was developed by the Building Research Establishment in 1990 in the UK. It is one of the leading global sustainability assessment methods for master planning projects, infrastructure, and buildings. BREEAM categories include management, health and wellbeing, energy, transport, water, materials, waste, land use and ecology, pollution, and innovation. The maximum value of BREEAM is 150 divided into six rating levels with a percentage score: Outstanding (85%), Excellent (70%), Very Good (55%), Good (45%), Pass (30%), and Unclassified (<30%) [19,28,62].
• LEED (Leadership in Energy and Environmental Design) was the most cited tool among the studies evaluated: This is a rating system developed by U.S. Green Building Council in 2000 for assessing and evaluating sustainable performance in five areas: (1) Building design and construction, (2) Interior design and construction, (3) Building operations and maintenance, (4) Neighborhood development and (5) Homes. LEED categories include location and transportation, sustainable site, water efficiency, energy and atmosphere, materials and resources, indoor environmental quality, innovation, and regional priorities. The LEED uses prerequisites before calculating the credits, and the maximum score is 110. The buildings can be classified into four levels: Certified (40e49 points), Silver (50e59 points), Gold (60e79 points), and Platinum (80e110 points) [19,58,63].
• Green Star was the third most cited tool together with CASBEE among the evaluated studies: This is an internationally recognized sustainable rating tool launched in 2003 by the Green Building Council of Australia. It is one of the most used systems and included: design, as-built, interiors, communities, and performance operation. Green Star categories include management, indoor environment quality, energy, transport, water, material, land use and ecology, emissions, and innovation. Performance rating tool can achieve the following scores: Minimum practice (1 star), Average practice (2 stars), Good practice (3 stars), Best practice (4 stars), Australian excellence (5 stars) and World leadership (6 stars) [19,61,64].
• CASBEE was the third most cited tool together with Green Star among the evaluated studies: The Comprehensive Assessment System for Building Environmental Efficiency (CASBEE) was developed in 2001 in Japan, where it is most used. It is a co-operative project between industry and government. CASBEE categories include the indoor environment, quality of service, outdoor environment on-site, energy, resources and materials, and off-site environment. The assessment result is calculated from the results of Q (Built Environment Quality) and L (Built Environment Load) and can achieve the following scores: C—Bad (1 star), B—Fairly bad (2 stars), B+—Good (3 stars), A—Very good (4 stars) and S—Excellent (5 stars) [19,61,65].
• SBTool (Sustainable Building Tool) was just the fifth most cited tool, and yet it was chosen for its different approach by providing an open framework in which regional users can insert new values to suit local building types. It was developed in Canada through the collaborative work of representatives from 20 countries under the direction of an International Framework Committee to assess the sustainability of buildings projects [66]. This is a generic assessment framework to rate sustainable lifecycle performance of sites and buildings projects. SBTool categories include: Site Location, Available Services, and Site Characteristics, Site Regeneration and Development, Urban Design and Infrastructure, Energy and Resource Consumption, Environmental Loadings, Indoor Environmental Quality, Service Quality, Social, Cultural and Perceptual Aspects, Cost and Economic Aspects. The total of all active criteria weights total 100% [19,35,66].

Tools identified above, and their structure of categories and scoring system will be considered for the development of the framework proposed in this paper.

2.5. Creation of a Structure of Categories for the Assessment Criteria

In this stage, the structure of assessment criteria categories from the BREEAM [62], LEED [63], Green Star [64], CASBEE [65] and SBTool [66] assessment systems were listed as displayed in

Table 4. Categories of the most relevant assessment tools.

LEED	BREEAM	Green Star	CASBEE	SBTool
L1—Location and transport	B1—Management	G1—Governance	C1—Indoor Environment	S1—Location, Services, and Characteristics of the land
L2—Sustainable space	B2—Health and wellbeing	G2—Habitability	C2—Quality of Service	S2—Regeneration and development of the land, Urban Design, and Infrastructure
L3—Rational use of water	B3—Energy	G3—Economic development	C3—Outdoor Environmental on- site	S3—Consumption of energy and resources
L4—Energy and atmosphere	B4—Transport	G4—Environment	C4—Energy	S4—Environmental loads
L5—Material and resources	B5—Water	G5—Management	C5—Resources and Materials	S5—Indoor Environmental Quality
L6—Indoor environmental quality	B6—Material	G6—Indoor environmental quality	C6—Off-site Environment	S6—Quality of service
L7—Regional priorities	B7—Waste	G7—Energy		S7—Social, cultural and perceptive aspects
L8—Innovation	B8—Land use and ecology	G8—Transport		S8—Cost and Economic Aspects
	B9—Pollution	G9—Water
	B10—Innovation	G10—Material
		G11—Land use and ecology
		G12—Emissions
		G13—Innovation

below.

Subsequently, from the concepts contained in each of the tools, a new structure was proposed incorporating concepts that were amalgamated from analyzing the benefits of all the individual systems identified.

Table 5. Structure of proposed categories.

Proposed Categories	Influence on the New Categories from the International Tools 1
P1—SOCIAL, LOCATION AND TRANSPORT ASPECTS	L1, L2, L7 | B4 | G2, G4, G8 | C3, C6 | S1, S2, S7
P2—MANAGEMENT	B1 | G1, G5 | C2 | S3, S6
P3—COST AND ECONOMIC ASPECTS	G3 | S8
P4—SOIL USE AND ECOLOGY	L2 | B8, B9 | G4, G11, G12 | C3, C6 |S1, S2, S4
P5—INDOOR ENVIRONMENTAL QUALITY	L6 | B2 | G6 | C1, C2 | S5, S6, S7
P6—ENERGY	L4 | B3 | G7 | C4 | S3, S4
P7—MATERIALS AND RESOURCES	L5 | B6, B7 | G10 | C3, C5 | S3, S4
P8—WATER	L3 | B5 | G9 | C5| S3, S4
P9—INNOVATION	L8 | B10 | G13

¹ For symbols, please refer to Table 4.

presents the new proposed structure and the full and partial influence on the new categories from the assessment tool categories.

The following descriptions detail what each of the proposed new categories assess. It establishes the parameters that enable the comparison between the 874 criteria extracted from the 33 publications reviewed.

• P1—Social, location, and transport aspects: Assesses macro aspects for carrying out a project, its urban context, and the points that vary according to region. It considers the environmental, social, and economic differences of each location and promotes their development.
• P2—Management: Assesses administration techniques (set of tools conceived to support companies in decision-making) aimed at sustainability of different phases of the work (viability, project, construction, use and maintenance, and dismantling), including legal aspects, transparency, ethics, the involvement of interested parties, training (community and industry), resilience, quality policy, human resources, commissioning and planning practices and construction, operation, and maintenance control.
• P3—Costs and economic aspects: Assesses issues related to costs and economic aspects, including prosperity and productivity of the region, financial accessibility to habitation, job opportunities, cost of capital, operational cost, lifecycle cost, the economic impact on the urban area and economic viability of commercial activities.
• P4—Soil use and ecology: This is limited to the area occupied by the work and assesses strategies for sustainable deployment of the project, including accessibility; reuse of constructions; reduction of environmental impact on the community and neighborhood; decrease in air, light and water blocks; heat island impediment; minimization of pollution; recovery of biodiversity; flexibility and endeavor in the use of soil.
• P5—Indoor environmental quality: Assesses issues related to the internal environment of the construction, including accessibility; lighting, thermal and acoustic comfort; wellbeing, health, and safety; air quality, low emission of VOC (volatile organic compounds); external view; natural, hybrid and mechanical light and ventilation (cold and hot); monitoring of external air quality; functionality and efficiency; flexibility and adaptability of the construction; maintenance of environmental quality.
• P6—Energy: Assesses the design, sustainable use and efficiency of energy, including the incorporation of efficient systems and equipment; generation of renewable energy and energy from alternative sources; reduction of CO2 emissions, greenhouse gases and cooling gases (non-use of CFCs); energy management of the construction and use of the work; reduction in demand; analysis of incorporated energy and operational energy.
• P7—Materials and resources: Assesses the measures for reducing the impact of construction materials on the environment, including the responsible acquisition of materials with low incorporated ecological impact (extraction, processing and manufacture and recycling); waste reduction; use of certified timber; waste administration (correct destination, selective collection); reuse of discarded materials; lifecycle analysis of the material; minimization of the use of virgin material.
• P8—Water: Assesses the sustainable use of water in the operation of the building and its land, including a reduction in the consumption of potable water (indoor and outdoor); minimization of losses through leakage; utilization and treatment of rainwater and greywater (reuse for watering, cleaning, cooling facilities and toilets); adoption of wastewater treatment stations; incorporation of efficient water-saving devices and rainwater administration.
• P9—Innovation: Assesses the benefits to sustainability that go beyond that required by the assessment tools (BREEAM, LEED, etc.), including the deployment of innovative practices, processes, and strategies that promote sustainability in the built environment; incorporation of innovations in the projects and innovative professional participation.

2.6. Structuring of the Criteria

To obtain the structure of criteria necessary for the development of the framework, the steps detailed in the flowchart in Figure 3 were followed.

As shown in Figure 3, the first step links the criteria from Table 2 to the proposed categories. Second, the table is ordered by category. A check is then conducted to ensure that the criteria have been linked correctly; this procedure is repeated until there are no inconsistencies. Finally, identical or similar criteria are unified with a new text, consolidating the criteria framework (CSC).

2.6.1. Framing of the Criteria in Categories

Due to a large number of criteria, and the fact that the descriptions for a given criterion vary across studies, it was necessary to unify those with the same representations. Detailed analysis of the articles was also needed to identify what the authors intended to represent in the description of each criterion. This then permits the correct framing of the criteria within categories.

Framing also resulted in a high number of criteria concentrated in specific categories, making the unification process difficult. The creation of a secondary category was deemed necessary to improve the analysis.

As such, two columns were added to Table 2; one referring to the MAIN CATEGORY, already mentioned in the past studies, and the other relating to SECONDARY CATEGORY, which indicates subcategorization, as shown in

Table 6. List of criteria and categories.

CATEGORY		Description of the Criteria	Documents
MAIN	SECONDARY		1	2	...	32	33
PX—Category X	Sub-category XY	Criteria A	X
PY—Category Z	Sub-category ZY	Criteria J
...		...	...
PX—Category X	Sub-category XY	Criteria B		X
...		...		...
PX—Category X	Sub-category XY	Criteria C				X
...		...				...
PX—Category X	Sub-category XY	Criteria D					X
...		...					...

.

2.6.2. Reordering of the Criteria in the Categories

After distribution and ordering of the 873 preliminary criteria into the proposed principal and secondary categories, it was possible to distinguish and group identical or similar criteria and propose new wording for criteria that essentially carried the same meaning.

To carry out the next stage, a new column was inserted in which texts containing the new descriptions of the criteria were elaborated and positioned, resulting in

Table 7. List of criteria and categories with new descriptions of the criteria.

CATEGORY		Description of the Criteria		Documents
MAIN	SECONDARY	New	Original	1	2	...	32	33
P1 Category 1	Sub-category 1.1	New description 11	Criteria A	X
			Criteria J			X
			Criteria L					X
	...	...	...
P2 Category 2	Sub-category 2.1	New description 21	Criteria H			X
			Criteria F		X
			Criteria M				X
	...	...	...
...	...	...	...

. Verification of the framing of all the criteria was repeated until there were no inconsistencies.

2.6.3. Unification of the Same or Similar Criteria

At this stage, the structure of the criteria was consolidated with the new descriptions, and the original criteria eliminated. As a result of unification, a structure was obtained containing 214 criteria distributed across nine categories as per Table S1, which is the basis of the framework for the sustainability assessment of public works.

2.7. Linking of the Criteria to Different Characteristics of the Project

To facilitate the work of the assessor, the criteria of the CSC were linked to certain characteristics that differentiate one project from another, as shown in

Table 8. Characteristics of projects.

Aspects	Characteristic
Phases-Lifecycle	planning, project and bidding, construction, use, and deconstruction;
Types of intervention	construction or renovation; and
Types of construction	building, urbanization, transport infrastructure, and facilities infrastructure.

:

The 214 criteria that are part of the CSC cover all types of public works. However, to evaluate a specific project, several criteria need not be assessed. For example, when evaluating the sustainability level of building renovation in its use phase, some land use criteria are not adequate and would be discarded in such instances.

To make the tool easier to use, several columns are inserted into the CSC, representing the characteristics of Table 8 and later linked to each of the 214 criteria. Then, the spreadsheet is automated in such a way that the assessor, when entering the system information on the characteristics of the work, automatically eliminates the inappropriate criteria.

This preliminary step assists public administrators, even those without extensive technical knowledge on aspects of sustainability, in measuring the extent of the sustainability of a given construction.

The following items detail the different characteristics of construction works and how linking the criteria contained in the CSC occurred.

2.7.1. Linking of the Criteria to Phases in the Lifecycle of the Project

There is no universal consensus on the phases that make up the lifecycle of a public project. However, some stages reoccur in various studies, as listed below.

• Planning: This is the stage where you get the macro information about what you want to do. At this stage, it is verified whether it is economically and technically feasible to proceed with studies to materialize the enterprise. In this phase the following works are developed: conception, feasibility, pre-design, site selection [4,22,30,42,54,57,66];
• Project and Bidding: This is the stage where the necessary information for the execution of the work is developed, including the technical and legal aspects of formalizing the bidding. In this phase, the following works are elaborated: a preliminary study, pilot study, legal project, executive project, construction documentation [4,22,30,34,42,54,57,66];
• Construction: This is the stage where efforts are made to materialize the project, its facilities are tested, and the asset is made available for use. In this phase the following works are elaborated: execution (construction, renovation, retrofit, …), construction review, commissioning, post-delivery [4,22,30,42,54,57,66];
• Operation/Use: This is the stage in which the asset is used; it begins at the delivery of the work and ends at the end of its useful life, when the aging of the work occurs and the wear of the facilities. In this phase, the following works are elaborated: performance, maintenance, aging, operation, utilization, operational management [22,30,42,57,66];
• Deconstruction: It occurs at the end of the useful life of the work when the construction loses its usefulness and must be discarded or replaced. In this phase, the following works are elaborated: demolition, waste disposal, site recovery [30,66].

Figure 4 summarizes the phases of the construction lifecycle and what each phase contemplates.

The CSC was modified, with five columns representing the different phases of the construction lifecycle being inserted. Each of the criteria was separately analyzed and linked to one or more of these stages.

Table 9. Number of criteria distributed by category and phase.

Category	Planning	Project and Bidding	Construction	Use	Deconstruction
P1—SOCIAL, LOCATION AND TRANSPORT ASPECTS	14	16	8	10	4
P2—MANAGEMENT	6	31	40	32	28
P3—COST AND ECONOMIC ASPECTS	9	16	4	4	3
P4—SOIL USE AND ECOLOGY	6	29	17	10	6
P5—INDOOR ENVIRONMENTAL QUALITY	0	17	10	12	0
P6—ENERGY	0	16	14	14	4
P7—MATERIALS AND RESOURCES	0	35	36	32	8
P8—WATER	0	17	16	17	2
P9—INNOVATION	2	3	3	3	2
TOTAL	37	180	148	134	57

shows the number of criteria that were linked to each phase of the work lifecycle, distributed by category.

It can be seen that the Project and Bidding phase has the highest number of associated criteria, followed by the Construction and Use phases. This is the phase in which the Bid Notice is prepared, bidding is processed and, especially, where public agents have the chance to alter guidelines, change projects, specify equipment and materials, and determine new constructive solutions that incorporate more sustainable elements into the project.

2.7.2. Linking of the Criteria to Types of Intervention

Type of intervention is another aspect that differentiates one project from another. According to TCU [13], “Public work considers all construction, renovation, manufacture, recovery or extension of a public asset.”

Upon linking the sustainability criteria to the five types of intervention mentioned, it was found that only those that represent new constructions were differentiated from the others. Therefore, it was possible to group types of intervention into just two groups:

• Renovation—incorporates recovery, extension, and renovation itself. This type of intervention impacts pre-existing constructions.
• Construction—refers to new works and incorporates manufacturing and construction itself. This type of intervention is differentiated from the previous by impacting aspects of its location and soil occupation.

Once again, the CSC was expanded, this time with two new columns representing types of intervention. Each criterion was analyzed again and linked to one, or both, of the interventions.

2.7.3. Linking of the Criteria to Types of Public Construction

As shown in Table 8, the third aspect considered in the present study that differentiates one construction from another is its type:

• Buildings—administrative buildings, terminals, schools, hospital buildings, museums, research institutions, and various other types [67,68];
• Public spaces—roads, squares, sidewalks, allotment, beaches, … [68];
• Transport infrastructure—highways, railways, airports, ports, works of art, … [67,68];
• The infrastructure of hydraulic facilities—water and sewage supply and distribution networks, dams [67,68];
• The infrastructure of electric energy—electric energy generation and distribution work and telecommunications [67,68];
• Industrial infrastructure—power stations, factories, refineries [68].

As per the previous items, the CSC was increased with six new columns representing the types of construction and the linkage to all the criteria.

After the linkages, the Sustainability Assessment Spreadsheet (SAS) was obtained, Table S2, which will serve for assessment activities.

3. Development of the Logical Structure of the Framework for Sustainability Assessment of Public Works and Its Operationalization

The feasibility of the use of the framework is strongly influenced by the demanded procedures and the mechanisms of operationalization. Figure 5 presents the proposed framework.

Procedures for assessment of the degree of sustainability in public works were identified, and a logical structure of procedural relationships was defined. As a result, a framework formed of six stages that group logically related procedures was obtained. To facilitate the use of the framework, it was decided to develop a computerized instrument through an electronic spreadsheet containing logical filters and formulae.

SAS presented in Table S2 acts as the basis for the sustainability assessment. In stages 1–3, evaluators use spreadsheets that automate the framework and inform project characteristics, automatically reducing the breadth of SAS criteria. Only the criteria that have specificities and characteristics similar to those of the work being evaluated remain.

The result obtained is still a generic list, and the assessor, already equipped with knowledge of all the details of the project, begins stage 4 by selecting the criteria that do not fit the characteristics of the construction, consolidating the Assessment Table (AT) with only the relevant criteria.

Despite the framework automatically establishing weights equal to 1 for all the AT criteria, in stage 5, the assessor may value or devalue a given criterion in relation to the others. For example, to increase the weight of a criterion by 20%, simply substitute the value corresponding to 1 for 1.2 and, to decrease by 10%, alter the weight to 0.9. If the assessor does not possess benchmarks or the experience to change said values, the use of those generated by the system is recommended.

All sustainability assessment tools establish a weighting system for their criteria; however, there is no consensus as to which approach or method to use [64]. The weights may be considered by groups of criteria or individually and may vary according to the client, investor, or impacted communities; they are inherent to the systems themselves, and when not specified, they receive equal values [47,64], as suggested in the present study.

In stage 6, the assessor scores the criteria contained in the AT. The score has a subjective component, in which each assessor shall make considerations taking into account their experience and individual interests. As a suggestion,

Table 10. Orientation for scoring.

Score	Orientation for Scoring
0	The criterion is compatible with the venture but was not considered during phases of project/bidding, execution and use.
0 to 4	It was superficially considered in project/bidding and poorly executed.
4 to 7	It was planned, projected and implanted, but without much detail or depth to the theme.
7 to 9	Well planned, projected, and implanted, with detail and depth, but without the use of specialized professionals or companies.
9 to 10	Very well planned, projected and implanted and fully maintained, using specialized professionals and companies with a referential standard of excellence (Benchmark)

presents parameters based on the authors’ experience.

The sustainability index of the work is automatically obtained after all criteria have been scored and their respective weights defined. The values are calculated and converted to a scale of 0 to 100, as follows:

$$Si=\frac{100 \times {{\displaystyle \sum }}_1^n\left(CSn\times CWn\right)}{Smáx\times {{\displaystyle \sum }}_1^{n}CWn}$$

(1)

In Equation (1), Si is the value of the sustainability index of a work obtained by the weighted average of the CS scores (criteria score) attributed to each of the criteria and their respective CW weights (criteria weight). To convert the results to the 0–100 scale, the numerator was multiplied by 100 and the denominator by the maximum Smáx value (maximum score) that can be applied to the criteria.

Table 11. Example of obtaining the sustainability index.

Criterion Objective	Score	Weight
Envision sporting, social and leisure facilities in occupied places	10	1
Anticipate adequate infrastructure for facilities in occupied places	8	2
Promote actions to mitigate social risks—health, safety, and disasters, among others	4	1
Establish an appropriate density of habitations for the occupied places	0	1

shows an example of calculating the Sustainability Index using Equation (1) and the information in AT.

$$Si=\frac{100 \times \left(1\times 10+2\times 8+1\times 4+1\times 0\right)}{10 \times \left(1+2+1+1\right)}=60$$

In comparison, the BREEAM, LEED, Green Star and SBTool sustainability tools frame their final results and analyses into classification levels in a variety of ways:

• BREEAM—Unclassified-(<10); Acceptable * (10 to 24); Pass ** (25 to 39); Good *** (40 to 54); Very Good **** (55 to 69); Excellent ***** (70 to 84) Outstanding ****** (85 to 100);
• LEED—Unclassified (<40); Certified (40 to 49); Silver (50 to 59); Gold (60 to 79); Platinum (80 to 110);
• GREEN STAR—Assessed-(<10); Minimum Practice * (10 to 19); Average Practice ** (20 to 29); Good Practice *** (30 to 44); Australian Best Practice **** (45 to 59); Australian Excellence ***** (60 to 74) World Leadership ****** (75 to 100);
• SBTOOL—E (Pi < 0); D (Conventional practice) (0 to 0.10); C (0.10 < Pi ≥ 0.40); B (0.40 < Pi ≥ 0.70); A (Best practice) (0.70 < Pi ≥ 1.00); A+ (Pi < 1.00).

In the present study, the value of Si (sustainability index) determines, on a scale from 0 to 100, the extent to which a project is aligned with the precepts of sustainability. To facilitate the analysis of public managers, in

Table 12. Levels of sustainability.

Interval	Level	Description/Impact
0–39	Low	Unacceptable level for public works. Projects with negative environmental, social, and economic impact. The public agent must incorporate sustainable elements to improve the classification.
40–64	Medium	Acceptable level. Projects with a neutral impact to sustainability. Possesses a wide scope for the implantation of improvements. The public agent should incorporate elements to improve the classification.
65–84	High	Recommended level for public works. Projects with positive environmental, social, and economic impact. Constructions, at this level, should be the minimum goal for the public agent.
85–100	Very high	Advanced level. Projects with positive impacts on all aspects of sustainability, reaching the standard of excellence. Works of this level must have their solutions disclosed and awarded.

, we suggest the limits between the evaluation ranges considering the distribution practiced by the LEED, BREEAM, Green Star, and SBTool tools. To facilitate the analysis of public administrators and, following the model used by other assessment tools, the obtained results were framed into four levels, as per Table 12.

To facilitate and streamline the execution of the framework stages, a software was used for the elaboration of electronic spreadsheets (Excel) to create an electronic instrument consisting of four parts. The first part contains instructions, recommendations, and the steps to be followed to use the system. It is where the assessor defines the characteristics of the project (Figure 6). The second spreadsheet contains the AT, which is automatically generated after the definition of the characteristics of the project, and is where the assessment itself is given. On this spreadsheet, the assessor scores the criteria and alters their weights if necessary (Figure 7). The third spreadsheet presents the result of the assessment after all the scores have been given, indicating the sustainability index of the project (Figure 8). The fourth spreadsheet shows the SAS (Figure 9), based on the information in the framework.

The assessor defines the characteristics of the project using the first spreadsheet (Figure 6), indicating the phase of the work, the type of intervention, and the type(s) of construction, marking the corresponding cells with an “X”.

These definitions are automatically applied in column A of the SAS (Figure 9) with the use of logical formulae from the functions of Microsoft Excel and result in the selection of criteria with the indicated characteristics.

The second spreadsheet (Figure 7) is automatically assembled from the SAS through the use of research and reference formulae listing the selected criteria and other information for assessment. At this time, the assessor manually selects the criteria that do not fit in with the characteristics of the project, marking the corresponding cells in column F (SCORE), and applying a filter to eliminate the selected lines, consolidating the AT.

In the next step, the assessor scores all the criteria, inserting values from zero to ten in the cells in column F (SCORE) of the second spreadsheet (Figure 7). As the scores are being inserted for the criteria, the system automatically establishes, through logical formulae, standard weight equal to 1. At this time, the assessor may make individual alterations, manually modifying the pre-defined values in the corresponding cells in column G (WEIGHT).

The third spreadsheet (Figure 8) presents the final result of the assessment. The Si is obtained from the application of Equation (1) to the data contained in columns F (SCORE) and G (WEIGHT) of the spreadsheet in Figure 8. Framing occurs automatically through logical formulae of the Excel functions, linking possible Si results to the intervals in Table 4.

It should be highlighted that the assessor can only edit the cells in yellow or orange, all the others are protected against modifications, impeding the user from editing the formulae and losing the linkages.

4. Evaluation of the Framework

For the framework evaluation, we used a real case of a project in the bidding phase, because in addition to having a higher number of associated criteria (Table 9), it is the phase that allows public agents to incorporate more sustainable elements into the work. The project was selected at random, through consultation with the website of the Niterói city hall [69], using case bidding process no. 23/2018, whose objective is to benefit an area located in a low-income neighborhood with the construction of a social square, a playground for children, a football field with artificial grass, a grandstand, an outdoor gym for the elderly, and a building containing changing rooms and an office. The documents that are made available to all companies interested in participating in the bidding process include: (a) the main document with the body of the public notice containing the legal requirements and guidelines for proposers; (b) basic project containing the plans that guide the execution; (c) descriptive memorial detailing how the services are to be performed; (d) contractual draft; (e) budget spreadsheet containing quantities and prices of services; (f) physical-financial schedule; (g) calculation memory showing how the values of the services were obtained; (h) photographic report of the area before the interventions.

To evaluate the framework, professionals with experience in the inspection and execution of public works, including personnel with bidding processes, experience where sustainability assessments need to be carried out, and were invited to participate. Three civil engineers and an architect participated in the evaluation, along with four young professionals with at least one year of experience in this segment of work. The professionals had no contact with each other. The procedure adopted was:

• Step 1—We provide all the documents of the notice and advise them to read the material in detail and understand all aspects of the construction;
• Step 2—We briefly explain the concepts of sustainability of works, the operation of the spreadsheets detailed in item 3 (Figure 6, Figure 7, Figure 8 and Figure 9) of the framework, and where to look for the information to be able to evaluate correctly. We present the content with the concern that none of the participants would suffer any kind of influence and would not harm the evaluation process;
• Step 3—We provide the framework’s electronic spreadsheets to individually assess each of the criteria selected for the work, considering the phase, type of intervention and type of construction, finding the work sustainability index (Figure 5);
• Step 4—We collect the spreadsheets containing the evaluations and provide a data collection instrument (Questionnaire S1,

  Table 13. Structure of the framework evaluation form.

  General aspects	Importance of using sustainability assessment tools, such as that presented, in public works.
  	The propensity of the assessor to make modifications in the conception, project, or execution of a given work, after using the framework.
  	The utility of continual use of the framework, by public administrators, to improve the sustainability of public work.
  	Usage capacity of the system to enhance the expansion and promotion of knowledge on the sustainability of works.
  Specific aspects related to the framework	Ease of use.
  	Ease of understanding and scoring the criteria.
  	The relevance of the criteria to the measurement of sustainability of works.
  	The capacity of the system to indicate the correct level of sustainability of a project.
  	The necessity for previous experience in sustainability assessment.
  	The necessity for specific knowledge, such as that related to the sustainability of works.
  	Potential in the use of the obtained result to reflect the level of sustainability of a project.
  Specific aspects related to the instructions for the use of the framework	Clarity of information.
  	Completeness of information.

  ) to assess whether the framework has achieved the objectives of each item in the questionnaire, according to a five-point Likert scale, varying from “Very low” to “Very high”. We made available in the questionnaire a field for the participants, if necessary, to make additional comments about the research items;
• Step 5—We collect the research results and organize them in tables to be analyzed and presented.

For the definition of the questionnaire items, we interviewed four professionals, of which two also participated in the evaluation of the tool. The other two were selected because we found it important to have a view of those who were not from the area, but who had already participated in the development of assessment tools. Our question was: What are the main items that should be considered to assess the framework’s ability to be used in practice? After eliminating redundancies, we developed a version of the questionnaire that was submitted to all four professionals and which was considered appropriate by all. Figure 10, Figure 11, Figure 12 and Figure 13 synthesizes the results of the specialist’s evaluation.

We considered, as strengths of the system, the items in which the sum of the percentage of the “Very high” and “High” evaluations of the item was equal to 100% and, as weaknesses, when it was equal to or less than the sum of the other evaluations.

Thus, analyzing the graphs in Figure 10, we can see that among the aspects evaluated in the questionnaire in Table 13, which were considered as strengths, there are two items among the General Aspects: “The propensity of the assessor to make modifications in the conception, project or execution of a given work, after using the framework” and “The utility of continual use of the framework, by public administrators”; four are among the Specific aspects related to framework: “Ease of use”, “The relevance of the criteria to the measurement of sustainability of works”, “The capacity of the system to indicate the correct level of sustainability of a project” and “Potential in the use of the obtained result to reflect the level of sustainability of a project” and; all Specific aspects items related to the instructions for the use of the framework: “Clarity of information” and “Completeness of information”.

Two items related to General Aspects deserve attention because they signaled an improvement margin despite not being considered as weaknesses. They are: “Importance of using sustainability assessment tools, such as that presented, in public works” and “Usage capacity of the system to enhance the expansion and promotion of knowledge on the sustainability of works”.

Among the aspects considered as weaknesses, all are related to the Specific aspects related to the framework: “Ease of understanding and scoring the criteria”, “The necessity for previous experience in sustainability assessment” and “The necessity for specific knowledge, such as that related to the sustainability of works”.

It has been found that there is a necessity to improve the system, identifying and adding information to the criteria that make using the framework more difficult. The results also demonstrate alignment to that proposed in the present study, which is the continual use of this tool, which would increase and stimulate the search for knowledge directing public works towards sustainability.

5. Conclusions

The need for enhancing the sustainability of the built environment is evident in the literature [36]. Focus has mostly been on the sustainable assessment of private works, with few studies directed at the assessment of sustainability in public works. To cover this apparent gap, the present study proposed, through an extensive and detailed bibliographic research, a flexible and comprehensive framework for the assessment of public works. A practical tool that functions as a checklist applicable to all types and sizes of public works, at any stage of the lifecycle, was developed. The tool developed incorporates and synthesizes logically related concepts, guidelines, and criteria proposed by various researchers and institutions working with the theme, to be used in all types of work. The sustainability assessment framework achieves two principal objectives, namely the comprehensiveness of the assessment undertaken, and secondly, maintaining the practicality of the proposed approach.

The framework was validated by nine specialists that responded to a questionnaire and validated the system. They considered the framework as a practical tool, indicating its high level of easiness, and that the final score obtained from the assessment represents the true level of sustainability of the project. Historical data containing sustainability levels of public works, collected from the use of the framework, serve as a reference for administrators to direct their resources to achieve more sustainable works.

This research has the typical limitations of research using the literature review to support the results. Even though a comprehensive and detailed bibliographic search has been carried out, there is always the possibility that an important article has not been considered. Moreover, despite demonstrating the final sustainability level of work, the results produced by the proposed framework are as good as the level of detail of information incorporated. For example, if the environmental, social, and economic criteria are being met, what percentage, and which category would be more developed is something that the tool cannot demonstrate at this stage.

To improve the results of this research, we have some suggestions. Although the framework considers some criteria that assess subjective and intangible socioeconomic aspects, the vast majority is formed by measurable criteria for tangible elements/aspects, since in the researched literature, this is the main focus. Thus, studies that made it possible to incorporate new criteria with this characteristic would contribute to increasing the structure’s ability to consider socio-cultural implications. In addition, other structures such as WELL and ASHRAE standards could also be analyzed in order to identify new relevant indicators and, with this, expand the scope of the framework. The organization of co-creative workshops with industry, public sector professionals, academics, and engineers, would also collaborate to standardize the scores and weight of the criteria.

An important aspect being considered is the possibility of selecting only the most suitable criteria for the project to be analyzed and of establishing different weights for the criteria, which makes it difficult to compare the scores between projects. Assessments usually have, to a lesser or greater degree, a subjective character, as they depend on the assessor’s interpretation of what is being assessed. There is no way to eliminate possible deviations in the assessment. However, for the same public agency, the conditions under which the projects are developed do not vary significantly, which means that the weights attributed to the same work should not undergo great variations. We also believe that the continued use of this tool by the public sector will be able to establish evaluation standards that allow the comparison between different works and that, over time, there will be an evolution capable of incorporating more and more sustainable elements and concepts to the works. Regarding the comparison of assessments carried out by public bodies in different cities, they may vary, which may also mean that the specifics of the project and the conditions under which the projects will be developed are being considered.

We hope that this study can contribute to the improvement of sustainability in the construction sector, in particular those related to public works, and to stimulate the development of new tools that operationalize, objectively and comprehensively, the assessment of sustainability.