Evaluation of Results of City Sustainable Transformation Projects in the Fields of Mobility and Energy Efficiency with Real Application in a District in Valladolid (Spain)

Abstract

This paper presents a novel evaluation framework to assess the effectiveness of city transformation projects related to energy efficiency and sustainable mobility actions. The evaluation framework is part of an Urban Regeneration Model designed to accelerate the urban transformation toward the smart city concept, taking into account all aspects of sustainability. This model has been developed and validated in the three EU cities (Valladolid, Spain; Nottingham, United Kingdom; and Tepebaşı, Turkey) where interventions in the energy, mobility, and ICT fields have been deployed. This model relies on an Evaluation Framework to support its main phases considering two levels of evaluation: city level, to assess globally the smartness and sustainability of the city, and project level, to support the decision-making and assess the impacts of specific implementations. This paper is focused on the second level and its application through the evaluation-supporting tool STILE in the assessment of the energy efficiency and sustainable urban mobility actions implemented in the city of Valladolid. The assessment analysis has allowed assessing how the energy efficiency interventions carried out in the Valladolid district have reduced the energy consumption, increased the use of renewable energies, and reduced the CO₂ emissions. In addition, it has allowed evaluating other aspects such as the air quality, thermal comfort, and energy bill, which have been also improved for the residents. Considering the analysis of the sustainable mobility interventions, the evaluation framework supports the assessment of the reduction of emissions and air pollutants and how the actions have converted electro-mobility into a real option for citizens. The evaluation of results after the implementation of this kind of actions is key to ensuring that successful actions can be replicated in other places achieving smarter and more sustainable cities.

1. Introduction

Approximately 40% of energy consumption and 36% of greenhouse gas (GHG) emissions in the context of the EU are related to the building sector [1]. Within the existing buildings, around 30% of the building stock is more than 50 years old [2] and almost 75% is energy inefficient [3]. Therefore, energy efficient retrofitting of existing buildings is a key element toward contributing to the decarbonization of the economy as established by the EU [4]. Today, the annual renovation rate of the EU building stock varies from 0.4% to 1.2% in the Member States. This rate will need to at least double to reach the EU’s energy efficiency and climate objectives [5].

On the other hand, transport represents almost a quarter of Europe’s greenhouse gas emissions and is the main cause of air pollution in cities. To achieve climate neutrality, a 90% reduction in transport emissions is needed by 2050 [5]. The transport sector has not seen the same gradual decline in emissions as other sectors: emissions only started to decrease in 2007 and still remain higher than in 1990. Within this sector, road transport is by far the largest emitter, accounting for more than 70% of all GHG emissions from transport in 2014 [6]. Road transport is estimated to be responsible for up to 30% of particulate emissions (PM) in European cities. Even ignoring the global warming and environmental damage, the emissions are a direct threat to health.

With this in mind, the sustainable development of urban areas is a key challenge for Europe where the retrofitting of its buildings and the improvement of the sustainable mobility takes on special importance [7]. In Spain, more than half of the buildings are built without adequate thermal insulation and generation equipment [8], which means very high energy consumption and mostly from fossil fuels, which exacerbates the problem of external energy dependence. On the other hand, mobility is one of the major challenges that cities have been facing over the years [9]. In cities, high-quality mobility is a necessity for the success of other urban sectors and the creation of jobs and plays a key role in creating an attractive environment for residents and business. One of the challenges of urban sustainability is to satisfy the demand, reducing resources and improving the service through a holistic approach.

In order to deal with the described situations, projects such as REMOURBAN are demonstrating innovative, efficient, and accessible technologies and processes in districts whose energy problems are evident in order to achieve Near Zero Energy Districts that serve as a reference and allow the replicability of this type of actions in other similar neighbourhoods, improving the environment and the quality of life of citizens. These projects are also focused on the improvement of the urban mobility, which is considered as a service that should treat challenges and problems of urban areas in a more sustainable and integrative way, where citizens and their quality of life are key factors in the transformation process of the city. The new concept places particular emphasis on the involvement of citizens as stakeholders and the coordination of policies between sectors, authority levels, and new business models.

In order to help other cities to identify their needs and to establish the most suitable interventions for covering that demand and replicate the success of the project, REMOURBAN (Regeneration Model for accelerating the Smart Urban Transformation) has designed a methodology, the Urban Regeneration Model (URM), which covers all the phases of the transformation process with the purpose of accelerating the urban transformation toward the smart city concept taking into account all aspects of sustainability. The Urban Regeneration Model has been developed and validated in the three lighthouse cities of the project (Valladolid, Spain; Nottingham, UK; and Tepebaşı/Eskisehir, Turkey) where innovative technologies and organizational and economic solutions have been implemented to significantly increase energy efficiency, improve the sustainability of urban transport, and drastically reduce greenhouse gas emissions [10].

The Urban Regeneration Model (Figure 1) defines a holistic methodology composed of several phases and decision-making processes that aim to support the understanding of the city objectives and needs in order to implement a set of strategies for a sustainable and smartness-oriented regeneration of the city.

Three key priority fields have been identified for the Sustainable Urban Regeneration process, where REMOURBAN provides a catalogue of solutions aimed at packaging the technologies with their related financing and societal aspects. These fields are “Sustainable Districts and Built Environment”, “Sustainable Urban Mobility”, and “Integrated Infrastructure and Processes”. These three priority areas have been those selected and integrated into the model as energy efficiency in buildings, sustainable mobility, and their intersection with ICTs represent three of the main assets that need to be addressed when facing sustainable and smart projects in cities.

Management, Evaluation, and Finance are the three key frameworks defined within the overall model, which establish the main enablers for the city transformation, and last but not least, four phases are the guiding thread of the model that, in addition to guiding the user in the process of transforming the city from a methodological point of view, include tools designed to support this process and facilitate the transformation of cities into more sustainable and smarter environments.

The evaluation framework as mentioned before is one of the three frameworks defined in the Urban Regeneration Model and is seen as the main supporting mechanism throughout the various phases of the city transformation, where the proper links among the different levels are essential to ensure that the model is useful for cities to support their transformation process toward a smarter and more sustainable environment.

Thus, the framework considers two levels of evaluation, the city level to evaluate the smartness and sustainability of the city [11] and the project level to support the decision-making and assess the impacts of specific implementation aimed at achieving the city’s high-level goals.

The objective of this paper is to present the evaluation framework and the results of the interventions implemented in Valladolid as part of the REMOURBAN project. The description of the interventions implemented in Valladolid, the evaluation methodology considering the evaluation support tool (STILE), and final results are presented in the following sections.

2. Contextualization of the Project Level Evaluation Approach

An evaluation framework is needed if we want to assess the impacts achieved after the implementation of a set of actions of different action fields in specific areas of the cities, and this is the main objective of the Project Level Evaluation developed within REMOURBAN project.

In the process of achieving buildings with a higher energy efficiency and a more sustainable mobility in cities, establishing a reliable metric is a key point to support cities in the decision-making process through the evaluation of different alternatives and subsequent assessment of the impacts achieved after the implementation of the actions. Otherwise, what is not measured cannot be known and therefore the replicability potential of the actions would be much reduced. The evaluation of the results after the implementation of these kind of actions is key to ensure that successful actions can be replicated in other areas.

During the last decade, multiple initiatives have appeared to assess specific fields of action within a city such as energy efficiency in buildings, environmental impacts, economic impacts, etc., and most of them rely on a list of key performance indicators to evaluate different characteristics in an isolated way. Some of these examples are schemes of indicators developed to evaluate the Sustainability of Urbanistic Activities [12], schemes developed by EU-funded projects that aim at collecting and analyzing existing frameworks and developing integrated approaches [13], recommendations on KPIs for smart and sustainable cities related to the Sustainable Development Goals [14,15], indicators for measuring energy poverty proposed by the European Commission [16], existing ISO standards for city services and quality of life [17], or initiatives aimed at aggregating data from projects under the umbrella of Smart Cities and Communities [18]. All these existing schemes deal with a specific set of indicators that measure different fields with a focus on specific aspects as energy consumption, renewable production, CO₂ emissions reduction, social acceptance, and return on investment, among others.

With regards to project level evaluation frameworks, as reported by [19], there are different approaches that vary on both the indicators selected as well as the assessment methods. Some of these are based on a pure list of indicators [20,21] from different sources (most of those listed in the previous paragraph) while some others [22,23] integrate normalization and aggregation methods in order to deliver composite indexes. Additionally, in [19], a new approach to providing a unique index that represents the project performance and support decision-making is provided, which allows comparing the success against a set of pre-defined goals. Thus, the system is built on an objective vs. post-intervention evaluation approach instead on a baseline vs. post-intervention evaluation approach.

Therefore, there is a still a research gap that requires further investigation when addressing the need for developing city transformation strategies related to providing a systemic and integrated approach that can serve to compare alternatives for implementation actions and assess them against a set of objectives. The research presented within this paper deals with this aspect and has investigated on this field with the objective of designing a comprehensive model aimed at improving the whole process. The outcomes presented within this paper in the form of conclusions aim therefore at answering the following research question: would a systemic and integrated model of aggregated indicators improve the process of the smart and sustainable city transformation through providing a tool to evaluate and assess implementation actions?

The scheme proposed within this paper has been developed based on the analysis of several of the abovementioned existing schemes where the selected indicators offer a consistent framework aimed at integrating all relevant indicators that can measure and evaluate the impacts achieved after the implementation of several actions. At the same time, it offers a method for aggregating them into the composite indices such as the Demo Site Index that can support the decision making process and impact assessment process.

3. Description of the Interventions Implemented in the City of Valladolid (Spain)

The Urban Regeneration Model described in the previous section has been applied in the city of Valladolid in order to establish the most suitable interventions for covering the identified demand, both for achieving a Near Zero Energy District and for transforming its urban mobility into a more sustainable one. The main interventions carried out to achieve the expected objectives are described below. These interventions are separated into the ones improving the energy efficiency of the district and buildings and the ones improving the urban mobility.

3.1. Measures for Achieving a near Zero Energy District

In Valladolid, one of the lighthouse cities of the project, the FASA neighborhood was selected for the implementation of a set of interventions designed in order to become a Near Zero Energy District and contribute to the city transformation to a more sustainable environment. This neighborhood was built during the 60 s for the workers of the Renault factory in Valladolid, and it is composed by 19 blocks, a tower and a building that contains the thermal power station that supplies heating to the 398 homes that make up the neighborhood. These buildings presented severe deficiencies in their thermal insulation that resulted in lack of habitability and comfort, as well as low energy efficiency.

The heating system consisted of a district network supplied by two fossil fuel boilers (natural gas and gasoil), and it was divided into three different circuits that provided the 398 dwellings with space heating, whereas the domestic hot water (DHW) was individually produced in each dwelling with different technologies depending on the energy source in each particular case: natural gas, butane, and electricity.

In REMOURBAN, with the aim of turning the neighborhood into a Near Zero Energy District, a set of actions have been designed and implemented. The actions are described in following sub-sections:

3.1.1. Passive Measures: Façade and Roof Insulation

One of the main objectives considered for the design of the interventions was the reduction of the energy demand. For achieving the aim of decreasing it, it was necessary to improve the thermal isolation of the building envelope, including the roof and façade.

For the thermal insulation of the façades, the least intrusive solution that could solve the thermal bridges completely was taken into account. The external insulation was the final solution chosen, which consists of fixing an insulation board to the external side of the façade and later applying a finish over the board. In the FASA district, the installed insulation consisted of a four-layer scheme, based on a 60 mm expanded polystyrene (EPS) board fixed on the brick wall, a first layer of mortar, followed by a glass fiber mesh, and a second layer of mortar. Finally, a surface finish was applied for aesthetic reasons.

Regarding the insulation of the roofs, among the available options for their insulation, an intermediate insulation was chosen for the blocks and external insulation for the tower. The intermediate insulation offers a combination of best performance, easy installation, and no disturbance on the tenants. A total of 60 mm of sprayed foam (SPF) insulation was laid under the roof and over the last slab of the block. The tower roof insulation was improved by adding an external insulation over the existing asphaltic layer. The scheme was an inverted roof system consisting of 60 mm of extruded polystyrene (XPS) insulation, a geotextile layer, and gravel.

3.1.2. Active Measures

Once the energy demand was reduced thanks to the passive measures implemented, the next step was to retrofit the thermal facilities in order to improve their energy efficiency and integrate renewable sources to the system.

The existing district heating system was renovated. On one hand, one of the gas boilers was replaced by two biomass boilers with a capacity of 500 kW each, keeping the other existing gas boiler (3.7 MW) to cover the demand peaks and to allow more flexibility in the fuel supply. The 20 substations of the district were renovated too, including new equipment and adding the centralization of the DHW, which allowed the owners to dispose of their old individual electric/gas boilers. Last the distribution network was updated, changing the existing steel pipes with pre-insulated pipes to minimize heat losses and with new, more efficient, variable-flow pumps.

With this new system, the dependence of the system on fossil fuels was decreased because the system now depends fundamentally on renewable energy sources, as around 80% of the thermal energy consumption is provided by biomass. Another relevant advantage is the decrease in the CO₂ emissions, due to the CO₂ emissions factor for biomass, which is significantly lower than for fossil fuels.

In addition to the new biomass boilers, with the aim of increasing the energy supply through renewable sources, a photovoltaic installation has been carried out. This new PV system was built on the south façade of the tower, which has a deviation of 12º and no shading obstacles, and the ventilation effect reduces overheating during summer, improving the efficiency of the modules. The PV modules selected have a nominal power of 77.5 Wp. They were installed in two rows in the flat area of the façade, avoiding interferences with the existing windows, finally adding up to a total aggregated capacity of 27.435 kWp. The PV energy feeds the district heating network through eight electrical resistances that heat up a water tank that acts as a buffer.

Other intervention aiming to improve the energy efficiency of the district was the substitution of the incandescent lamps by LED in common areas of all buildings.

Last but not least, an energy management system (EMS) structured in three different levels has been implemented in the district. This EMS acts at three different levels: district (DEMS), building (BEMS), and home (HEMS). The DEMS is responsible for managing the district heating as a whole, controlling the boilers’ room, and the district heating network distribution. The BEMS is in charge of monitoring and controlling both the heating and DHW (domestic hot water) facilities in each of the 20 buildings substations. Finally, the HEMS monitors all 398 dwellings of the district. There are two different kinds of devices at home level: heat cost allocators, installed in each radiator to measure the individual heating consumption of each dwelling, and thermostatic valves to allow the tenants to adjust the temperature inside each room.

3.2. Measures for Improving Urban Mobility

Actions in REMOURBAN are focused on improving the efficiency of transport of people and goods, either by replacing conventional ICE (internal combustion engine) vehicles with more efficient (hybrid or pure electric) alternatives or changing the way vehicles are used (either reducing the travelled distance, travelled time, or required number of trips). These are the actions implemented in Valladolid in terms of sustainable urban mobility.

3.2.1. Replacement of ICE by Electric/Hybrid Vehicles

Several studies have already demonstrated that the increased energy efficiency of EVs (electric vehicles) makes them cleaner than comparable ICE vehicles, even in the worst-case scenario, when charged by coal-generated electricity [24]. One of the actions of the project was to partially fund the purchase of 45 EVs for industrial/commercial use through a monitoring program in which participants installed a monitoring device in the EV that allowed analyzing the performance of the vehicle for the duration of the project.

Additionally, the municipality of Valladolid purchased five hybrid diesel–electric buses, capable of operating in fully electric mode inside the low emissions area of the city.

Vehicles funded by the project travelled a total of over 1 M km during the monitoring period, which amounted to an estimated energy savings of exceeding 800 MWh over a period of two years.

3.2.2. Improvements to the Charge Infrastructure

A common argument for prospective purchasers when weighting the acquisition of an EV is the lack of a reliable and widespread charge infrastructure. However, the demand for a better charge infrastructure is also low due to the small number of electric vehicles, which creates a vicious cycle that perpetuates the situation.

In order to break the cycle, the charging infrastructure of the city was renovated. Apart from the increase in the number of public charging stations, several of the charging stations were upgraded to include more charging ports and also to increase the charging speed.

Additionally, the project encouraged the deployment of private charge stations for private and public use under a monitoring program similar to the EV purchase where the charging station activity is also monitored.

3.2.3. Additional Actions

Several activities have been performed to raise awareness about EVs, to promote the shift to clean and sustainable vehicles. Finally, a pilot for a car sharing system was tested with the municipality to explore additional means to improve the mobility efficiency. A smartphone and web application were developed to manage EV reservation and trip sharing in order to reduce the number of vehicles and trips necessary to support the mobility needs of the municipality.

4. REMOURBAN Evaluation Framework

The Evaluation Framework developed within REMOURBAN establishes the basis of the evaluation mechanisms, which is based on the definition of indices and indicators, as well as related protocols for their calculation, applicable at the different levels to ensure that the model is useful for cities to support their transformation process toward smarter and more sustainable environment. In addition, this evaluation framework implemented in the evaluation support tool STILE (see Section 6 of this paper) could support the evaluation of design alternatives by using simulation capabilities or changing the prioritization criteria and is envisaged as a decision-support tool for the various stakeholders that are part of a Smart City implementation project.

The framework defines two levels of evaluation: Project Level, to provide a clear identification of the project impact regarding interventions, and City Level, to assess both the sustainability and smartness of participating cities. Thus, the Sustainability (Su) and Smartness (Sm) indices defined at the framework aim at providing an assessment of the whole urban smartness and sustainability performance. At project level, the Demo Site Index (Ds) has been defined to support the evaluation of the impacts achieved through the implementation of interventions related to energy, mobility, ICT-integrated infrastructures, or non-technical actions.

This paper is focused on presenting the evaluation methodology and results obtained after the implementation of the energy and urban mobility actions in Valladolid as part of the REMOURBAN project through the calculation of the Demo Site Index (Ds). The Ds methodology and its implementation steps are presented below.

5. Methodology for the Evaluation at Project Level—Demo Site Index (Ds)

As described above, the Demo Site Index (Ds) is defined to evaluate the project level interventions. The aim of the Ds index is to provide a numerical representation of the status of the project actions before and after the implementation of the REMOURBAN interventions. Thus, the progress of the interventions is measured through the calculation of the Ds index at different stages of the project interventions.

The Ds index is calculated through the aggregation of normalized and weighted indicators by measurable objectives to assist the evaluation of the project impacts and assess the progress of each of the lighthouse cities interventions. The general scheme of the Demo site index is shown in the following Figure 2.

As shown in Figure 2, the indicators are the basis for the evaluation process, which are normalized, weighted, and aggregated to calculate the Ds global index. Project level indicators (showed in Table 1) are weighted to estimate partial indices (Measurable Objectives) defined for each of the areas of intervention (Buildings and District, Urban Mobility, ICT Integrated Infrastructures, and Non-Technical Actions). This framework of indicators, sub-indices, and project evaluation index constitute a valuable supporting tool for the evaluation of the impact and expected result of the REMOURBAN project.

The steps followed for the application of the methodology at project level and thus for the calculation of the Demo Site Index of Valladolid after the demonstration phase of the REMOURBAN project are presented below:

5.1. Scope Definition

The scope includes the four areas of intervention (energy, mobility, ICT, and non-technical) of the city of Valladolid in Spain, although in this paper the results after the implementation of the energy and urban mobility actions are shown in Section 7.

5.2. Baseline Period Definition

It is the timeframe chosen to represent the initial status of the project level indicators that is used as reference for comparison in order to measure the impact due to the implementation of the project interventions in the different areas. In REMOURBAN, this baseline period represents the status of the different areas previous the project interventions.

5.3. Reporting Period Definition

It should encompass at least one complete normal operating cycle in order to fully characterize the effectiveness of the actions. Depending on the specific implementation timings for each of the actions in each demo site, a specific reporting period was defined for each one. In REMOURBAN, the reporting periods of the energy and mobility actions implemented covered at least the last year of the project, but in most cases, this period was longer, exceeding 24 months.

5.4. Data Collection and Analysis

The collection of data is one of the most challenging tasks of the process, and at the same time, the quality and amount of data used for calculating the indicators is one of the most critical issues to obtain a reliable index. The quality of available data is a key aspect in the index construction process; also, a considerable amount of information is required to calculate an index based on the aggregation of indicators.

Most of the data required for the calculation of the indicators at project level are gathered directly from direct measurements, statistical information, and in some cases also from simulations. Data are collected and processed in the Local ICT Platform specific to each city and then sent to the REMOURBAN Global ICT Platform. The Global ICT Platform exposes a REST API service, so that the evaluation support tool STILE (described in Section 6), which is in charge of the indices calculation, can retrieve the variables stored in the global platform as inputs to calculate the indicators and indices.

In the case of the Valladolid demo site, the Smarkia platform has been used as the Local ICT Platform, where the whole data structure for Valladolid city has been created. This includes a full breakdown of entities and variables according to the monitoring plan established for this city (demo area).

The time series of values that make up variables’ updated data in the Local ICT Platform are provided from the local metering devices and data source systems, by means of CSV (comma separated values) files. These CSV files are uploaded into Smarkia FTP (file transfer protocol) server, and each of the partners in charge of variables’ monitoring and updating have their own account in the FTP server for this purpose.

Once the file is received in Smarkia FTP server, a periodic job runs a process of import and reading of the files’ data in order to create the updated time series of the corresponding variable with the information of value and date/time specified within the file.

5.5. Calculation of the Index

Once the indicators have been calculated, it was needed to establish a method for their normalization. The selected method is based on the establishment of interval values for each indicator. These intervals are defined with the minimum and maximum values. Therefore, the minimum values are substituted by 1, while the maximum values are replaced by 10, all the intermediate measures being relative values in the interval from 1 to 10. The min–max normalization functions depend on whether the objective value is the highest (as is the case of renewable energy penetration rate) or whether it is the lowest (as is the case of CO₂ emissions). The first group of indicators are defined as “positive indicators” while the second group are defined as “negative indicators”.

The weighting method considered for the calculation of the Ds index and sub-indices is the equal weights. The indicators are first weighted within their measurable objective and then aggregated to obtain intermediate indices for each measurable objective (Buildings and District, Urban Mobility, ICT Integrated Infrastructures and Non-technical Actions). Then, weights need to be established for each measurable objective to be aggregated into the final global index; in this case, equal weights is again considered for the measurable objectives. Regarding the aggregation, the geometric method is considered for the aggregation of the weighted indicators. This method is based on the product of the weighted and normalized indicator scores.

The computer-based evaluation tool STILE (described in following section) has been defined and developed in the REMOURBAN project to implement the framework. Thus, the STILE tool calculates and normalizes the indicators and weights and aggregate them in order to calculate the Ds index in an automatic way based on the methodology and algorithms implemented within the tool.

5.6. Evaluation of the Results

At this point, it is possible to perform the comparison and detailed analysis of the reporting period index results (after the implementation of the actions) and the baseline period index results (the status before the implementation of the actions).

In Section 7, it is possible to see the evaluation results of the Demo Site Index for Valladolid and the specific evaluations for each of the main areas of interventions presented in this paper (Energy and Mobility).

5.7. Project Level Indicators

In the following tables, the set of indicators defined for the calculation of the Ds index are presented:

Table 1. Project level indicators by Measurable objective for the Ds Index calculation.

Measurable Objective	KPI	Unit	Measurable Objective	KPI	Unit
Buildings and Districts	Energy Demand [25]	kW/m2	Urban Mobility	Energy consumption	MW/year
	Energy Consumption	kW/m2		CO2 emissions [17]	tCO2
	Primary Energy Consumption	kW/m2		NOx emissions [17]	Kg NOx
	Useful Energy	kW/m2		Hydrocarbon (HC) emissions [17]	Kg HC
	Renewable Energy Production	%		Particulate Matter (PM) emissions [17]	Kg PM
	CO2 Emissions [13,17]	tCO2/m2		EV Penetration Rate	%
	Thermal Comfort [26]	%		EV Charging Points	Number of EV charging stations
	Indoor Air Quality Comfort [27]	n/a		Total kWh Recharged	Kilowatt hour in a period
	Energy Bill	€		Energy Bill	€
	Investment	Million €		Investment	Million €
ICT	Smart Electricity Meters [26]	Number of meters	Non-Technical	Initiatives of Public Incentives	Number of incentives per period
	Visualizing Real-time Information	%		Awareness Raising Campaigns	Number of awareness initiatives per 100 inhabitants
	Modes of Transport Integrated on Smart Cards	%		Learned solutions for Non-technical Barriers	Number of learned solutions
	Rate of Trips using Smart Cards	%		Channels Used for Citizen Engagement	Number of channels
	Location Tracker Sensors	Number of Location Tracker sensors		Visits to Project Information	Number of visits, views
	Meteorological Sensors	Number of ICT meteorological sensors		Social Media Accounts	Number of REMOURBAN social media accounts
	Air Quality Sensors [28]	Number of ICT air quality sensors		Marketable Products	Number of marketable products
	Indoor Sensors [26,27]	Number of ICT indoor sensors		Innovative/Green Public Procurement [13]	%
	Web Applications and Services	Number of project apps/services		Papers for Innovative Actions	Number of papers
	Visits/Access to Webs/Services	Number of visits of project app/services during project execution		Cities Interested in being Followers	Number of cities
	Registered Users	Number of registered users of project app/services during project duration		Investment	Million €
	App Downloads	Number of apps developed within the project download during project duration
	Investment	Million €

Table 1. Project level indicators by Measurable objective for the Ds Index calculation.

Measurable Objective	KPI	Unit	Measurable Objective	KPI	Unit
Buildings and Districts	Energy Demand [25]	kW/m2	Urban Mobility	Energy consumption	MW/year
	Energy Consumption	kW/m2		CO2 emissions [17]	tCO2
	Primary Energy Consumption	kW/m2		NOx emissions [17]	Kg NOx
	Useful Energy	kW/m2		Hydrocarbon (HC) emissions [17]	Kg HC
	Renewable Energy Production	%		Particulate Matter (PM) emissions [17]	Kg PM
	CO2 Emissions [13,17]	tCO2/m2		EV Penetration Rate	%
	Thermal Comfort [26]	%		EV Charging Points	Number of EV charging stations
	Indoor Air Quality Comfort [27]	n/a		Total kWh Recharged	Kilowatt hour in a period
	Energy Bill	€		Energy Bill	€
	Investment	Million €		Investment	Million €
ICT	Smart Electricity Meters [26]	Number of meters	Non-Technical	Initiatives of Public Incentives	Number of incentives per period
	Visualizing Real-time Information	%		Awareness Raising Campaigns	Number of awareness initiatives per 100 inhabitants
	Modes of Transport Integrated on Smart Cards	%		Learned solutions for Non-technical Barriers	Number of learned solutions
	Rate of Trips using Smart Cards	%		Channels Used for Citizen Engagement	Number of channels
	Location Tracker Sensors	Number of Location Tracker sensors		Visits to Project Information	Number of visits, views
	Meteorological Sensors	Number of ICT meteorological sensors		Social Media Accounts	Number of REMOURBAN social media accounts
	Air Quality Sensors [28]	Number of ICT air quality sensors		Marketable Products	Number of marketable products
	Indoor Sensors [26,27]	Number of ICT indoor sensors		Innovative/Green Public Procurement [13]	%
	Web Applications and Services	Number of project apps/services		Papers for Innovative Actions	Number of papers
	Visits/Access to Webs/Services	Number of visits of project app/services during project execution		Cities Interested in being Followers	Number of cities
	Registered Users	Number of registered users of project app/services during project duration		Investment	Million €
	App Downloads	Number of apps developed within the project download during project duration
	Investment	Million €

6. Evaluation Supporting Tool: STILE

A valuable computer-based tool, named the Smartness and Sustainability Evaluation Tool (STILE,) has been developed as one of the core services that form part of the REMOURBAN ICT solutions. STILE was conceived as the service to support, automate, and help to achieve the objectives set out in the Evaluation Framework. Therefore, in line with the Evaluation Framework, this tool allows for a quantified measurement of the cities’ progress on the way to sustainability and smartness on one hand and the performance of REMOURBAN project in terms of efficiency and effectiveness of its interventions on the other hand. This way, STILE arises as the cornerstone to reinforce the communication between stakeholders and decision-makers in the cities.

STILE enables running evaluations for any of the REMOURBAN lighthouse cities at any moment. When an evaluation is launched, STILE takes the set of monitored variables stored in the Global ICT Platform for that city and the corresponding period of time.

The tool, at a first step, calculates a set of indicators taking those variables as inputs by applying the formulas defined in the Evaluation Framework. Then, the set of formulas and calculations designed in the Evaluation Framework to obtain the Measurable Objectives from the indicators were programmed as part of the tool and, finally, by implementing the corresponding formulas from the Measurable Objectives, the indices are obtained.

The key benefit of using STILE is not only the quantification of the indices, but a powerful presentation of the whole data set behind the final value of the index that goes from the set of variables to an index value, with several intermediate calculation levels in between, all depicted in a graphical way, making it easier for the user to have full information at a glance.

The way for the user to run a new evaluation by means of a simple form and the way results are depicted are aspects that were carefully designed under usability requirements to make the tool really easy for the user to use while providing full information through self-explanatory charts.

From a practical and visual perspective, Figure 3 shows the STILE tool, allowing us to select the city for which we want to run an evaluation:

Using the STILE visualization solution to represent the whole data set from variables to the final index makes it easier to understand information, because all figures are displayed on just one screen, quantified and depicted in a hierarchical way for a deeper insight into grouping levels and dependencies, as can be seen in Figure 4.

Additionally, the user can dig into any level or branch to obtain more information, just by clicking on each of the elements in a fully interactive way, which helps them to better understand the final value of the indices, based on its indicators and measurable objectives.

This way, the main objective of the STILE tool implementing the Ds index is to help in the assessment of the effectiveness of the demo site interventions in cities, supporting decision-making when some new interventions or improvement of the existing ones is being under discussion or evaluation in the city.

STILE has also a simulation capability that allows modifying the weights used for the generation of the index. STILE user can assess the relevance of the indicators when compared by pairs. Therefore, STILE implements a tool that combines the human knowledge to provide a pair comparison of indicators for the different measurable objectives through a matrix that lets the user set a value for each of the pairs of indicators arranged along the x and y axis, and a set of math algorithms that calculate the resulting vector of weights using the values of that comparison as input. The following Figure 5 shows the matrix enabled to calculate the weighting scheme for Ds index in a weighting simulation:

There is a kind of convention to carry out pair comparison by using a value from a set of predefined values:

• 0: not assessed.
• 1: equal importance. Two indicators contribute equally to the objective.
• 3: moderate importance. Experience and judgement slightly favor one indicator over another.
• 5: strong importance. Experience and judgement strongly favor one indicator over another.
• 7: very strong importance. Dominance of one indicator over another demonstrated in practice.
• 9: extreme importance. The evidence favoring one indicator over another is of the highest possible order of affirmation.

According to this method, the pairwise comparison matrix have to be built in order to establish the importance of each indicator at the different hierarchy levels of measurable objectives to achieve the defined goal toward Ds calculation.

When comparison values are provided, the weight of each indicator is calculated through solving the eigenvector. The weighting vector is automatically calculated this way, and it is used in the simulation as weighting scheme to be applied on the indicators.

7. Evaluation of Valladolid Demo Site

The Ds index is defined to assist on the assessment of the impacts of the overall project in each of the demo cities. This section presents the results of the calculation of the Demo Site Index (Ds) of the interventions in Valladolid and specifically the results of the Buildings and District and Urban Mobility sub-indices.

As can be seen in Figure 6, the demo site index of Valladolid has increased from 1.89 to 4.84, showing the great impact of REMOURBAN interventions in the different areas of the city.

In Figure 7, it can be appreciated that all the interventions areas (Buildings and District, Urban Mobility, ICT Integrated Infrastructures and Non-technical Actions) have increased their values after the project actions. Since this paper is focused on energy and mobility interventions, looking at the Buildings and District and Urban Mobility sub-indices, it is possible to conclude that both sets of actions implemented in Valladolid have had a positive impact. The results show how the Buildings and District measurable objective has improved from a baseline value of 3.76 to a value of 7.15 after the interventions, and the Urban Mobility objective has changed from 2.52 to 4.51, demonstrating the great impact achieved in the intervention areas.

7.1. Evaluation of Buildings and District Indicators for Valladolid Demo Site

The main aim of the REMOURBAN project within the area of Buildings and District is to improve the efficiency in the use of energy and to change the current energy sources by decarbonizing the energy supplies and increasing the share of renewable at the same time as improving the user’s comfort and reducing the energy bill.

The Buildings and District sub-index is composed of a group of indicators that allow assessing the impact of the specific actions and interventions of the project, i.e., energy demand, energy consumption, renewable energy production, thermal comfort, etc., comparing the situation before and after the interventions.

Figure 8 and Figure 9 show the comparison of the Valladolid demonstrator indicators of Buildings and District results in the baseline and the reporting periods. They show an overall improvement in the district.

As Figure 8 and Figure 9 show, the most affected indicator is the renewable energy production. In the baseline period, the energy produced for the heating and the domestic hot water came from non-renewable sources. The heating was produced with natural gas in two old boilers and the DHW with individual boilers in each dwelling, fed with natural gas, butane, or electricity. The intervention mainly included the replacement of one of the existing gas boiler with two new biomass boilers (renewable), the centralization of the DHW in more than 50% of the dwellings, and the renovation of the distribution network. Furthermore, a new PV system was installed as a support for the thermal plant. These measures implied that the renewable energy production was around 80% during the reporting period, making the value of the indicator increase up to 7.15.

Both energy demand and energy consumption improved almost in parallel, as they have a direct relationship with each other. The energy demand was reduced due to the improvement in the insulation of the buildings carried out during the retrofitting and the use of the new thermostatic valves installed in the dwellings. A reduction of 30% in “Thermal Energy Demand” was reported (mainly due to the façade insulation carried out) and a 28% reduction in “Energy Consumption” (as the efficiency in the District heating network improved too). As buildings now have lower energy losses, the energy demand was reduced and the indicator improved. Together with this indicator goes the energy consumption. A reduction in the energy demand also means a reduction in the consumption, which, considering also the higher increase in the systems efficiency, implies the improvement in this parameter.

Furthermore, as buildings now have better thermal responses to thermal fluctuations, the parameter “Thermal Comfort” increased by 12%, showing better thermal behavior of insulated dwellings.

The primary energy consumption is also related to the energy consumption, though it involves the typology of fuel too. The use of biomass as the main fuel and the new PV system implied a variation on the fuel share and, thus, a variation on the primary energy factors, being the primary energy factor for biomass and PV lower than the one for natural gas and electricity from the grid, respectively. The combination of a lower (better) primary energy factor together with lower energy consumption implied an important improvement in the primary energy consumption indicator.

The useful energy is a parameter that improved too. As the energy demand was reduced due to the insulation and the use of the thermostatic valves, the useful energy necessary to heat the dwellings was also reduced, meaning that this indicator improved.

The energy bill for the tenants has also been reduced due to several factors. On one hand, the energy demand is lower, which means a lower consumption. On the other hand, the fuel changed from natural gas/diesel to biomass/natural gas. The cost of biomass is lower compared with the other two, and the biomass share is around 80%. The PV contribution to cover part of the heating needs of the district should also be considered, as it is reducing the use of biomass/natural gas and therefore reducing the operational costs. These two factors, energy consumption reduction and RES contribution, justify the improvement on this indicator.

It is important to highlight the great reduction in CO₂ emissions too, where a reduction of 70% was achieved, mainly due to the high increase in renewable energy use, and the improvement achieved with the building’s insulation and higher efficiency systems.

7.2. Evaluation of Urban Mobility Indicators for Valladolid Demo Site

In this subsection, the main results of the evaluation focused on the mobility actions are presented. Figure 10 and Figure 11 show the comparison of the Valladolid demonstrator indicators’ results in the baseline and the reporting periods of Urban Mobility.

The switch from ICE (Internal Combustion Engine) to electric vehicles had a significant effect on the reduction of emissions of all types of pollutants (CO₂, NOx, PM, and HC). Energy consumption in both buses and cars has also been reduced, although there is still room for improvement, particularly in the buses, where the hybrid buses have been operating in non-electric mode for part of the monitoring period.

REMOURBAN has demonstrated the viability of electric transportation in the low emissions area of the city, particularly enabling the buses of Line 7 to operate in full electric mode for the part of the trip that takes place in the city center, while still maintaining the ICE power train for additional reliability of the vehicles, and which has actually allowed deploying those same buses as backup for different services where no charging stations are available, so they were operating all the time in hybrid mode when required.

The energy efficiency actions in the mobility aspect of the project have raised the public awareness of electric vehicles and increased the perception of the citizens of EV as an alternative to traditional ICE vehicles for urban mobility. The project also established a monitoring methodology allowing the evaluation of the impact of the mobility actions. Key findings of the project include:

• Energy consumption of the cars participating in the project has been cut in half.
• Tailpipe emissions of the cars participating in the project have been removed completely (although there are still some PM emissions of the brakes and wheels, it still is lower than ICEs due to regenerative braking).
• Well-to-wheel emissions of the EV participating in the project have also been significantly reduced; they have been shifted from the city center to the generation site and will further improve as the grid generation capacity turns to a cleaner energy mix.
• Although the number of public charging stations has not increased significantly, several charging stations have been upgraded to fast charging points.
• The number of kWh recharged has not improved significantly, mostly due to the shift from free charging to a pay-per-use model.
• Additional private charging stations have been deployed to complement the public network.
• Despite more than doubling for the monitoring period, the penetration rate of EVs in Valladolid is still very low. The project has contributed to increase visibility and public awareness, but cars are goods with a long lifetime, so a single year is too short of a period to see a significant change in the vehicle distribution of the city.

8. Discussion

As presented within this paper, the evaluation of the results is key to assessing the achievement of the expected impacts but also brings an essential mechanism to foster replication of the solutions developed, which, indeed, is one of the strategic elements of the project. To ensure the replicability of the actions, it is necessary to create a consolidated and consistent reference of impacts.

The framework presented aims at covering an existing gap within the current state-of-the-art where, as shown within Section 2, several approaches exist but not building on a baseline vs. post-intervention comparison approach through the aggregation of indicators toward building a unique index that allows supporting both decision-making and assessment of the effectiveness of the actions implemented.

This framework has been implemented in three EU cities being the case of the city of Valladolid the one presented in this paper, where the results of the application of the evaluation framework through the tool developed for this aim are shown.

The interventions carried out in the buildings (both passive and active) have reduced all forms of energy (demand, consumption, primary and useful). The increase in the use of renewable sources has contributed to achieving a very low dependence on fossil fuels through the implementation of solutions such as biomass boilers and PV system. Interventions in buildings have not only reduced CO₂ emissions, but they have also improved indoor air quality and thermal comfort for people living in these buildings. From an economic perspective, the energy bill per household has been reduced considerably thanks to the combination of all the energy measures.

Regarding mobility interventions, the objective of the interventions has been to reduce emissions and transform urban mobility into a better and more sustainable mobility. As has been shown, CO₂ emissions and other pollutants have been drastically reduced, but it should be noted that in these five years, electric mobility has become a real option for citizens, gradually increasing the EV penetration ratio.

In order to make the URM more useful and easily replicable, some supporting tools have been developed, one of them being the STILE tool, which is focused on the application of the evaluation framework both at city and project level. The STILE tool requires a great amount of data—which have been collected thanks to the monitoring of the actions—for calculating the indicators defined in each category of the evaluation framework. Data collection was one of the most challenging tasks, taking into account the need for a great amount of data with enough quality to achieve accurate results.

The calculated values though STILE for the city of Valladolid and the comparison between the baseline and the reporting periods are shown within this paper, and their results have been analyzed focusing mainly on the energy and mobility actions.

From the experience gained in this project through the applicability of the Urban Regeneration Model to the three local contexts covered by the project (Valladolid, Spain; Nottingham, UK; and Tepebaşı, Turkey), it is possible to affirm that if similar actions in terms of energy efficiency and sustainable urban mobility were implemented in other urban areas of other cities, similar positive impacts could be achieved. However, it is important to remark that each specific local context has its specific particularities (weather conditions, social context, legislation, etc.) and therefore different levels of impacts could be achieved where another set of solutions may be the optimal one to be applied.

Thus, the evaluation framework presented within this paper is able to capture these local and boundary conditions, which will affect the decision regarding the best solutions to implement. At the same time, the adaptability to these local conditions is essential in order to ensure an appropriate assessment of the actions implemented. This differs from other approaches, which can be found in the literature based on comparing and benchmarking cities, which may introduce uncertainties to the process.

Additionally, in the same way, the STILE tool, which has been developed to deliver an automatic implementation of the evaluation framework, is easily adaptable to any context and has been demonstrated to be a very useful tool for the assessment of actions implemented as well as to support processes to plan and evaluate future interventions.

The analysis of the different levels of indicators, sub-indices and final index in a user friendly way allows assessing the impacts achieved through the interventions carried out within the scope of the project not only from a global perspective but also in an independent way. Thus, it can be affirmed that this framework brings a new step within the current state-of-the-art in the field of evaluating the city transformation.

Additionally, the authors acknowledge that both the evaluation framework and its implementation through the STILE tool developed in this context could be further investigated and improved through, for example, the integration of other aspects such as materials, water, or waste analyzing the district not only in terms of energy efficiency but also from a circular economy perspective. In addition, considering the current COVID situation, specific indicators could be defined and integrated in the tool in order to analyze in detail how buildings and urban mobility could influence in the reduction of the pandemic risks based on certain actions.

9. Conclusions

This paper presents a project evaluation framework developed as part of an Urban Regeneration Model that has been developed and validated within three European cities toward supporting the city transformation into a smarter and more sustainable environment.

This Urban Regeneration Model builds on several elements where one of them is a city transformation evaluation method since the evaluation is sought as the main supporting mechanism throughout the various phases of the city transformation process. This framework considers two levels of evaluation: the city level, to assess both sustainability and smartness of the city as a whole, from a comprehensive and integrated perspective, and the project level, to provide a clear identification of the impact of implementation of technologies and solutions on the key priority areas (sustainable districts and built environment, sustainable urban mobility and integrated infrastructures and processes) aimed at achieving the city high-level goals.

As the study shows, the project evaluation framework is based on the aggregation of indicators toward evaluating a set of measurable objectives and a composite index that reflects the overall project performance. This framework supports both decision-making during the design process as well as assessment of the impact of actions after the project implementation on a baseline vs. post-intervention comparison approach accompanied by the appropriate protocols in order to adequately measure and aggregate the required data for the calculation of the indicators and indexes.

The adaptability of both the framework and the tool developed to automate the calculation process to the context and boundary conditions is essential in order to ensure the appropriate assessment of impacts and avoid the uncertainties introduced when comparing cities or areas where the conditions differ considerably.

This framework and the overall Urban Regeneration Model have been designed, implemented, and validated in three European cities, which has allowed demonstrating their usefulness in supporting the city transformation process in the fields of energy efficiency and sustainable mobility while constituting a great basis for adding new elements and further expanding its scope with new aspects of city sustainability.