Analyzing the Tagging Quality of the Spanish OpenStreetMap

Abstract

In this paper, a framework for the assessment of the quality of OpenStreetMap is presented, comprising a batch of methods to analyze the quality of entity tagging. The approach uses Taginfo as a reference base and analyses quality measures such as completeness, compliance, consistence, granularity, richness and trust . The framework has been used to analyze the quality of OpenStreetMap in Spain, comparing the main cities of Spain. Also a comparison between Spain and some major European cities has been carried out. Additionally, a Web tool has been also developed in order to facilitate the same kind of analysis in any area of the world.

1. Introduction

OpenStreetMap (OSM) [1] is a Volunteered Geographic Information (VGI) system [2] aimed to create and distribute free geographic data for the world. The assessment of the quality of OSM maps is crucial to consider OSM as a reliable source of geographic data. A number of methods (see [3] for a survey) and tools (https://wiki.openstreetmap.org/wiki/Quality_assurance) for quality assurance have been developed during the last years in order to report bugs and errors on OSM maps as well as to assist, monitor and visualize OSM contributions.

Quality measures such as completeness, consistency, positional/temporal/thematic accuracy, and quality indicators, such as the purpose, usage, and lineage, are used to assess OSM quality. It fits well with standardized criteria for the assessment of quality in geographic data: ISO/TC 211 (Technical Committee (https://www.iso.org/committee/54904.html) which establishes the principles for describing the quality of geographic data (ISO 19157:2013 (https://www.iso.org/standard/32575.html). It defines components for describing data quality, specifies components and content structure of a register for data quality measures, describes general procedures for evaluating the quality of geographic data as well as establishes principles for reporting data quality.

OSM provides a simple and open tagging mechanism of OSM elements with key-value pairs. Contributors can add whatever they want to as key-value pairs, but there are best practices to follow. For instance, some keys are intended to classify OSM entities into classes: building, highway, amenity, shop, etc., while other keys play the role of attributes: name, maxspeed, created_by, etc., Values are used to set attribute values, while values for classes are used to classify class members into categories: residential, hotel, monument, etc., As observed in [4], some of the contributed entities can be assigned to wrong or implausible classes, due to (1) individual interpretation of the submitted data, or (2) misunderstanding about commonly used classes. Ambiguous nature of entities, large number of users with diverse motivations and background can be the causes of such situations. The open and loose mechanism for tagging in OSM leads to many incompletely classified or wrongly attributed entities.

In order to facilitate the understanding of OSM data by contributors/users as well as the correct visualization and processing of OSM data by tools, the use of keys (and their recommended set of values) should be subject of agreement between OSM partners. Information about the so-called folksonomy [5], i.e., the tags used to thematically describe OSM elements, has been collected and aggregated by several websites such as Taginfo (http://taginfo.openstreetmap.org), which uses a full OSM database to report its statistics, and Tagfinder (http://tagfinder.herokuapp.com).

Focused on the conceptual classification of OSM entities, the work of [6] establishes several parameters for assessing the quality of OSM: Accuracy: Distance between conceptualization and domain knowledge. It can be seen as the degree of correctness in the classification of features into classes; Granularity: Level of thematic description present in the data, moving from very abstract to very specific concepts; Completeness: Coverage in the conceptualization of the features of interest. A distinction exists between class completeness and attribute completeness; Consistency: Degree of homogeneity in the descriptions of geographic features; Compliance: Degree of adherence of an attribute, a feature, or a set of features to a given source, ranging from non compliance to full compliance; and Richness: Amount and variety of dimensions that are included in the description of the real-world entity.

In this paper a framework for the assessment of the quality of OpenStreetMap is presented, comprising a batch of methods in order to analyze the quality of entity tagging. Our approach uses Taginfo as reference base. Taginfo recollects the most used keys by OSM contributors, the most used combinations of keys, as well as the most used values for keys. Taginfo is used, in our approach, to assess the quality of the tagging process of the study area.

Our approach follows the same categorization of quality indicators proposed by [6], adapted to the OSM item tagging process as follows:

• Completeness: Assuming navigational and Point of Interest (POI) queries can be considered the main usage of OSM, the occurrence of certain attributes is crucial for these queries. For navigational purposes, the occurrence of name, direction flows (oneway) and maximum speed (maxspeed) of highways and, house numbers in buildings, are essential for a precise navigational information. For POI retrieval, for instance, the occurrence of name of POIs, and a more or less detailed description of the POI is required: opening hours, phone, etc.
• Compliance. The quality of a certain area is measured in terms of the conformance of tagging process of a certain entity in the study area to most common practices, according to Taginfo.
• Consistency: Our approach analyzes the contributor’s agreement about the tagging process of a certain entity in the study area. Here the standard deviation of the number of attributes used for describing the entity is analyzed.
• Granularity: Our approach assesses the average and median of the number of attributes used for describing an entity. A greater number of attributes means a more detailed information.
• Richness: Here our approach assesses the quality of the classification of a certain entity into categories (i.e., values associated to classes). A greater number of categories means a richer classification.
• Trust. Assuming the “many eyes principle” [7], the quality is also evaluated in terms of the number of versions of entities as well as the local (in the study area) and global (in the planet) experience of contributors.

Our approach aims to provide a number of indicators/scores to measure the quality of any OSM dataset with independence of the zone, country or continent to be analyzed. It permits to compare any pair (or set) of cities. On the other hand, our approach assumes that the main goal of quality assessment is to ensure a suitable query processing focused on POI search and navigational queries. Some expected query results can be discarded due to missing, unusual or heterogeneous information. While contributors are free to label OSM maps following some (local-regional-country) agreements, it complicates query formulation and comparison. For instance, some local communities can agree to assume maxspeed and oneway default values and omit them. The goal of the approach is to evaluate, for instance, completeness, with independence of local common practices, assessing the occurrence of certain tags in certain entities. Thus, completeness analysis can be intended in two ways: the lack of information (missing/unknown) which can be crucial for the usage of OSM datasets and most common practices of OSM contributors in a given area. Be able to distinguish missing/unknown information from a local common practice is not always possible. Moreover, our approach takes Taginfo as basis of agreement points, which has been elaborated from OSM contributions in all the world. Taginfo is used to evaluate compliance of the OSM dataset in terms of the percentage of commonly used keys as well as commonly used combinations of keys. Consistency (i.e., agreement in the number of tags for specific entities) is also analyzed. Again, this indicator is crucial for POI search purposes. Granularity and richness analyze the quality of classification and description of entities that again guarantees filtering of answers when queries are formulated. Finally, trust proposes the analysis of the experience of contributors as well as the number of revisions of OSM items.

The framework has been used to evaluate the quality of the current status (i.e., at the time of the study) of OSM in Spain. We have selected a set of Spanish cities, with different size (and population), as well as with historical, economic and cultural background. Also, we have compared Spanish cities with some major European cities.

We have also developed a Web tool called QXOSM, available at http://xosm.ual.es:8080/qxosm, enabling the quality analysis of the tagging process in any area of the planet. QXOSM has been built on top of XOSM, also a Web system (http://xosm.ual.es/XOSM/), developed by our group in the last years [8,9,10,11]. XOSM allows the querying of OSM with XQuery database query language (https://www.w3.org/XML/Query/). XOSM is equipped with an XQuery library of operators enabling spatial and keyword based queries as well as aggregation queries. The quality analysis of any OSM dataset is carried out by executing a batch of XOSM queries against the dataset, wherein each XOSM query represents a quality indicator. The Web tool permits to select any area of the OSM planet, and to assess the quality of the selected area in real time. The dataset of the selected area is retrieved with the help of the overpass API (https://wiki.openstreetmap.org/wiki/Overpass_API). Once retrieved, the Web tool analyzes completeness, compliance, consistence, granularity, richness and trust of the selected area. Moreover, the Web tool offers a great flexibility allowing the selection of the entities, categories and attributes to be analyzed. Entities, categories and attributes are retrieved with the help of the Taginfo API (https://taginfo.openstreetmap.org/taginfo/apidoc). The results of the analysis are shown in two forms: aggregated (numeric results) and disaggregated (charts). For instance, in the granularity analysis, the average and median of the number of attributes of a certain entity is reported, but also a pie chart is used to show the percentage of entity instances for each set of attributes. The back-end of the tool has been implemented in XQuery, while the front-end has been implemented in Vaadin (https://vaadin.com/). The Web tool catches fetched data in order to improve the answer time of the analysis tasks. Unfortunately, for performance reasons, the Web tool has limitations on the size of the selected area. The analysis presented in the paper for some cities has been made off line.

1.1. Related Work

In the literature, there are several works whose goal is to analyze the quality of OSM for a specific country (or group of countries). As far as we know, there does not exist such study for Spain. Unfortunately, a standard method to analyze the OSM quality does not exist.

From the proposed methods, we can distinguish those carrying out an extrinsic quality analysis by using an authoritative dataset in order to compare it with OSM data, and that acts as ground truth dataset [12,13,14,15,16,17,18,19,20,21]. The main goal of existing extrinsic methods is to validate the accuracy of OSM geometries with regard to real world objects. It is important, for instance, for using OSM maps for navigational purposes, and the study of the accuracy of OSM geometries is mainly focused on OSM ways. There is also a strong interest on the precision of OSM nodes, which has a high impact when OSM maps are used for locating POIs. However, as pointed by several authors (see, for instance, [6,22]), a dataset may contain highly accurate geometries, but if the description of the entities and their attributes is not clear, articulate, rich and complete enough, the value of the data for consumers will be severely curtailed. On the other hand, extrinsic analysis is not always possible, and new approaches explores the possibility of assessing the quality using intrinsic dimensions [23].

A number of methods for intrinsic quality analysis has been proposed in which OSM data are validated assuming some rules of quality. The newest intrinsic quality evaluation methods [24] use the number of contributors [7], heavily edited objects [25] and combinations of factors: number of versions and number of users, as well as confirmations, tag corrections and rollbacks [26] to assess the quality of OSM. In some cases the history of OSM contributor’s updates is used for the analysis. While the analysis of the history updates is interesting, its extraction is currently a difficult and time-consuming process. On the other hand, the concept of “crowdquality” [27] identifies two characteristics: the quality of the user and the quality of the geographic information. In this line, the authors of [26] investigate which indicators influence trust, focusing on intrinsic properties that do not require any comparison with a ground truth dataset. High numbers of contributors, versions and confirmations are considered as positive indicators, while corrections and revisions have a negative influence on trustworthiness. The “many eyes principle” [7] is crucial for trust, where the quality is more likely to be higher if more people have worked on a feature. Trust is closely related to reputation (a subjective perception of trustworthiness), inferred from the historical behavior of a certain contributor. Related to trust, credibility comprises accuracy, authority and competence in addition to trustworthiness.

Taginfo has been previously used as reference base in some works. For instance, in [28] the OSM Map Features (https://wiki.openstreetmap.org/wiki/Map_Features) suggestions and recommendations are analyzed for 40 cities, revealing that it is generally average or poor. They selected the 30 most frequently occurring keys in Taginfo. They state that co-occurrence of keys suggested by the OSM Map Features does not always happen. Tagging process is also analyzed in [22], carried out on 25,000 objects from the OSM databases of Ireland, United Kingdom, Germany and Austria. The selected objects are the so-called heavily edited, having 15 or more versions. They analyze the number of unique tags and values assigned to name, highway and landuse tags.

A more elaborated and extrinsic analysis can be found in [13], in which the authors study the quality of the French OSM dataset with regard to geometric accuracy (positioning and geometry resolution from the ground reality), attribute accuracy (accuracy of quantitative attributes, correctness of non-quantitative attributes and classification of features), completeness (absence of data –omission– and the excess of data –commission–), logical consistency (internal consistency: modeling rules, specifications, integrity constraints, distinguishing intra-theme consistency versus inter-theme consistency), semantic accuracy (correspondence with real world objects), temporal accuracy (actuality of the objects relative to changes in the real world), lineage (capture and evolution of objects) and usage (how well the database fits for the use that will be made). They conclude that the number of tags linearly increases with the number of contributors. Therefore, the more contributors, the better quantitative attribute quality is. Additionally, they conclude that smaller objects are more likely to be missing, and contributors are more focused on capturing attractive objects. They also detect that territories are best represented in rich areas, and/or areas with a young population, and completeness becomes very problematic in rural areas.

Intrinsic analysis using the OSM history of contributor’s updates can be found in [26], in which a subset of the entities in the study area is selected based on the number of versions/editions that the entity has undergone. Trust assessment is based on the provenance of the entity, as well as on the number of users involved in creating an entity. Also indirect confirmations are taken into account by looking at all revisions that have been made in the immediate vicinity of an entity after the last revision of an entity. Also tag corrections and rollbacks (when the value for a certain tag is changed) decrease trustworthiness.

OSM history of contributor’s updates is also used in [29] for intrinsic analysis. They highlight several quality measures via the iOSMAnalyzer tool (https://github.com/zehpunktbarron/iOSMAnalyzer) according to six categories: (1) general information on the study area: development of OSM entities and tags, currentness of data, comparison of newly created and edited objects, syntactic attribute accuracy, positional accuracy of junctions; (2) user information and behavior: number of contributors, contributor activity, distribution of contributors, user profiles; (3) routing and navigation: road network completeness, attribute accuracy of roads, road network currentness, logic consistency and positional accuracy of the road network, roads without a name or route number; (4) map-applications: geometric polygon representation, untouched OSM features, invadid polygons, logical consistency of landuse polygons, development of selected polygons; (5) geocoding: development of address information, completeness of address annotations, completeness of house numbers tagged to buildings; (6) points of interest-search: development of POIs, average number of POI tags, attribute completeness of POIs. iOSMAnalyzer is mainly focused on geometries (categories (3) and (4)), but it includes some elements of completeness analysis (categories (1), (5) and (6)), and contributor analysis (category (2)).

There are some other studies about OSM quality analysis in specific areas of the world. For instance, in [30] an extension of Quantum GIS (QGIS) (https://www.qgis.org/) processing toolbox is presented enabling to assess the completeness of the spatial data using intrinsic indicators, proposing a heuristic approach to test the road navigability of Punjab (India). This is also the case of [31], in which some areas of Brazil are studied in terms of length of rural roads, density of urban roads, number of buildings, percentage of classified roads, number of days since last edition, number of versions/editions, and a comparison is stablished with economic and developmental variables.

There is also a number of developed tools with different goals, ranging from tagging process improvement (recommender and data validation systems), to visualization of OSM data and quality metrics. Among tagging process improvement tools, the tag recommender system called OSMantic (https://sites.google.com/site/osmanticjosm/) [24], developed as a JOSM (https://josm.openstreetmap.de/) plugin, automatically suggests relevant tags to contributors during the editing process. It suggests tags that could be added to better describe map entities, and by detecting tags associated to the same feature that appear too dissimilar. Relations between tags are computed based on the semantic similarity between tags and the number of times a tag has been used in the database. Taginfo and OSM semantic network [32] are used by OSMantic.

Data validation systems are, for instance, Keep Right (https://keepright.at/) which performs several data consistence checks, and shows them in a map, including geometry checks: non-closed areas, dead-ended one-ways, etc., as well as some tag related checks: deprecated tags, missing tags, point of interest without name, etc. It has mechanisms for reporting false positives and for labelling a bug as fixed. Osmose (Open Street Map Oversight Search Engine) (http://osmose.openstreetmap.fr/es/map/) is similar to Keep Right but offers a larger number of data consistence checks, including missing and wrong tags. JOSM Validator integrated with JOSM editor checks data loaded into the editor, highlights errors and warnings, and some automatic fixes are done by request. It checks all objects modified in a session by the user, reporting errors even though the user is not responsible of them. Several checks are done including bad keys and values, untagged ways, missing and duplicated names. OSM inspector (https://tools.geofabrik.de/osmi/), is an error debugging tool which takes part of the GeoFabrik tools. It shows a map with several layers: geometry, routing, tagging, places, highways, areas, coastline, addresses, water, public transport stops and public transport routes. In the case of tagging, empty tag key, empty tag value, tag key with space, unusual character, unusual key length, tagged with “FIXME”, and name/description without feature tags are detected. For highway and addresses several data consistence checks are carried out. Finally, in [4] two methods are proposed based on constraint checking and machine learning, in order to check the integrity of VGI data: hierarchical consistency and classification plausibility. The methods can be applied to check validity of data during contribution or data correction.

Visualization of OSM with the aim to analyze quality of maps has been studied by [33], whose system OSMatrix (http://koenigstuhl.geog.uni-heidelberg.de/osmatrix) is a Web-based approach to visually explore quality metrics of OSM. MVP OSM (https://github.com/napo/mvp-osm) [34] is a tool designed to highlight areas where contributors provide a high level of spatial detail. MVP OSM scores local knowledge (ignoring extraction from aerial imagery), mapping experience (time spent: number of edits and months) and community recognition (frequency in which contributor updates data within a given area). OSM Tag History tool (http://taghistory.raifer.tech) visualizes the usage of a tag in the OSM database by a line chart. OSMstats website (http://osmstats.neis-one.org) examines even other statistical data about the OSM datasets and the users, and visualizes the data by line charts. The geospatial distribution of elements tagged as buildings or roads can be examined by OpenStreetMap Analytics (http://osm-analytics.org).

1.2. Comparison with Related Work

Our approach can be considered an extension of the analysis proposed in [22], since we have carried out the analysis of a large number of entities, and also this analysis has been extended to consistency, granularity, richness and trust. Our study can be considered similar to the proposed in [13], focused here on the Spanish OSM dataset, but without the use of an authoritative dataset. As we will show, it does not prevent to make a detailed analysis of tagging process. With the exception of the accuracy, quality aspects as completeness, compliance, consistence, granularity and richness can be analyzed in our approach. Moreover, some other aspects related to trust as number of versions, local and global experience of contributors, can be analyzed in our approach. Instead of using OSM Map Features suggestions and recommendations like in [28], our approach compares the tagging process of Spanish cities with Taginfo in two cases (1): occurrence of keys, and (2) co-occurrence of keys. In (1) our approach analyzes whether the most used keys (Taginfo top 300 keys), are also used in Spain. The occurrence of rare keys (from 300 and beyond) is considered as non-compliant tagging. In (2) our approach analyzes the use of the top 300 combinations of keys in Taginfo. The occurrence of unusual key combinations is considered as non-compliant tagging. We have decided to carry out an intrinsic analysis of the current status (at the time of study) of Spanish OSM dataset, assuming that it may impose some limitations to the analysis. However, a reference base is used (i.e., Taginfo) to assess the tagging process. In our opinion, even though OSM data are constantly evolving, the analysis of a certain instant of the dataset can offer an adequate picture of data quality. Moreover, our main aim is to compare the data quality of a group of cities, and to establish data quality indicators for current and future analysis. Additionally, it permits to develop a Web tool that, in real time, will be able to analyze pieces of OSM planet. The number of quality indicators we are able to analyze in real time justifies, in our opinion, this decision. Given that our approach is intrinsic, the thematic accuracy (i.e., degree of correctness in the classification of entities with regard to real world objects) cannot be measured. Finally, trustworthiness is analyzed like in [26], but in terms of the number of versions and the local and global experience of the contributor. We assume that the more experience of the contributor, the better the reliability of the entity is. Number of versions also increase reliability.

The structure of the paper is as follows. Section 2 will present the framework of OSM quality analysis, and the main results of the analysis of Spanish cities. Section 3 will compare Spanish cities with some major European cities. Section 4 will present the Web tool. Finally, Section 5 will conclude and present future work.

2. Analyzing the Tagging Quality of the Spanish OpenStreetMap

In this section, the framework for the intrinsic analysis of the tagging quality in OpenStreetMap is described.

2.1. Datasets

Figure 1 summarizes the main features of the analyzed cities: number of OSM elements, nodes, ways and relations. Cities have been selected according to their size (the size is measured in terms of number of OSM elements) ranging from small cities such as Almería, Bilbao, Granada, Oviedo, Santander, Segovia and Vigo to big cities like Madrid and Barcelona, including medium-size cities such as Sevilla, Valencia and Zaragoza. We only measure the number of tagged elements/nodes/ways/relations (i.e., with at least one tag). Some cities have been selected due to their touristic nature. This is mainly the case of Segovia and Granada, but it also happens in big cities, specially Madrid and Barcelona. As Figure 1 shows, usually the number of ways is greater than the number of nodes. However, Bilbao, Santander, and specially Zaragoza have a greater number of nodes. We have analyzed these three cases. In the case of Bilbao, from 11,424 tagged nodes, 6166 nodes have only one tag, and from them 2486 have only house number as tag. 3350 from the 6166 nodes are added by three users. In the case of Santander, 14,928 from 16,980 tagged nodes are added by two users, sourced from open data of the local government. In the case of Zaragoza, the number of nodes is similar to big cities, and the number of tags of them is usually (38,384 from 51,408) greater than three, sourced from open data of Spanish government (38,055 from 38,384). As conclusion, the use of certain practices by OSM contributors can increase the number of nodes with regard to ways; for instance, the use of nodes to annotate house numbers (as in the case of Bilbao) as well as the use of sourced data from Spanish government (as in the case of Zaragoza).

OpenStreetMap has a enthusiastic community in Spain (https://wiki.openstreetmap.org/wiki/ES:Spain, https://lists.openstreetmap.org/pipermail/talk-es/). The Spanish government offers through WMS services information about the Spanish territory, via the Spanish cadastre (http://www.catastro.meh.es), the Geological Survey of Spain (IGME) (http://www.igme.es/), the Spanish Geographic Institute (IGN) (http://www.ign.es/), and the Spanish Infrastructure of Spatial Data (IDEE) (http://www.idee.es/).

In the following, we will analyze completeness, compliance, consistency, granularity, richness and trust of this group of cities. The analysis is made from the current status of the dataset (datasets were captured in June 2017).

2.2. Completeness

Assuming the usage of OSM mainly focuses on navigational and POI queries, the occurrence of certain keys is crucial for this purpose. In particular, for navigational purposes (1) the occurrence of the house number (i.e., addr:housenumber) in buildings allows to locate addresses; (2) the occurrence of the attribute name in streets, represented by the key highway, is vital; (3) for routing (shortest path between two places) the attributes oneway and maxspeed can be crucial to build the graph. With regard to POI search, the keys public transport, amenity, shop, tourism, religion and historic can be considered essential and all of them should be named, but amenities and shops should also include opening hours and phone.

We have selected a number of values for each key (see

Table 1. Selected entities, attributes and categories of the completeness study.

Purpose	Entity	Attributes	Categories
Navigational	building	addr:housenumber	yes, house, residential, apartments
Navigational	highway	name, oneway, maxspeed	primary, secondary, tertiary
POI search	public_transport	name	platform, stop_position, stop_area, station
POI search	amenity	name, opening_hours, phone	restaurant, bar, cafe
POI search	shop	name, opening_hours, phone	convenience, supermarket, clothes, hairdresser, bakery, car_repair, yes
POI search	tourism	name	information, hotel, attraction, viewpoint, picnic_site, guest_house, camp_site, artwork, museum, hostel, motel
POI search	religion	name	christian, muslim
POI search	historic	name	memorial, archaeological_site, ruins, yes, monument, castle, building

), representing the main OSM categories (according to TagInfo classification). Even though more values for the selected keys can be considered, the goal of the analysis is to measure in which degree the main OSM categories are complete for navigational and POI search purposes.

With regard to navigational queries and building completeness, we should take into account that OSM contributors usually use three different mechanisms to provide house numbers into buildings: (a) to directly tag the building with the key addr:housenumber; (b) to tag one of the nodes of the building geometry with the key addr:housenumber; (c) to tag a node, overlapping the building, with the key addr:housenumber. The cases (a) and (b) are easier to detect while case (c) involves a costly spatial query. In order to avoid this spatial query, we have decided to make an estimation of occurrences of house numbers in buildings. Basically, the number of occurrences addr:housenumber and buildings are computed. The results are shown in Figure 2. As we can see, only Santander has a proportional number of house numbers with respect to buildings, while Zaragoza has a higher number of house numbers. In the other cases, the number of house numbers is considerably lower than the number of buildings (especially Granada and Segovia).

We have individually analyzed the case of Zaragoza, which highlights from the given results. We have concluded that open data of the local government has been used to enrich Zaragoza OSM dataset. There are 36,615 nodes in this city with addr:housenumber, and from them 35,432 have as data source the open data of the local government. 268 nodes have the tag building, and from them, 260 have the attribute addr:housenumber, and 245 have the open data of the local government as data source. However, ways with the tag building are 3074, and from them, 640 have the tag addr:housenumber, coming from open data of the local government a total number of 250. On the other hand, 95 ways have the tag addr:housenumber but are not labeled with the tag building. Thus, there are a high number of nodes with house number but not labeled as building, and there do not exist enough ways (labeled with building) using/overlapping such nodes.

With regard to navigational queries and highway completeness, Figure 3 shows that the inclusion of the attribute name has to be improved in Almería, Granada, Oviedo, Segovia, Sevilla, Valencia and Vigo (less than 75%). The same can be said for oneway, which usually has similar rate, being better in some cases like Almería, Segovia, Sevilla and Valencia, and considerably lower in Oviedo, Santander and Vigo. The maxspeed rate is usually lower, specially in Granada, Oviedo, Valencia, Vigo and Zaragoza, with less than 15%. As was commented in the introduction, some local communities can agree to omit oneway and maxspeed, assuming “yes” and 50km/h by default. This can be the reason of low rates in some cities. However, as show in Section 3, this agreement is not always taken in other European communities.

With regard to POI search purposes and amenity completeness, the results are shown in Figure 4. Here we can see that even though the name has been usually added to amenities, opening hours and phone (with the exception of phone in Santander) are usually missing. The shop completeness results are shown in Figure 5. Here, Santander is very well tagged. Opening hours and phone have a low rate in all the cities, while name is present in a high percentage. There is no compelling reason why the rate of name tagging for amenities and shops is lower in some cities. One would expect that touristic cities are better labeled in these elements, but we generally suspect that the main reason for a complete labeling is the presence of a high number of active contributors or few but highly active contributors in a certain area.

In order to get a more detailed view, we have analyzed the occurrence of shops in Barcelona and Madrid. With regard to Madrid we have detected that from 4572 shop items, there are 3706 nodes with name, and the rest (866 items) without name. From these 866 items, most of them represent small shops (127-kiosk, 102-convenience, 32-supermarket, 109-hairdresser, 45-butcher and 56-bakery) and 205 are “vacant”. In the case of Barcelona, we have a similar behavior. Barcelona has 3438 shops items, 2782 items with name, and 656 items without name. Again, these unnamed elements represent small shops (145-kiosk, 78-convenience, 38-supermarket, 93-hairdresser, 30-butcher, 75-bakery) and 85 are “vacant”. Therefore, even when the percentage of unnamed shops is higher in big cities, most of them correspond to small shops.

Figure 6 shows the results of naming for public transport, tourism, religion and historic keys. Public transport has a lower rate of naming in Granada, Madrid, Segovia, Sevilla and Vigo (less than 75%). This is a special case, because according to OSM Map Features, name in “platform” and “stop_position” is recommended if no “public_transport=stop_area” exists, else optional. However, according to Taginfo, the most used values of public_transform are platform (55%), stop_position (32%), stop_area (9%) and station (1%) (see https://taginfo.openstreetmap.org/keys/public_transport#values) and an 82% of public_transport items has an associated name (see https://taginfo.openstreetmap.org/keys/public_transport#combinations). Thus, the observed differences in cities could potentially be attributed to local communities interpreting “optional” Wiki guidelines differently. With regard to tourism, naming should be improved in Almería, Barcelona and Vigo (less than 75%), but they can be still considered well-tagged. In the case of religion with the exception of Segovia (less than 75%), the naming task is right. Finally, Santander and Valencia should improve the naming of historic entities (less than 75%), but they can be still considered well-tagged. As previously, there is no compelling reason why the name tagging in tourism, religion and historic entities is lower in some cities. One would expect that touristic cities can be better labeled in tourism, religion and historic items, but we believe that the main reason for a complete labeling is a high number of active contributors or few but highly active contributors in a certain area.

2.3. Compliance

Compliance is crucial for two reasons. Firstly, OSM partners should agree in how OSM items are tagged. Secondly, OSM tools (mainly those for visualization of maps), require a common language to be able to properly draw maps. The quality of an OSM dataset is now measured in terms of percentage of OSM items with commonly used keys according to Taginfo. Basically, we use the ranking of Taginfo of most used tags for checking the conformance of OSM datasets. The use of less commonly used keys is a symptom of heterogeneous or faulty tagging. It is also a symptom of individual interpretation of the submitted data or misunderstanding about commonly used classes. Figure 7 shows the results of the compliance analysis. The 300 most used keys of Taginfo have been selected for the analysis. The most compliant cities are Almería, Granada and Vigo, while the less ones are Oviedo, Santander, Sevilla and Valencia.

We have individually analyzed the case of Oviedo, which highlights from the given results. Oviedo has 2945 elements with less commonly used keys. We found that there are three contributors with a high level of contributions (415, 1182 and 267, respectively) tagging elements with less commonly used keys. The use of a group of keys, those with prefix is_in, and, in particular, two of them: is_in:region and is_in:province (which are out the 300 most used keys of TagInfo) is the cause of the lack of conformance. On the other hand, there are 307 elements in Oviedo obtained from Spanish open data including a less commonly used key called ref:RRG.

The second part of the study of compliance is about the co-occurrence of keys, more concretely, the percentage of OSM items with commonly used combinations of keys (see Figure 8). Taginfo offers a ranking for each key of most used combinations with other keys. Our analysis has been limited to three cases: highway, shop and tourism. Highway is, in most of the cities, compliant with Taginfo (more than 90% have common combinations of keys), with the exception of Oviedo (77%), Santander (85%) and Valencia (88%). Shop is also compliant with Taginfo, even better than highway (all the cities have a 90% of commonly used combinations of keys). This is also the case of tourism, in which Santander and Zaragoza fail, with a 41% and 76%, respectively, of less commonly used combinations of keys. Again, we have individually analyzed the case of Santander (for the tag tourism), in which 43/50 elements have been introduced by two contributors, using the key check_date, which is an unusual key for tourism entities.

As conclusion, even when a high number of tags can be considered a positive quality indicator, the use of non compliant tags, especially non compliant group of tags, tends to get worse the quality of the dataset. Usually, such group of tags are consequence of imports from open data of local or regional governments.

2.4. Consistency

Consistency analysis has as goal to assess the degree in which a certain entity has a similar representation in the same area. A certain entity can be described by a number of keys (and values). However, usually, the number and type of keys to describe a certain entity varies from one to another. It can be due to multiple contributors, different provenance, missing information, etc., of the same kind of entity. It can be usual in different areas of a country (information provided by local and regional government), but for a specific area should be similar. The consistency is measured by the standard deviation of the number of keys. Thus, low score means homogeneous tagging, whereas high score means heterogeneous tagging. Our approach assumes that the main goal of quality assessment is to ensure a suitable query processing, for which consistency is crucial. Some expected query results are discarded due to heterogeneous information. Figure 9, Figure 10 and Figure 11 show the results of the analysis for the entities highway, shop and tourism. In our opinion, consistency must be analyzed for each category of entities in isolation because each category can be described by a different subset of tags. For this reason, we have individually analyzed three categories of highway, shop and tourism. In the case of highway the cases of primary, secondary and tertiary; in the case of shop the cases of convenience, supermarket and clothes; and in the case of tourism we have considered information, hotel and attraction.

In general terms, highway information is less homogeneous than shop and tourism ones. Almería, Madrid, Segovia, Sevilla, Valencia and Vigo have more homogeneous information about highways, whereas Barcelona, Oviedo and Santander have more heterogeneous information. Bilbao, Segovia and Zaragoza have more homogeneous information about shops, while Oviedo and Santander have more heterogeneous information. Finally, Almería, Granada, Santander and Vigo have more homogeneous information about tourism, while Barcelona, Bilbao, Madrid, Oviedo, Valencia and Zaragoza more heterogeneous information.

We have analyzed the following cases which highlight from the results. In Barcelona, we focused on highway-primary, in which from 570 elements, 66 items have less than 5 tags, 59 elements submitted by four contributors, but 87 elements have more than 15 tags, 74 of them send by one contributor. In Oviedo, we focused on highway-secondary, in which from 176 elements, 20 elements (14 submitted by two contributors) have less than 5 tags, while 21 elements have more than 13 tags. In the latter case, 7 elements from them use the group of keys is_in: is_in:city, is_in:continent, etc. The same happens in Oviedo for the case of highway-tertiary. We have also analyzed the case of shop-supermarket in Santander. From 36 elements, 7 elements (which are nodes) have less than 3 tags, and 6 elements have been added by the same contributor. However, 5 have more than 13 tags. Here the use of the groups of keys addr and is_in increases the number of tags, and the data source is the cadastre of Santander. In the case of Oviedo for tourism-hotel, from 29 elements, 5 elements have less than 4 tags, and 4 elements (which are ways) have more than 20 tags. Again the use of groups of keys addr and is_in increases the number of tags (17 tags addr/is_in in some cases). In the case of Madrid (tourism-attraction) from 90 elements (which are nodes), 15 elements have less than 5 tags and 7 ways (of the same contributor) have more than 20 tags. The source of data is the local government of Madrid, including the group of keys addr and source (in some cases up to 12 tags).

As conclusion, low consistency is a consequence of the use of group of tags, obtained from open data of local or regional governments. While imports enrich datasets, they partially cover the elements of certain entities, and some other elements of the same entities are poorly labeled with regard to imported ones. Also a higher number of versions, specially in more attractive objects, causes a low consistency.

2.5. Granularity

Granularity analysis has as goal to assess the quality of the information provided to entities. More detailed information means more useful map. Let us remark that granularity and consistency are complementary. For instance, an unique tag for all the items is highly consistent (however few information is available and thus few granularity). The same ten tags for all the items is also highly consistent, and also granularity can be considered high.

In this case the average and the median value of the number of keys used for describing entities is analyzed. In particular, we have analyzed building, shop and tourism. Here, we are taking into account the number of keys describing an entity with independence of the conformance to Taginfo. Figure 12 shows the results of the study.

The distribution of number of keys is not normal (i.e., the average and the median value are not the same in many cases). Santander is the only city where the median number of tags is greater than the average (mainly in the case of tourism), which indicates very different tagging behavior. In general terms, buildings have lower level of granularity than shop and tourism items, which can be caused by contributors that usually focus their attention on most known places of the city. Santander provides better granularity (in all the items), while Madrid and Oviedo highlight in tourism items (an average of 6 tags or higher). In the rest of cases, tourism items have from 3 to 6 tags. Shop items have a greater granularity in Madrid and Oviedo (an average of 4 tags or higher). In the rest of cases, shop items have from 2 to 4 tags. We suspect that the main reason is the import of open data from local and regional government sources.

Santander example is specially interesting. In the case of shop, there exist 1855 elements (nodes and ways). From them, 1667 elements have more than 10 tags (89%). Again, the groups of keys addr and source increase the number of tags. Most of them have been included by three contributors (1460 elements). In the case of tourism, from 85 elements (nodes and ways), 51 have more than 9 tags (60%). Again the groups of keys addr and source increase the number of tags, most of them (concretely, 41 elements) submitted by the same contributor.

2.6. Richness

Richness analysis has as goal to assess the quality of the entity classification in categories. We have analyzed three cases of entities: building, shop and tourism. Here, the number of distinct categories used for each city has been computed. The results are shown in Figure 13. We found here that Barcelona, Madrid and Sevilla have the richest classification, while Oviedo and Segovia the poorest one. Shops are the richest entities, while building and tourism entities are poorly classified. Let us remark that the total number categories in Taginfo for each one (with a percentage of use greater than 0.1) is 53, 186 and 36, respectively. Here, we believe that higher richness in big cities is expected due to a wider variety of shops and tourism places.

2.7. Trust

The quality is also evaluated in terms of the average number of versions of the entities and the local/global experience of contributors. With regard to version average, the study reveals that it ranges from 2.02 (Zaragoza) to 2.49 (Oviedo). Let us remark that in OSM, for nodes, each geometry and tagging change results in a new version. However, for ways, only edits that change the referenced nodes (e.g., deleted or added node to the way), and tag changes result in new version numbers.

Local experience evaluates the average number of contributions in the study area by contributor in the current status (at the time of the study), and the global experience the average number of contributions in the planet by contributor. The higher is the value of these indicators, the better is the trust on the data of the study area. While the use of a full history dump is better to analyze local experience, it has a high computation cost. Obviously, with our approach, some contributors can be ignored if the number of revisions of their contributed items is high. Instead, global experience takes into account the changesets count by contributor obtained from OSM Web site (see for instance https://www.openstreetmap.org/api/0.6/user/7406). Using the changeset count to assess global experience is a little misleading since the number of edits (i.e., features having a user as last contributor) highly varies between changesets. It also depends on the editor (e.g., iD users tend to submit more changesets with less changes, while JOSM users submit less changesets with more way edits).

Figure 14 and Figure 15 shows the results of the analysis. The contributors with a higher local experience are found Madrid, Santander and Zaragoza, while the lower local experience is found in Granada and Vigo. Finally, contributors with a higher global experience can be found in Almería, Madrid, Santander and Segovia, while the contributors of the rest of cities have similar global experience. We have not found a compelling relation between contributor experience and other quality indicators. For instance, even when we could expect that the higher contributor global experience, the better compliance or richness, as well as the higher contributor local experience, the better completeness and consistency, we cannot conclude that in our study.

3. Comparison of Spanish and European Cities

Now we show the same kind of analysis for a group of European cities: Athens, Berlin, London, Paris, Rome and Vienna, and they are compared with Spanish Madrid, Barcelona, and Santander (the most representative in terms of OSM data in Spain). Data for all the cities are taken in April 2018 (see Figure 16).

As we can see in Figure 17, the proportion house numbers/buildings is lower in Spain (Madrid and Barcelona) than in Berlin, Paris and Vienna, being similar to London, but higher than in Athens and Rome. As Figure 18 shows, the rate of occurrence of highway names in Spain is similar to other European cities. The tag oneway is used in Spain at the same rate as most of cities (with the exception of Athens and London). In the case of the tag maxspeed, the cities Berlin, London, Rome and Vienna are better labeled, while Athens and Paris are considerably worst labeled, much closer to Santander.

The rate of amenity naming in Spain (see Figure 19) is similar to other European cities, while opening hours tagging in Spain is better than Berlin and Vienna. The tag phone has a higher rate in Santander, however Madrid and Barcelona rates are more similar to other European cities. The rate of shop naming in Spain (see Figure 20) is similar to other European cities, in which Santander highlights from the rest of European cities in the use of opening hours and phone. However, with the exception of Berlin and Vienna, in Barcelona and Madrid the tag opening hours can be found at the same rate as other European cities. The tag phone, however, is still missing in a high rate in most of the European cities. The naming in Spain of public transport, tourism, religion and historic (see Figure 21) can be considered similar to most of the European cities. With the exception of Athens-public transport, Paris-historic, Rome-historic, the rate of occurrence in the European cities is greater than 70%.

Conformance to commonly used keys (see Figure 22) can be consider in Spain similar (or even better) to most of European cities. Conformance to commonly used combinations of keys (see Figure 23) can be also considered similar to most of European countries, where Santander has a lower rate in the case of tourism entities.

Comparing consistency of Spain and European cities (see Figure 24, Figure 25 and Figure 26), we find that highways in Barcelona and Santander, and shops in Santander, are less consistent. Santander highlights in the consistency of tourism. In the rest of cases, Madrid, Barcelona and Santander are similar to other European cities. Analyzing granularity (see Figure 27), we can find similar results for Madrid and other European cities, being lower (in most of cases) in Barcelona, and higher (in all the cases) in Santander. Similar results of richness (see Figure 28) to other European cities (with the exception of Berlin, London and Vienna) can be found in Barcelona and Madrid.

With regard to version average, most of the European cities have a higher version average (Barcelona-2.12, Madrid-2.15, Santander-2.06, London-2.47, Paris-2.53, Rome-2.49 and Vienna-2.52, and specially Berlin-3.46), while local experience of contributors (see Figure 29) is higher in Madrid and Santander (with the exception of Athens). With respect to global experience (see Figure 30) Madrid, Barcelona and Santander contributors have a medium, low and high level of experience, respectively.

4. Web Tool of QXOSM

In order to facilitate the analysis of any area of the world with the same quality indicators proposed in our framework, we have developed a Web tool available at http://xosm.ual.es:8080/qxosm. This Web tool enables the selection of a certain area of the OSM planet and from the menu (see Figure 31) to run each type of analysis. The Web tool offers a summary of the layer including the number of elements (ways and nodes), number of contributions (taking into account the number of versions for each element), number of contributors and average number of versions. The Web tool enables to customize the analysis in such a way that for each type of analysis permits the selection of entities (and categories/attributes) for which the analysis is carried out. For instance, in the case of completeness (analysis of missing attributes), the tool enables the selection of the entity and the group of attributes for which the occurrence is analyzed. Moreover, the Web tool shows the results using charts reporting for each bar the percentage/number of items. The Web tool shows both aggregated and disaggregated data. For instance, the chart of granularity analysis shows the average number of keys, as well as a pie chart in which groups of keys and their percentage of use, is shown. The same can be said for richness analysis, in which the percentage of items for each category is shown in a pie chart as well as the number of different categories. The Web tool also enables the analysis of a certain area by number of versions and date of creation/modification. Finally, the user contributions (local and global experience) can be analyzed, and charts are used to visualize number of contributions, classified by user experience.

5. Conclusions and Future Work

In this paper we have presented a framework for the quality analysis of the tagging process of OSM. We have defined a batch of quality indicators which measure several aspects of data quality. While completeness analysis aims to ensure the usage of OSM data for specific purposes, compliance analysis is intended to measure that OSM data fit well with standardized criteria of tagging. Consistency analysis has as goal to assess the homogeneity of OSM data. Granularity and richness analysis focus on classification and labeling processes. We have applied our framework to analyze OSM data of Spain, where as far as we know, there does not exist previous studies. However, we have extended our analysis to some major European cities in order to compare the current status of the Spanish OSM with nearby areas. Even though our study has not been exhaustive with respect to set of entities, categories and attributes, our main aim was to have a picture of the current status of the Spanish OSM data. Additionally, we have developed a Web tool in order to facilitate to carry out the same kind of analysis in other areas of the world.

As conclusion, the current status of the Spanish OSM can be considered satisfactory in some indicators (compliance and consistency), while in some others (granularity and richness) should be improved.

Completeness is still a matter of further improvement, since mainly navigational information is still away to be satisfactory. The proportion of house numbers with respect to buildings is low (with the exception of Santander and Zaragoza). The inclusion of the attribute name has to be improved in Almería, Granada, Oviedo, Segovia, Sevilla, Valencia and Vigo. The same can be said for oneway, which usually has similar rate, being better in some cases like Almería, Segovia, Sevilla and Valencia, and considerably lower in Oviedo, Santander and Vigo. The maxspeed rate is usually lower, specially, in Granada, Oviedo, Valencia, Vigo and Zaragoza. The tag name is usually included in amenities and shops, but it is still incomplete. The same can be said for public transport (with the exception of Granada, Madrid, Segovia, Sevilla and Vigo), tourism (with the exception of Almería, Barcelona and Vigo), religion (with the exception of Segovia) and historic entities (with the exception of Santander and Valencia). Current tends on geographic information systems, requiring the availability not only of navigational information, but more frequently of detailed information about POIs. POI information can be considered better, even though some elements are still missing. For instance, shopping and amenity places should include opening hours, phone, etc, and specific categories (restaurants, hotels) more detailed information (prices, cuisine, stars, etc). In our analysis, we have found that they are generally missing. Opening hours and phone have a very low rate in all the cities.

Compliance is generally achieved, but some cases of less commonly used keys like Oviedo, Santander, Sevilla and Valencia, and less commonly used combinations of keys like (highway–Oviedo, Santander and Valencia–, tourism–Santander and Zaragoza) should be improved. With regard to consistency, some items are described more homogeneously than others. For instance, highway information is less homogeneous than shop and tourism ones. Barcelona, Oviedo and Santander have heterogeneous information about highways, Oviedo and Santander about shops, and Barcelona, Bilbao, Madrid, Oviedo, Valencia and Zaragoza about tourism. The use of group of keys (addr, is_in and source) by some few contributors coming from sourced data (open data of government) affects both compliance and consistency.

Granularity can still be improved; for instance, tourism entities have a greater granularity than shop and building ones, which can be caused by contributors that usually focus their attention in most known places of the city. Santander provides better granularity (in all the items), while Madrid and Oviedo highlight in tourism items (an average of 6 tags or higher). In the rest of cases, tourism items have from 3 to 6 tags. Shop items have a greater granularity in Madrid and Oviedo (an average of 4 tags or higher). In the rest of cases, shop items have from 2 to 4 tags. In addition, we found that Barcelona, Madrid and Sevilla have the richest classification, while Oviedo and Segovia the poorest one. Shop are the richest entities, while building and tourism entities are poorly classified. The contributors with a higher local experience are found Madrid, Santander and Zaragoza, while the lower local experience occurs in Granada and Vigo. Finally, contributors with a higher global experience can be found in Almería, Madrid, Santander and Segovia, while the contributors of the rest of cities have similar global experience. We have not found a compelling relation between contributor experience and other quality indicators. For instance, even when we could expect that the higher contributor global experience, the better compliance or richness, as well as the higher contributor local experience, the better completeness and consistency, we cannot conclude that in our study.

As future work, we plan to extend our study to some other areas of Spain Also we could extend our study to rural areas in order to measure the absence or presence of specific information for rural areas. We also plan to incorporate new quality indicators to our work and the Web tool.