Semantic Trajectory Analytics and Recommender Systems in Cultural Spaces

Abstract

Semantic trajectory analytics and personalised recommender systems that enhance user experience are modern research topics that are increasingly getting attention. Semantic trajectories can efficiently model human movement for further analysis and pattern recognition, while personalised recommender systems can adapt to constantly changing user needs and provide meaningful and optimised suggestions. This paper focuses on the investigation of open issues and challenges at the intersection of these two topics, emphasising semantic technologies and machine learning techniques. The goal of this paper is twofold: (a) to critically review related work on semantic trajectories and knowledge-based interactive recommender systems, and (b) to propose a high-level framework, by describing its requirements. The paper presents a system architecture design for the recognition of semantic trajectory patterns and for the inferencing of possible synthesis of visitor trajectories in cultural spaces, such as museums, making suggestions for new trajectories that optimise cultural experiences.

1. Introduction

Visitors of cultural spaces (e.g., museums, archaeological sites) are usually offered a rather static and less personalised experience, e.g., a group-organised guided tour of exhibits in museum rooms. To overcome this problem, there have been numerous studies that utilise advancements in recent technologies, such as IoT and pervasive computing technologies, to monitor and analyse visitor movement and interactions within cultural spaces [1,2]. Analysed data (with high volume, velocity, and variety) gathered from sensors (streaming/dynamic data) and datastores/databases (historical/static data) are used to recognise/infer visitor preferences and personal interests to propose and eventually deliver an enhanced cultural experience. This is achieved either by providing personalised and enriched content or by suggesting personalised navigation in the cultural space. A related example is found at the Rijksmuseum Amsterdam, which offers a real-time routing system for implementing a mobile museum tour guide for personalised tours [3].

An already effective approach for discovering preferences and needs for moving users in cultural spaces is through the analysis of their trajectories (big movement data), as they contain rich explicit and implicit information and knowledge. Due to the evolution of mobile computing, wireless networking, and related technologies, such as GPS, mobile applications can monitor and share information about user position during movement, e.g., while the user is visiting a cultural space. The existing infrastructure enables applications to produce a vast amount of streaming data that include information not only about locations and places that users are visiting but also the paths/routes/trajectories the users are following, as an aggregation of connected spatial points in specific time-lapses [4,5,6,7].

Semantic Web technologies offer powerful representation tools for pervasive applications. The convergence of location-based services and Semantic Web standards allows easier interlinking and semantic annotation of trajectories, resulting in semantic trajectories. Trajectory-based operations, which involve spatiotemporal data of moving entities, are becoming increasingly important in related studies and applications, as they provide insights about human movement and the ability to extract patterns and predict future behaviours. As described in [8], a semantic trajectory-based recommender system (RS) is designed on the basis of the observation that users with similar trajectories would have similar preferences for the available objects, and outperform traditional recommendation methods that do not consider trajectory or environment information.

The motivation for this research is to explore human movement and behaviour in cultural spaces to provide optimised cultural experiences. Based on this motivation, we propose a framework that represents visitor movement as enriched semantic trajectories to extract useful information required as input to a recommender system that would provide optimum alternatives to their cultural experiences. The main requirements of the framework (described in Section 5) are summarised as follows:

• Exploitation of raw spatiotemporal trajectory data
• Semantic segmentation and annotation of the trajectory
• Trajectory description using suitable ontologies
• Semantic trajectory enrichment with Linked Open Data (LOD)
• Semantic annotation of cultural spaces and points of interest (POI) to provide context and capability for semantic integration with user trajectories
• Trajectory analytics for pattern recognition and classification
• Future location prediction
• Dynamic user profiling
• Integration of User Knowledge Graph (UKG)
• Integration of Cultural Space (CS) and POI Knowledge Graph (KG)
• Integration of KG-Based recommender system (RS) for path-based and KG-based recommendations
• Integration of context-aware RS
• Integration of hybrid RS
• Integration of collaborative filtering RS
• Inference and proposal of a possible synthesis of visitor trajectories

The specific set of requirements was derived from an extensive study of the related work (pros and cons of related approaches/systems), as well as from our motivation to select the most appropriate techniques and methods related to the synthesis of both paradigms, i.e., semantic trajectories and recommender systems. The aim was to develop a novel framework that will eventually (a) enhance the functionality and efficiency of a recommender system for visiting cultural spaces, and (b) connect user trajectories with optimised cultural experiences.

The aim of this paper is two-fold: (a) to present a systematic literature review of state-of-the-art approaches related to semantic trajectories and recommender systems, with an emphasis on the cultural domain, and (b) to introduce a high-level framework for the recognition of trajectory patterns, inferencing possible syntheses of visitor trajectories in cultural spaces and the combination of trajectory analysis results with visitor dynamic profiling data. The RS is used to suggest optimal alternative trajectories in real-time to enhance the visitor experience. The framework is presented through a use case scenario of a trajectory that takes place in the city of Athens and the Acropolis Museum.

The structure of this paper is as follows: Section 2 describes the basic concepts of Semantic Trajectories, Recommender Systems, and Knowledge Graphs. Section 3 discusses the survey methodology and the decisions made for the final selection of related studies. Section 4 presents the reviewed state-of-the-art related work regarding Semantic Trajectories and Recommender Systems. Section 5 critically discusses the related work based on a set of requirements towards a framework that enhances cultural experiences. Section 6 presents the proposed system architecture design. Finally, Section 7 concludes the paper and reports on future work.

2. Preliminaries

2.1. Semantic Trajectories (STs)

A trajectory is defined as the composition of the sections of connected traces and points that express a meaningful movement in space and time by an object or entity of interest. The study of trajectories is fundamental for the comprehension of moving object/entity behaviour, as there is a plethora of useful information in the path/route the object/entity follows to navigate between start and end points. The behaviour of a trajectory is the sum of the characteristics that identify the essential details of a moving object/entity or a group of moving objects/entities. A set of such unique characteristics creates a short description of a group of trajectories which are called patterns [9,10]. Data analytics based on trajectories of moving objects/entities, such as trajectory clustering and construction, could provide advantages in the solutions of several common or more complex problems [11,12,13]. Trajectory analysis can be performed either using raw spatiotemporal data or semantically annotated movement data. Although data mining plays a significant role in this domain, as these algorithms are used for the extraction of trajectory patterns, most of the mining algorithms are developed for raw data implementation and, as a result, they are not effective in recognising patterns for specific domains. Moreover, the exclusive usage of spatiotemporal data provided by movement tracking sources lacks significant information about the context of the movement [4,14].

One of the main issues is the difficulty to correlate the patterns with movement behaviours to extract and expand the knowledge about them. The challenge is in the enrichment of the spatiotemporal data of the trajectories with semantic, context-based information, relevant to the moving objects and the contextual association of the trajectories with related low- or high-level events. One way to address this challenge is the use of ontologies and linked data (LD) to semantically connect the trajectory patterns and behaviours with the broader context of the movement. As stated by Dodge et al. [15], the movement behaviour depends on the general context in which it takes place, as every movement has a specific meaning in the moment and in the space/environment that it is happening. Semantic trajectories are the trajectories that have been enriched with semantic annotations and one or more complementary segmentations [10]. Annotations of segmented parts of a trajectory (episodes) could be “stop” or “move”, or, in other cases, could be points or regions of interest. An example semantic trajectory of a touristic walk is depicted in Figure 1.

Knowledge discovery tasks can be performed in semantic trajectories to extract patterns based on characteristics, such as changes or stops, POIs, or specific behaviours that can be recognised in a single or a group of trajectories and used to create classes and classify or identify future trajectories [9].

2.2. Recommender Systems (RS)

RSs are software tools or AI applications that are designed to predict the user’s interests and preferences based on statistics, data mining algorithms, and machine learning techniques, to suggest/recommend products, services, locations, routes, etc. RSs handle the problem of information overload that users normally encounter and affect the way users make decisions through the recommendation of suitable actions or objects of interest. An example of a Cultural Recommender System is depicted in Figure 2. The main tasks of a RS are to (a) gather and process data, (b) create models based on the available data, (c) apply the models to existing and future data, and (d) receive feedback and re-evaluate the models. Common characteristics that a RS should possess for it to be considered efficient are: accuracy, coverage, relevance, novelty, serendipity, and recommendation diversity [16]. Most recommendation strategies are defined based on three main relation types: (a) User–Item relation, (b) Item–Item relation and (c) User–User relation.

▪

User–Item relation: This relation is based on the user profile and the explicitly documented preferences of the user towards a specific type of Item.

▪

Item–Item relation: This relation occurs based on the similarity or the complementarity of the attributes or descriptions of the Items.

▪

User–User relation: This relation describes the Users that possibly have similar tastes with respect to specific Items, such as mutual friends, age group, location, etc.

User preferences are calculated with the use of explicit ratings provided by user evaluations or feedback, and implicit ratings that are inferred from the users’ interactions with the products or the aforementioned relations.

Depending on the available data, there are two main methods of categorisation of objects (supervised and unsupervised) based on the similarity of their features. Classification is the supervised method. It uses predefined tags and classes to categorise a group of inputs. Commonly used algorithms for classification are K-Nearest Neighbours (kNN), Decision Trees, and Naïve Bayes. In the case of Clustering, the unsupervised method, labels or categories are unknown in advance and the task is to effectively categorise given inputs and discover similarities based on certain criteria. Examples of clustering algorithms are K-Means Clustering and DBSCAN (Density-based Spatial Clustering).

One of the most common issues that RSs deal with is the rating sparsity, known also as the cold start problem, which causes inefficiency and low accuracy on recommendations. RSs face that condition when the number of available items, i.e., candidates for recommendation, is much greater than the number of users rating them. This situation occurs especially when a RS has recently started collecting ratings or has not been exploited by a lot of users [17].

Popularity-based RSs are often used to bypass the cold start problem that recommend the most selected products by other users. Demographic RSs, use information like age, gender, etc., to classify users for future recommendations. Demographic techniques are sometimes included in Hybrid RSs to increase robustness [18].

Collaborative Filtering (CF) RS use models that focus on User–User relations or Item–Item relations to infer ratings about products. The methods used in CF are equivalent to those of a classifier that creates a training model from labelled data. The basic idea of CF methods is that unspecified ratings can be imputed because the observed ratings are often highly correlated across various users and items. The main challenge in those methods is that the matrices of the user ratings are very sparse, as the users usually rate a fraction of the available products. When the user preferences are specified, the model tries to discover similarities to other users. If the similarity discovery is successful, the ratings of similar users are used to infer values to complete the rating matrices [16].

Content-based RSs are based on the idea that the user will prefer products similar to those already used and rated highly. This type of RS creates representations of products based on their features and descriptions and matches them to similar attributes of other products to suggest them to the target user. The content-based methodology calculates user profiles and specifies the interests and preferences to compute a relevance score that predicts the user’s level of interest in a specific product. The product attributes for representations are usually extracted by metadata or textual descriptions, but there is an increasing interest in the advantages of Semantic Web technologies to approach content-based recommendations. As there is a plethora of open knowledge sources that provide semantic information, recent research studies shift from keyword-based to concept-based representations of products and users [19].

Apart from the Item ratings, context may be anything that might affect the desirability of particular recommendations at the time of the generation of the recommendations [19]. The context-aware RS (CARS) is a new trend in recommender systems. The CARS considers user profiles as dynamic and evaluates user preferences and interests along with other factors that may occur in the current situation of the target user, like the user’s location, the user’s companionship, the weather, etc. The CARS aims to provide personalised recommendations according to both user profiles and their current contextual conditions [20].

The knowledge-based RS utilises domain knowledge either provided by experts in the form of domain-specific rules and ontologies or by the usage of the knowledge available on the Web as structured LOD. KGs can be used to exploit explicit connections between user entities and product entities or infer implicit connections to create suggestions for the users. As mentioned in [21], there are several studies on ontology-based, LOD-based, path-based, and KG-based recommendation approaches that perform better than traditional recommendation approaches, especially in cases with small amounts of sample ratings and sparse rating matrices.

The hybrid RS exploits the advantages of different approaches, like the effectiveness of KG-based RSs in sparse data, or the collaborative methods when multiple user ratings are available. The hybrid RS leverages the strengths of several approaches, allowing recommendation methods to produce separate ranked lists of recommendations, and then merging their results to produce a suggestion list.

2.3. Knowledge Graphs

KGs are increasingly getting attention from academic and industry organisations as they provide several advantages compared to relational databases, regarding the representation and management of big and heterogeneous data. As defined by Hogan [22], a KG is a graph of data intended to accumulate and convey knowledge of the real world. The nodes represent entities of interest, and the edges represent relations between these entities. While there is a conceptual overlap between KGs and ontologies, because both are formed to be “an explicit specification of a conceptualisation”, KGs can be considered more as “a graph of data with the intent to compose knowledge” [23,24].

As described in [24], the requirements for a graph to be considered a KG are:

• Knowledge graph meaning is expressed as structure.
• Knowledge graph statements are unambiguous.
• Knowledge graphs use a limited set of relation types.

In terms of inferencing and entity representation, a KG can accumulate simple statements as edges in the data graph, but for more advanced tasks, more expressive ways are required, such as the use of ontologies and rules [22].

A KG stores and manages relations about entities. These relations could be expanded, and new relations could be created with the use of inference algorithms, e.g., rule-based reasoning, OWL/RDFS reasoning, or combinations of these approaches, that can infer knowledge and enrich the KG.

Figure 3 depicts an example KG that includes a user sub-graph describing the visitors, a POI sub-graph describing museums and exhibits in the city of Athens, and their connections. A sample set of RDF triples [25] of this KG is the following:

:Peter       rdf:type    foaf:Person.

:Peter       foaf:knows    :Mary.

:Mary      :visited      db:Parthenon.

db:Parthenon  rdf:type   :POI.

db:Parthenon  dbo:location  dbr:Athens.

Apart from KG enrichment, several techniques can be applied to a KG to provide insights and perform analysis of the encoded data. The most widely used techniques and approaches for analysis are listed below:

• Centrality: discovers the nodes with the most connections and the biggest impact in the graph.
• Community Detection: discovers sub-graphs that are more closely connected internally, compared to the rest of the graph.
• Connectivity: evaluates the quality of the connections in the graph, in terms of resilience, reachability, etc.
• Node Similarity: measures which neighbour nodes are in a specific area of the graph, based on their features and connections.
• Path Finding: discovers possible reachable paths between predefined terminal nodes.
• KG Embeddings: transforms graph representations to a low-dimensional vector space (graph embeddings), to allow ML applications to handle them efficiently.
• KG Recommendations: KGs, by design, provide the technical means to integrate various heterogeneous information sources, for instance, POIs and user preferences. Thus, feature similarity discovery algorithms can be applied to enhance recommendation techniques.

3. Survey Methodology

The research methodology followed in this paper focuses mainly on the collection of information sources related to semantic trajectories and cultural recommender systems. The research was conducted in a period of six months, examining academic articles, relevant literature, and web resources published between 2016 and 2021 (5 years).

The research for articles was conducted on academic web portals such as Scopus, Google, and Semantic Scholar, ResearchGate, ACM Digital Library, IEEEXplore, and SpringerLink.

The specific search terms used in various combinations were:

• semantic recommender systems
• trajectory-based recommender systems
• semantic trajectory-based recommender systems
• cultural recommender systems
• semantic trajectories
• trajectory annotation
• trajectory segmentation
• POI extraction and annotation
• trajectory enrichment
• human movement trajectories
• cultural semantic trajectories

The papers that were collected were mainly related to the domains of museum studies, cultural computing, and computing.

We have included articles that mostly cover human movement and trajectories that are primarily related to the domains of our field of study. Although a significant amount of the published research is on semantic trajectories and trajectory analytics in a wide range of scientific fields, we did not include works applied to other domains, such as the works related to human road safety or autonomous vehicles. However, we have taken into consideration a selection of non-human trajectory-related studies that propose noteworthy approaches regarding the semantic annotation and management of trajectories [13,26,27,28,29,30,31,32,33,34,35,36,37].

RSs are increasingly applied to a variety of use cases. Therefore, there are numerous articles about their use in several domains. We have limited the selected works to those that utilise knowledge and semantic approaches for recommendations and those that focus on cultural applications and trajectories [2,8,38,39,40,41,42,43,44,45,46,47,48,49,50,51].

4. State of the Art in STs and RSs

4.1. Semantic Trajectories (ST)

4.1.1. Annotation of Trajectories

In de Graaff et al. [26], a novel algorithm for automating the detection of visited POIs is proposed. They describe a POI as “a location where goods and services are provided, geometrically described using a point, and semantically enriched with at least an interest category”. A Polygon of Interest (POLOI) has a similar definition to a POI, except that the location is described by a polygon. The proposed method tries to identify visited POIs both in outdoor and indoor trajectories from raw smartphone GPS data but is specifically designed for urban indoor trajectory analysis. It focuses on detecting stops that take place at known and predefined POIs. The challenges overcome by the proposed algorithm are the non-detection of indoor visited POIs and the false positive detection due to lack or instability of the GPS signal. Because of the unavailability of the GPS signal for the indoor segments of a trajectory, the proposed algorithm selects points before and after the users get into a building and projects them to a polygon. The POI visit extraction algorithm considers the accuracy of the location, reductions in speed, changes in direction, and projection of signals onto polygons to extract the staypoints (the centroids of stop sequences) from a trajectory. An experiment with students in the city of Hengelo was set up to validate the proposed approach and concluded that the algorithm outperformed several existing approaches.

Another method for annotating trajectories, following a different approach, is the one presented by Nogueira et al. [13]. In this work, it is stated that the focus is on modelling mobile object trajectories in the context of the Semantic Web, and it is based on an ontological approach to extract episodes from semantic trajectories. An episode is defined as the smallest semantic unity in Semantic Trajectory Episodes (STEP) [52]. Episodes encapsulate values that a Feature of Interest (FOI) or Contextual Element (CE) may assume during the trajectory. They expand the already published STEP ontology to represent generic spatiotemporal episodes. It is claimed that, with the use of Semantic Web technologies, it is possible to integrate various data sources, data, and metadata and also support reasoning by inference, in order to enrich movement data. To validate the expanded ontology, a framework called FrameStep is proposed. The framework contains a graph–object mapping layer that enables the creation of episode instances that can be then transformed and serialised into triples. It utilises the ontology along with annotation algorithms to form semantic trajectories from raw GPS data by detecting episodes and enriching the trajectories. The framework also integrates with LD cloud and OpenStreetMap [53] to retrieve information about the context and the environment of the analysed trajectories.

In the work of Chen et al. [30], the challenge of organising raw spatial trajectories and fusing them with semantic data is examined. They present a structured and self-described way to annotate trajectory episodes with sentiments, events, or topic words. To achieve this goal, a model is implemented, where each user has multiple trajectories divided into episodes. Each episode contains semantic information like events or sentiments extracted by a combination of spatial and temporal information, with the context semantic information derived from posted texts by the users, using Natural Language Processing (NLP).

4.1.2. Semantic Trajectory Management

The work of AL-Dohuki et al. [31] focuses on an approach to interact with trajectory data through visualisations, enriched with semantic information about the trajectories. The approach was designed and evaluated for taxi trajectories. The trajectories are converted to documents through a textualisation transformation process where the GPS points are mapped to street names or POIs, and the speed is described quantitatively. After the transformation, each document is described by a meta-summary and indexed to enable queries over a text-based search engine. The system is data-structure agnostic, and the results are integrated with visualisations and interactions to promote easy understanding. The evaluation of a prototype claimed to be successful, and its ease of use is demonstrated appropriately.

Motivated by the need to exploit data from disparate and heterogeneous sources in an integrated manner to increase the predictability of moving object trajectories, Santipantakis et al. [32] propose a framework for semantic integration of big mobility data with other data sources, providing a unified representation and support analysis tasks by exploiting trajectories at various levels of analysis. One of the major challenges of this work is to enrich surveillance data providing meaningful information about moving entity trajectories, and annotating trajectories with related events, therefore creating enriched trajectories. To meet these challenges, they introduce the SPARTAN approach for providing enriched streams of mobility data, incorporating online compression, data transformation, and link discovery functionality. The domains of interest are marine and aviation. Streaming spatiotemporal data are the input to the proposed framework. It performs data cleaning and summarisation, transforms data to Resource Description Framework (RDF) [25] in compliance with a generic ontology (namely, datAcron ontology [54]) for trajectories, and performs integration with other streaming and archival data sources. The output of the framework is a stream of linked RDF data that contains enriched trajectories of moving objects. The experimental evaluation of the proposed framework demonstrates efficiency and scalability when using large, real-life datasets.

The first step to semantic annotation and enrichment of the trajectories is to perform the necessary trajectory segmentation. This can be done either with supervised or unsupervised ML techniques. In the first case, the criteria are predefined, and the segmentation can be implemented with ML techniques trained with labelled segments or by applying algorithms that segment the trajectory based on a threshold. In the latter case, the applied methods must discover similarities and homogeneity without specified criteria, based on point density or a cost function. In the work of Amilcar Soares Júnior et al. [33], a semi-supervised algorithm is proposed that implements the Minimum Description Length (MDL) principle to measure homogeneity inside segments. This algorithm uses a small set of labelled trajectory data during the trajectory segmentation task to drive the unsupervised segmentation of unlabelled trajectory data. The semi-supervised approach takes advantage of both the supervised and unsupervised techniques, so the user must annotate a small set of trajectories to help the algorithm recognise meaningful segments for the rest of the trajectories. As mentioned in the study, the main advantage of this method, compared to pure supervised ones, is that it reduces the human effort to label the number of trajectories. A few examples can be used to target the segmentation task to specific domains. The proposed algorithm is characterised as reactive, for automatically defining the values of the input parameters by analysing the results of previously provided examples. As stated in the study, for the experiments conducted using real-world datasets, the proposed algorithm outperformed the state-of-art competitors.

4.1.3. Semantic Trajectory Modelling

Semantic modelling is the step after the annotation for utilising semantic trajectories. In Vassilakis et al. [34], authors present SemMR, a semantic framework for modelling interactions between human and non-human entities, managing reusable and optimised cultural experiences towards a shared cultural experience ecosystem that may seamlessly accommodate mixed reality experiences. The proposed framework is based on the concept of cultural experience as a semantic trajectory (eX-trajectory). It is designed to utilise Mixed Reality (MR) technologies and applications for creating and managing eX-trajectory content and tools. Methods are proposed for monitoring and analysing user interactions in MR spaces for behaviour pattern extraction for optimising eX-trajectories at runtime by enriching and augmenting them with useful information from heterogeneous sources. To evaluate the framework, simulation experiments ran with artificial users as inputs. As stated, the eX-trajectories that were generated by the system were parsed and evaluated for appropriateness to each user profile and visitor path and were found to be in alignment with the user visiting style and preferences.

The related work for modelling human movement behavioural knowledge from GPS traces for categorising mobile users [35] is motivated by the challenge of using extracted knowledge from trajectory analysis of a specific place to cluster users and discover user behaviour patterns in a target place, without analysing it. That work proposes a framework that forms trajectories from raw GPS data, creates a movement behaviour model, and applies pattern mining methods to transfer knowledge regarding the human movement to geographical regions. To achieve the knowledge transfer, they cluster user behaviour in a specific region of interest and apply the results to semantically similar regions. To model the user movement, a Bayesian network is used that can encapsulate measures of the randomness of movement. The classifier of the Bayesian network provides the probability of a target user to match multiple categories that are predefined based on the features and the trajectory analysis. As an experimental evaluation, a real-life dataset of monitored GPS trajectories was used to transfer knowledge to a region with insufficient data and the results were close to those of the analysed area.

As mentioned by most of the reviewed related works, the mainstream trajectory representation methods focus mainly on the spatiotemporal information of the trajectory and not on the context or the semantic information that could be extracted from them. Gao et al. [36] present a novel representation of semantic trajectories that takes into account domain knowledge, in addition to spatial and temporal data, in order to enhance semantic trajectory retrieval. They propose a synchronisation-based clustering model to transform raw GPS points to multi-resolution Regions of Interest (ROIs). They deploy a tree-shaped hierarchical network that captures each ROI in a set of GPS trajectories. This leads to the replacement of raw trajectories with sequences of ROIs. A hierarchical embedding model transforms the ROI sequences to continuous vectors, based on geographical and semantic trajectory features in a way that the similarity measures between two trajectories can be computed by the Euclidean distance of two vectors directly. The embedding model emphasises their context of movement to extract semantic relations among target objects. The results presented after evaluation experiments showed that the proposed method had superior performance in semantic ROI/trajectory retrieval tasks compared to state-of-the-art methods and deep network embedding models.

Most of the studies in the field of trajectory representation and analytics focus on GPS data of outdoor activities and movements, while those that capture indoor POIs present them as parts or stop points of a broader trajectory. Motivated by the lack of indoor trajectory research, Kontarinis et al. [37] combine aspects of semantic outdoor trajectory models with a semantically-enabled hierarchical symbolic representation of the indoor space, in alignment with the OGC IndoorGML standard [55]. The proposed model for enriched indoor semantic trajectories utilises a standardised indoor space modelling framework that contains the semantic trajectories alongside the semantically enriched representations of indoor space. The indoor space is described as a layered multigraph. The nodes of the multigraphs represent spatial regions, while the edges represent the topological relations between spatial regions. The model was deployed and evaluated for a dataset of spatially aggregated timestamped points of visitors of the Louvre Museum, to present its expressiveness. The study also presents the application of standard and advanced pattern mining methods to provide a formalisation for indoor trajectory mining and express the combination of semantic and spatiotemporal data.

In the work of Krisnadhi et al. [56], a pattern for modelling semantic trajectories is presented. This model adds a spatiotemporal expansion to a previously published model for the representation of trajectories. The work is motivated by the lack of published patterns for simple spatiotemporal extents. The pattern indicates that a trajectory should be modelled as a sequence of fixes that are connected by segments and that every fix should have a spatial and a temporal extent. The model follows a set of axiomatic rules. Specifically, every trajectory must have one starting and one ending fix, trajectories for the same spatiotemporal extent cannot have temporal overlap, and every spatiotemporal extent must have at least one trajectory.

4.1.4. Semantic Trajectory Analytics

While there are several works in trajectory analysis and enriching trajectories and managing them as semantic trajectories provides better insights into the target movement behaviour, limited research has been conducted on the use of Convolutional Neural Networks (CNNs) in connection with modelling human movement patterns. In Karatzoglou et al. [27], a novel CNN-based approach for representing semantic trajectories and predicting future locations is introduced. The CNN approach design was based on an NLP use case similar to the proposed, explaining that the data were also in one-dimensional format and that relevant use cases with the use of CNNs have already been studied in NLP. The CNN takes semantic trajectories as input and assigns a unique index to every semantic location. Trajectory indexes are passed to a hash table which assigns a feature vector to them. These feature vectors are task-specific representations that the system extracted in the training phase from the available data by discovering the optimal semantic location representations. The results are used as input to the core model for predicting the next semantic location. To evaluate the approach, they have worked with semantically enriched data of a single-user model and multiuser models that contained the trajectories of 100 users. The evaluation results showed that, although the proposed model is sensitive to sparse data, it outperforms other reference systems in terms of accuracy and is capable of modelling semantic trajectories and predicting future semantic locations.

In Zhang et al. [28], a system is proposed to extract the semantic trajectory patterns of data produced by users’ positioning devices. As stated in the paper, the already proposed mined trajectory patterns are unable to reflect the semantic information hidden in the trajectory because a user’s trajectory not only contains the physical movement track but also embodies the user’s purpose for moving. Motivated by that, they propose a probabilistic generative model that annotates and clusters the pre-processed trajectory. Raw spatiotemporal data with no semantic information are divided into two parts: the moving and the stop data. They formulated a spatiotemporal threshold and clustering-based method to extract the stopover points. A probabilistic generative model was implemented to identify starting and ending points on a trajectory, connect them to POIs and, by annotating those points, discover the visiting purpose of the trajectory. Finally, the PrefixSpan algorithm [57] is applied to discover patterns over trajectories by finding suffixes for multiple prefix sequences. Evaluating experiments of the proposed method, in contrast to widely used algorithms in pattern mining like K-Means and DBSCAN, showed that it outperforms the other methods in all cases.

Several privacy issues can emerge when tracking and analysing human movement, especially when enriching the movement with context information and discovering patterns that are frequently followed. In order to reduce the risks of invasion of privacy, the work of Khoroshevsky and Lerner [29] is based on the assumption that there is no data sharing among users and that the user data are stored on the client. By performing the analysis on the user device, they prevent the exchange of sensitive location information between servers and users. An algorithm that combines spatial and semantic information for movement pattern discovery and location prediction is proposed. They retrieve semantic data from the OpenStreetMap API to specify semantic places around points that are clustered in geographic locations. To achieve point clustering in semantic locations, they propose two clustering evaluation metrics. After point clustering, the terminal and intermediate stop points form the user location history, which can be transformed into a sequence of visit locations. To cluster trajectories, they use a similarity metric for string sequences. To discover trajectory patterns, each sequence is assigned to the closest detected cluster of trajectories. Experiments that compared the proposed algorithm with previous works revealed that it provides accuracy for a reasonable number of location sequences.

The work of Liu and Wang [8] is motivated by the challenge of community detection over a set of trajectories. In contrast to most proposed methods that evaluate clustering by proximity-related metrics, they propose a framework that exploits semantic information from multiple sources, where the trajectories in a specific cluster exhibit similarity in one or more movement-related features. They defined the difference between clustering and community detection as the difference between a set of objects related purely through spatial proximity and a set of objects whose proximity or movement similarity is likely a manifestation of some underlying mutual interaction or shared relationship. The proposed framework leverages information markers underlying the raw trajectory data to detect groups based on movement information of users and semantic information of the space where the movement takes place. The framework learns the consistent graph Laplacians by constructing the multi-modal diffusion process and optimising the heat kernel coupling on each pair of similarity matrices from multiple information sources. It models the trajectory similarity based on semantic-level movement, spatiotemporal proximity, and velocity, then computes similarity measurements to determine the communities derived from the computed values. After communities are formed, they apply a collaborative filtering recommendation method based on the observation that similar users that belong to the same community have similar preferences. After experiments were conducted on selected datasets, it is stated that the community detection system and the recommendation method outperformed other clustering and recommendation algorithms.

4.2. Recommender Systems (RS)

4.2.1. Cultural RS

In Amato et al. [44], a methodology that combines recommendation with agent-based planning techniques to implement a planner of routes within cultural spaces is proposed. The problem of finding a scheduled path of visitors in a cultural space or site is handled as a reachability problem and uses multi-agent models to achieve the goal of accessing POIs within certain deadlines. First, the approach analyses user preferences to provide an accurate list of cultural items. Then, the multi-agent planning methods calculate the paths that follow sequence steps to meet the goal of visiting the suggested items. For each pair of users and items, the recommender can compute a rank that measures the expected interest of the user in an item, using a knowledge base and a ranking algorithm. The ranking algorithm integrates information about preferences and past behaviours of the target user and the user community, user feedback, and contextual information, to create the list of suggested items. The browsing system is represented as a directed labelled graph and depicts the sequence of chosen items to increase the similarity measure between them. Finally, the agents compute and recommend the path that meets the requested goals or state that it is unreachable.

Su et al. [45] propose and develop a Big Data architecture that leverages edge intelligence in addition to cloud computing for a more scalable and user-centric analysis of the cultural data. The architecture consists of a data ingestion stage, a knowledge base, a data process stage, and applications. Apart from the architecture, a user-centred recommendation edge-intelligence strategy is proposed for cultural recommendations. The RS relies on a context-aware hybrid recommendation strategy deployed on a multilayer architecture based on Big Data and edge computing technologies. It considers user preferences, location, and items’ semantic features of POIs, to generate touristic routes as a sequence of POIs. The recommendation algorithm relies on features of cultural items, the user’s past itineraries, and behavioural information captured by the stream processing from social networks. As a proof of concept, they developed an application for suggesting POIs, such as museums, to city visitors. Evaluation results demonstrated that the recommender system outperformed other proposals. This is explained using the Knowledge Base that stores information about objects and users, combining them with those obtained from social networks during the ranking phase.

The work of Cardoso et al. [46] presents the implementation of an application that suggests routes for cultural heritage visits. The application is designed with an adaptive user interface where routing and augmented reality are connected to acknowledge the needs of user categories, such as elders, kids, experts, or general users. The proposed application aims to suggest the optimal route of POIs between terminal nodes in a cultural environment. The suggested route is computed considering the maximum visit time and a vector of the user preferences. The design of the proposed application is based on the optimisation problems of user preference extraction, the number of visiting POIs, and time spent exploring them. The methods designed for the navigation problem were based on the ant colony optimisation algorithms and weighted function strategy. They computed the optimal paths inside a network of POIs, in near real-time, considering the user preferences and given limitations. The application is in the final stages of development and under testing with real users in a real museum environment.

The system presented in Smirnov et al. [47] deals with tasks related to the info-mobility concept: user action analysis, preference revealing, and cultural heritage recommendation based on the preferences and current situation. The system suggests possible touristic paths for visiting cultural POIs. The application architecture is based on the Smart-M3 information sharing platform and consists of a set of joined services by a smart space that provides semantic-based information exchange. The RS calculates the item ranking based on the CF method to provide the list of POIs. The ranking algorithm considers the ratings set by all the users of the system concerning the similarity to the current user, the user preferences, the current situation in the target location, and the reachability of the POIs. The user situation while using the application is modelled by ontology-based context. The application provides detailed information about cultural heritage POIs that are retrieved from internet sources in real-time. It also estimates the reaching path and presents it in an interactive map. The evaluation shows that the application is efficient in finding the cultural POIs in the user’s area in a reasonable time and can be used for on-the-fly tourist support during a trip.

An interesting research challenge is how the cultural factors influence RS efficiency, whether cultural differences could be a technology barrier, or whether there are universal factors for RSs. Motivated by that challenge, Hong et al. [48] proposed the novel concept of cross-cultural contextualisation and a model to compute the cross-cultural factor affecting user preferences. They propose a contextualisation model for computing the cross-cultural factor, which influences user preferences in RSs by using Matrix Factorisation and clustering techniques. As mentioned in the study, a systematic analysis of the dataset and the experimental results suggested that individual users could be considered as country-wise groups for the model to analyse the cross-cultural factors. The users’ cultural preferences are modelled to a rating matrix that contains vectors of cultural preferences on different items. Matrix factorisation is applied to interpolate the sparse matrix. Experiments using a real-world dataset, which contains ratings by visitors from different countries, showed the effectiveness of the proposed model and supported that there are cultural factors that influence user rating behaviour in recommendations.

In Loboda et al. [49], the authors demonstrate the impact of the museum RS and implement a content-based RS to generate personalised museum tours to enhance visitors’ experience by providing a personalised way to engage with museum collections. The feedback received from a study conducted to evaluate the improvement of a visit by a RS focused on the provided information about the objects and the accessibility of the museum collection. The developed RS was based on the content-based filtering method regarding the feature similarities across the items of the collection. To address the cold-start problem, the users had to select preferred items from a list before the application provides a personalised tour. The application provides detailed information about the recommended items. Evaluation by real visitors reported that the RS made the visit more structured and helped the discovery of interesting objects. Another aspect of the findings of the user evaluation was that diversity in museum RSs might be favoured over accuracy.

In related works about social recommendation services for cultural heritage [50], it is stated that we can guess the affinity of an artwork choice between two users from the users’ artwork-watching histories and the artwork features. Motivated by that, Hong et al. presented a novel recommender system based on a method for discovering and exploiting social affinity between users based on artwork features and user experience. The system is designed based on a use case scenario where the museum that exploits the RS is equipped with an IoT system relying on intelligent sensors and services that can identify the user behaviour during the visit. The RS follows a hybrid recommendation approach, where the suggestions are computed from the results of a social-based and context-based recommendation method. The social recommendation ranks the list of artworks based on social affinity by requests of individuals or groups. The affinity between users is estimated using an affinity graph, which is created from filtering the information of the user history and the features of the artworks. For artwork similarity measurement, the Jaccard similarity coefficient and Euclidean distance were calculated. The context-based method recommends artworks based on user interaction. The proposed approach is about to be evaluated through a developed application in a museum in Naples that offered the necessary infrastructure.

4.2.2. Semantic and Knowledge-Based Recommender Systems

Interactive RSs are modelled as a multistep decision-making process to capture the dynamic changes of user preferences. Zhou et al. [38] present a recommendation approach that utilises reinforcement learning methods and KG to provide semantic information to an Interactive RS. Reinforcement learning methods face an efficiency issue when provided with a small sample of data. To address the issue, they leverage prior knowledge of the item relations in the KG for better candidate item retrieval, enrich the representation of items and user states, and propagate user preferences among the correlated items. Interactions between the user and the system last for a defined time period. At each period, the system dynamically generates a list of items based on historical interaction data and item similarity from the KG, suggests them to the user, and receives feedback in order to update the recommendations. The introduced model consists of a graph convolution module, a state representation module, a candidate selection module, and the Q-learning network module. Evaluation experiments demonstrated that the proposed approach outperformed the state-of-the-art method.

In Sansonetti et al. [2], the authors introduce a hybrid RS empowered by social media interactions and LOD. The first step in the recommendation method is the collection of data relevant to the user by analysing the social profiles to retrieve preferences and past interactions with other users and with cultural places. The collected information is stored in a Neo4j [58] graph database and disambiguation tasks are applied to the graph through LOD tools. The proposed recommendation approach integrates both social and semantic recommender methods. The social recommendation method performs CF to suggest items that users with similar interests from the social network prefer. Semantic recommender leverages the DBpedia [59] and Europeana [60] knowledge bases for suggesting cultural places that share similar semantic features. The system also considers the contextual information of the suggested places, such as accessibility and weather conditions. The recommended POIs are formed in an itinerary with user preferences and contextual constraints. The routes are modelled as a directed graph where nodes represent the POIs, and the weighted edges represent the time required to reach the next POI in the sequence. The graph is filtered based on the location of the POIs. POIs are annotated with information from the LinkedGeoData [61] dataset using SPARQL [62] queries. Experimental results on real users showed the effectiveness of the modules of the proposed RS.

Minkov et al. [39] propose a graph-based recommender framework, to help museum visitors deal with information overload. A KG is constructed to model the museum environment into classes of entities. The nodes represent entities of users, multimedia presentations, physical positions of artworks, or semantic themes, while edges represent structured relations between entities or viewed relations between users and artworks. To infer similarity between nodes, the Personalised Page Rank (PPR) algorithm [63] with a random walk is applied to the graph to rank items based on their relevance to the target user profile. The user profile is generated by using the previous interactions of the user with other entities in the graph. So, user feedback is given on viewed multimedia presentations or POIs while not-yet-watched presentations are ranked up according to the user interactions and preferences. As reported in the study, the results of experiments conducted using data collected at the Hecht Museum showed that graph-based recommendation using the PPR measure outperformed a set of classical collaborative and content-based recommendation methods, justifying the superiority of the graph-based approach.

Qassimi and Abdelwahed [51] present a graph-based recommendation approach that utilises semantic information extracted from collaborative tagging of cultural heritage places to enhance cultural heritage visits and suggest semantically related places that are most likely to be of interest to the visitors. The recommender system is based on the emerging graphs representing the semantic relation of similar cultural heritage places and their related tags. The emerging graphs form a multilayer graph. Its nodes represent the cultural heritage places. The edges represent their relations. Descriptive metadata are extracted by exploring a folksonomy of shared resources to augment the cultural places. The augmented places are clustered based on the tags used to annotate them. The user is provided with suggestions of similar places to those previously visited, tagged, or rated. Evaluation of the system shows that it achieves better results compared to content-based approaches.

4.2.3. Trajectory-Based Recommender Systems for Cultural Spaces

Rodriguez-Hern et al. [40] introduce a trajectory and user-based collaborative filtering approach and implement a context-aware recommender system in order to provide the user with visiting routes in a museum. The system considers several contextual aspects, such as user preferences, choices of other visitors, time constraints, current location, and trajectory. To evaluate the proposed approach, they used a real dataset of the artwork of the Museum of Modern Art collection and reproduced the layout of six floors by converting map images available on the Web to graph structures by using the tool WebPlotDigitizer [64]. The DataGenCARS [65] tool was used for user rating generation, producing ratings provided by synthetic users for artworks already visited, based on the users’ profiles, as well as random trajectories that were assigned to the users. The exchange of opinions among visitors relies on a central information service that allows feeding the recommendation process with data to pro-actively suggest changes in the recommended route in the museum. The recommendation approach detects visitors with similar preferences to the user and uses their ratings to estimate the potential ratings of the user for various artworks. If those ratings exceed a threshold, an artwork is considered a candidate for suggestion. The highest-rated items are ordered in a way that minimises the overall distance and the generated path that consists of their optimal sequence is suggested to the user.

DeepTrip is an end-to-end neural network method for understanding the underlying human mobility and modelling of the POI transitional distribution in human moving patterns. DeepTrip is proposed by Gao et al. [41] as an implementation of a trajectory embedding approach for a low dimensional representation of POI contextual features. A trip encoder that leverages a recurrent neural network is responsible for the route embeddings and a trip decoder for reconstructing the routes. A Generative Adversarial Network (GAN) is defined as a learning model consisting of a generator and a discriminator that compete in a two-player min-max game. The framework incorporates a GAN to enhance the generation ability of the trip decoder for POI sequence recommendations. Evaluation experiments show that DeepTrip outperforms the state-of-the-art baseline resulted from various evaluation metrics, although it expends more effort in understanding human mobility through learning implicit trajectory distributions.

Geo-tagged photos can be used to construct users’ trajectories as they contain spatiotemporal information in sequential order and are useful for mining patterns of human movement. Cai et al. [42] present an itinerary RS based on semantic trajectory pattern mining from geo-tagged photos in order to provide a suggestion of POIs. Sequences of geo-tagged photos that capture POIs provide spatial and contextual information like weather conditions and travel duration. Semantic itineraries are built by annotating raw trajectories with application-dependent contextual and spatial semantics extracted from geo-tagged photos and environmental data like day, type, time, weather conditions, and place names. Then, the semantically enriched trajectories are mined by performing a variation of the Prefixspan algorithm for patterns of frequent sequences of semantic stops in the progress of the trajectories. The system consists of the offline trajectory pattern mining part and the online itinerary recommendation part. The recommendation part suggests itineraries based on a user query by filtering and ranking candidate itineraries from the semantic trajectory pattern database. The experimental results on real datasets from Flickr [66] support the effectiveness and efficiency of the proposed system over traditional approaches.

In Xu and Han [43], the authors introduce a framework for next location recommendation based on trajectory analysis and user behaviour. The framework is based on a Recurrent Neural Network (RNN) and a Similarity-based Markov Model (SMM) that combines inferred user behaviour and spatial information to suggest future locations. Raw trajectory data are converted offline to semantic sequences. Then, the users are added to clusters based on semantic similarity of their trajectories. The neural network is trained from the trajectory features of the user clusters. For the online location prediction part, the trajectory of the target user is transformed to a semantic sequence by applying the Word2vec algorithm [48] to embed spatiotemporal and semantic information into a universal space, and the user is attached to the most similar cluster. The SMM creates a state transition matrix from the correlation of the user with others in the cluster and historical trajectory data. The output of the model is an ordered sequence of candidate locations, where users can choose preferably. The proposed framework is reported to be superior to other tested models in terms of prediction performance.

5. Evaluation and Discussion

In

Table 1. Comparative table of reviewed related works.

*	a	b	c	d	e	f	g	h	i	j	k	l	m	n	o
[26]	x	x
[13]	x	x	x	x
[30]	x	x
[31]	x	x
[32]	x	x	x	x
[33]	x	x
[34]	x	x	x	x	x	x		x
[35]	x	x				x
[36]	x					x
[37]	x	x	x		x	x
[56]		x	x
[27]						x	x
[28]	x	x				x
[29]	x	x		x		x	x
[8]	x	x				x
[44]								x						x	x
[45]				i	x			x	x	x		x		x	x
[46]			ii		x			x				x		x	x
[47]								x				x		x	x
[48]								x						x
[49]								x							x
[50]								x	x			x	x	x
[51]					x			x		x	x
[38]								x		x	x
[2]				iii	x			x	x	x		x	x	x	x
[39]								x	x	x	x	x
[40]	x							x		x		x		x	x
[41]	x				x	x	x	x							x
[42]	x	x			x	x									x
[43]	x	x						x						x

i: enrichment of POIs with LOD, ii: ontological description of the context, iii: enrichment of POIs with LOD. * Based on the order presented in Section 4.

, the evaluation of related works based on the requirements of the proposed framework is presented. The columns represent the requirements (as listed below) and the rows show the referenced studies. The order of the presented works in Table 1 is based on the structure and order of their presentation in Section 4. This rationale, along with the arrangement of the evaluation criteria (first half are related to ST, second half are related to RS), are the reasons for the effect in Table 1, i.e., quadrants I and III are comparatively empty.

The main requirements of the framework are listed as follows:

• Exploitation of raw spatiotemporal trajectory data: raw trajectory data include useful spatial information combined with time-specific stops, speed, and direction of the visitor, needed for the initial segmentation.
• Semantic segmentation and annotation of the trajectory: for raw trajectories to be converted to semantic trajectories and analysed as such, segmentation of the trajectory and annotation of the parts are necessary.
• Trajectory description using suitable ontologies: Ontologies provided a structured and unified way of semantically describing instances of entities and fuse them with domain knowledge
• Semantic trajectory enrichment with linked open data (LOD): LOD grant a plethora of continuously updated information from different data sources regarding the context of the trajectory
• Semantic annotation of cultural spaces and points of interest (POI) to provide context and capability for semantic integration with user trajectories: semantically described POIs and spaces can make trajectory segmentation and recommendations more effective, and the interlinking of POIs and trajectories possible
• Trajectory analytics for pattern recognition and classification: the main goal of the process is the ability to discover features and recognise patterns in trajectories to categorise them and extract meaningful information about visitor movement
• Future location prediction: effective trajectory analysis and classification can lead to future location prediction, which is a useful input for the RSs
• Dynamic user profiling: updating user profile based on explicit and implicit feedback of user behaviour
• Integration of User Knowledge Graph (UKG): describes user profiles semantically and represents them as nodes in a KG
• Integration of Cultural Space (CS) and POI Knowledge Graph (KG): represents semantically annotated and enriched POIs and CS as a KG
• Integration of KG-Based recommender system (RS) for path-based and KG-based recommendations: performs path finding and connectivity methods for discovering possible recommendation lists in the optimal ranking order
• Integration of context-aware RS: provides suggestions considering contextual information to enhance final recommendations
• Integration of hybrid RS: merges multiple recommendations to achieve maximum efficiency and accuracy
• Integration of collaborative filtering RS: leverages user similarity to produce meaningful suggestions
• Inference and proposal of a possible synthesis of visitor trajectories: evaluation and combination of RS suggestions with respect to user preferences for generating and proposing optimised trajectories

By reviewing the state-of-the-art method in semantic trajectories and recommender systems in recent literature, it appears that a significant amount of related work is partially related to the topics of semantic trajectories and semantic cultural recommender systems. However, as our research indicates, none of these related works fully exploit the semantic information and the insights of trajectory analytics to (a) enhance the functionality and efficiency of a recommender system for cultural spaces, and (b) connect user trajectories with cultural experiences. Some of the related works [40,41,46] propose an integration of RS approaches with trajectory extracted information, but either there is no semantic annotation or analysis of the trajectories, and the focus is only on GPS points, or the recommendation approach uses only the user current position and not the full semantic representation of user trajectory to define the context. Last but not least, in most of the related works [2,40,41,42,44,45,46,47,49], the outcome is a suggestion of a path generated based on user interest. That path usually is the shortest path of connected POIs, without considering the user visiting style.

6. Proposed System Architecture Design

In this paper, we present the architectural design of the proposed framework (Figure 4) along with a real-life use case scenario. The framework is designed to overcome the limitations of existing related work and fully meet the requirements to provide recommendations for optimum alternatives to visitor cultural experiences based on semantic trajectory analytics, as these were discussed in Section 5.

The framework is designed to be trajectory-centred and to express the integration of user experience and movement in an enriched semantic trajectory. Results of analytics on that trajectory and the use of information of similar trajectories in the same cluster will provide valuable input for meaningful recommendations. Furthermore, the suggestions for the future visiting locations and the information about POIs are designed to be tailored to user preferences and continuously re-evaluated based on user choices and their physical location. As a trajectory evolves, it can be classified in a more effective manner and can be more accurately matched to the visiting style(s) of a user.

6.1. Use Case Scenario

The recording of the purpose of a cultural trip to a city is a semantic annotation at the trajectory level, while the recording of the presence of a person at a specific location, such as a visit to a temporal art exhibition, is a semantic annotation at the position level. Movement analysis of a tourist who engages in cultural related activities in Athens results in a trajectory for the whole movement in the city as a ‘tourist inside Athens’. One or more distinct trajectories of daily cultural experiences, such as a tour of the Museum of Acropolis on Friday morning, are also recorded. The distinct areas of the museum and the exhibits are semantically described. Visitor data, such as age and gender, personal preferences regarding art, music, types of museums a user visits, and the ways of touring in an exhibition space have been collected and evaluated. During the visit to the Acropolis Museum, that information is combined to produce semantically richer and more accurate trajectories, as well as to suggest routes and alternatives to improve the visitor experience. Trajectory-based personal recommendations are provided via a smartphone application on a user’s phone screen, or in embedded screens near exhibits or artworks. The recommendations provide either short or detailed descriptions based on calculated user profiles and suggestions for relevant and related exhibits along with the most efficient routes to reach them.

The tourist in our scenario uses the application on a smartphone device. The application records the GPS signal to monitor the current location of the user (Trajectory Monitoring). The user has provided personal information and interests to build an initial version of a profile. As depicted in Figure 1, the user starts the walk from Plaka and, after a while, stops for a coffee break. This information is stored and used to annotate the trajectory with preferences in sightseeing and social activities (Figure 4: Trajectory Segmentation, Trajectory Annotation). As the user moves in Athens, the trajectory is taking shape and it is compared with stored trajectories in order to be grouped in a cluster with trajectories that have similar characteristics (Figure 4: Trajectory Clustering). The analysed trajectory is classified as a ‘touristic walk inside Athens’ (Figure 4: Trajectory Classification). The user gets a notification from the application to continue the walk to Acropolis to visit the Parthenon and receives information about this POI. The user provides feedback for the suggestion and evaluates the related information (Figure 4: explicit User Feedback). The system discovers that similar trajectories follow the path to Acropolis Museum, so it suggests the museum as the next visiting location (Figure 4: Collaborative Filtering Recommendations). The user is provided with a basic path that covers the main exhibits of the museum, based on their current profile preferences. Inside the museum, Bluetooth beacons, installed near the exhibits, provide information to the application about the proximity of the user and the time spent near them. That information, along with the interaction of the user to the provided contextual details about the targeted exhibit, are recorded as interesting information for them (Figure 4: Implicit User Feedback, Dynamic Profiling). The system is actively computing new recommendations and provides them to the user. The recommendations are based on the user preferences, the available time span, and the visiting style. The visiting style is evaluated by comparing the current trajectory with others stored in the KG. Data from the beacons and the application show that the user expresses more interest in statues rather than other exhibits. The application suggests that visitors with this preference follow a route from the Archaic Acropolis on the 1st floor to the Caryatids and the Athena Nike on the 2nd floor (Figure 4: Collaborative Filtering Recommendations). The recommendations consider avoiding crowded exhibits and rearrange the order of the suggestions (Figure 4: Context-Aware Recommendations). The calculated tour in the museum is estimated to last approximately two hours. After the first hour, the user is detected to behave differently, i.e., not devoting enough attention to the exhibits, bypassing most of the system suggestions. This behaviour is recognised as boredom or exhaustion (Figure 4: Behaviour Monitoring, Experience Evaluation). The application re-evaluates the suggested route and proposes the following (in sequence): (a) to skip the following exhibits on the 2nd floor, (b) to move to the 3rd floor, (c) to take a 20-min break at the museum café, and (d) to visit the temporal exhibition of the Acropolis Museum. The suggestion about the 3rd floor, where the artworks from the Parthenon are exhibited, was based on the interest and positive feedback received from the user about the Parthenon earlier in the tour (Figure 4: Knowledge-based/Context-Aware Recommendations).

6.2. System Architecture Modules and Tasks

In Figure 4, the proposed system architecture design for the proposed framework is depicted. The framework is divided to interconnected modules and information exchangers. The distinct modules are Monitoring, Pre-processing, Semantic Representation, Profiling, Semantic Trajectories, Trajectory Analytics, and Recommendation.

The Monitoring Module includes trajectory and user behaviour monitoring tasks. It is assumed that the cultural space is equipped with proper monitoring infrastructure (e.g., IoT devices such as Bluetooth beacons or RFIDs, and cameras) for tracking the visitor movement and behaviour. The Trajectory Monitoring task is responsible for collecting information about the physical movement of the user. This information is generated from GPS signals of portable devices such as smartphones and smartwatches, and proximity signals from Bluetooth beacons or RFID tags. The User Behaviour Monitoring task collects information about the behaviour of the user. This information is collected from the interactions of users with POIs, recorded by the developed application or by the evaluation of the user’s mood (e.g., bored, interested). Movement and facial expressions can also be recorded by cameras in inner (e.g., museum) or open spaces (e.g., open museums/archaeological sites).

The Pre-processing Module receives data from the Trajectory Monitoring and Behaviour Monitoring tasks. At this module, the data is cleaned and integrated for further processing. Data integration is implemented by semantically describing the data with suitable ontologies. For instance, a data model for the system could use the datAcron Ontology [54] to describe trajectories, EDM [67] or CIDOC-CRM [68] to describe artworks, and FOAF [69] or User Profile Ontology [70] to describe users. Data is cleaned and transformed to RDF triples with tools like Karma [71] and eventually stored as a KG. This data conversion is necessary to (a) handle the heterogeneous data in a unified manner, and (b) for further enrichment and linking with related LOD.

The Semantic Trajectories Module is responsible for the transformation of raw trajectories to semantic trajectories. After data transformation, the trajectories formed by raw spatiotemporal data are segmented based on stop/move parts, velocity, or predefined POIs. The segments are semantically annotated with contextual information (day, time, place name, weather, etc.) and domain knowledge (e.g., abstract concept about the artwork). The semantic trajectories are also enriched with LOD (e.g., cultural POIs linked with a DBpedia or Europeana entity), complementary information stored in the KG, such as data provided by the Semantic Representation Module, or features visiting style evaluated by the comparison of stored trajectories in the KG.

The Semantic Representation Module is responsible for the representation of the cultural space. To provide predefined POIs and ROIs for more efficient trajectory segmentation, the cultural space and its exhibits must be described in a manner that is in alignment with the movement data and persist useful features of artworks and their peripheral area. POIs and ROIs are then semantically described with a suitable ontology that covers semantic and spatial information, converted to RDF, and stored as main entities to the cultural space KG. The entities are now linked and enriched with external information from LOD. Furthermore, they are interlinked with the POIs recognised in the trajectory segmentation part.

The Trajectory Analytics Module is responsible for the analysis and categorization of semantic trajectories. It consists of two tasks: the “online” Real-Time Trajectory Classification and the “offline” Trajectory Clustering task. The stored semantic trajectories are then analysed for pattern extraction and for the formation of semantically similar trajectory clusters. Each cluster contains trajectories with discovered visiting styles that are spatially or semantically similar. For example, one may spend the whole visit exploring all artworks and artifacts of a temporary exhibition while another is selectively visiting the essential and mainstream exhibits of a museum and spend the rest of the visit in the museum cafe. Clusters are necessary for the “online” part, where the partially constructed trajectories are assigned to the most similar one in real-time and provide to the semantic segmentation and annotation part useful information about entities’ visiting style and preferences.

The Profiling Module is responsible for updating the user profile and evaluating user state. It consists of the Feedback task and the Dynamic Profiling and Experience Evaluation task. Implicit (user actions in the monitored space, visited exhibits and time spent near them) and explicit (direct ratings through forms/questionnaires) user feedback, alongside the classified trajectory, provide essential information for dynamic profiling. The Dynamic Profiling & Experience Evaluation task combines information about user preferences with feedback and behaviour monitoring data to evaluate user experience and dynamically update the user profile. The recording profiles, containing personal information and provided or inferred preferences, are stored in the User KG.

The Recommendation Module consists of the Collaborative Filtering task, the Knowledge-based/Context-Aware recommendations task, and the Trajectory synthesis and Recommendation of optimal trajectories task. CF RS methods applied to the User KG provide suggestion lists based on user similarity. A Knowledge-based/Context-aware RS provides suggestions based on semantic object similarity measurements and user preferences while considering contextual information like crowd density, weather, and minimum time needed to explore a region. This module is also responsible for providing complementary semantic and multimedia information about the visited artifacts. For instance, the user (e.g., Peter in Figure 3) is interested in a specific artwork that is described in the KG (Caryatid). The artwork is linked with information and related entities in the KG, such as the creator and the museum that it is exhibited at. The museum (Acropolis Museum) is related to the city (Athens) that it is located, while the city is related to its POIs. If other users with similar preferences (Mary) have shown interest in POIs (Pantheon) in this city, a recommended path will occur from one POI to the other, based on the feature similarity, community detection, and reachability derived from the KG.

The Trajectory synthesis and Recommendation of optimal trajectories task integrates a hybrid RS that merges the recommendation provided by the CF RS and the Knowledge-based RS. This RS leverages the merged recommendations to achieve trajectory synthesis and provide optimal personalised routes that guide visitors to preferred exhibits, according to the preferred visiting style and the available timespan. A crucial part of the described architecture is the continuous feedback and the experience evaluation that affect the RS state and create the potential to dynamically update the provided recommendations and the complementary information.

The proposed architecture is designed to meet the abovementioned requirements and cover the partial absence of ST exploitation for empowering RS towards cultural experience optimisation. Furthermore, the architecture is designed to capture the entire process of handling raw spatiotemporal data, converting them to enriched ST, clustering and classifying ST based on features, creating user and cultural space KG, and integrating different types of RS to effectively recommend enhanced trajectories.

7. Conclusions and Future Work

Cultural spaces like museums are increasingly emphasising a more personalised, optimised, and enhanced visiting experience. A way to achieve that is through efficiently and effectively understanding human movement in cultural spaces. Movement can be effectively represented and evaluated by semantic trajectory analysis, while personalisation can be realised by user/visitor profiling and by providing meaningful and interesting suggestions utilising specialised, software-like recommender systems. In this paper, we conducted a systematic review of the state-of-the-art related work, focusing on the intersection of the semantic trajectories and recommender systems, and on the advantages of the Semantic Web Technologies and KGs in both research fields. Subsequently, a framework and a system for the collection, annotation, and analysis of trajectory data, along with the integration of a hybrid knowledge-based RS, is proposed for optimising cultural experiences.

Future plans for this work involve the implementation of the proposed system by developing a set of methods and tools that meet the presented framework requirements by (a) the effective transformation of raw trajectories to semantic trajectories, (b) the performing of analytic tasks to extract meaningful information about visitors, and (c) the integration of a hybrid RS that combines results from a KG-based, a CF-based, and context-aware RS, for optimal suggestions. Future work will also include the design and evaluation of a use case scenario and experimentation with real-life users to receive useful feedback on the efficiency of the framework.