TravelRAG: A Tourist Attraction Retrieval Framework Based on Multi-Layer Knowledge Graph

Abstract

A novel framework called TravelRAG is introduced in this paper, which is built upon a large language model (LLM) and integrates Retrieval-Augmented Generation (RAG) with knowledge graphs to create a retrieval system framework designed for the tourism domain. This framework seeks to address the challenges LLMs face in providing precise and contextually appropriate responses to domain-specific queries in the tourism field. TravelRAG extracts information related to tourist attractions from User-Generated Content (UGC) on social media platforms and organizes it into a multi-layer knowledge graph. The travel knowledge graph serves as the core retrieval source for the LLM, enhancing the accuracy of information retrieval and significantly reducing the generation of erroneous or fabricated responses, often termed as “hallucinations”. As a result, the accuracy of the LLM’s output is enhanced. Comparative analyses with traditional RAG pipelines indicate that TravelRAG significantly boosts both the retrieval efficiency and accuracy, while also greatly reducing the computational cost of model fine-tuning. The experimental results show that TravelRAG not only outperforms traditional methods in terms of retrieval accuracy but also better meets user needs for content generation.

1. Introduction

The gradual recovery of the global economy has been accompanied by a robust rebound in the tourism industry [1]. With the growing number of social media users, sharing and documenting personal experiences on social platforms has become an integral part of daily life for many. Content shared on social media is not only up-to-date, but also rich in geographic information [2,3]. The rapid growth in users of short-video platforms, alongside the coming of age of a curious younger generation, has allowed many lesser-known attractions to reach wider audiences through recommendation algorithms. This trend has encouraged people to select travel destinations based on content with significant social influence [4,5]. Special-interest attractions differ from mainstream sites primarily due to the short-term influx of visitors and the lack of detailed travel planning. Moreover, travel notes shared on social platforms by the younger generation have become a major reference [6]. These travel notes typically include information on the location of attractions, activities, personal experiences, and nearby accommodations and dining options. Rapid analysis and accurate retrieval of travel notes from social platforms are essential for adapting to current tourism trends.

Accurately and comprehensively extracting geographic information from texts is essential for analyzing user behavior. Imagine a traveler writing in a review: “Walk along the cobblestone path for five minutes, and you’ll see a blue cottage on your left. The café next to it has the best matcha latte”. Humans can easily understand that this is a recommendation for a café, recognize that the café is located beside the “blue cottage”, and know that it can be reached by following the “path”. However, it remains challenging for a machine to accurately interpret the geographic information embedded in such descriptions. The system would need to parse vague location references like “blue cottage” and “path”, understand that “matcha latte” implies a popular item, and interpret “on your left” as a directional cue. These nuanced yet pivotal geographic data are intuitive to human readers but can often surpass a machine’s current comprehension capabilities. If machines could better extract and interpret these details, recommendation accuracy would improve, helping travelers more easily locate their ideal destinations. For the tourism industry, this could mean a more personalized and seamless user experience. As internet culture has evolved, a form of content known as “memes” has emerged, characterized by humor and cultural references; this viral content is disseminated online rapidly and is continuously evolving and being integrated into popular online culture [7]. Memes are primarily characterized as being “entertaining and memorable” and their format often differs significantly from conventional language used in daily conversation. For example, on Chinese social media platforms, media articles with attention-grabbing titles often begin with “Who understands, my family members?”. Such titles are used to emphasize the tone but usually do not convey the actual content being discussed in the article. Such memes introduce noise into the text, challenging a model’s ability to accurately extract information. Removing this noise while precisely extracting tourism-related geographic information from the text is crucial for constructing a comprehensive tourism knowledge base.

With the rapid advancement of large language models (LLMs), like the mainstream model ChatGPT [8], they have become valuable tools in everyday life, and numerous AI applications have emerged. LLMs require a lot of data in the training stage, and the latest date generated by these data is usually the time when the training starts, and thus the training data do not cover the latest content. This causes LLMs to “hallucinate” when answering questions outside the scope of their acquired knowledge [9]. Moreover, this training process requires significant investment and takes a long time, often consuming millions of dollars in financial resources, making it a huge challenge to inject up-to-date data into the LLM [10]. The emergence of Low-Rank Adaptation of Large Language Models (LoRA) reduces the cost of fine-tuning the model, and the core idea behind its implementation is to carry out the low-rank decomposition of the updated part of the model parameters to reduce the number of parameters in the calculation, optimize the computational complexity and reduce the hardware requirements [11,12]. Although the use of LoRA can effectively improve training efficiency, the model’s underlying parameter count remains substantial. Consequently, this fine-tuning approach still incurs high costs in scenarios that demand frequent knowledge updates.

Text retrieval aims to find relevant information resources in response to users’ queries, while knowledge-intensive retrieval tasks use deep learning methods [13] to map both queries and documents into dense vectors through neural networks and to automatically learn queries and document representations from data. This approach enables more precise matching of information needs. However, with the increasing model complexity and growing demands for frequent knowledge updates, it continues to face challenges in balancing efficiency and cost. To address the problems of outdated information and hallucinations in LLMs, integrating external knowledge bases has become a solution to enhance the performance of knowledge-intensive retrieval tasks. The integration of retrieval techniques with LLMs has given rise to Retrieval-Augmented Generation (RAG) [14], a method that leverages an LLM to retrieve relevant content, thereby improving the quality of generated text.

Recently, many researchers have focused on combining knowledge graphs with LLMs [15,16,17,18]. Knowledge graphs store information in a structured format, allowing for the better organization and management of internet data. GraphRAG uses knowledge graphs as an information source for LLMs, enhancing model performance and effectively addressing issues such as hallucinations, gaps in domain-specific knowledge, and outdated information [15].

In this paper, we propose a framework called TravelRAG to improve the query performance of large language models in the field of travel. Based on the idea of GraphRAG, this framework provides the latest social platform information and multi-level scenic spot knowledge for the large language model by constructing a tourism knowledge graph, effectively reducing the hallucination problem. We further optimized the retrieval process in RAG by constructing a knowledge graph of scenic spot classification for communities with different interests and generating relevant reports for interested communities to improve the retrieval efficiency and the response accuracy.

To assess the performance of TravelRAG, we conducted a comparative analysis with a traditional RAG pipeline. The results indicate that TravelRAG significantly improves both the retrieval efficiency and accuracy of tourism information retrieval, while also substantially reducing the costs associated with model fine-tuning. Our contributions are as follows:

• We developed an automated pipeline for constructing knowledge graphs based on large language models.
• Adapting to current advancements, we converted a substantial amount of the unstructured text into the multi-layer knowledge graph, utilizing the knowledge graph as the retrieval source in RAG.
• The RAG pipeline, which uses a knowledge graph as its retrieval source, demonstrated a superior retrieval accuracy compared to traditional RAG methods.

2. Related Works

2.1. Knowledge Graph

The knowledge graph is a knowledge base that represents entities in the objective world and their relationships in the form of a graph, enabling intelligent systems to acquire the ability to solve complex tasks [19,20]. The knowledge graph is a network knowledge structure composed of the “entity–relation–entity” triplet and the form of entities and their related attribute–value pairs [21]. The key operations of knowledge graphs include the following three parts: (1) entity and relationship extraction, semantic analysis and other machine learning and natural language processing methods and algorithms; (2) knowledge representation, graph database and knowledge fusion methods and technologies for storing knowledge graphs and (3) data integration and knowledge reasoning in the application of knowledge graphs. Knowledge graphs can be divided into two categories considering the construction process. One construction process is early ontology, which is used in WordNet [22], HowNet [23] and so on. Most of these knowledge maps are handmade by experts, whose knowledge is of high quality and can ensure accuracy and completeness, but their disadvantage is that they are difficult to form on a large scale. The other process is to automatically extract entities and relationships from open internet information to construct the knowledge graphs, as is achieved through Yet Another Great Ontology (YAGO) [19] and DBPedia [24]. The scale of this kind of knowledge graph is large; however, the complexity of the data source and the inaccuracy of the information extraction algorithm may lead to a lot of incomplete information and noise. Knowledge graphs have a better ability to organize, manage and understand information from the internet, and can be used for semantic searches [25,26], intelligent question answering [27] and personalized recommendations [28,29], with certain value in specific fields. The application of knowledge graphs is the current research hotspot in the information field, and they are also a basic technology that can be used to promote the development of artificial intelligence.

Large language models show impressive performance in a variety of natural language processing tasks. These models are pre-trained using a large corpus of text to generate responses that meet user requirements. In order to enable large language models to grasp and copy necessary structural models when exposed to specific context information, many researchers use in-context learning (ICL) [30]. Context learning entails the researcher giving appropriate prompts to the language model and constantly correcting incorrect answers, involving making the large language model answer questions in a specific format or output information in a structured form. The construction of domain-specific knowledge maps requires a large number of hand-labeled corpora, which is very expensive work. Therefore, because large language models can output structured information, some researchers believe that they can be used to produce knowledge graphs. Hu [31] noted that ChatGPT has excellent learning ability with few samples, uses a small amount of data to fine-tune the model, performs subject classification for texts to be processed and extracts entities and relationships to create knowledge graphs. Wei [32] proposed a Knowledge In Context with GPT (KICGPT) framework, a knowledge graph completion framework inherited from a large language model and based on triples. This framework can achieve a better knowledge graph completion effect without additional training overhead. The framework of Attack Knowledge Graph (AttacKG+) proposed by Zhang [33] can extract the defined information effectively and accurately.

Although the performance of large language models is impressive in various tasks, LLMs have great limitations and hidden dangers in the face of complex tasks such as knowledge reasoning. Answer generation is used in large language models to predict the next word on the basis of the last word generated, which means that the generated answers are determined according to probability, so the whole process is uncertain. In addition to using large language models to construct knowledge graphs, some researchers also use knowledge graphs to enhance the ability of large language models. In order to solve the instability caused by LLMs, Agrawal [34] believes that taking a knowledge graph as the external information source of LLMs could effectively solve their hallucination problem. Petroni [35] took a knowledge graph as the knowledge source of a language model, while Dai [36] stored the knowledge graph as a neuron in the transformer [37] architecture of the language model and Choudhrary [38] and Wang [39] used the advantages of knowledge graphs to enhance the reasoning ability of LLMs.

The theme of tourism involves a comprehensive knowledge of geography, history, cuisine, user experience and other aspects. However, current knowledge retrieval on tourism topics usually only results in knowledge in a single field, lacking effective correlation between different fields [40]. Experts and scholars are researching how to extract the tourism-related information scattered in the text and use this information to build a tourism knowledge map. DBtravel [41] is a knowledge graph for English travel generated by the collaborative travel website Wiki-travel. Zhang [42] constructed a Chinese knowledge graph and extracted the tourism-related knowledge graph from the existing textual corpus, while Xiao [43] obtained extensive data from popular travel websites and extracted travel knowledge from them. Liu [44] constructed a knowledge graph of tourist attractions, combined the traditional Q&A model with the fine-grained knowledge graph Q&A model based on BiLSTM-CRF and constructed a knowledge Q&A system. Tan [45] adopted the method based on translation to train a knowledge graph, achieving knowledge vectorization of knowledge graphs to establish the knowledge graph in the tourism field.

2.2. Retrieval Augmentation Generation

The information retrieval (IR) model aims to find information on a large scale and from often unstructured data sources to meet the needs of users; it is often applied to search engines or Q&A and recommendation tasks. IR can be divided into sparse retrieval and dense retrieval [46]. In dense retrieval, documents and queries are usually represented in the form of a tape model; that is, each word is treated as an independent feature and the relevance of the documents and queries is determined mainly by matching their shared terms [47]. However, the sparse retrieval method ignores the semantic relation between words, which may lead to inaccurate retrieval results and semantic mismatch.

The dense retrieval method maps the query and document to a fixed dimension vector space so that they can be represented by a fixed-length single vector [48]. Semantic information is taken into account in the formation of these vectors, solving the semantic mismatch problem to a certain extent. However, this method of mapping text of varying lengths and complexities into a single dense volume may lead to the loss of semantic information, especially when dealing with long text or text with complex sentence structures, where a single vector may struggle to adequately capture the multi-dimensional and multi-level semantic features of the text.

In order to solve this problem, Huang [13] proposed a method to obtain text semantics by using deep learning technology to map queries and documents into dense vectors and automatically learn the representation of queries and documents from the labeled data. A matching function (such as cosine similarity or the dot product) is used to calculate the correlation between queries and documents. Compared with the traditional bag-based retrieval model, this method can achieve semantic-based matching to a certain level. Guo [49] et al. proposed the Deep Relevance Matching Model (DRMM), and they believed that the key to the retrieval task was to evaluate the semantic relevance between queries and documents. These works are considered preliminary explorations of dense retrieval models.

With the advent of Bidirectional Encoder Representations from Transformers (BERT) [50], models’ understanding of language complexity has improved due to its excellent text representation ability and deep semantic understanding ability. It also provides a solid foundation for the development of dense retrieval models. On this basis, researchers have proposed many IR methods based on pre-training models [51]. Karpuhin [52] proposed a Dense Passage Retrieval (DPR) model suitable for open-domain Q&A, while Khattab [53] proposed ColBERT and Santhanam [54] proposed ColBERTv2, which greatly surpass traditional retrieval methods in terms of retrieval accuracy and can calculate the semantic association between documents and queries more accurately.

Although dense searching has achieved great success, it still faces two major limitations [55,56,57]: (1) the Dense Retrieval (DR) model uses an indexed search pipeline with fixed search procedures (MIPS), which makes it difficult to jointly optimize all modules in an end-to-end manner, and (2) learning strategies are often inconsistent with pre-training objectives, which makes it difficult to make the full use of knowledge in pre-trained language models [58]. This kind of traditional retrieval model focuses on “retrieval”, which can obtain accurate information from the entire internet (or other information sources), but usually does not conduct in-depth analyses of the search results. When the user requires relatively complex information, they need to browse multiple results to obtain the required information.

With the excellent understanding and generation ability of large language models, the paradigm of information retrieval has gradually changed to a generative approach [57,59]. This paradigm has had great success, but encounters a “hallucination” problem [10] when processing queries beyond its training data or when current data are required. Therefore, some experts and scholars have proposed using the retrieval-enhanced generation (RAG) method to solve this problem. RAG is a method that combines retrieval-based and generative models [14]. RAG enhances the LLM’s capabilities by retrieving relevant documents from an external knowledge base through semantic similarity calculations. By referencing external knowledge, RAG effectively reduces the problem of generating factually incorrect content [60]. Early search enhancement systems integrated search and generation into an end-to-end system [14,52] using customized datasets to fine-tune small-scale models [50,61]. By designing different prompts, Yu [62] showed that LLMs could retrieve existing knowledge inside the model through prompts. Khattab [63] proposed DSP (Demonstration–Search–Predict), a complex pipeline that relies on a frozen language model and a retrieval model to solve knowledge-intensive tasks. These RAG methods are collectively referred to as traditional RAG frameworks.

Current RAG implementations rely on dense vector similarity searches as a retrieval mechanism. However, this approach of dividing the corpus into blocks of text and relying solely on a dense retrieval system proved inadequate for complex queries [60]. Luo [64] mixed sparse and dense searchers to effectively improve the retrieval effect. However, this type of approach is still limited by the limited range of source data defined in advance by the developer. In addition, achieving the granularity necessary to find answers to complex queries in similar blocks of vector spaces remains a challenge [65]. This inefficiency is due to the method’s inability to selectively locate relevant information, resulting in the retrieval of a large amount of block data that may not directly contribute to answering queries. The best RAG systems will accurately retrieve only the necessary content, minimizing the inclusion of irrelevant information. This is where knowledge graphs (KGs) can help, providing a structured and unambiguous representation of entities and relationships that is more accurate than retrieving information using vector similarity [66].

There has been a lot of research on the intersection of graph-based techniques and LLMs. Giarelis [16] proposed a unified LLM-KG framework to assist fact-checking on public deliberative platforms, which could help solve the problems of hallucination and indecision currently displayed by LLMs. Logan [17] built a KGLM to generate factual sentences by selecting facts from a knowledge graph using the current context. Luo [18] proposed a novel approach known as graph reasoning (RoG), which uses knowledge graphs as a relational path; based on this, interpretable reasoning can be performed by LLMs. The GraphRAG method was proposed by Hu [15]; this method uses an LLM to create a knowledge graph, combining the knowledge graph with RAG and setting different cue enhancement methods in the query. GraphRAG has shown significant advantages in processing hierarchical data and is superior to traditional RAG methods.

3. Methods

From the perspective of administrative divisions, all geographical entities are organized according to a certain level. Specifically, this organizational structure is ordered from macro to micro, that is, “continent–country–province–city–county–township”. The main advantage of this layered approach is its ease of management and retrieval. For example, when planning a travel itinerary, an individual usually takes an interest in a specific scenic spot first, and then gradually refines their interest to the region to which the scenic spot belongs, the specific activities within the region, and the monetary and time costs required for these activities and other elements of different granularity [67]. Different tourists have different requirements for tourism resource retrieval granularity, which requires us to develop new methods to provide more efficient and effective service recommendations with increases in user retrieval granularity. This has always been a core issue in the field of service computing [68].

Further, given the personalized habits of users when working with LLMs, it has become particularly important to meet the needs of different users regarding search granularity. Considering the combination of LLMs and knowledge graphs, the integration of different levels of travel information is crucial to improving user experience. When querying tourism information, users may show diversified needs on a spatial scale. For example, they may pay attention to the spatial relationships between scenic spots, between scenic spots and hotel facilities and between scenic spots and food facilities. Therefore, it is particularly crucial to conduct multi-level tourism resource retrieval according to different use intentions. This multi-level retrieval can not only meet the personalized needs of users for tourism information but also improve the accuracy and efficiency of service recommendations.

The overall structure of the proposed TravelRAG framework is shown in Figure 1. The implementation of TravelRAG involves several key steps: document segmentation, entity extraction, entity linking, and community construction. In this process, we systematically extract entities such as attractions, hotels, restaurants, and activities from documents, link these entities and then organize them into communities based on their respective types. Subsequently, community reports are generated. During the retrieval phase, TravelRAG enables the LLM to efficiently search within these constructed communities based on user queries, allowing for the generation of accurate responses to tourism-related inquiries.

3.1. Travel Graph Construction

3.1.1. Document Chunking

As shown in Figure 2, tourism-related documents often take the form of long texts, where entities mentioned across the document are typically spread over multiple paragraphs and are closely interrelated. To maintain as much contextual integrity as possible, we divide the text into segments that align with the optimal context length for efficient LLM processing. Traditional segmentation methods typically split documents into fixed-length chunks based on predefined values, with no shared context between these segments. This can lead to errors in entity recognition by the LLM, especially when the same entities appear across different segments with shared meanings. To mitigate this issue, we implemented a sliding window approach that incorporates overlapping context paragraphs between segments. This method significantly improves the LLM’s ability to consistently recognize the same entities across different sections of the document.

3.1.2. Entity Extraction

As shown in Figure 3, after completing document segmentation, the LLM will extract entities from the source text as graph nodes. At this stage, we design various prompts to guide the LLM to extract as many different types of entities as possible. Given the variety of tourism-related entities, we need to create a list of entity types to help the LLM extract categories such as attractions, hotels, food, and recreational facilities.

Because the pre-trained model has not been specifically fine-tuned for extraction tasks, we provide several example cases to the LLM as demonstrations. Demonstrating entity extraction examples can significantly improve the LLM’s performance in this task. In addition to extracting entities, the LLM will analyze the context surrounding the entities and generate a summary description, which will serve as the basis for the subsequent entity linking step.

3.1.3. Entity Linking

Knowledge graphs often encounter issues of data sparsity and imbalance. Designing triples for entities with hierarchical relationships is a crucial component of knowledge graph construction, as it can enrich the semantic information of entities and relationships [69]. To address this, researchers leverage the latent semantic information among hierarchically related entities to learn their representations [70]. For example, TransC [71] models entities of different hierarchical levels within the same embedding space, while JOIE [72] assigns separate dimensions to individual instances. Li [73] proposed a three-layer DIK knowledge graph architecture, which adopts a systematic approach to expand and update knowledge at each layer of the knowledge graph.

As shown in Figure 4, when constructing a tourism knowledge graph, the diverse range of entity types makes integrating all these entities into a single, large knowledge graph a daunting and complex task. Moreover, having an excessive number of graph nodes at the same level can complicate the LLM’s retrieval process. For example, if a user query focuses specifically on attractions, searching within a single layer that aggregates all tourism-related information may increase the risk of generating hallucinations.

In the context of same-level entity linking within a knowledge graph, the primary objective is to merge entities that reside at the same hierarchical level and possess similar content or closely related meanings. This approach entails calculating similarity metrics to facilitate the merging or association of analogous entities within this level, thereby aiding in the identification of duplicate information and standardizing the representation of comparable concepts. Such a methodology not only mitigates redundant information but also enhances the overall compactness of the knowledge graph, ultimately leading to more consistent and accurate retrieval outcomes, the formula for which is shown in Equation (1):

$$E=\left\{(n,t,ct)|{\displaystyle \frac{\varphi \left(C_e\right)·\varphi \left(t\right)}{∥\varphi \left(C_e\right)∥·∥\varphi \left(t\right)∥}}\ge {\delta }_{r_1}\right\}$$

(1)

where E represents a collection of entities of the same type; $C_e$ represents the relevant content of the entity, whose composition is expressed as $C_e=[n,ct]$; t represents the candidate entity type; n indicates the name of the entity; $\varphi$ denotes the function that converts the content into a vector and ${\delta }_{r_1}$ is the predefined threshold value for cosine similarity.

3.1.4. Relationship Linking

As shown in Figure 5, after aggregating different entities of the same type, the next step is to link these aggregates with higher-level entities. For example, once we form a “hotel” aggregate from individual hotel entities, we then establish relational links between this aggregate and attraction entities at the first level.

Cross-level associations in knowledge graphs serve to establish semantic links between entities across different levels of abstraction. For example, a tourist attraction can be connected to its broader geographic location or to specific activities offered within it by calculating cross-level semantic similarity. When the semantic similarity between entities at different levels meets a predefined threshold, they are linked to indicate a hierarchical or semantic relationship. These hierarchical associations enable the knowledge graph to construct a well-defined parent–child structure, allowing users to navigate from high-level information down to detailed, specific data. This provides a richer query context and deeper information for more comprehensive insights. In contrast to entity linking, relationship linking focuses on the connections between entities across various hierarchical levels. Consequently, semantic similarity is assessed between entities at different levels rather than between an entity and its corresponding type, the formula for which is shown in Equation (2):

$$R_{e^{L_j}}^{e^{L_i}}=\left\{(e^{L_i},e^{L_j})|{\displaystyle \frac{\varphi \left(C^{L_i}\right)·\varphi \left(C^{L_j}\right)}{∥\varphi \left(C^{L_i}\right)∥·∥\varphi \left(C^{L_j}\right)∥}}\ge {\delta }_{r_2}\right\}$$

(2)

where R denotes a collection that aggregates entities at different levels; L represents the level at which the entity is located and e represents an entity in a collection of different entity types. In order to reduce entity link errors at different levels, we use the entity name n, entity source context $ct$, and its composition, $C=[n,ct]$. $\varphi$ denotes the function that converts the content into a vector and ${\delta }_{r_2}$ is a predefined threshold. By calculating the similarity of entities at different levels, relationships will be linked if they reach the threshold.

In this process, we calculate the degree of both the original and target entities, where the degree refers to the number of edges connected to each entity in the graph. After determining these degrees, we introduce a new variable called “rank”. This rank is computed by summing the degrees of the original and target entities, allowing us to assess the total connectivity between the two entities linked by each edge. This step is crucial for understanding the overall connectivity and significance of relationships within the knowledge graph.

3.1.5. Community Construction and Summary Reports

When a scenic spot entity is connected with its surrounding facility entities, this group of interconnected entities is considered the “community” of that scenic spot. These scenic spot communities are constructed using the Leiden algorithm [74], a method particularly suited for detecting and refining community structures within complex networks. The calculation formula is shown in Equation (3):

$$H={\displaystyle \frac{1}{2m}}{\sum }_c(e_c-\gamma {\displaystyle \frac{K_c^2}{2m}})$$

(3)

where H represents the module in the community; $e_c$ denotes the actual number of edges within interest community c and the expected number of edges is given by ${\displaystyle \frac{K_c^2}{2m}}$, where $K_c$ represents the total degree of the nodes within community c and m is the total number of edges in the entire network. The parameter $\gamma$ controls the resolution: a higher resolution leads to a greater number of communities, while a lower resolution results in fewer communities.

This algorithm is employed to generate a hierarchical structure of entity communities, clustering the entities constructed in the previous step. This approach is a method of navigating knowledge at various levels of granularity. After the different types of scenic spot communities are formed, the LLM is used to summarize these communities. The LLM reviews the relevant information associated with the nodes in each community and generates a report. This report includes a brief introduction to the scenic spot and an overview of the primary supporting facilities within the community, such as hotels, restaurants, and recreational amenities.

3.2. Greedy Matching Retrieval Method

The overall structure of the retrieval pipeline is shown in Figure 6. In the previous section (document chunking and entity extraction), we created the document vector database and entity vector database simultaneously. The LLM extracts the possible entities from the query conditions, and then queries the entity data with the highest semantic similarity in the vector database, as shown in Equation (4):

$$Sim(q_i,c_j)={\displaystyle \frac{q_i·c_j}{∥q_i∥c_j\parallel }}$$

(4)

where $Sim(q_i,c_j)$ denotes the similarity between the query $q_i$ and the candidate $c_j$; $q_i$ presents the i th vectorized query entity and $c_j$ presents the jth vectorized candidate entity.

In the matching process after querying, the greedy matching strategy is applied to select the entity vector from the database that is most similar to the query entity as the optimal matching item. At the same time, considering the importance factor, the impact of specific words on the overall query semantics is determined, the matching score is calculated, and then the most matched items are returned. The calculation process is shown in Equations (5) and (6):

$$Score(q_i,c_i^{*})={\omega }_i·Sim(q_i,c_i^{*})$$

(5)

$$TotalScore(Q,C)={\sum }_{i=1}^n{\omega }_i·Sim(q_i,c_i^{*})$$

(6)

where $Score(q_i^{*},c_i)$ presents the matching score of $q_i^{*}$ and $c_i$; $c_i^{*}$ is the best match of $q_i$; ${\omega }_i$ presents the importance factor of q; $TotalScore$ presents the overall matching score; $Q=\{q_1,q_2,\dots ,q_n\}$ presents the set of all vectorized query entities; $C=\{c_1,c_2,\dots ,c_n\}$ represents the set of entity vectors in the entity vector database.

After receiving the query result, the LLM obtains node data related to these entities from the knowledge graph, including the node itself and the node hierarchy. Then, the communities, text units and relationships most relevant to these node data are found in community reports, text databases, and knowledge graphs. Finally, the LLM presents the query information to the user as a fluent natural language answer.

This method improves the efficiency of the LLM’s search process, improves the hit rate of answers by prioritizing the search community and effectively reduces the hallucination phenomenon.

4. Experiment and Results

4.1. Experimental Data

4.1.1. RAG Data

Tourism has become a popular leisure activity, and there are many factors that influence destination choice, including advertising activities of the destination country, promotional activities of travel agencies, airfare discounts, movies or TV programs, personal preferences and so on [75]. Due to the prevalence of network technology, the internet has become the main channel for people to find and disseminate information. Text data, as one of the main formats of internet data, are a form in which users express opinions and evaluations effectively and widely. A large number of online reviews for tourism websites, hotels, and services indicate this [76]. The travel decisions of tourists are greatly influenced by the travel experiences of others, which are presented in the form of travel reviews or blogs [77]. These texts can provide valuable insights to potential tourists and help them optimize their destination choices and explore travel routes [78]. It is worth noting that these articles often include locations and details not covered by traditional tourist routes, making them valuable resources for revealing little-known tourist attractions and information [79]. Therefore, we collected travel blogs on “Wuzhen” from domestic tourism websites such as Ctrip and MaFengWo in China, which were written by real tourists who had visited Wuzhen.

We used a crawler to retrieve text from the website, as internet text usually has no fixed format, we need to use relevant code to clean the text, delete unnecessary text and incorrect characters, and retain the basic structure of the article, namely “title-author-time-content”. Finally, we sorted the completed dataset parameters as shown in the following

Table 1. The number of different types of questions.

Content Structure	Number of Articles	Total Word Count
Title–author–time–content	100	80,983

.

4.1.2. Test Data

We employed a large language model to assist in generating the test sets. Our prepared RAG dataset was input into the model, allowing it to read and process the data and subsequently generating questions, answers and the corresponding source text from which the answers were derived. The Q&A in the test sets are categorized into four types, Simple, MultiContent, Reasoning, and Conditional, each representing different levels of complexity and requirements for the model’s understanding and response generation. The distribution of the number of problems is shown in

Table 2. The number of different types of questions.

Type	Simple	MultiContent	Reasoning	Conditional
Number	50	20	20	10

.

• Simple—This type of question answering indicates that the LLM can directly extract the answer from the context.
• MultiContent—This type of question answering indicates that the LLM needs to examine multiple documents to synthesize an answer.
• Reasoning—This type of question answering indicates that after receiving the query, the LLM must perform some reasoning based on the source text to provide an answer.
• Conditional—This type of question answering indicates that the LLM may need to respond under specific constraints.

4.2. Environments

Our experimental environment is shown in

Table 3. The table includes the hardware model and software version used.

Hardware and Software	Configuration	Detailed Information
Hardware	CPU	Intel Core i9-12900KF
	Memory	64 GB
	Graphics Card	NVIDIA GeForce RTX 3090 Ti 24 GB
Software	System	Ubuntu 20.04.6 LTS CUDA 11.8
	Software	Python 3.10.13 PyTorch 2.1.2

.

4.3. Models

We used the Qwen2 [80] and GPT-4 [81] series models as benchmark models. The Qwen2 series, developed by Alibaba Cloud, consists of large-scale pretrained language models with parameter sizes of 7 B, 57 B and 72 B. A significant portion of the training data for this series consist of Chinese corpora, resulting in strong performance on Chinese tasks. The GPT-4 series, developed by OpenAI, represents the latest generation of large-scale AI language models, featuring 200 B parameters. The GPT-4 model has enhanced capabilities for understanding long texts, enabling the generation of more coherent and consistent outputs. When constructing the traditional RAG method as a baseline, we selected the BGE-embedding model for text-to-vector conversion in the text embedding stage [82]. The BGE-embedding model outperforms all other community models in terms of semantic retrieval accuracy and overall semantic representation in both Chinese and English. Furthermore, it supports processing at multiple levels of granularity, accommodating input text with varying degrees of detail.

4.4. Evaluation Metrics

We used RAGA (Retrieval Augmented Generation Assessment), a reference-free assessment framework for RAG pipelines [83]. Evaluating RAG architectures is challenging because there are several aspects to consider: the ability of the retrieval system to identify relevant and focused contextual paragraphs, the ability of the LLM to utilize these paragraphs accurately, and the quality of the generation itself. We evaluated the retrieval effectiveness of large language models using the following four metrics.

Faithfulness evaluates the factual consistency of the generated answer with the provided context, based on the annotated ground truth and the context retrieved by the LLM from the original text. The calculation formula is shown in Equation (7):

$$Faithfulness={\displaystyle \frac{\left|I\right|}{\left|C\right|}}$$

(7)

A higher score indicates a greater alignment between the answer and the retrieved context. In this formula, I represents the number of claims in the generated answer that can be inferred from the given context and C denotes the total number of claims in the generated answer. This metric is based on the annotated ground truth and the context retrieved by the LLM from the original text.

Answer Relevancy is the relevance of the answer and is evaluated based on how closely it matches the user-provided query. A higher score indicates a closer alignment between the answer and the query. This score is calculated using cosine similarity. The calculation formula is shown in Equation (8):

$$AnswerRelevancy={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}cos\left(E_{gi},E_o\right)$$

(8)

where $E_{gi}$ represents the embedding of the generated question i, $E_o$ represents the embedding of the original question and N denotes the total number of generated questions.

Contextual Precision assesses whether the factually relevant entries presented in the context are ranked higher. Ideally, all relevant document segments should appear at the top. This metric is calculated based on the question and its corresponding contexts, with values ranging from 0 to 1, where a higher score indicates greater precision. The calculation formulas are shown in Equations (9) and (10):

$$ContextualPrecision@K={\displaystyle \frac{{\sum }_{k=1}^K\left(Precision@k\times v_k\right)}{\left|\mathrm{Relevant}\mathrm{Items}\mathrm{in}\mathrm{Top}\mathrm{K}\right|}}$$

(9)

$$Precision@K={\displaystyle \frac{\mathrm{True}\mathrm{Positives}@\mathrm{k}}{\mathrm{True}\mathrm{Positives}@\mathrm{k}+\mathrm{False}\mathrm{Positives}@\mathrm{k}}}$$

(10)

where $ContextualPrecision@K$ represents the precision metric that assesses the ranking of factually relevant entries within the top-K context results. This precision metric emphasizes placing relevant document segments at higher ranks. K denotes the number of chunks in the context considered and $v_k\in \{0,1\}$ indicates whether the k-th ranked chunk is relevant (1) or not (0).

Context Recall measures the alignment between the retrieved context and the ground truth. Its values range from 0 to 1, where a higher score indicates better performance. In order to estimate context recall based on the ground truth, each sentence in the ground truth is analyzed to determine whether it can be matched with the retrieved context. Ideally, all sentences in the ground truth should be matched with the retrieved context. The calculation formula is shown in Equation (11):

$$ContextRecall={\displaystyle \frac{\left|A\right|}{\left|T\right|}}$$

(11)

A higher score indicates better recall performance. In this formula, A represents the number of claims in the ground truth that can be matched with the retrieved context and T denotes the total number of claims in the ground truth. Ideally, all claims in the ground truth should have corresponding matches within the retrieved context.

4.5. Baseline

To evaluate the effectiveness of our proposed model, we compare it with the NaiveRAG framework. The NaiveRAG research paradigm (which we call “traditional RAG” in this article) represents the earliest approach, which follows the traditional process of indexing, retrieval, and generation, also known as the “search–read” framework [84]. The traditional RAG construction process can be summarized in the following three points:

Indexing: Raw data in formats like PDF, HTML, Word and Markdown are cleaned, converted to plain text and divided into smaller chunks. These chunks are encoded into vectors using an embedding model and stored in a vector database, enabling efficient similarity searches.

Retrieval: The system encodes each user query as a vector and computes the similarity with stored vectors, retrieving the top K most similar chunks as context for the response.

Generation: The query and retrieved chunks are combined into a prompt for a large language model, which generates a response. In a conversation, previous dialogue history can be included to support multi-turn interactions.

4.6. Results

4.6.1. Travel Knowledge Graph

As illustrated in Figure 7, we constructed a knowledge graph of tourist attractions using the previously described method to improve the reasoning capabilities of large language models in retrieval tasks. This knowledge graph was constructed using GPT-4. The knowledge graph was visualized using Gephi (https://gephi.org/, accessed on 12 November 2024, version 0.1.0), which is an open-source software for graph and network analysis. It works well with large network diagrams and is an ideal tool for studying complex network structures. Due to its multi-layer structure, it was projected into a two-dimensional graph, with entities at different hierarchical levels represented by circles of varying sizes. Since the input corpus consists of travel notes about Wuzhen, the top-level entity in the knowledge graph is “Wuzhen”, while the interest groups related to tourist attractions within Wuzhen are distributed across subsequent levels, and the specific entities within these attractions are represented at even lower hierarchical levels.

Table 4. The results of extraction.

Type	Numbers
Entities	2081
Relations	20
Relational Triples	1187

shows the number of entities, relationships and triples extracted. Figure 8 presents the entities extracted from the travel notes corresponding to the figures, as well as the relationships among these entities. During the extraction process, the LLM also provides detailed descriptions for the extracted entities. When performing entity linking, the LLM simultaneously analyzes and summarizes the relationships among the entities to enhance the retrieval efficiency in future tasks.

4.6.2. Metric Comparison

Our experimental results are shown in

Table 5. Performance comparison between TravelRAG and the traditional RAG pipeline.

Method	Size	Faithfulness	Answer Relevancy	Context Precision	Context Recall
Qwen2-RAG	72 B	0.55	0.59	0.70	0.79
GPT-4-RAG	180 B	0.65	0.67	0.80	0.87
TravelRAG	base on GPT-4	0.76	0.87	0.85	0.60

. Since the source text is in Chinese, we used the Qwen2 model, known for its strong support for Chinese, and GPT-4 as the foundation models in the comparative experiments. Compared to the traditional RAG pipeline, the proposed TravelRAG framework demonstrates significant improvements in retrieval performance, indicating that leveraging a tourism knowledge graph enables the LLM to enhance the retrieval effectiveness for tourist attractions. However, TravelRAG underperforms in the context recall metric, which warrants further investigation. During the retrieval process, the LLM first queries the descriptions of entities and their relationships stored in the tourism knowledge graph. These descriptions are summaries generated by the LLM after reading the travel notes, rather than direct excerpts from the original text. This discrepancy results in a lower context recall value compared to the traditional RAG pipeline. Detailed case analyses will be provided later in Section 4.6.4.

4.6.3. Ablation Study

We conducted ablation experiments with varying model parameter sizes to investigate the impact of parameter size on the performance of TravelRAG. The experimental results, as shown in the

Table 6. The table presents the experimental results of the TravelRAG framework constructed using models with varying parameter sizes, with the results evaluated using four metrics.

Method	Size	Faithfulness	Answer Relevancy	Context Precision	Context Recall
Qwen2-TravelRAG	7 B	0.32	0.21	0.32	0.15
Qwen2-TravelRAG	57 B	0.57	0.43	0.63	0.38
Qwen2-TravelRAG	72 B	0.69	0.78	0.75	0.71
GPT-4-TravelRag	180 B+ 1	0.76	0.87	0.85	0.60

¹ The exact parameter size of the model is not officially reported and has been inferred through estimates made by other researchers [85].

, indicate that as the number of model parameters increases, the accuracy of retrieval performance improves progressively across the four evaluation metrics. Analyzing the construction process of TravelRAG, we find that the need to provide lengthy prompts and sample outputs to the LLM before each stage of knowledge graph construction presents significant challenges for the model’s comprehension. For instance, during the entity extraction phase, the LLM must extract entities and their attributes from unstructured text and output them into a structured JSON format. Smaller models may struggle to extract entities comprehensively, and models not fine-tuned for generating data in JSON format may produce incorrect outputs, potentially leading to failure in subsequent stages of the construction process that depend on specific data formats.

Due to the varying proportions of languages in the pre-training corpora of large language models, their retrieval performance for Chinese queries differs. To investigate the differences in retrieval results between models with different language capabilities, we designed four categories of questions: Simple, MultiContent, Reasoning, and Conditional. The test structure of the two methods on the four types of questions was determined by assessing the accuracy of the answers, which involves measuring how closely the generated answers match the ground truth answers. This metric is derived by calculating answer accuracy, as detailed in Equations (12)–(14):

$$Score=Similarity(Answer,Groundtruth)0\le Similarity\le 1$$

(12)

$$f\left(score\right)=\left\{\begin{array}{c}TrueAnswer,score\ge 0.5\hfill \\ FalseAnswer,score<0.5\hfill \end{array}\right.$$

(13)

$$p={\displaystyle \frac{NumberofTrueAnswers}{NumberofQuestions}}$$

(14)

where $Score$ indicates the degree of similarity between the answer and the ground truth, ranging from 0 to 1. A threshold is set to filter the results, with a score of 0.5 or higher considered a true answer, and a score below 0.5 considered a false answer. p is used to denote the accuracy of the answers.

As shown in

Table 7. The experimental results were evaluated on four types of questions, based on models pre-trained with varying proportions of Chinese in their corpora.

Method	Size	Simple	MultiContent	Reasoning	Conditional
Qwen2-TravelRAG	72 B	0.89	0.57	0.65	0.54
GPT-4-TravelRag	180 B	0.88	0.43	0.72	0.78

, for simple questions, the performance gap between the RAG systems built with the two models is small, but for other categories, the results are mixed. In MultiContent retrieval, the RAG system built using Qwen2 performs better, while in Reasoning and Conditional retrieval, the GPT-4-based system performs better. Thus, we conclude that models pre-trained with a higher proportion of Chinese corpus are more effective when answers need to be retrieved and aggregated from multiple sources, whereas models with larger parameter sizes excel in tasks involving logical reasoning and multi-constraint queries.

4.6.4. Case Analysis

As illustrated in Figure 9, we conducted a case study to thoroughly analyze how the proposed multi-layer tourism knowledge graph guides the retrieval process in the LLM. We posed the same query, which included conditional constraints, to both the traditional RAG pipeline and TravelRAG framework; the search was limited to attractions “suitable for families and students”.

When a user submits a query, the traditional RAG pipeline searches its vector database for the most relevant match but lacks the ability to perform reasoning-based retrieval that aligns with the user’s specific needs. As shown in the Figure 9, the response from the traditional RAG system is a full sentence but contains factual inaccuracies, with incorrect terms highlighted in red. In contrast, TravelRAG’s response is more logically structured. After constructing the tourism knowledge graph and using it as a source of information, TravelRAG retrieves data step-by-step from the multi-layer knowledge graph, guided by the specified conditions. It begins by searching for tourist attractions within the Xitang community and then lists relevant spots that meet the search criteria, explaining the details of each site to help users determine whether the results match their needs. Experimental results show that the TravelRAG framework can effectively respond to queries based on specific constraints.

5. Conclusions

A framework for tourist site retrieval is presented in this paper that integrates knowledge graphs into a large language model (LLM). This framework leverages an LLM to construct a knowledge graph for the tourism domain from unstructured text gathered from social platforms. By organizing entities into hierarchical communities, where each community contains all relevant entities, the framework builds a multi-layer knowledge graph. This knowledge graph also acts as the retrieval source for the LLM, guiding the model to generate answers based on this structured information. This approach enhances the accuracy of the LLM in responding to tourism-related queries and addresses the issue of hallucinations or fabricated information that are often produced by LLMs. Travelogues shared by users on social platforms often do not have a standardized format, and their writing styles vary widely, posing substantial challenges for smaller language models in terms of reading comprehension. Additionally, LLMs struggle to update their internal knowledge base promptly in response to the rapidly increasing availability of information online. Compared to traditional Retrieval-Augmented Generation (RAG) pipelines, the proposed TravelRAG framework demonstrates significant improvements in retrieval accuracy. The experimental results show that, with the support of the multi-layer knowledge graph, TravelRAG provides users with more accurate, detailed and contextually relevant responses to their queries. The main limitations of TravelRAG are as follows: on one hand, although LLMs have improved in understanding long contexts, they still struggle to accurately parse content in travelogues, which are rich in complex semantic information. On the other hand, the construction of the knowledge graph depends on manually crafted prompts, which increases the labor cost of creating prompt templates.

The focus of future work will be exploring how to further improve the reasoning ability of LLMs under complex conditions and for multi-hop problems when a knowledge graph is used as their information source.