Dental Loop Chatbot: A Prototype Large Language Model Framework for Dentistry

Abstract

The Dental Loop Chatbot was developed as a real-time, evidence-based guidance system for dental practitioners using a fine-tuned large language model (LLM) and Retrieval-Augmented Generation (RAG). This paper outlines the development and preliminary evaluation of the chatbot as a scalable clinical decision-support tool designed for resource-limited settings. The system’s architecture incorporates Quantized Low-Rank Adaptation (QLoRA) for efficient fine-tuning, while dynamic retrieval mechanisms ensure contextually accurate and relevant responses. This prototype lays the groundwork for future triaging and diagnostic support systems tailored specifically to the field of dentistry.

1. Introduction

The delivery of best practice dental care is challenged by operator-dependent biases and variable approaches to clinical decision-making. Inter-operator agreement on diagnoses and prognoses ranks low to moderate for routine cases and even lower for complex clinical assessments [1]. Agreement is further hindered when clinical record keeping is not standardized such as in private dental practices operating under different constitutions. A practitioner’s experience and geographical practice location greatly influence these variations [2]. Historically, access to epidemiological data and equitable resource allocation has been limited by geographical boundaries and monetizing incentives, preventing a globalized approach to dental practice [3].

The dot-com bubble revolutionized access to the internet, and consequently to scholarly articles and treatment guidelines, providing a first step toward standardizing dental care [4]. However, relying on these resources in real time can undermine patient confidence, especially for younger practitioners. Over time, experience helps to bridge knowledge gaps, but it remains a challenge to quickly extract relevant insights from the vast amount of available literature.

With the introduction of large language models (LLMs) such as ChatGPT and Microsoft Co-pilot in 2021, practitioners gained access to real-time, conversational tools trained on the scholarly literature, best practice guidelines, and large datasets. These generative artificial intelligence (AI) models offer diagnostic aids and help to educate patients by providing data-driven prognoses. Despite their potential, the field of dentistry lacks effective methods to extract and apply the vast amounts of available data. Moreover, real-time access to these tools is hindered by the lack of advanced computational hardware in underserved regions, where dental care is most needed [5].

Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing, excelling in understanding contextual nuances [6,7,8,9]. However, their integration into dentistry, particularly guided by evidence-based best practices, remains unexplored. The present study applied the LLM LLama2 [6], trained on a corpus of dental guidelines and published materials, to create a chatbot for dentistry. Using branching logic in simulated conversations between practitioners and patients, coupled with billions of open-source parameters, this is one of the first studies to implement a dental chatbot framework for a scalable autonomous support system. The current research documents the preliminary development of the framework which is an open-source expert system with an integrated LLM. The Chatbot is a part of the Dental Loop Research Initiative and follows the success of software-based Dental Loop SnP research, which focused on quantitative assessments of jaw movement during speech [10].

Table 1. Metadata.

Nr	Code Metadata Description	Source Details
C1	Current code version	V1.0
C2	Permanent link to code/repository used for this code version	https://github.com/faisalahmedsifat/Dental-Loop-Chatbot (Accessed on 17 December 2024)
C3	Permanent link to reproducible capsule	https://codeocean.com/capsule/6580331/tree/v1 (Accessed on 17 December 2024)
C4	Legal code license	Apache 2.0
C5	Code versioning system used	none
C6	Software code languages, tools and services used	Python, Llama-Index, HuggingFace
C7	Compilation requirements, operating environments and dependencies	Operating System: Windows
C8	If available, link to developer documentation/manual	N/A
C9	Support email for questions	[email protected] [email protected]

provides details on the metadata for the developed framework.

2. Literature Review

The use of artificial intelligence in medicine began gaining traction in the early 2000s with decision-support systems that employed basic branching probabilistic models to assist in diagnostics [11]. However, these systems lacked a conversational interface [11]. By the 2010s, decision-support tools were primarily developed using Visual C# and relied on simple macro buttons and input–output system architectures [12]. These tools assisted clinicians and researchers by retrieving pharmacological data and medical guidelines, streamlining access to information that previously required extensive memorization. The introduction of specialized search engine chatbots further improved efficiency [12,13].

A significant shift occurred in the late 2010s as researchers began transitioning from rule-based systems to AI-powered, self-improving models. This evolution addressed the limitations of fixed-output text generation that characterized earlier medical chatbots and decision-support systems [14,15]. The early 2020s marked another turning point with the rapid advancement of large language models (LLMs) and natural language processing (NLP) systems like ChatGPT (http://chat.openai.com/) [16].

These modern AI models demonstrated remarkable improvements in accuracy and reliability. A recent investigation into the application of LLMs in medicine found their responses to be “nearly all correct” or “completely correct”, highlighting their potential as robust tools for medical decision-making [17].

AI-driven chatbots in dentistry emerged significantly later than their counterparts in medicine, primarily because earlier dental chatbots were rule-based decision-support systems. These systems gained traction during the COVID-19 pandemic, as the dental industry faced widespread closures, which were far more severe than those in medicine. However, their functionality was largely limited to promoting oral hygiene education [18]. Even as late as 2023, most dental chatbots relied on decision trees with predefined outcomes [19].

The first dental chatbots leveraging AI-driven natural language processing (NLP), specifically using GPT-4, were introduced in 2024 to process dental imaging guidelines [20]. Despite this progress, the field remains nascent. By 2024, reputable journals were primarily publishing pilot studies exploring the application of general large language model chatbots, such as Google Bard and ChatGPT, in dentistry. These studies highlighted significant limitations, as the chatbots often performed poorly in effective communication when compared to human dental specialists [21].

Some proprietary systems developed in 2024 featured NLP architectures with undisclosed or closed-source designs. These systems produced standardized responses and were trained on approximately 1300 internal sources of dental information, focusing narrowly on implants and dentures [22]. More recent efforts have involved using browser-based, low-code NLP models with multi-factor authentication, but these remain limited in scope [23].

This documentation represents one of the first implementations of a dental chatbot based on the open-source Llama language model. This approach enhances accessibility and ensures the system is specifically trained on comprehensive general dentistry guidelines, making it more generalized compared to previous systems that focused on narrow aspects of dental care. By addressing the lack of trust in AI systems, which are often criticized for their “black box” nature and lack of transparency, this model aims to build practitioner confidence. Notably, feedback from 2024 community engagement sessions demonstrated user reservations about chatbot decision-making skills and the perceived opacity of AI systems [24].

3. Materials and Methods

The Dental Loop Chatbot is designed to provide real-time, evidence-based clinical guidance for dental professionals. The system is powered by the Llama2 language model, initially fine-tuned with a curated dataset of authoritative dental guidelines. These include standards from reputable organizations such as the Australian Dental Association [25], the British Dental Association [26,27,28], the Scottish Dental Clinical Effectiveness Programme [29], the Royal College of Surgeons of England, the National Institute of Health Library, and the International Association of Dental Traumatology [30,31,32].

Preliminary results indicate that integrating additional guidelines and textual resources can further enhance the system’s accuracy and versatility. Since most local guidelines are adaptations of internationally recognized standards, the model’s framework is inherently adaptable. This adaptability ensures that the system can align with localized guidelines, even when they deviate from established international protocols, offering a robust foundation for broader applicability.

The chatbot operates on a fully implemented interactive pipeline, allowing users to input natural language queries and receive detailed, context-aware responses. It combines its fine-tuned model with Retrieval-Augmented Generation (RAG) [33,34], pulling relevant documents from its knowledge base to provide precise answers to even complex dental queries. The system is further optimized using the QLoRA (Quantized Low-Rank Adaptation) fine-tuning technique, enabling effective performance in resource-limited environments [35]. Its modular design allows for customization and expansion, enabling users to adapt it for specific clinical needs or integrate additional datasets. Responses can be exported as ‘.csv’ files for documentation or review.

3.1. Software Architecture

Figure 1 illustrates the complete workflow of the Dental Loop Chatbot, showcasing the processes involved in knowledge acquisition, model fine-tuning, response generation, and evaluation. The workflow begins with the collection of authoritative dentistry guidelines and textbooks that are widely followed by dental practitioners. These texts are extracted, pre-processed, and cleaned to create a structured text corpus. The corpus serves as the foundational knowledge for fine-tuning the chatbot and forms the basis for evaluating its performance in adhering to dental guidelines.

The chatbot utilizes LLaMA2 7B and 13B, two versions of an open-source large language model (LLM). These models are selected for their strong natural language understanding capabilities. LLaMA2 7B is lightweight and resource-efficient but performs less effectively in complex reasoning tasks compared to the larger LLaMA2 13B model, which demonstrates improved accuracy and reliability. Both models are fine-tuned using Quantized Low-Rank Adaptation (QLoRA), a technique that updates a minimal number of parameters while keeping the majority of the pre-trained weights frozen. This enables efficient adaptation to domain-specific knowledge without requiring significant computational resources. The result of this process is the Knowledge Fine-Tuned LLM, which incorporates essential dental knowledge directly into its parameters.

To enhance the system’s ability to retrieve relevant information dynamically, a Retrieval-Augmented Generation (RAG) system was employed. The text corpus was divided into smaller chunks, and vector representations of these chunks were generated using the all-mpnet-v2 embedding model. These embeddings were stored in a vector database, where similar texts were positioned closer together in the vector space. During query processing, the RAG system retrieved the most relevant chunks from this database and fed them to the LLaMA2 model, enabling it to generate accurate and contextually informed responses. In one configuration, the RAG embeddings were not fine-tuned, relying instead on the pre-trained embeddings to retrieve relevant chunks.

An alternative approach involved fine-tuning the all-mpnet-v2 embedding model on the same text corpus to improve the relevance of the retrieved chunks. This ensured that the RAG system was better aligned with domain-specific knowledge, allowing for more precise retrieval and ultimately improving the chatbot’s ability to answer queries. The effectiveness of both configurations, i.e., RAG without fine-tuning and RAG with fine-tuned embeddings, was assessed.

To measure how well the chatbot performs in adhering to the knowledge derived from the texts, we developed a benchmark called DentFactCheckBenchmark15 (Figure 2). This benchmark consists of 15 randomly generated questions derived from the original dental guidelines stored as text corpus. The chatbot’s responses to these questions were assessed to evaluate its ability to retain and apply factual knowledge from the training data. After training the chatbot using various techniques, the generated answers were evaluated by human experts and a superior LLM to ensure factual accuracy and alignment with the guidelines. These evaluations were scored, and the results are presented in a performance table in the results section.

3.2. Software Functionalities

• The system combines parametric knowledge from the fine-tuned LLaMA2 model with real-time document retrieval through Retrieval-Augmented Generation (RAG), ensuring that responses are backed by up-to-date dental knowledge.
• The LLaMA2 model is fine-tuned using QLoRA, a method that reduces memory overhead, enabling the chatbot to function efficiently even in resource-limited environments while maintaining accuracy in dental domain responses.
• The RAG system retrieves relevant documents from a pre-indexed dental knowledge database using a fine-tuned all-mpnet-base-v2 embedding model, ensuring that the chatbot provides contextually accurate answers to specific dental queries.

4. Results

The factual accuracy of the Dental Loop Chatbot under various fine-tuning configurations was cross-referenced over 15 random facts extracted from international guidelines. For the LLaMA2 7B model, base instruction tuning scored 3/15, while QLoRA fine-tuning achieved 2/15. The RAG system with pre-trained embeddings improved the performance to 5/15, and fine-tuning the embeddings raised the score to 7/15. The combination of QLoRA fine-tuning and RAG embedding fine-tuning delivered the best performance at 10/15. For the LLaMA2 13B model, RAG achieved 5/15, improving to 7/15 with fine-tuned embeddings.

5. Discussion

The significance of the Dental Loop Chatbot lies in its pioneering application of large language models to the dental field, aiming to bridge the gap in evidence-based clinical decision support. This research focuses on the efficacy of fine-tuning techniques, such as QLoRA and RAG, to ensure that the chatbot retains and delivers accurate dental knowledge drawn from guidelines and textbooks. The research also highlights how these advanced LLM techniques can help to extend dental expertise, even in resource-limited environments, providing accessible decision support to practitioners across diverse healthcare settings, including underserved regions.

The initial goal was to develop a diagnostic chatbot for dentistry, but as the project progressed, it evolved into a broader study exploring the potential of enhancing LLM capabilities for dental decision support. By injecting domain-specific knowledge from dental guidelines and textbooks, the research aimed to determine how well LLMs like LLaMA2 could be fine-tuned to retain and accurately present this information. With further data integration and model refinement, the chatbot has the potential to become a real-time triaging tool for clinics, offering evidence-based reassurance to both practitioners and patients [36,37]. Incorporating the chatbot into mobile systems could help identify, categorize, and prioritize dental emergencies, potentially leading to a fully standardized and autonomous triaging system.

Currently, the Dental Loop Chatbot functions as a chat-based rapid information retrieval tool, focused on delivering evidence-based answers sourced from dental texts. Some limitations were identified during evaluations: the quantized 7B model exhibited challenges with reasoning and struggled with consistent responses, whereas the 13B model showed more reliable accuracy. One of the key limitations highlighted is the absence of fine-tuning using Retrieval-Augmented Dual Instruction Tuning (RA-DIT), which could further enhance the chatbot’s retrieval capabilities and contextual awareness [38]. Future research will explore these additional fine-tuning methods to improve the system’s overall performance.

5.1. Addressing Data Privacy Concerns

At its current prototype stage, the framework serves as an interactive pipeline for fine-tuning LLMs, refining RAG adapters, and evaluating responses to dental queries. By design, no patient-specific data can be stored when processed, thereby minimizing privacy concerns. However, we recognize that any future transition to a scaled product would need to address significant privacy and security aspects, which must be implemented with appropriate ISO certifications.

While blockchain has been proposed as a potential solution for ensuring data privacy, it is not applicable to the current prototype due to its focus on real-time processing of data. Instead, the system employs secure software development practices, restricts access to its knowledge base, and adheres to privacy-preserving retrieval methods.

5.2. Limitations

The primary limitation of this study is the inability to conduct a validation test, primarily due to the lack of a scalable knowledge base and a user-friendly interface for practitioners to use in real time while feedback is collected on the backend. As highlighted, most studies in 2024 continue to explore large language models (LLMs) developed by well-funded corporations with expansive databases supporting their knowledge. Future recommendations include conducting real-time validation once appropriate ethical approvals are sought and integrating a more extensive knowledge base of patient records in collaboration with the local dental industry.

6. Conclusions

The Dental Loop Chatbot serves as an initial step for providing real-time, evidence-based dental guidance to practitioners. Its modular design allows for scalability and continuous improvement. The system is designed to perform in resource-limited settings such as remote clinics and serves as a foundational step toward automated triaging and clinical support.