Enhancing Anesthetic Patient Education Through the Utilization of Large Language Models for Improved Communication and Understanding

Abstract

Background/Objectives: The rapid development of Large Language Models (LLMs) presents promising applications in healthcare, including patient education. In anesthesia, where patient anxiety is common due to misunderstandings and fears, LLMs could alleviate perioperative anxiety by providing accessible and accurate information. This study explores the potential of LLMs to enhance patient education on anesthetic and perioperative care, addressing time constraints faced by anesthetists. Methods: Three language models—ChatGPT-4, Claude 3, and Gemini—were evaluated using three common patient prompts. To minimize bias, incognito mode was used. Readability was assessed with the Flesch–Kincaid, Flesch Reading Ease, and Coleman–Liau indices. Response quality was rated for clarity, comprehension, and informativeness using the DISCERN score and Likert Scale. Results: Claude 3 required the highest reading level, delivering detailed responses but lacking citations. ChatGPT-4o offered accessible and concise answers but missed key details. Gemini provided reliable and comprehensive information and emphasized professional guidance but lacked citations. According to DISCERN and Likert scores, Gemini had the highest rank for reliability and patient friendliness. Conclusions: This study found that Gemini provided the most reliable information, followed by Claude 3, although no significant differences were observed. All models showed limitations in bias and lacked sufficient citations. While ChatGPT-4o was the most comprehensible, it lacked clinical depth. Further research is needed to balance simplicity with clinical accuracy, explore Artificial Intelligence (AI)–physician collaboration, and assess AI’s impact on patient safety and medical education.

1. Introduction

The swift progression of Artificial Intelligence (AI) has given rise to Large Language Models (LLMs), sophisticated algorithms that leverage deep learning techniques and extensive datasets to comprehend and generate information across a wide spectrum of topics, presenting it in a manner that closely emulates human communication [1,2]. The capabilities of LLMs have prompted evaluations of their potential application within the healthcare sector, where their aptitude for learning and pattern recognition presents promising opportunities for tasks such as diagnostic formulation, genotype analysis, and treatment planning [1,3]. Patient education and counseling represent a particularly promising domain for the application of LLMs, owing to their advanced natural language processing (NLP) capabilities, enabling them to generate human-like responses by predicting words based on vast training data [2,3].

Anesthesia is a topic that can induce anxiety and apprehension in patients, primarily stemming from a limited understanding of the underlying procedures, fear of mortality, and concerns about potential side effects [4]. This sense of unease is compounded by global studies revealing widespread confusion among patients regarding the role of anesthetists, the common risks associated with anesthesia, and enduring misconceptions about postoperative side effects such as postoperative pain [5,6,7,8]. Such uncertainty can contribute to considerable perioperative anxiety, which poses a substantial challenge given its potential to negatively impact postoperative outcomes and increase morbidity [4].

This underscores the critical importance of patient education, which, when delivered effectively during the perioperative period, can provide essential psychosocial support and alleviate preoperative anxiety, ultimately leading to improved postoperative outcomes [9]. Anesthetists’ requirements to provide patient education and relieve preoperative anxiety can be limited by the time constraints of other clinical and non-clinical duties. The advent of LLMs offers a potential solution to mitigate these pressures, enabling the efficient delivery of patient education without overburdening anesthetists. This study seeks to explore the potential of LLMs to provide comprehensive and patient-friendly information to patients regarding anesthetic and perioperative care.

2. Materials and Methods

Three language models—ChatGPT-4o (Open AI, San Francisco, CA, USA), Claude 3 (Anthropic, San Francisco, CA, USA), and Gemini (Google, Mountain View, CA, USA)—were rigorously assessed by using a set of fifteen distinct prompts (

Table 1. Prompts input into each LLM.

Prompt Number	Prompt
1	Why should I fast before anesthesia?
2	How long before my anesthesia should I give up smoking?
3	How many standards of alcohol per day is acceptable before my anesthesia?
4	What medications should I stop taking before my anesthesia? And how long before my anesthesia should I stop taking them?
5	I am an insulin-dependent diabetic, should I still take my insulin before anesthesia?
6	I have atrial fibrillation, is it safe for me to have anesthesia?
7	How is the anesthetic administered?
8	Will I have pain after the procedure with anesthesia?
9	How long will it take for the anesthesia to wear off?
10	What is the difference between anesthesia and sedation?
11	Is there anything I should avoid after having anesthesia?
12	What are the risks of anesthesia?
13	Will anesthesia stop me from breathing?
14	What if I wake up in the middle of the anesthesia?
15	How will anesthesia affect my pregnancy?

). The models were selected for their extensive popularity among users, each attracting millions of monthly visitors [10,11]. Additionally, their free-to-use accessibility makes them the most probable LLMs to be utilized by the general public. The authors Ratnagandhi and Lim (Junior Medical Officers) collaboratively determined that these prompts encompassed the most frequently asked questions from patients. To minimize potential bias related to prior online behavior, this study employed incognito mode in the browser. Consistency in the testing procedure was ensured by having a single author present all prompts on the same day, utilizing the same device and account.

A set of evaluation metrics was considered—for assessing readability, we utilized the Flesch–Kincaid Grade Level (FKGL), Flesch Reading Ease Score (FRES), and Coleman–Liau Index (

Table 2. Readability and reliability Scores.

LLM	Prompt No.	FKGL	FRES	Coleman–Liau Index	DISCERN Score
ChatGPT-4o	1	10.7	36.9	16.9	45
	2	12.9	30.1	17.7	47
	3	15.3	15.4	19.3	57
	4	8.9	43.1	15.5	39
	5	12.5	33.7	16.7	40
	6	14.9	16.3	19.3	43
	7	9.6	44.4	12.6	35
	8	12.0	40.5	14.9	42
	9	7.7	59.4	11.5	38
	10	9.5	48.7	15.5	44
	11	9.6	41.3	15.9	36
	12	14.2	25.4	18.1	38
	13	14.2	16.3	19.5	47
	14	11.8	38.1	15.9	46
	15	10.6	39.0	15.5	38
	Mean ± St Dev	11.6 ± 2.4	35.2 ± 12.6	16.3 ± 2.3	42.6 ± 5.5
Claude 3	1	12.4	36	16.6	56
	2	25.8	2.1	16.7	54
	3	18.4	13.1	17.1	46
	4	13.5	30.1	16.8	51
	5	19.2	20.7	15.6	53
	6	19.1	3.9	20.2	48
	7	19.4	6.0	18.1	42
	8	14.4	31.2	16.3	48
	9	15.4	25.3	15.3	46
	10	19.5	0.0	21.0	44
	11	14.9	22.0	19.0	41
	12	18.6	0.0	23.7	44
	13	13.3	31.4	16.6	47
	14	12.3	26.9	17.2	55
	15	11.8	31.1	17.3	44
	Mean ± St Dev	16.5 ± 3.9	18.4 ±13.6	17.8 ± 2.3	47.9 ± 4.7
Gemini	1	11.8	37.4	16.7	55
	2	11.4	45.8	15.4	50
	3	14.3	21.3	18.5	53
	4	12.3	32.2	17.6	53
	5	16.5	19.8	17.8	51
	6	12.2	31.9	16.9	58
	7	10.2	44.8	13.7	47
	8	14.0	31.0	16.7	44
	9	13.2	34.1	15.2	49
	10	11.8	39.7	15.5	50
	11	11.3	40.6	15.5	52
	12	16.5	9.4	20.7	52
	13	11.1	41.2	15.8	49
	14	16.3	18.4	17.9	53
	15	15.0	21.6	18.3	47
	Mean ± St Dev	13.2 ± 2.1	31.3 ± 10.9	16.8 ± 1.7	50.9 ± 3.4

) [12,13,14].

The Flesch Reading Ease score ranges from 1 to 100, with higher values indicating easier readability. The Flesch–Kincaid Grade Level metric was employed to gauge the educational attainment needed for understanding, where a score of 8 signifies suitability for individuals at an eighth-grade education level in the United States. The Coleman–Liau Index spans from 0 to infinity, correlating with the US school grade necessary for comprehension; for example, a score of 6 indicates appropriateness for a sixth-grade reading level. Scores between 13 and 16 correspond to college-level understanding, while those exceeding 16 are categorized as a professional level. The quality of the responses generated by the LLMs was evaluated using the DISCERN score (Table 2) and a Likert Scale (

Table 3. Likert Scale of the LLMs’ responses.

Likert Scale
Criteria	ChatGPT-4o	Claude 3	Gemini
Clarity	3	5	5
Comprehension	3	5	4
Readability	5	3	3
Patient friendliness	5	5	5
Informativeness	3	5	4
Total	19	23	21

), with ratings ranging from 1 to 5 across several criteria: clarity, comprehension, readability, patient friendliness, and informativeness [15,16]. The DISCERN and Likert Scale scores were derived by an anesthetist (Vittalraj) and junior medical officers (Ratnagandhi and Lim). A paired t-test (p < 0.05) was also conducted to compare the LLMs’ scores against each other (

Table 4. T-test of LLMs’ scores against each other.

	ChatGPT-4o	Claude 3	Gemini
ChatGPT-4o	-	−0.42, p-values = 0.67	0.016, p-value = 0.99
Claude 3	0.42, p-values = 0.67	-	0.99, p-value = 0.33
Gemini	−0.016, p-value = 0.99	−0.99, p-value = 0.33	-

).

3. Results

According to Table 2, the Flesch–Kincaid Grade Level Analysis indicates that responses generated by Claude 3 necessitate the highest level of educational attainment in the United States (US) for comprehension, with an average score of 16.5 ± 3.9, equivalent to a postgraduate level of education [17]. In contrast, outputs from ChatGPT-4o required the lowest US grade reading level, with an average score of 11.6 ± 2.4, placing it below Gemini, which scored 13.2 ± 2.1. This trend is further corroborated by the Flesch Reading Ease Score Analysis, where Claude 3 again demanded the most advanced reading proficiency (18.4 ± 13.6), followed by Gemini (31.3 ± 10.9) and ChatGPT-4o (35.2 ± 12.6). Consistent findings emerged from the Coleman–Liau Readability Index Analysis, with Claude 3 being identified as the most challenging for readers, recording an average score of 17.8 ± 2.3. Conversely, ChatGPT-4o exhibited the highest level of accessibility according to this index, achieving a score of 16.3 ± 2.3, while Gemini recorded a score of 16.8 ± 1.7.

The DISCERN score assessment indicated that Gemini delivered the most reliable information in response to the prompts, achieving a score of 50.9 ± 3.4. This was succeeded by Claude 3, which received a score of 47.5 ± 4.9, while ChatGPT-4o registered a mean score of 42.6 ± 5.5.

The Likert Scale (Table 3) scoring system evaluated Claude 3 as the preeminent LLM, achieving an aggregate score of 23 and securing a perfect score of 5 across all domains, with the exception of readability. This was closely followed by Gemini, which attained perfect scores in the categories of clarity and patient friendliness, aggregating a score of 21. Conversely, the Likert Scale assessment identified ChatGPT-4o as the lowest-performing LLM, with a total score of 19; nonetheless, it was the sole model to achieve a perfect score of 5 in readability.

3.1. ChatGPT-4o

ChatGPT-4o provided responses that effectively addressed each prompt, employing straightforward language and occasionally utilizing bullet points to segment information into manageable portions. This approach earned it a perfect score of 5 in the patient friendliness category on the Likert Scale (Table 3) However, in comparison to Claude 3 and Gemini, its answers were somewhat superficial, lacking the depth and elaboration present in their responses. This discrepancy was particularly apparent in the model’s response to prompt 5, where it failed to adequately address the potential risks associated with the mishandling of insulin prior to anesthesia. This shortcoming was reflected in its DISCERN score of 40, a significant deviation from the higher scores of Claude 3 and Gemini, which were 53 and 51, respectively. Similarly, for prompt 8, ChatGPT-4o received a DISCERN score of 42, compared to Claude 3’s score of 48, which can be attributed to the omission of critical information, particularly regarding red flags to monitor during the postoperative period, as well as the lack of detailed pharmacological options for pain management. Additionally, it is worth noting that ChatGPT-4o’s responses were typically more concise than those of the other two LLMs, often offering significantly shorter answers. Additionally, ChatGPT-4o did not reference any sources for the information and advice presented. While the responses from ChatGPT-4o were accessible and easy to comprehend, they were ultimately unremarkable in terms of content depth and nuance.

3.2. Claude 3

Although Claude 3 did not employ the same linguistic simplicity as ChatGPT-4o, as evidenced by its consistently inferior FKGL, FRES, and Coleman–Liau Index scores (Table 2), its responses were markedly more detailed and nuanced. The model distinguished itself through its tendency to provide supplementary information. This was reflected in its response to prompt 1, not only articulating the need for fasting but also presenting quantifiable and actionable fasting guidelines for patients. Similarly, in prompt 12, the model outlined specific prevention strategies designed to mitigate risks associated with anesthesia. These qualities were clearly reflected by Claude 3’s perfect Likert Scale (Table 3) score of 5 for both informativeness and comprehension, as opposed to ChatGPT-4o’s score of 3 for the same categories.

Moreover, Claude 3 regularly employed bullet points to organize its responses, enhancing the clarity and efficiency of its information delivery. Additionally, Claude 3 demonstrated a unique feature in its response to prompt 2, proactively offering to provide patients with further reading material and resources. Unfortunately, this was only demonstrated in that single response. However, despite the substantial depth of clinical information, Claude 3 consistently lacked citations and references to substantiate the data presented. Overall, Claude 3’s responses provided patients with detailed and expansive information to their questions.

3.3. Gemini

Similar to Claude 3, Gemini consistently delivered detailed and comprehensive responses to prompts, often providing additional insights beyond the scope of the questions asked. For instance, in its response to prompt 12, Gemini outlined a broad spectrum of risks associated with anesthesia, categorizing them by threat level, while also offering practical recommendations to mitigate these risks. This approach enabled Gemini to notably outperform both Claude 3 and ChatGPT-4o, achieving a DISCERN score of 52, compared to their respective scores of 44 and 38 on the same prompt. Likewise, in response to prompt 11, the model not only addressed the direct query by listing precautions to take prior to anesthesia but also suggested healthy lifestyle habits for patients to adopt in the lead-up to the procedure.

Gemini also employed bullet points effectively to organize its responses, presenting complex information in a clear and digestible format. However, like Claude 3, it did not provide sources or citations to substantiate the clinical information presented.

Furthermore, echoing Claude 3, Gemini underscored the critical role of healthcare professionals in guiding patient care. Throughout its responses to prompts 4, 5, 6, 13, and 15, the model emphasized the necessity of involving anesthetists or other qualified medical practitioners in decision-making, stressing that patient treatment should always be informed by professional expertise.

In conclusion, while Gemini lacked citations for its information, it provided in-depth and nuanced responses enriched with additional practical advice.

4. Discussion

Anesthesia has been an indispensable cornerstone of modern medicine for over 175 years, playing a critical role in facilitating a wide range of surgical and diagnostic procedures [18]. Its development and widespread use have revolutionized the ability to perform complex operations and interventions that would have been otherwise impossible for patients to endure, and it is involved in over 230 million procedures globally each year, underscoring its ubiquity and essential role in contemporary healthcare [19]. Despite its widespread use, anesthesia carries significant risks, ranging from minor complications to life-threatening events. This necessitates meticulous preoperative evaluation and a detailed informed consent process. Given the complexity and volume of cases, the time investment is considerable, impacting both healthcare providers and patients. The patient education process must strike a delicate balance between ensuring comprehensive understanding and maintaining the efficiency of medical workflows.

The evaluation of the three LLMs revealed significant concerns regarding clinical accuracy and potential bias, as reflected in the subjective DISCERN scores. All models consistently scored poorly on questions assessing bias detection and understanding of information sources. Notably, none of the models disclosed sources or citations for the provided information. This lack of transparency is particularly troubling given that LLMs rely on internet-sourced data, which may include inaccurate or biased information [20]. The “Black Box” problem is a significant hurdle in the development and deployment of LLMs within healthcare. It refers to the opacity of these complex systems: while we can observe the inputs and outputs, the internal processes by which LLMs transform data into responses remain largely hidden. This lack of transparency has profound implications for their understanding, trust, and safe implementation in clinical practice.

The “Black Box” opacity of LLMs raises concerns about biased outputs and hinders clinical integration. LLMs trained on limited data, such as from a single continent, may produce inaccurate recommendations for diverse populations [21]. Additionally, the inability to scrutinize the training data (guidelines, textbooks, and research) and provide real-time updates on medical information or guidelines unless prompted by the user necessitates medical practitioners’ input, potentially leading to distrust and reluctance to adopt these tools [22,23,24].

To safely utilize LLMs in healthcare, the “Black Box” problem must be addressed. This involves increasing data transparency regarding training sources, developing explainable AI (XAI) to make LLM reasoning clearer, and implementing human-in-the-loop systems where clinicians retain oversight of LLM recommendations [21,25].

All three models consistently recommended consulting an anesthetist or healthcare professional to mitigate bias in patient care decisions, as reflected in their high scores for “Does it provide support for shred-decision making?” in the DISCERN evaluation. This underscores the critical role of healthcare professionals in patient education and further emphasizes the need for human-in-the-loop systems in complex clinical decisions to ensure patient safety.

Claude 3 and Gemini achieved relatively higher scores in the DISCERN assessment, largely due to the breadth and depth of clinical information incorporated in their responses. Both models consistently delivered answers that extended beyond the immediate scope of the prompts, offering comprehensive insights that anticipated potential follow-up questions. This was reflected in both LLMs’ responses to prompt 11, “Is there anything I should avoid after having anesthesia?”, where Claude 3 and Gemini additionally provided practices that patients could incorporate to alleviate the effects or minimize the risks of anesthesia. This proactive approach is particularly valuable as patients may not always possess the knowledge to formulate the right questions. By providing detailed and anticipatory responses, these models help fill knowledge gaps that patients may not even know about. However, while this aspect of the responses is commendable, there is room for improvement, particularly in the inclusion of external resources, citations, and recommended readings. This would lend greater credibility and validation to the information provided, enhancing the overall reliability of the responses.

In terms of readability, ChatGPT-4o consistently recorded the lowest required reading grade across multiple readability assessments, including the Flesch–Kincaid Grade Level Analysis, Flesch–Kincaid Reading Ease, and Coleman–Liau Index. These results were further corroborated by its performance on the subjective Likert Scale, where it achieved a perfect score of 5 in the readability category, making it the only model to do so. This raises a critical question: Should LLMs prioritize delivering detailed and technically accurate information or should they focus on simplifying that information to make it more accessible? A 2014 Canadian study highlighted the difficulties patients face with information recall, underscoring that even well-meaning and detailed explanations can be difficult for patients to retain [26]. A related study in Spain revealed that 21% of patients did not read the consent forms they signed, and two-thirds could not recall the information given by their anesthetists [27]. These findings suggest that a range of factors, including cultural and functional illiteracy, contribute to patients’ difficulty in processing and remembering medical information [26].

From this perspective, ChatGPT-4o’s approach of emphasizing clear and patient-friendly language that is easier to comprehend might significantly enhance patients’ engagement in medical education and improve the retention of essential information. However, the trade-off lies in the potential oversimplification, which could mean that patients are not receiving enough clinical detail to make fully informed decisions. Given that many patients already struggle with complex medical terminology and detailed explanations, there is an inherent need for LLMs to strike a delicate balance between Claude 3 and Gemini’s depth of information and ChatGPT-4o’s readability. Too much detail may overwhelm patients, while too little may undermine their ability to make informed decisions. Achieving this balance is crucial if LLMs are to be trusted as reliable tools for independent patient education with minimal intervention from healthcare professionals. Further research could help determine the optimal level of detail, with patient feedback being central to understanding how these models can best serve as educational resources. Specifically, studies assessing patients’ experiences with LLM-generated responses to anesthesia-related queries could offer valuable insights into how these tools can be refined to maximize efficacy and minimize time constraints for healthcare professionals.

The use of LLMs in patient education offers a range of compelling benefits that can enhance both the quality and accessibility of healthcare information. LLMs’ proficiency in linguistics, vocabulary, and grammar allows for tailored explanations of complex medical concepts in a manner that is clear and easily understood by patients [28]. This can be particularly valuable in situations where patients need to comprehend intricate procedures, risks, or treatment options. Many patients have a limited understanding of what anesthesia entails, and this lack of knowledge can lead to heightened stress, misconceptions, and reluctance towards treatments and even increase risks of intraoperative and postoperative complications [29,30]. By providing real-time, tailored, and on-demand educational support, LLMs can help bridge knowledge gaps, reduce patient anxiety, and foster a sense of empowerment [31,32,33]. LLMs can also improve accessibility by overcoming language and literacy barriers, making anesthesia education more accessible to diverse patient populations [33,34].

By improving comprehension and reducing misunderstandings, LLMs can play a critical role in enhancing patient safety and satisfaction in the perioperative setting whilst also alleviating the cognitive load on anesthetists and other healthcare professionals responsible for delivering information in person.

While AI holds significant promise in the healthcare sector, offering the potential to enhance efficiency and alleviate the work burden on physicians, this study highlights the substantial modifications and improvements that remain necessary before AI can be seamlessly and safely integrated into clinical practice. Beyond technical challenges, it is crucial to consider the broader implications of AI’s role in the healthcare workforce, particularly the risk of displacing physicians. Additionally, the widespread adoption of AI could inadvertently diminish career satisfaction within the medical profession by reducing the intellectual stimulation traditionally derived from diagnostic problem-solving [25]. As discussed, the implementation of human-in-the-loop systems, where physicians actively engage in overseeing AI activities, would not only mitigate safety concerns but also preserve the central role of doctors in patient care [21,25,35]. Far from replacing medical professionals, such an approach could expand opportunities within the field, positioning doctors as pivotal figures in guiding and supervising AI applications in healthcare, thereby enhancing both patient outcomes and career prospects [35,36,37].

5. Conclusions

This study demonstrated that Gemini delivered the most reliable information among the three LLMs, as evidenced by its superior Likert scores and average DISCERN scores. However, no statistically significant differences were observed between the three models (Table 4). It was followed by Claude 3, which exhibited comparable performance across both subjective evaluation metrics. Despite this, all three LLMs revealed significant limitations regarding bias, primarily due to the lack of external resources and citations to substantiate the information presented.

ChatGPT, while emerging as the LLM that produced the most easily comprehensible responses, was found to lack sufficient clinical depth in its answers. This raises the need for further research to explore the optimal balance between simplicity in language and the necessary clinical detail required for informed patient decision-making. Such studies could help determine how to best tailor LLM-generated content to ensure both accessibility and accuracy, ensuring that patients are not only able to understand the information but are also equipped with enough clinical insight to make informed choices regarding their healthcare. Further research is required to evaluate the capacity of physicians to effectively collaborate with AI as the implications of such collaboration on the delivery of medical education can affect patient safety. In addition, it would be beneficial to investigate physicians’ satisfaction with the integration of AI into clinical practice, exploring influences on workflow, decision-making processes, and overall professional experience.