Reengineering eADVICE for Long Waitlists: A Tale of Two Systems and Conditions

Abstract

Long outpatient waiting times pose a significant global challenge in healthcare, impacting children and families with implications for health outcomes. This paper presents the eHealth system called eADVICE (electronic Advice and Diagnosis Via the Internet following Computerised Evaluation) that is designed to address waiting list challenges for paediatricians. Initially designed for children’s incontinence, the system’s success in terms of health goals and user experience led to its adaptation for paediatric sleep problems. This paper focuses on user experiences and the development of a working alliance with the virtual doctor, alongside health outcomes based on a randomised controlled trial (N = 239) for incontinence. When reengineering eADVICE to sleep disorders, the promising results regarding the reciprocal relationship between user experience and building a working alliance encouraged a focus on the further development of the embodied conversational agent (ECA) component. This involved tailoring the ECA discussion to patient cognition (i.e., beliefs and goals) to further improve engagement and outcomes. The proposed eADVICE framework facilitates adaptation across paediatric conditions, offering a scalable model to enhance access and self-efficacy during care delays.

1. Introduction

Long waiting times for outpatient clinics (attended by patients who are not hospitalised but are being managed through hospitals) are a problem affecting healthcare systems across the globe, including in Organization for Economic Co-operation and Development (OECD) countries [1]. The waiting time for paediatricians plays a critical role in the overall healthcare experience of children and their families and has substantial implications for health outcomes. The experience of waiting for a paediatrician’s appointment extends beyond a mere inconvenience; it holds the potential to shape immediate and long-term consequences, impacting patient satisfaction, healthcare access, reduced costs including preventing deterioration in the condition which is more costly to treat, treatment adherence, and ultimately, the overall health and well-being of young patients [2].

While the challenge of extended outpatient waiting times has long captured the attention of healthcare policymakers, the efficacy of suggested solutions, such as the provision of allied health services during wait times or increasing general practitioner (GP) hours, remains notably conditional on specific target groups and geographical areas, often unintentionally excluding those from low socioeconomic backgrounds [3,4]. Alternatively, eHealth-delivered treatments emerge as promising, user-friendly, and cost-effective alternatives, poised to offer an approach to address the consequences of long waiting lists [2,5].

Going beyond merely an outpatient portal, eADVICE (electronic Advice and Diagnosis Via the Internet following Computerised Evaluation) allows patients waiting to see a specialist to gain online access to assessment, diagnosis and tailored treatment advice. Patients remain under the care of their GP, who can also receive treatment recommendations and information sheets for their patients [6]. Essentially, eADVICE provides a communication channel between child, parents, referring doctor and specialists facilitated by an embodied conversational agent (ECA) which they can interact with to guide them in following their recommended treatment.

Adherence to treatment advice is a key factor for successful healthcare outcomes; however, a notable proportion of patients, reaching 50%, fail to adhere to their prescribed treatment plans [7]. Despite the potential benefits of eHealth-delivered treatments in addressing the adherence problem [8,9,10], the potential of these portals are still not well understood and further investigation is needed to confirm their value in improving treatment adherence and health outcomes [7]. In the case of eHealth-delivered treatments, adherence is complicated by the need to follow the technology as well as the health recommendations like traditional face-to-face treatment modalities [11]. It underscores the critical importance of bridging the gap between technological solutions and ensuring patients’ commitment to their health and well-being.

Typically, pre-specialist treatment information, such as articles on the Internet, tends to be generic and there is a lack of discussion. In contrast, eADVICE is unique in delivering personalised treatment advice before seeing the specialist. Nevertheless, it is important to note that adherence to treatment advice from a program may hold less authority than advice given by a human specialist who had seen the patient first, and therefore adherence may especially be a problem here.

The eADVICE framework specifically targets the adherence issue by allowing patients to discuss their treatment with the ECA and hence develop a working alliance with the ECA. Developing a working alliance with a doctor is the biggest predictor of treatment adherence [12]. The portal offers the possibility of documenting some of the treatment adherence interactions, thus providing additional feedback (similar to CPAP machines documenting usage at home). Based on evidence from a pilot study with 79 paediatric patients showing that eADVICE improved health outcomes for patients on the waiting list [6], we received funding to conduct a randomised controlled trial (RCT) with 239 patients and also redevelop eADVICE for a different chronic health condition. In the project, our aim was to test eADVICE in both the original chronic condition of paediatric incontinence and to expand it to paediatric sleep disorders, because clinics for these conditions typically have waiting times of over 12 months.

Due to the profound link between working alliance and adherence [13], in this paper and as part of our review of the RCT of the eADVICE system for incontinence in children, we analyse the users’ experience and their development of a working alliance with the ECA. In seeking to further improve the user-ECA working alliance, we enhance and articulate the ECA’s conversations into dialogue patterns in the context of reengineering the original eADVICE system to a novel health condition, sleep disorders in children. As part of the adaptation activity, we uncover insights into the fundamental processes involved in extending its utility to address waiting list challenges across various health contexts. However, we are not limiting our examination and contributions to these two paediatric conditions; rather, we pose a broader question:

How can we create a scalable framework for the reengineering of eADVICE to address the long waiting issue in other chronic health conditions in children?

Drawing on our experience in designing and implementing the two eADVICE programs, we offer in this paper our lessons learned and the eADVICE framework which supports the development and implementation of eADVICE to provide a sustainable model of outpatient care for other chronic conditions.

2. Related Literature

This literature review aims to explore critical issues in healthcare, focusing on access, adherence to treatment advice, and the role of technology, specifically empathic ECAs, in enhancing healthcare delivery. It, first, discusses the pervasive challenge of extended waiting times in public healthcare systems and their impact on patient outcomes and treatment adherence. The review then discusses how factors such as the quality of patient-healthcare provider communication, which are crucial for adherence, are also influenced by the systemic challenges of access and waiting times. Following this, the review discusses the emerging field of ECAs, particularly their application in healthcare for adults and children, showcasing their potential to improve adherence and health outcomes. Finally, the review considers the concept of a working alliance between patients and healthcare providers, both human and ECA, and its significance in fostering better treatment adherence and overall healthcare effectiveness. The interconnections between these topics underline the comprehensive approach needed to address the multifaceted challenges in healthcare.

2.1. Access and Adherence to Healthcare Advice

Extended waiting times for non-emergency treatment are a widespread challenge within healthcare systems burdened by capacity constraints, where demand often outstrips available resources [14]. In public healthcare settings, this problem disproportionately impacts patients who cannot afford private care, placing them on lengthy waitlists for specialist intervention. These delays pose a significant concern, raising questions about the potential negative consequences for health outcomes and the compromised benefits of timely treatment [1,15].

In the Australian context, where direct free access to specialist care is typically unavailable for patients except in acute situations through the emergency department, outpatient clinics play a critical role in providing public access to diagnosis and treatment of chronic health problems and post-inpatient care. However, the rising pressures on healthcare systems driven by factors such as population growth, longer lifespans, and the prevalence of chronic illnesses create a complex dilemma regarding the adequacy of healthcare funding and the efficiency of service delivery [16,17].

Although the precise relationship between waiting times and enduring patient outcomes remains a topic of ongoing research, a prevailing consensus acknowledges the adverse influence of extended waiting periods on patients’ physical and emotional well-being [18,19]. This underscores the imperative requirement for the identification and execution of innovative strategies aimed at enhancing the efficiency and accessibility of publicly funded healthcare systems. In this context, the pursuit of solutions becomes vital to ensure timely and high-quality healthcare delivery, aligning with the needs of an evolving healthcare landscape [20].

Further complicating the issue is the challenge of treatment adherence, even when medical advice is available. Studies across various conditions and delivery methods (human or technology-mediated) show that average adherence hovers around 50% [21,22]. This highlights the dual challenge of not only providing timely treatment advice but also fostering patient engagement and active adherence to the recommended treatment advice. There are many factors that influence treatment adherence [23], such as cost, frequency of use, ease of following the treatment and the quality of patient-healthcare provider communication and, when delivered by technology, usability, eHealth literacy and technology acceptance [24,25]. Research suggests that face-to-face interaction with healthcare professionals, particularly when a strong working alliance is established, can significantly improve adherence [26].

2.2. Conversational Agents, Health and Children

To address the intertwined issues of access, waiting times, and treatment adherence, empathic ECAs have emerged as a promising solution. Empathic ECAs have been found to deliver healthcare support deemed as less judgmental than a human [27,28] and, for certain health conditions, achieve higher levels of adherence than a human provider, though further research is needed [25,29]. For instance, Bickmore and colleagues [30] demonstrated the successful development of a companion ECA, called “Hospital Buddy”, which interacted over one to three days with eight patients aged 33–55. The study showed development of a good working alliance (4.4 to 6 on a 7-point Likert scale) between the “Hospital Buddy” and patient. With ageing populations and an increasing focus on ambient monitoring, applications like Virtual caregiver [31] and work by Tsiourti, Moussa [32] indicate that utilising ECAs alongside other technologies, such as serious game and virtual reality technology for occupational therapy exercise [33], are likely to become more prevalent in healthcare. ECAs acting as mentors and coaches, mostly for adults, have also been used for health education and behaviour change (e.g., [34]).

Despite the obvious potential of engaging game-based and VR technology involving the use of avatars to improve adherence in children [35], the use of empathic ECAs in paediatric healthcare remains largely unexplored. Existing preliminary studies, such as the ECA used to assist children with autism in developing social skills [36] and “Pain Buddy” [37], aimed at enhancing pain management in children with cancer, showcase the promising potential of this approach in paediatric settings.

We do, however, see the use of text-based, but not embodied, conversational agents in a range of applications relating to children. A review of the last decade of HCI work with children using voice-based conversational agents [38] identified a focus on anthropomorphising conversational agents for learning, play and storytelling, often in home and family contexts and for children with differing abilities.

Specifically concerning health, we see a range of different uses of text-based conversational agents (chatbots) for children and adolescents. Text-based conversational agents have been used for diagnosis, such as the Teenchat conversational agent by Huang, Li [39] that was used to listen to adolescents’ conversations for the diagnosis of stress. The agent was found to have a diagnosis precision rate of 78.34%. The study by Mujeeb, Javed [40] used the Aquabot conversational agent to predict Achluophobia—the fear of darkness and autism disorder—and achieved a precision rate over 87% for diagnosing autism in three patient groups, each with 10 patients aged between 1–7.

Health education, with the goal of behaviour change, also makes use of conversational agents. The chatbot created by Crutzen, Peters [41] enabled adolescents to ask questions about sex, drugs and alcohol. The study with 929 adolescents reported an average of 11 conversations per participant, with an average duration of 3:57 min. The chatbot was rated highly by participants, particularly compared with the use of search engines or other information sources.

Conversational agents for behaviour change in overweight adolescents have been evaluated in multiple studies. In a study with 22 adolescents, Heldt, Büchter [42] found that 67% of patients had more than four daily interactions with the study’s conversational agent over a period of 5.5 months, showing high compliance. In the study, 43% of adolescent participants successfully completed prescribed daily challenges. Similar results were achieved with 20 adolescents in a study involving conversational agents by l’Allemand, Shih [43]. Another study by Stasinaki, Brogle [44] with 21 obese adolescents found significant improvement in five out of eight physical performance tests for the treatment group compared with only two of the eight tests for the control group. In yet another study with 15 overweight children, Kowatsch, Nißen [45] found that a socio-emotional bond was developed with the agent by the end of the study, with nearly 70% of patients having at least 4 daily conversational turns and 13 reporting enjoying their chats.

Conversational agents have also been used for management of chronic conditions in children and adolescents. Rhee, Allen [46] conducted a study with 15 adolescent-patient dyads for the self-management of asthma. They found a high response rate (81–97%) to daily text messages from the chatbot, with a higher response in the morning compared with bedtime, and an average of 19 self-initiated messages per adolescent sent to the chatbot mostly discussing symptoms. As well as developing a sense of partnership between the adolescent and parent, the chatbot increased the understanding of symptoms and triggers, sense of control and adherence. Kowatsch, Schachner [47] evaluated the MAX conversational agent, which was developed for 10–15-year-olds with asthma and designed to support the child–parent team as well as the healthcare professional. The study found that MAX acted as a mediating role between all parties towards improving the patients’ health outcomes. A study involving children aged 7–14 with Type 1 diabetes who used a conversational agent for 6 months demonstrated an increase in self-care scores, but there was no effect on quality of life related to diabetes [48].

A recent meta-analysis of conversational agents [49] for chronic health conditions found that users were becoming comfortable with their use but that evidence of their efficacy for chronic health conditions was lacking due to an insufficient description of their technical details. By clarifying the components of eADVICE in this article and our lessons learned in modifying the eADVICE-Continence for the condition of sleep, we seek to make our approach transparent for adaptation to other conditions.

2.3. Working Alliance to Address Adherence

A key mechanism through which ECAs can improve adherence is through building a strong working alliance. In eADVICE, the goal of the ECA is to improve adherence to the treatment advice. As the main predictor of treatment adherence [12], our conversations were designed (Section 3.2) and redesigned (Section 4.2) to build a working alliance: the positive collaborative relationship between the therapist and the patient. Mathieu, Heffner [50] describe a working alliance as the patient’s motivation to agree to follow the advice recommended by the therapist. The working alliance has also been called a therapeutic alliance, alliance, therapeutic relationship, or ‘helping alliance’ [51], with varying measures depending on the specific conceptualisation. The literature on psychotherapy, pharmacotherapy and placebo-therapy reports that a good working alliance is critical to deliver a good treatment outcome [52,53,54].

Bordin [55] is attributed with coining the term working alliance, which comprises three elements. Firstly, there is the mutually agreed goal of the patient and therapist. Towards achieving this goal, we have the tasks which are the behaviours the patient needs to perform. We may also consider these tasks to be plans of the patient’s intended behaviours. Additionally, there is a bond, which is a sense of caring and trust, where the patient believes the therapist is acting benevolently and in their best interest. To achieve shared goals and plans, the two parties work together using positive communication and developing a sense of involvement leading to adherence [56]. Furthermore, there needs to be communication of the planning of the patient’s intention to do the task/s required by the treatment [57].

Drawing on the belief–desires–intention computer-based agent theory of Bratman [58], a plan does not require knowledge of the actions to be performed but intention to take the actions and replan when needed. This view is very compatible with behaviour change (e.g., adherence), which requires commitment and also a revision of goals and trying different approaches when the original plan was not followed or found to be useful. Grosz and Kraus [59] notion of SharedPlans further extends this idea, emphasising the importance of collaborative planning and adaptability within therapeutic relationships. This perspective is particularly relevant for creating effective human–agent interactions, where the ECA must understand and adapt to the patient’s needs and responses.

Conversational agents, particularly their dialogue, play an integral role in achieving human–agent mutual understanding and shared planning [60], which can lead to the development of a working alliance [61]. This has been demonstrated in a number of tools such as Collagen [62] and Dtask [63], where a dialogue engine was used to promote a working alliance and health-related behavioural change.

While most studies into the effect of a working alliance have concerned adults, a growing number of studies have investigated its relationship with psychotherapy outcomes in children and adolescents [64,65]. For example, a study by Lindqvist, Mechler [66] involving 272 participants aged 15–19 years diagnosed with depression found an interplay between working alliance and emotion regulation in the context of internet text-based treatments. Also, concerning internet-based treatments for depressed adolescents, Mortimer, Somerville [67] found that a strong working alliance developed togetherness, agency and hope. All of these studies and reviews concern working alliances in the context of adolescents and their mental health, unlike our work which involves a working alliance to improve outcomes in the context of chronic conditions.

The human–machine working alliance has been identified as conceptually different from the human–human alliance [68]. In a study involving users (N = 20) of mental health apps, Tong, Lederman [68] found that both digital and human working alliances have similarities; however, there were differences in terms of flexibility and emotional experience. Also of note, they found it was not essential to have a human–machine shared goal in the case of a digital working alliance.

In summary, whilst the challenges of extended waiting times and low adherence pose significant obstacles in healthcare, innovative technology solutions like empathic ECAs that have the potential to build a working alliance offer promising avenues for improving patient outcomes and ensuring equitable access to quality care. The predominant use of voice or text-based only conversational agents with children raises questions for us because they do not provide a face-to-face conversation found to be important for adherence [11,26] and do not provide a visual game-like form of interaction that should be more engaging for children. We assume that voice or chatbot style agents have been used because users may be more familiar with them, and they were easier to implement and study. From our own experience, we recognise the added technical difficulties experienced by parents when needing to access our ECAs. Further research and development in this area are crucial to fully realise the potential of ECAs in transforming paediatric healthcare and enhancing the efficiency and effectiveness of healthcare systems overall.

3. Tale 1—eADVICE for Incontinence

3.1. Problem Domain

While extended waiting times and adherence challenges present significant hurdles across public healthcare, urinary incontinence in children poses a unique and often under-addressed problem domain. Affecting approximately 10% of school aged children (5–15 years), incontinence can dramatically impact their physical and emotional well-being, social participation, and academic performance [69,70].

Current challenges in paediatric incontinence management further intensify these consequences. Long wait times for specialist appointments at tertiary children’s hospitals can worsen the emotional and physical consequences for children and their families, leaving them navigating the problem for extended periods with limited support [71]. Limited access to specialised therapists contribute to inconsistent care and suboptimal outcomes, hindering effective management of the condition [72]. Finally, children may struggle with adhering to complex treatment plans, particularly those involving behavioural modifications or medication regimens, further complicating long-term success [73].

Emerging technologies including empathic ECAs offer a promising avenue for addressing these challenges in paediatric incontinence management. By providing personalised support, education, and behaviour change reinforcement, ECAs can potentially bridge the gap between diagnosis and treatment, enhance treatment adherence, and empower children and families to cope with the problem more effectively.

3.2. eADVICE System

eADVICE conceptually comprises three main components: (1) an interactive website containing a set of (triaging) questions to capture data about the child including their health status, particularly focusing on the medical history and current state of health for the specific health condition; (2) algorithms that use the inputs from the website to determine the appropriate treatment/s and (3) an interactive conversation with an ECA to encourage the development of a working alliance (see Dr. Evie (eVirtual agent for Incontinence and Enuresis) in Figure 1) for each specific treatment. A database stores the history of each session including health status, treatment/s recommended and interactions with the virtual character. Further consideration of the conceptual design of the eADVICE can be found in [74]. These three components must be created for each specific health condition.

As previously described in Richards and Caldwell [6], GPs could invite their patients to use eADVICE whilst on the specialist waiting list. After giving consent, families could answer 15 questions covering the child’s age and weight, fluid intake, incontinence frequency and severity, and bladder and bowel habits. If the child was interacting with the system, they could use the children’s version of the questions which also included a related colourful graphic. The child’s data are used to determine the recommended treatment. The system also monitors the child’s condition through recording of each visit in a MySQL database. If eADVICE recommends medication or is unable to provide a recommendation, the family are asked to speak with their GP.

Based on the patient data, eADVICE recommends one or more treatments out of the six available treatments: Caffeine Advice; Fluid Advice; Bowel Program; Timed Voiding; Enuresis Alarm Training and Medication Discussion. The recommended treatments can be accessed by the patient as well as their GP, in case GP support is needed. Dr. Evie delivers these treatment plans to patients and parents/carers in a conversational style following the dialogue flow presented in Figure 2a. Dr. Evie starts the conversation by greeting the user and reminding them of the goal of the current treatment before delivering the treatment plan and then closing the conversation. Each treatment advice has a specific fixed dialogue which is the same dialogue for all users, but the actual conversation will vary based on options chosen that are delivered to all the patients needing that specific treatment. The dialogues of the six treatments were designed and reviewed by incontinence specialists to deliver evidence-based behavioural advice typically offered in face-to-face consultations, ensuring age-appropriateness, clarity, and motivational language.

For a period of up to six months (according to the RCT design), every two weeks the family may seek another assessment. Families are expected to implement the treatment during the two-week period and are able to discuss the treatment with Dr Evie as often as they wish in the intervening period.

3.3. The Outcome of eADVICE—Incontinence RCT

Participation in the RCT was open to children aged 5 to 18 years who had been referred by their doctor to the Continence Services at three tertiary children’s hospitals in NSW, Australia (The Children’s Hospital at Westmead, Sydney Children’s Hospital or John Hunter Children’s Hospital). In addition, GPs from the Western Sydney Primary Health Network who accessed Urinary Incontinence in the Children HealthPathways Program (a web-based information portal designed to support primary care clinicians in their workplace) were able to directly refer their patients using the HealthPathways’ referral form. Children who had daytime incontinence (at least twice per week) or enuresis (at least three times per week) were eligible and children with an organic cause for urinary incontinence, including defects of the central nervous system (i.e., global developmental delay) or major urological abnormalities were ineligible for the study. Families needed to have internet access to be eligible. Only one child per family was allowed to participate to minimise contamination.

As illustrated in Figure 3, the participants were randomly assigned to either the intervention group, where participants were granted immediate access to eADVICE for 6 months upon consenting to participate, or the wait list control group, where families were placed on a waiting list for six months before gaining access to eADVICE. The evaluation of eADVICE was undertaken 6 months after using eADVICE for both groups.

We compared the experiences of participants in the two study groups at 6 and 12 months, using patient reported experience measures (PREMS), usability and the user experience of the eADVICE-Continence program through a survey. For user experience evaluation, we measured the extent of the working alliance with the program using the short version of the Working Alliance Inventory (WAI-SR). The 12-item WAI-SR comprises three components each with 4 items: goal (agreement on goals), task (agreement on tasks to achieve the shared goals) and bond (sense of trust and caring).

3.3.1. Participants

A total of 239 participants were recruited and randomly assigned to either the intervention or control arms of the treatment trial. The intervention arm comprised 120 participants with a mean age of 8.8 (SD = 2.3) and 60.8% were male. In the control arm, there were 119 participants, with a mean age = 8.8 (SD = 2.1) and 63% were male. There was no significant difference in terms of health literacy score between the two groups, with both averaging 3.77 (SD = 0.64). However, only 34 participants from the control arm and 50 from the intervention arm completed the post-trial surveys including user experience and working alliance surveys.

3.3.2. Health Outcome Evaluation

At 6 months post-baseline, a comparative study between the two groups showed that eADVICE had notable efficacy. The intervention group, who had access to eADVICE, achieved higher daytime continence (75.0% dryness) and nighttime continence (23.9% dryness) compared to the control group (61.2% and 12.2%, respectively). Further analysis revealed an overall greater reduction in incontinence frequency and severity in the intervention group relative to the control group (who do not have access to eADVICE at this point). When the control group was given access to eADVICE from 6–12 months, they exhibited improvements at 12 months and increased dryness. Readers can find more details on the health outcome in [75] and the economic analysis from the perspective of the healthcare funder.

Despite significant clinical improvements in all domains measured (e.g., daytime wetting, nighttime wetting, reduced frequency and severity of wetting and better quality of life), many patients still wanted to see the doctor (78% in the intervention group and 84% in the control group (p = 0.3)).

3.3.3. Working Alliance and User Experience Evaluation

The working alliance questionnaire demonstrated high internal consistency (Cronbach’s α = 0.95). For testing the between-groups differences, we utilised Bayesian analysis with t-distribution as a prior distribution. As shown in

Table 1. Working alliance statistics.

Control Group (N = 34)	Intervention Group (N = 50) µ1 (sd)	µ1–µ2 [95% CI] µ2 (sd)	Means for Combined Data (N = 83)
Task	2.36 (1.13)	2.19 (0.90)	0.13 [−0.36, 0.61]
Goal	3.04 (1.27)	2.60 (1.13)	0.44 [−0.13, 1.0]
Bond	3.12 (1.23)	2.76 (1.13)	0.37 [−0.20, 0.93]
Working alliance	2.81 (1.09)	2.50 (0.94)	0.31 [−0.18, 0.80]

, we report the mean difference and the 95% credible intervals (CIs), which if it includes zero, the difference is likely not statistically significant. Hence, the three dimensions of the working alliance showed no significant differences between the intervention and control groups when the control groups eventually had access to eADVICE. This indicates a similar working alliance developed in both groups with Dr. Evie at both timepoints. This alignment with expectations is attributed to the fact that, at the time of measurement, both groups had been engaged with eADVICE for a duration of 6 months. Therefore, data from the two arms were combined for the overall analysis reported hereafter. The working alliance means of the combined data are reported in the last column in Table 1.

The user experience questionnaire comprised 13 items detailed in

Table 2. User experience statistics ordered by the correlation of the item and the working alliance. The correlation was computed using N = 80.

Item	Low on the Scale			High on the Scale			Correlation (r) of the Item with Working Alliance [95% CI]
	n1	µ1	σ1	n2	µ2	σ2
I felt confident using the eADVICE program	19	1.70	0.57	61	2.90	0.92	0.56 [0.40–0.71]
I think that I would continue to use the eADVICE program to manage my child’s wetting	43	2.20	0.93	37	3.00	0.86	0.53 [0.36–0.70]
How supported did you feel in managing your child’s wetting problem by the eADVICE program? *	18	2.00	1.00	36	3.20	0.81	0.48 [0.29–0.64]
I thought that eADVICE was easy to use	13	1.77	0.95	67	2.80	0.82	0.46 [0.27–0.64]
I found that the various functions on eADVICE were well-arranged	16	1.90	0.78	64	2.80	0.97	0.39 [0.19–0.59]
How did you find the quality of the communication with the eADVICE program? *	11	2.20	1.20	51	2.90	0.95	0.36 [0.15–0.55]
It was easy to read the screen	6	1.90	0.71	74	2.70	0.99	0.33 [0.13–0.53]
eADVICE is designed for all levels of users	14	2.00	0.62	66	2.80	1.00	0.32 [0.08–0.53]
How did you find access to the eADVICE program? *	8	2.20	1.30	51	2.80	0.99	0.30 [0.084–0.49]
I was able to easily download and save the information regarding the treatment plan	16	2.40	1.10	64	2.70	0.98	0.23 [0.01–0.44]
The pop down explaining questions in my child’s terms was useful	8	2.10	1.30	72	2.70	0.95	0.16 [−0.08–0.40]
The information was well organised	9	1.50	0.62	71	2.70	0.94	0.11 [−0.13–0.34]
How did you feel about the time you spent waiting to access the online eADVICE program? *	18	2.70	1.20	35	2.70	0.86	0.03 [−0.19–0.26]

* The item was measured on 5-point scale with the middle point being neutral. Those who scored neutral were not included in the low or high on the scale statistics.

: 9 items were assessed on 4-point scales, while 4 items were evaluated on 5-point scales, indicated by an asterisk (*) in Table 2. In examining participants’ scores on these scales, we determined the number who scored low and high (selected the low or high two scores on the Likert scale), along with the average for each item (refer to Table 2). For 5-point scales, we excluded those who scored neutral from either the high or low groups, considering only those who expressed clear preferences. Additionally, to explore the correlation between user experience and the user–agent working alliance, we calculated Spearman correlations between the user experience items and working alliance, as detailed in Table 2. It is essential to note that the correlation analysis involved the entire participant set (n = 80).

3.3.4. Discussion

The results provide substantial evidence supporting the potential utilisation of the eADVICE system and ECA to address challenges related to extended waiting lists and alleviate the burden on the health system as well as patients and their families. The system provides timely health advice and effectively engages individuals, encouraging them to adopt beneficial changes in their routines to address their health issues. Notably, the results indicate that following 6 months usage of eADVICE, there were significant improvements in participants’ health outcomes. Interestingly, although the patients were statistically significantly better and drier, it did not significantly affect their desire to see the specialist, which suggests that people want human to human interaction despite the child’s condition being significantly better or gone.

Focusing on the paper’s central theme, user experience, as outlined in Table 1, a noteworthy observation is the lack of substantial evidence, with the 95% confidence regarding a difference in working alliance between the treatment and control arms after six months of system usage. As no significant difference in working alliance is observed, the findings suggest that immediate or delayed access has no influence on the development of the working alliance and that the system is a potential solution to be provided at any timepoint for those who are on the waiting list. However, this conclusion is limited to developing a working alliance and not health outcomes as the previous results indicated that immediate access could gain better outcomes.

Table 2 provides insights into the dynamic interaction between working alliance and user experience. It is apparent that the working alliance and user experience reciprocally influence each other. Significant correlations, as emphasised by Cohen et al. [76], indicate that a robust working alliance substantially boosts user confidence and the intent to use the system in the future (r > 0.5). Additionally, moderate correlations (0.3 < r < 0.5) suggest that user support and communication quality are also impacted by the therapeutic bond. Notably, factors within the system’s design seem to impact the working alliance itself, as demonstrated by the moderate correlation between ease of use, readability, and the functions arrangement. These findings highlight the potential for optimising eADVICE’s design to further enhance user experience and engagement, ultimately leading to improved health outcomes.

4. Tale 2—eADVICE for Sleep

4.1. Problem Domain

In this tale, we targeted another common area with long wait lists related to sleep disorders in children, taking into consideration the profound impact of sleep on their overall well-being. The peaceful sleep children enjoy is not just a time for rest, but a powerful force that helps them develop in healthy ways. Yet, sleep problems affect many children around the world, with estimates exceeding 30% [77,78].

The impact of sleep problems in children is not limited to the bedroom. These issues can cast a long shadow on their physical health, mental clarity, and emotional stability, and the challenge of accessing specialised care for these invisible struggles further amplifies the impact [79]. Obtaining the right help for children with sleep problems is often far from easy. Long waiting lists, sometimes lasting months or even years, add another layer of stress for families already dealing with the many challenges caused by sleep issues [80]. This lack of access makes it harder to address the underlying health and developmental problems, trapping children in a cycle of sleep deprivation and its negative consequences.

Research efforts focused on developing effective non-pharmacological interventions such as technology-based solutions to address the long waiting for specialist appointments hold promise for providing accessible and evidence-based options for families [81]. Specifically designed sleep apps for children provide educational resources and behavioural advice such as relaxation exercises, guided meditations, and bedtime stories, with gamification elements enhancing the patients’ engagement [82,83,84]. While challenges like long-term efficacy, cost-effectiveness, accessibility, and parental engagement exist, those technology-based solutions show significant potential to enhance sleep hygiene and overall well-being in children. Therefore, to minimise the long-term impact of sleep problems on children and families, early identification and intervention is key. The reengineering of eADVICE-Continence to eADVICE-Sleep targets behavioural changes to promote healthy sleep habits.

4.2. eADVICE-Sleep and SAM

With the promising findings from eADVICE for incontinence, to target the sleep problem in children we developed the three components of the eADVICE-Sleep system: website, algorithms and treatment conversations (see Section 3.2). Overall, eADVICE-Continence and eADVICE-Sleep use the same pattern to ask questions and calculate treatments. Of course, the eADVICE-Sleep website includes questions more specific to the child’s sleep behaviour rather than continence. Additionally, questions in eADVICE-Sleep were age-specific and required three different sets of questions to be created for different age groups: infant, child, and teen. This in turn has a knock-on effect to the code changes required to the internal files, where a child’s age is first acquired, then the relevant age-group questions, answers and treatment calculations are used. Another main difference is the nature and method of capture of the supplementary information, with eADVICE-Continence using the Time–Volume Chart and eADVICE-Sleep using a Sleep Diary.

The sleep-specific algorithms use the data to identify the appropriate treatment. Paediatric sleep specialists identified six treatments that target the most common behavioural problems: sleep diary, sleep hygiene, settling routines, snoring, night terrors and caffeine consumption. Users follow the same established process of eADVICE outlined above in Section 3.2 to receive their treatments with adjustments tailored to the context of sleep problems. As well as sleep-specific conversations, one for each treatment, a new conversational agent, Sleep Adherence Mentor (SAM), was created for this condition (Figure 4). While the primary dialogue flow of each treatment follows the same flow as Dr. Evie (Figure 2a), SAM tailors the delivery of the treatment plan to the user’s cognitive state, including beliefs and/or goals (Figure 2b), motivated by our desire to further improve working alliance and adherence and to improve health outcomes compared to the first implementation with Dr. Evie.

SAM utilises the patients’ cognition including beliefs, desires (goals) and intentions (BDI) to tailor the treatment plan of each user and to help patients to follow its advice by providing a tailored explanation according to the available cognition elements.

The design of the treatment plan dialogue is inspired by the complex connection between how we think, our beliefs and our goals, which in turn influences behaviour change [85,86]. Theories, such as the Theory of Reasoned Action, suggest that our cognitive state influences how we interpret information, subsequently determining the actions we take [87]. People who find a connection between what they believe and/or what they want to achieve are more likely to adopt strategies for behaviour change [88,89]. This close link between thinking, belief, and goal setting highlights the importance of not just addressing visible behaviour but also tackling the underlying cognitive and motivational aspects that steer our actions.

As a result, SAM’s dialogue starts each initial conversation with a user by asking about their beliefs (commonly perceived as barriers) and goals (commonly seen as motivators). These beliefs and/or goals govern SAM’s dialogue and advice, ensuring the advice is tailored to the user’s cognition. Moreover, to incorporate educational elements into the conversation, users have the opportunity to ask SAM about the reasoning behind each piece of advice. The explanation is designed to begin by referring to the previously elicited user cognition (beliefs and/or goals) and explaining how the advice aligns with them before providing additional information about why the user should follow the recommendation. For example, SAM might explain why a teenager should limit their caffeine consumption: “You find caffeine can boost your energy (user’s belief) but that is not the only way to boost your energy. There are healthier alternatives that contain less sugar or caffeine. I also recommend you eat whole foods that are high in complex carbohydrates, healthy fats, and vitamin B. Add to this regular exercise, lots of water and at least nine hours of sleep at night”.

4.3. Preliminary Assessment of eADVICE—Sleep

As part of the development process, we received initial anecdotal feedback from parents and children, as well as other health professionals at the hospital. Interestingly, the specialists from the incontinence study commented on the superiority of the sleep dialogues in terms of structure, content and comprehensiveness, such that Dr. Evie lacks a clear connection between barriers (beliefs) and motivations (goals), and does not provide explanations which refer to the user’s beliefs and goals in order to provide relevant education and guidance.

The analysis of the informal user feedback on the system reveals a predominantly positive user experience, commended for its clarity, ease of use, and effective communication of information in manageable sections. For example, a healthcare professional praised the system’s transparency with the explanation and use of understandable language. Interestingly, a health care professional and father noted that his children perceived the system as an educational tool rather than a medical one. Another healthcare professional shared their child’s interest in the system, indicating potential for further content expansion to address specific topics of interest. However, constructive feedback, including suggestions for incorporating pictures and diagrams, highlights opportunities for improvement. Formal evaluation has not been undertaken to date due to lack of funding. We are in the process of addressing this.

5. Lessons Learned—eADVICE Development Process

Through having developed eADVICE-Continence and eADVICE-Sleep, we have identified and validated an overall eADVICE framework and development process that can be applied to other health conditions. Firstly, we note that in developing the three main components of eADVICE (website, algorithms and treatment conversations, please refer to Section 3.2), it involves extensive and iterative discussion and interaction with specialists with expertise in the specific health condition, as well as patients and their families. Aligning with the three components: (1) the website requires the identification of the data that needs to be captured from the patient and their family; (2) algorithms must be specified to capture the heuristics (rules) that the specialist uses which identifies appropriate treatment/s based on the data collected by the website; and (3) conversations designed to develop a working alliance for each treatment will need to be written to include both the questions and statements made by the ECA as well as the set of response options available to the patient considering their age group and preference that may affect the language they prefer to be addressed with.

The process to develop the questions, algorithms and conversations can follow both a top-down and bottom-up approach. Both approaches will be needed to gather all of the necessary information. In addition to guiding the selection of appropriate treatments, these questions also serve as a health-risk assessment, identifying any potential concerns that might require urgent attention from a healthcare professional. Following a top-down approach involves identifying the common treatments prescribed to deal with the health condition. This can be then used to identify what data are needed, the rule/s for recommending that treatment and the information (educative aspect, typical questions asked and explanations) and the issues (motivations, barriers, concerns) that the patient may want to discuss with the ECA.

Following a bottom-up approach involves the elaboration of the lower-level elements of these three components. For the website, the health team will need to elaborate on the questions that are typically asked at face-to-face or telehealth consultations to gather information from patients. These questions aim to diagnose health conditions and identify which treatments are most suitable. Additional data such as the patient’s medical histories or family medical histories and records of behavioural patterns (such as sleep diaries for eADVICE-Sleep and time and volume charts for eADVICE-Continence) that cover a specific period of time, such as one or two weeks, may also be essential for formulating the correct treatment. Setting time frames for taking action such as data collection and following treatments and receiving (new) treatments provides clarity and consistency in terms of system interactions and also seeks to motivate the patient/family to engage regularly with both the program and the treatments. For the algorithms, the specialist will need to articulate the data and reasoning process they use to decide when a treatment is appropriate. For the conversations, the health team, in consultation with patients and their families, will need to consider what questions, concerns, clarifications, barriers, etc., may be raised by the patient or parents/carer. A review of leaflets and fact sheets that are supplied to patients and parents may help to identify some of this content. The literature can potentially identify barriers to adherence to treatments that can be addressed in conversations. Of importance in designing conversations is identifying and using language that is age-appropriate, empathic, inclusive, warm and friendly while also being professional.

In contexts where the health team and/or the technical team are less experienced in knowledge engineering or acquisition and less aware of the data, algorithms and conversations that exist in their domain, it is recommended that existing patient consultations be recorded and analysed, with HREC approval. This will delay the implementation of the system as analysis can be time intensive, but collecting the necessary data, obtaining the treatment algorithms correct and writing conversations that increase adherence is essential for the eADVICE approach to be successful. The analysis of the interviews can follow a similar method described in Salman, Richards [90]. The piloting and validation of all components and the complete system will be essential before it can be used on patients on waiting lists.

Further support for the development of empathic conversations can be found in Richards and Caldwell [91]. The consideration of design elements can be found in [23,92].

6. Limitations and Future Work

The key limitation of our proposed generalised framework is that we have not yet run a study with eADVICE-Sleep. To address this issue, we recently obtained funding to commence the evaluation of eADVICE-Sleep in the context of children with neurodisability. This necessitates further work with specialists and stakeholders to adapt eADVICE-Sleep to this specific cohort of children and their families. The lessons learned from adapting from continence to sleep, as described in this paper, will be valuable for this new project, and we anticipate this for other health conditions too.

Some limitations of eADVICE will be condition specific. For example, eADVICE-Sleep cannot treat all sleep disorders such as obstructive sleep apnoea. In such cases, it becomes important for eADVICE to triage certain disorders and direct patients and their families to gain earlier access to the clinic.

When compared with most trials reported in the literature involving children and adolescents [93,94], our RCT, which recruited 239 paediatric patients and captured data over an 18-month period with a 6-month waitlisted control, 6-month treatment and follow-up 6 months later, can be considered a large paediatric study over an extended period. Nevertheless, limitations concerning the RCT with eADVICE-Continence include study numbers, the short six-month duration of the trial and limited power to detect significant effects for sites with shorter waiting times. The complexity of conducting a trial in children is a further limitation.

The use of handcrafted dialogues may be seen as both a strength and a limitation. The dialogue templates presented were designed to support automation. Our state-based dialogue engine allowed us to use parameters to adapt the ECA’s responses based on the patient’s responses, particularly their beliefs and goals. However, it was not possible to ensure that the templates would produce well-structured, grammatically correct sentences and thus, for the sake of usability, comprehensibility and acceptability, it was necessary to provide handcrafted wording. We anticipate that the advances in GenerativeAI will provide an imminent solution to this issue. With the increasing use of GenerativeAI and tools such as ChatGPT, we are seeing the deployment of chatbots in many areas, including health. With this deployment are many concerns regarding the accuracy of the advice being provided and also the resultant sharing of sensitive health data used by the Large Language Model (LLM) to grow its knowledge and database. As a key goal of eADVICE is to provide evidence-based treatment and guidance while patients are awaiting a specialist appointment, it is essential that the content is fully reviewed by domain experts rather than generated by AI. In the future, we might consider the use of an LLM to tailor the expression used in the dialogues to the individual, but identifying such preferences remains open research questions [95,96]. As a step in this direction, in two safer contexts, we have used an LLM to modify dialogues to imbue relational cues to further increase working alliance [97] concerning health and well-being advice delivered using a digital cognitive coach, physical coach and dietician [98,99], and to imbue personality cues in non-player characters who offer different ethical perspectives [100] in a serious game for the ethics training of cybersecurity professionals [101].

7. Conclusions

The eADVICE system emerges as a promising solution to the challenges of extended outpatient waiting times in paediatric healthcare. Through successful implementations targeting incontinence and sleep issues, it demonstrates significant improvements in health outcomes and positive user experiences. The stability of the working alliance over time between ECAs and patients and parents underscores the system’s reliability and user engagement. The reciprocal relationship between the working alliance and user experience highlights opportunities for further optimisation in design, enhancing overall therapy outcomes.

In contrast to the increasing use of artificial intelligence in medicine where the machine’s algorithms make decisions and generate natural language conversations, the eADVICE framework captures the knowledge of the domain expert, including their ability to accurately diagnose and empathically discuss appropriate treatments with patients. With the current excitement and fear around ChatGPT and large language models, the advantage of our system is that it is embedded in a health care system and not likely to be influenced by false information, something that can not be guaranteed by ChatGPT. It is endorsed by health professional and is therefore safe. The paper concludes by proposing the eADVICE framework as a scalable model for various chronic subacute conditions, emphasising the need for ongoing research and development in this transformative field of paediatric healthcare. When current GenerativeAI technologies significantly improve their ability to guarantee accuracy, not to hallucinate and are also able to ensure patient data remain secure, we anticipate improved scalability and personalisation, and the potential adaptation to new health conditions following the framework outlined in this paper.