Transforming Telemedicine: Strategic Lessons and Metrics from Italy’s Telemechron Project (Telemechron Study)

Abstract

Background: The Telemechron project addresses critical needs in telemedicine by evaluating technology assessment frameworks and key performance indicators (KPIs), among other objectives. Effective frameworks and KPIs are crucial for assessing telemedicine services, especially in terms of their impact on patient outcomes and service efficiency. Methods: This study adopted a dual approach to assess the contributions of the Telemechron project. Firstly, it applied a technology assessment framework to quantitatively evaluate telemedicine services, focusing on measurable improvements and systematic analysis. Secondly, it investigated a set of predefined KPIs using specific metrics and parameters to evaluate their applicability and limitations in telemedicine settings. Results and Discussion: The technology assessment framework demonstrated significant utility by providing structured, quantifiable improvements across key aspects of telemedicine services. It was effective in evaluating the alignment of services with strategic goals and their integration with existing healthcare systems. The predefined KPIs, although not developed within this study and not directly used by the different operational units (OUs)—which established their own context-specific KPIs—still provided valuable insights into telemedicine performance. However, the study revealed that these KPIs highlighted a need for further customization to enhance their relevance across various contexts. While the KPIs may offer useful performance indicators, their general applicability necessitated adjustments to better address the specific needs of telemedicine applications. The analysis model for the KPI set, in terms of metrics and parameters, is exportable and generalizable to other national and international telemedicine contexts. Conclusions: The study confirms the effectiveness of the framework in delivering measurable improvements in telemedicine services and underscores the importance of adapting KPIs for specific contexts. Future research should focus on further applying and expanding the framework and metrics, refining KPI models, integrating new technologies, and conducting cross-national comparisons to enhance the applicability and effectiveness of telemedicine evaluations.

1. Introduction

1.1. Technology Assessment and Telemedicine

Technology assessment (TA) plays a crucial and strategic role across various sectors, with its importance being particularly pronounced in the healthcare field. Telemedicine, characterized by its complexity and heterogeneity, involves numerous interconnected components. This complexity makes the task of conducting effective technology assessment both particularly challenging and indispensable.

The complexity of telemedicine systems necessitates a thorough and nuanced approach to TA. Given the varied and evolving nature of telemedicine technologies, there is a pressing need for structured and comprehensive assessment frameworks. Such frameworks are essential for evaluating the efficacy, safety, and overall impact of telemedicine interventions, ensuring they meet the diverse needs of patients and healthcare providers effectively.

Various studies have addressed technology assessment (TA) in telemedicine from different perspectives, exploring various domains and highlighting diverse needs [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]. The necessity for a structured TA framework is evident from multiple aspects of these studies.

For example, Jacob et al. [2] and Smits et al. [4] underscore the importance of rigorous evaluation frameworks for eHealth tools. They advocate for a comprehensive approach that captures multiple dimensions of technology performance and user experience.

Incorporating stakeholder preferences is another area where a structured framework is crucial. Von Huben et al. [3] emphasize that including stakeholder feedback in TA processes enhances the relevance and acceptability of digital health solutions. This indicates that a structured framework is essential to ensure all relevant perspectives are considered, leading to more effective interventions.

The shift from digital health to digital well-being further reflects the need for a structured framework. Smits et al. [4] highlight that evaluating the impact of digital health technologies on overall well-being requires a framework that goes beyond traditional health metrics, capturing a broader range of effects.

Targeted evaluations of specific health interventions, such as telehealth-based cancer rehabilitation [5], cardiac rehabilitation [13], and telemedicine with clinical decision support for critical care [1], reinforce the importance of a structured framework. These studies show that assessing digital interventions in specialized healthcare domains benefits from an approach that addresses unique needs and challenges.

The application of TA frameworks is a common theme. Von Huben et al. [6,8] and Kitsiou et al. [15] discuss the need for systematic reviews and assessments to analyze the role of digital health in managing chronic diseases. A structured framework facilitates this process, ensuring evaluations are comprehensive and based on evidence.

Innovative approaches, such as the “Sandbox Approach” proposed by Leckenby et al. [7], reflect a proactive stance towards developing structured methodologies. This approach suggests that a flexible yet structured framework can adapt to the evolving landscape of healthcare technology.

Cost-effectiveness analysis, highlighted by Bonten et al. [9] and Jiang et al. [11], demonstrates the importance of incorporating economic considerations into TA. A structured framework that includes cost-effectiveness helps understand the financial implications of digital health interventions and supports informed decision-making. The systematic review by Vis et al. [10] highlights the critical role of health technology assessment frameworks specifically designed for eHealth. Their work emphasizes the need for tailored approaches that can address the unique challenges and opportunities presented by digital health technologies, reinforcing the importance of structured and comprehensive assessment methodologies in this rapidly evolving field.

The focus on peer-to-peer interactions in digital interventions for psychotic disorders [12] emphasizes the importance of including social dimensions in the TA process. A structured framework that considers these aspects ensures a more comprehensive evaluation of digital health interventions.

Critical assessments of home telemonitoring interventions [15] and specific respiratory interventions [16,17,18] highlight the importance of methodological rigor. Evaluating the quality of existing reviews and exploring evidence-based practices benefit from a structured framework that captures both strengths and limitations.

The evaluation of diagnostic tools, such as portable monitoring devices for diagnosing obstructive sleep apnea [19], and the longitudinal perspective on telemedicine applications [20] further support the need for a structured approach. Longitudinal evaluations and ongoing assessments are crucial for identifying effective applications and addressing evidence gaps over time.

Overall, the studies advocate for a comprehensive and structured approach to TA in digital health and telemedicine. They emphasize the importance of systematic evaluation frameworks that incorporate stakeholder preferences, address a broad range of health impacts, and consider economic and social dimensions. Developing and implementing such frameworks supports a holistic understanding of the strengths and limitations of current digital health evidence, leading to more effective and evidence-based solutions.

1.2. Role of Key Performance Indicators in Telemedicine Tecnology Assessment

Key performance indicators (KPIs) are increasingly discussed for their usefulness in measuring strategic parameters within complex systems, providing tangible and rigorous metrics that are valuable across multiple levels and domains (key performance indicator [21]). These indicators are becoming more commonly applied in the health domain, a rapidly evolving sector (KPIs in the health domain [22]). KPIs offer a structured way to measure and evaluate various parameters, making them essential tools in systems that require precise and multi-level assessment.

Telemedicine, a part of the health domain, represents a heterogeneous and complex system where KPIs can play a significant role in technology assessment across various fields of application, particularly where ongoing digitalization drives continuous innovation [23]. KPIs are fundamental in this context, as they provide objective quantifiable metrics, allowing for a clear assessment of how well telemedicine systems function. They enable the identification of both strengths and areas needing improvement, offering valuable insights for technology management and innovation within these dynamic environments. However, it is important to note that KPIs are often very specific and not universally standardized, which presents both opportunities and challenges. The specificity of KPIs means they must be carefully tailored to the unique aspects of each application or domain. For example, in the study by Canning et al. [24], clinical pharmacy quality indicators were found to lack uniformity and fail to measure outcomes directly, illustrating the difficulty of applying standardized KPIs in complex healthcare settings. Despite efforts to establish a consensus on pharmaceutical care bundles and outcome measures, only a limited number of indicators achieved consensus, highlighting the challenge of creating universally applicable KPIs in such a diverse field. Similarly, Garbelli et al. [25] demonstrated that while the implementation of standardized KPIs can lead to significant improvements in patient outcomes, such as reduced mortality, the need for standardization and structured implementation is critical for consistent results. The study found that increased KPI target achievements were linked to better patient survival rates, emphasizing the impact of standardized clinical practice.

In telehealth, Jackson et al. [26] identified variations in the reporting of performance metrics across different telehealth classes and study stages, further underscoring the challenges of KPI standardization. Clinical outcomes and patient satisfaction were commonly reported, but other performance metrics were less frequently documented, reflecting inconsistencies in KPI application.

Similarly, Baughman et al. [27] compared the quality of care between telehealth and in-person office-based care, finding that telehealth exposure led to better performance in specific KPIs related to testing and counseling. This study highlighted the favorable association between telehealth and quality of care, particularly in chronic disease management and preventive care, but also reflected variability in KPI implementation and measurement.

In analyzing the broader implementation of telehealth programs, Avanesova et al. [28] emphasized the importance of consistent KPI collection and routine analysis to support telehealth capacity building and international legislation. This study highlighted the role of KPIs in evaluating patient outcomes and ensuring effective telehealth interventions.

Further studies, such as those by Berlet et al. [29] and Kidholm et al. [30], have shown the importance of KPIs in evaluating new technologies and frameworks. Berlet et al. focused on the usability of a 5G-enabled telemedical system, using KPIs to assess technical performance and clinical applicability. Kidholm et al. assessed the face validity of the Model for Assessment of Telehealth, underscoring the importance of KPIs in evaluating telehealth effectiveness and suggesting areas for improvement.

Other research, including that by Schröder et al. [31] and Vo et al. [32], explored the impact of telehealth on emergency medical services (EMSs) and the integration of telehealth within incentive programs and the related specific KPIs. Schröder et al. [31] highlighted improvements in EMS operations through tele-EMS systems, while Vo et al. proposed an Integrated Telehealth Model (ITM) to incentivize telehealth adoption and quality improvement. Both studies illustrate the role of KPIs in optimizing healthcare delivery and assessing the impact of telehealth interventions.

Overall, while KPIs are integral to technology assessment in telemedicine—driving performance evaluation, supporting strategic decisions, and fostering ongoing enhancements—their specificity and lack of standardization require thoughtful design and implementation. As demonstrated by recent studies, the challenge of applying universally applicable KPIs across diverse healthcare settings remains significant, emphasizing the need for continuous refinement and standardization to enhance their effectiveness.

1.3. Study Context Within a National Project and Key Objectives

In the context of a national Italian project, detailed below in terms of its architecture, roles, and activities, significant emphasis was placed on TA activities and the role of key performance indicators (KPIs) within this framework.

The Italian National Institute of Health (ISS), in accordance with its institutional role within the health domain, collaborated on the project through various supporting activities, particularly focusing on TA and model integration. Among its contributions, the ISS conducted a thorough review of the scientific and documentary literature, which provided essential insights for guiding project partners.

In addition, the ISS developed a well-structured tool for technology assessment tailored to the project’s needs. This tool was designed to systematically evaluate various aspects of telemedicine projects, ensuring a comprehensive and rigorous assessment.

The first review [33], focusing on TA studies in telemedicine, identified the key characteristics necessary for an effective TA tool. Drawing on these insights, a TA tool was developed and validated, as detailed in [34]. This tool consists of five forms with subsections that project development units must complete, ensuring a thorough and consistent evaluation across all project components.

A second review [35], which concentrated on KPIs in telemedicine, highlighted the application-specific nature of KPIs and the general lack of standardization. This review revealed that KPIs are often tailored to specific applications, resulting in variability in their use and reporting. It also exposed the diverse and inconsistent KPI proposals from various national and international organizations. This variability in KPI standardization poses challenges for comparing and integrating KPIs across different telemedicine initiatives and emphasizes the need for more uniform KPI development and application.

As the project nears its conclusion, this study aims to achieve several interrelated objectives, with a focus on dissemination across the entire project and team. The first objective is to disseminate detailed information about the project’s structure, including partner architecture, roles, and activities. Building on this, the second objective is to quantitatively assess the improvements in TA practices within the project using the validated tool described in [34]. The third objective addresses insights gained from the analysis of KPIs, focusing on their limitations and the need for a structured validation process. Consequently, a KPI model proposed in the national Italian guidelines was examined, and a generalized evaluation model was applied to assess the adequacy of this KPI model based on relevant parameters and metrics.

By achieving these objectives, the study aims to enhance the overall effectiveness of the project, provide valuable insights for future initiatives, and contribute to the standardization and improvement of TA and KPI practices in telemedicine.

2. Methods

In alignment with the proposed objectives, the study design is organized into the following sections:

• Section 2.1: Project Architecture
  This section offers a comprehensive overview of the national project’s structure. It details how the project was conceptualized and organized, including the integration of the framework used. This part of the study describes the key components of the project, the roles of various participants, and the overarching goals that guided the implementation of the framework. It emphasizes the strategic alignment between the project’s objectives and the framework’s application, illustrating how the framework was operationalized to address the specific needs and challenges of the project.
• Section 2.2: Framework and KPI Tabular Schema Application for Enhanced Assessment Structure
  This section (a) briefly recalls the framework used in this study for the final assessments and (b) reports a detailed tabular schema outlining the KPIs proposed for assessing its application in the project. It includes a list of categorized KPIs in a schema arranged in a structured format for assessing the applicability or non-applicability of each KPI within the context of the project’s activities.

2.1. The Project Architecture

The Telemechron project (Telemedicine for Home-based Management of Patients with Comorbidities) was presented under the 2018 targeted research funding call as a network initiative (see funding section).

The Telemechron project represents a significant effort in targeted research focused on the integration of telemedicine in the management of chronic patients. It is a network-based initiative funded by the Ministry of Health, various regional authorities, and an autonomous province. The formal agreement for this project was established on 9 July 2020 under the research code NET-2018-12367206, with a project duration of 36 months, starting on 1 October 2020.

The project involves multiple healthcare institutions, including USL Toscana Nord Ovest in Tuscany, IRCCS Maugeri of Lumezzane in Lombardy, and the Azienda Provinciale per i Servizi Sanitari (Provincial Health Services Company) of the Autonomous Province of Trento. In addition, the Istituto Superiore di Sanità (ISS) is actively collaborating, especially in overseeing quality assurance and technology assessment.

The 36-month project is structured into four main activities, each corresponding to an operative unit (OU) and assigned a specific work package (WP) as follows:

• OU 1 → WP1: USL Toscana Nord Ovest focuses on the use of telemedicine in managing chronic kidney failure patients. This activity aims to enhance the management of patients through remote monitoring and intervention strategies.
• OU 3 → WP3: IRCCS Maugeri of Lumezzane addresses the needs of chronic heart failure patients with comorbidities such as diabetes through telemedicine solutions. The goal of this activity is to provide continuous monitoring and personalized care to improve patient outcomes.
• OU 4 → WP4: Azienda Provinciale per i Servizi Sanitari of the Autonomous Province of Trento works on telemedicine for managing type 2 diabetes mellitus. The activity focuses on home-based care solutions to support diabetes management and to improve long-term health outcomes.
• OU 2 → WP2: ISS (Istituto Superiore di Sanità) plays a connector/mediator role and leads the technology assessment efforts. This activity is essential for integrating and assessing telemedicine technologies across different domains, ensuring that clinical governance and regulatory standards are met. ISS is also responsible for the development and validation of multidimensional clinical governance indicators to monitor and ensure quality across the project’s activities.

By aligning each OU with a WP, the Telemechron project ensures that different aspects of telemedicine for chronic disease management are thoroughly explored while fostering collaboration among healthcare entities. The project aims to build a robust scalable model for the future of healthcare, addressing technological, regulatory, and clinical needs.

Figure 1 reports a sketch of the architecture of the project. There are three operational units (OU1, OU3, OU4) assigned to project development work packages (WP1, WP3, WP4), with one of them (OU1) also responsible for the overall project coordination. Additionally, there is another operational unit (OU2) assigned to WP2, responsible for quality assurance in the integration of the telemedicine model, which includes dynamic and iterative multidimensional technology assessment.

Overall, the Telemechron project represents a significant initiative aimed at enhancing home care for chronic patients through the implementation of telemedicine solutions, contributing to the development and innovation within the healthcare sector. Each partner in the project plays a distinct and complementary role, ensuring the success and effective execution of the initiative. Below is a detailed description of each partner’s role.

• USL Toscana Nord Ovest (Tuscany) (OU 1/WP1)
  Role: Coordination of the entire network project and implementation of activities in the Tuscany region.
  Focus: Study and experimentation of telemedicine services for patients with chronic kidney failure.
  Objective: To coordinate project activities to achieve integration goals and improve home care for patients with chronic kidney failure. This includes evaluating the effectiveness, efficiency, and sustainability of telemedicine solutions.
• IRCCS Maugeri of Lumezzane (Lombardy) (OU 3/WP3)
  Role: Study and implementation of telemedicine services within the Lombardy region.
  Focus: Patients with chronic heart failure and comorbidities.
  Objective: To explore the use of telemedicine to enhance the management and home care of patients with chronic heart failure and associated conditions.
• Azienda Provinciale per i Servizi Sanitari of the Provincia Autonoma di Trento (OU 4/WP4)
  Role: Study and implementation of telemedicine services in the Provincia Autonoma di Trento.
  Focus: Patients with type 2 diabetes mellitus.
  Objective: To test and evaluate telemedicine solutions that improve the monitoring and management of type 2 diabetes in the context of home care.
• ISS (Istituto Superiore di Sanità) (OU 2/WP2)
  Role: Coordination of technology assessment activities, multidimensional evaluation, and methodological support to all project partners.
  Focus: Technological and multidimensional assessment of telemedicine services.
  Objective: To provide an integrated and comprehensive evaluation of the telemedicine services implemented by the other partners, focusing on clinical effectiveness, safety, organizational impact, and financial sustainability.

Together, these competencies and responsibilities aim to cover a broad spectrum of chronic medical conditions, making a significant contribution to the progress and innovation in telemedicine and home care. The collaboration among partners enables a holistic and integrated approach to addressing challenges and maximizing the benefits of the Telemechron project.

2.2. Framework and KPI Tabular Schema Application for Enhanced Assessment Structure

2.2.1. Framework Used for the Assessment

The framework developed by the Italian National Institute of Health (ISS) is a central tool for conducting comprehensive evaluations across various domains of telemedicine services. It provides a structured approach to assess the planning, implementation, and outcomes of these initiatives (see in Box 1 the structure). Designed to address a broad range of necessary areas for successful telemedicine delivery, the framework allows for a thorough review of strategic alignment, organizational design, technical requirements, financial considerations, and service performance.

The framework serves as the primary tool for quality control, ensuring consistency and standardization across all project units. This systematic approach allows all participating units to report on key criteria in a consistent and standardized format, which is essential for maintaining objectivity, transparency, and alignment with national healthcare priorities.

Each OU within the project, while adhering to the overall framework structure, provides its own detailed framework based on the specific context and needs of its telemedicine service. The framework is divided into various sheets (Box 1), each corresponding to different components/aspects of the service, and they are further subdivided into subsections that guide the OUs in providing in-depth analysis and reporting. These sheets cover areas such as service functionality, target population, expected outcomes, and operational needs.

For example, the following can be observed:

• OU 1 (USL Toscana Nord Ovest) focuses on chronic kidney failure and provides a framework for this patient group.
• OU 3 (IRCCS Maugeri of Lumezzane) develops a framework addressing chronic heart failure and comorbidities.
• OU 4 (Azienda Provinciale per i Servizi Sanitari of the Provincia Autonoma di Trento) provides its own framework for telemedicine services for type 2 diabetes mellitus patients.

Each OU’s specific framework ensures that the ISS can carry out a detailed and systematic evaluation, as it gathers data and insights directly from the units working on the ground. This collaborative approach ensures that all critical components of the telemedicine services are evaluated and understood in context.

Once the project units complete and submit their responses using their frameworks, the ISS conducts a thorough evaluation, including a detailed audit to confirm the accuracy and completeness of the information. The audit process cross-references the data with established guidelines and best practices, ensuring that the services are both effective and efficient.

After the audit, the ISS provides feedback in the form of a detailed report. This feedback not only serves as an evaluation but also acts as a tool for continuous improvement. The report identifies areas of strength and areas needing improvement, offering actionable recommendations for enhancing the telemedicine services.

The iterative feedback loop fosters a culture of continuous quality improvement, enabling the project units to refine and adjust their services in response to ongoing evaluations. This ensures that telemedicine services remain adaptable and aligned with the evolving needs of healthcare delivery, thus enhancing the long-term success and sustainability of these services.

The framework is not just an evaluation tool; it serves as a dynamic dossier that documents all essential aspects of a telemedicine service. The sheets within the framework act as data collection forms to be completed by the project teams. Each sheet corresponds to a specific component/aspect of the service, such as service profiling, strategic alignment, design and implementation, adoption processes, and evaluation.

The detailed structure allows for a comprehensive documentation process, ensuring that all relevant information is systematically collected and clearly presented. This ensures consistency across all project units, facilitating the analysis of the effectiveness, scalability, and potential for the broader implementation of telemedicine services. The framework also serves as a reference for future projects, helping to identify best practices and key success factors.

Ultimately, this dossier-style framework (Box 1) enables a clear, organized approach to documenting, evaluating, and improving telemedicine services, supporting their effective implementation and fostering innovation and continuous learning throughout the project.

Box 1. Architecture of the framework.

Sheet No. 1: Profile of the Telemedicine Service or Suite of Telemedicine Services

   1.1

Name of the telemedicine service or suite of services

   1.2

Healthcare organization and reference region or autonomous province

   1.3

Operational unit

   1.4

Service or telemedicine suite manager

   1.5

Type of telemedicine service or services in the suite

Sheet No. 2: Strategic Framework of the Telemedicine Service or Suite of Telemedicine Services

   2.1

SWOT analysis of healthcare before introducing the telemedicine service or suite

   2.2

Care needs addressed by the telemedicine service or suite

   2.3

Functional description of the telemedicine service or suite

   2.4

Target population and therapeutic and socio-economic indications

   2.5

Evidence from the scientific literature

Sheet No. 3: Design and Implementation of the Telemedicine Service or Suite of Telemedicine Services

   3.1

Organizational and managerial requirements gap analysis

   3.2

Technical and technological requirements gap analysis

   3.3

Economic financial evaluations and procurement strategy for acquiring goods and services from the market

   3.4

Methodology adopted and stakeholders involved in the design and implementation

   3.5

Regulatory and compliance aspects for implementation

   3.6

Risk management

   3.7

Best practices

Sheet No. 4: Adoption of the Telemedicine Service or Suite of Telemedicine Services

   4.1

Criteria and procedures for service activation and management

   4.2

Methods and indicators for monitoring service quality and minimum performance thresholds (service level agreement)

   4.3

Service pricing and cost-sharing rules

   4.4

Change management

   4.5

Activation date of the telemedicine service or suite

   4.6

Volume of services delivered

Sheet No. 5: Evaluation of the Telemedicine Service or Suite of Telemedicine Services

   5.1

Dimensions and indicators for service evaluation (expectations, KPIs, etc.)

   5.2

Methods for collecting and analyzing data for service evaluation

   5.3

Results of service evaluation

   5.4

Lessons learned, critical success factors, and recommendations for large-scale adoption or transferability of the experience to other contexts

Both in specific phases and at the conclusion of the project, each item of the framework (for example, items 1.1 in the Box 1) was evaluated through both quantitative and qualitative assessments. The quantitative evaluations (objective of this study) involved the use of questions rated on a 6-point Likert scale, which are the focus of this project report. These ratings provided a systematic approach to measuring various aspects of the project based on predefined criteria. Each item was assessed on a scale ranging from “Strongly Disagree” to “Strongly Agree”, allowing for nuanced feedback and a comprehensive analysis of the project’s performance and outcomes.

In addition to the quantitative evaluations, a detailed report was prepared for submission to the Italian Ministry of Health. This report included a thorough qualitative assessment of the project, offering insights into the implementation processes, challenges encountered, and overall effectiveness. It presented a narrative analysis that contextualized the numerical data, providing a richer understanding of the project’s impact and contributions.

2.2.2. Tabular Schema of KPIs with the Structure of Assessment

Participants in the project were given the freedom to define specific key performance indicators (KPIs) based on the project’s objectives. This approach aligns with the literature review conducted in [23], which emphasizes the importance of tailoring KPIs to the specific goals of each project. A methodology was proposed to assess the applicability or non-applicability of nationally available KPIs, as proposed in a guideline [36] after comprehensive analysis performed by the ISS working group (WP2).

To facilitate this evaluation, a tabular grid was developed, allowing each work package (WP1, 3, 4) involved in project activity to indicate the priority level of the selected KPIs and further feedback This structured approach enabled a systematic comparison and prioritization of KPIs based on their relevance and importance for each component of the project. Each work package could then rate the KPIs using a defined priority scale.

Experts from the project work packages (WP1, WP3, WP4) and the technology assessment work package (WP2) evaluated the applicability of the proposed key performance indicators (KPIs) within the framework of a national guideline [36]. The goal of this evaluation was to assess how well the KPIs align with and support the objectives and standards outlined in the national telemedicine framework.

WP2 was responsible for developing the evaluation tables and specific metrics. The project units, which had defined specific KPIs for their respective areas, provided valuable input based on their practical experience. Their contributions were crucial in evaluating how effectively the proposed standard KPIs could be integrated into their projects and align with national guidelines. This collaborative approach ensured that both theoretical and practical considerations were addressed, leading to a more thorough assessment of the KPIs.

The tabular grid (

Table 1. Evaluation metrics for telemedicine services: indicator applicability and prioritization by work packages.

Category	(CODE)/Short Key (Italian Tongue)	Indicator	Applicable for WP (YES/NO)—If NO, Propose an Alternative	Priority for WP (1–5, 1 = min; 5 = max)
DIMENSION Volume of services provided
	(DIM01)/DIM-Dimensione assoluta	Absolute size: number of users followed/12 months
	(DIM02)/DIM-Dimensione assoluta dinamica	Dynamic aspect of the size indicator: number of users followed in the last 12 months/number of users followed in the previous 12 months
	(DIM03)/DIM-Copertura del target	Target coverage in %: % of users followed compared to the total number of users affected by the service’s targeted condition in the area of interest
	(DIM04)/DIM-Dimensione media	Average size: average number of contacts/month
	(DIM05/DIM-Dimensione media/utente	Average dimension: average number of contacts per month per user (applicable for telehealth and televisits in telemonitoring)
CONTINUITY IN TIME Duration and stability of the service
	(CON01)/CONTIN_TEMP-Durata	Duration: number of months of activity since service activation
	(CON02)/CONTIN_TEMP-Stabilità	Stability: dispersion indices (e.g., standard deviation, range) of the average dimension indicator over a reference time period
COMPLEXITY Organizational complexity of the service
	(COMPL01)/COMPL-Indicatore qualitativo	Qualitative indicator: type of professionals involved in service delivery (GP, specialist doctor, nurse, etc.)
	(COMPL02)/COMPL-Indicatore quantitativo	Quantitative indicator: number of operators involved in service delivery (person–months)/number of users
QUALITY Service response standards and performance
	(QUAL01)/QUAL-Standard di servizio	Service standard: standard response time
	(QUAL02)/QUAL-Performance di risposta	Response performance: number of services within the standard time
EFFICIENCY Service cost
	(EFFI01)/EFFICIEN-Costo totale annuo	Total annual cost of maintaining the service (staff, equipment, etc.)/number of users followed
EFFECTIVENESS Comparison with the population of users affected by the condition targeted by the telemedicine service but followed in a conventional manner in the area of interest
	(EFFE01)/EFFICA-Riduzione mortalità	Reduction in mortality: % of deaths in the last 12 months among users followed in telemedicine/% of deaths in the last 12 months among users followed in a conventional manner
	(EFFE02)/EFFICA-Riduzione incidenza re-ospedalizzazioni	Reduction in the incidence of re-hospitalizations among users: % of re-hospitalizations in the last 12 months among users followed in telemedicine/% of re-hospitalizations in the last 12 months among users followed in a conventional manner
	(EFFE03)/EFFICA-Riduzione n° giorni degenza	Reduction in the number of hospital days: number of hospital days in the last 12 months per user followed in telemedicine/% of hospital days in the last 12 months per user followed in a conventional manner
	(EFFE04)/EFFICA-Riduzione tempo trascorso in emergenza	Reduction in the time spent by users in emergency services and number of emergency room visits: time (hours) spent in the last 12 months in emergency services per user followed in telemedicine/time (hours) spent in emergency services in the last 12 months per user followed in a conventional manner
	(EFFE05)/EFFICA-Miglioramento qualità della vita	Improvement in quality of life: standard measures of quality of life, possibly modified ad hoc (e.g., SF health surveys, SF36, SF12 questionnaires)
USER SATISFACTION User satisfaction (patients and caregivers)
	(SAT01)/GRADIM-Indicatore qualitativo	Qualitative indicator: specific questionnaires administered to users (patients, caregivers)
	(SAT02/GRADIM-Drop-out assoluto	Absolute quantitative indicator (drop-out): number of users who voluntarily leave the telemedicine path/12 months (drop-out)
	(SAT03) GRADIM-Drop-out relativo	Relative quantitative indicator (drop-out): number of users who voluntarily leave the telemedicine path/12 months/number of users followed (drop-out)

) organizes various indicators into categories for evaluating the performance of telemedicine services.

This method ensured a collaborative and thorough approach to KPI selection, allowing each work package to contribute their insights and expertise. The resulting prioritization helped in focusing efforts on the most relevant and impactful KPIs, ultimately enhancing the project’s ability to test in a defined national KPI environment.

3. Results

In line with the study’s objectives and utilizing the proposed methodology, this section presents the results obtained within the project.

Through the collaboration of all work packages (WP1–WP4), as outlined in Section 2.1 of the project architecture, both the operational units (OUs) focused on specific projects (WP1, WP3, WP4) and the OU (OU2) responsible for technology assessment (TA) contributed to the project’s outcomes. These results are reflected in the evaluation of improvements through the framework validated in [34], which is summarized and referenced here, and in the development and application of a KPI evaluation model using defined and adaptable metrics and parameters within the field of telemedicine.

Section 3.1, “Quantitative Outcomes from the Framework”, details the final assessment of the three projects and the measure of improvement compared to the initial project position.

Section 3.2, “Quantitative outcome from the Tabular schema and metrics”, focuses on the definition and application of the KPI evaluation model, detailing the metrics and parameters used. The collaborative approach facilitated effective teamwork. The project OUs operate in three distinct areas of telemedicine—diabetes, cardiology, and nephrology—covering significant sectors of fragility and chronic conditions, providing direct experience with the practicality of KPIs. The TA OU offers mediation and connections between the three domains and intervention models.

3.1. Outcome from the Framework

3.1.1. Quantitative Outcome

Framework Development and Iteration:

Each work package (WP1, WP3, WP4) adhered to this framework (see Box 1), ensuring a comprehensive evaluation of every aspect of the telemedicine service. This included its design, which focused on the service’s conceptual framework and operational features, its implementation, which assessed the deployment processes, technical integration, user adoption strategies, and its evaluation, which involved analyzing performance metrics, user satisfaction, clinical outcomes, and overall impact. This approach provided a holistic view, ensuring that both the technical and human-centered aspects of the telemedicine service were thoroughly examined to support its optimization and scalability.

After each initial specific framework is developed, it undergoes several rounds of improvement. Each round involves a cycle of feedback and adjustments to refine the content, ensuring that the framework remains aligned with the overall objectives of the project and incorporates any necessary updates.

Review Cycles:

• Round 1: In the first cycle, each partner submits their initial feedback after compiling their part of the framework. This initial round helps identify any gaps or areas that need clarification or improvement.
• Round 2: After modifications are made based on the feedback from Round 1, the revised framework is reviewed again. Partners provide further input, helping to further refine the methodology and the structure of the framework. This round is focused on making sure the framework is comprehensive and that all elements are well defined.
• Round 3: In the final review round, all previous feedback is consolidated, and any remaining issues or inconsistencies are addressed. This round ensures that the framework is finalized and ready for the next steps.

Final Evaluation:

Once the final version of the framework is complete, it undergoes a quantitative evaluation. This evaluation uses a six-level Likert scale (see as an example the explanation in https://uk.surveymonkey.com/mp/likert-scale/ (accessed on 8 January 2025), a commonly used tool to measure responses across different criteria. In this case, all the items of the framework are assessed.

The six-level Likert scale used in the evaluation ranges from one to six, where

• one represents the lowest score (minimum evaluation);
• six represents the highest score (maximum evaluation).

To interpret the results, a satisfaction threshold is defined as

$$\mathrm{Threshold}={\displaystyle \frac{(1+6)}{2}}$$

(1)

• Scores above 3.5 reflect positive feedback, with satisfaction increasing as values approach six.
• Scores below 3.5 indicate negative feedback, with dissatisfaction intensifying as values approach one.

The evaluation was conducted systematically across all sheets and their respective items. For illustration, Box 2 highlights the items for sheet 4 assessed by means of the Likert scale.

During each iteration (as explained later), the ISS team performed both analytical and qualitative evaluations using the six-level Likert scale for individual items (e.g., 4.1–4.6). Following this process, an audit feedback report was generated to summarize the findings and provide constructive recommendations.

Box 2. Sheet 4 with the single items assessed using the six-level Likert scale.

4. Adoption of the Telemedicine Service or Suite of Telemedicine Services:

   4.1

Criteria and procedures for service activation and management

   4.2

Methods and indicators for monitoring service quality and minimum performance thresholds (service level agreement)

   4.3

Service pricing and cost-sharing rules

   4.4

Change management

   4.5

Activation date of the telemedicine service or suite

   4.6

Volume of services delivered

Reporting:

The evaluations and measurements were conducted by a team of four experts from WP2, ensuring a comprehensive and objective assessment aligned with the project’s objectives.

All experts from WP2 at the ISS brought extensive expertise in technology assessment and had significant experience in health technology assessment (HTA) across multiple domains. Their competencies spanned regulatory aspects, medical devices, and economic and social considerations, ensuring a multidimensional and robust evaluation process.

After the final evaluation is completed, an analytical report is prepared based on the final version of the framework and the feedback received throughout the review cycles. This report synthesizes the evaluation results and outlines key findings, areas for improvement, and recommendations for future adjustments. The report is then submitted to funding and supervising bodies for further review and to ensure that the project aligns with its intended goals.

In summary, this process ensures that the framework is constantly refined through a series of feedback loops, with partners working together to improve its quality. The iterative nature of this approach helps to identify and address issues early, ensuring that the final framework is a robust and effective tool for assessing telemedicine services. The quantitative evaluation provides an objective measure of its effectiveness, while the final reporting ensures transparency and accountability to the funding and supervising bodies.

Figure 2 illustrates the original butterfly diagram in Italian, serving as an example of the evaluation for one work package (WP) based on the six items in sheet 4 during the final round. This statistical graphical tool provides a clear representation of the evaluation outcomes, highlighting both the high level of acceptance and the absence of a tail below zero for any item.

The butterfly diagram, characterized by its symmetrical layout, displays the distribution of responses for each item on a Likert scale, with positive evaluations represented on one side and negative evaluations on the other. The absence of a negative tail indicates that no item received scores falling below the threshold of 3.5, underscoring the overall positive reception.

This visualization is particularly useful in showcasing the convergence of evaluations, as it emphasizes not only the collective agreement on the high scores but also the uniformity in the positive assessment across all items. Such a tool aids in effectively communicating the results, ensuring clarity for stakeholders and facilitating informed decision-making.

A similar situation was consistently observed across all five sheets and throughout the three project work packages (WPs). In each case, the evaluations demonstrated a high level of acceptance, with no significant deviations or negative assessments noted for any of the items analyzed. This uniformity across the various WPs highlights the robustness and effectiveness of the methodological approach employed, ensuring that the framework adequately addresses the unique requirements and objectives of each WP while maintaining a cohesive and systematic evaluation process.

Table 2. Average score recorded for each sheet and average increase compared to the first round. The data pertain to all projects (WP1, WP3, WP4) as a whole.

Sheet	Average Score at the End of the Evaluation Process (Round 3)	Average Increase Compared to the Initial Evaluation (Round 1)
No. 1: Personal Data of the Telemedicine Service or Suite of…	5.6	∆ = 1.8
No. 2: Strategic Framework of the Telemedicine Service or Suite of…	5.2	∆ = 1.9
No. 3: Design and Implementation of the Telemedicine Service or Suite of…	5.2	∆ = 1.9
No. 4: Adoption of the Telemedicine Service or Suite of…	5.3	∆ = 2.0
No. 5: Evaluation of the Telemedicine Service or Suite of…	5.3	∆ = 2.1

presents the average scores recorded for each sheet, along with the average improvement compared to the first evaluation round.

The data pertains to all WPs (WP1, WP3, WP4) as a whole. The iterative and collaborative nature of this process ensures that the framework continuously evolves to meet the project’s needs and expectations effectively.

3.1.2. Further Qualitative Considerations

Family involvement is essential in managing chronic conditions through telemedicine. In cases of chronic kidney failure, telemedicine provides families with tools to assist in managing dialysis, fluid balance, and dietary restrictions, enabling better at-home care and reducing hospital admissions. For type 2 diabetes, it supports families in monitoring blood glucose levels, ensuring medication adherence, and adopting necessary lifestyle changes, leading to improved blood sugar control and fewer complications. In chronic heart failure, telemedicine enhances family participation in tracking symptoms, adhering to treatments, and promoting healthier habits, resulting in better overall condition management.

By offering remote monitoring, real-time access to healthcare professionals, and continuous education, telemedicine empowers families to actively engage in the care process. This involvement improves disease control, minimizes hospitalizations, and enhances patient satisfaction, ultimately leading to a higher quality of life for patients and their families.

However, considering the specific focus of our study, it is important to highlight how the framework, with its tailored items, has contributed and can continue to contribute in these critical areas. Through a structured approach, the framework facilitated a comprehensive understanding of how telemedicine services can support the involvement of families in managing chronic conditions, such as chronic kidney failure, type 2 diabetes, and chronic heart failure.

In the results, it is crucial to highlight that the framework, especially through Sheet No. 2: Strategic Framework of the Telemedicine Service or Suite of Telemedicine Services, has significantly contributed to improving the management of patients with chronic conditions, such as chronic kidney failure, type 2 diabetes, and chronic heart failure, within the family setting. The framework facilitated a deeper understanding of the essential role that families play in managing these complex health conditions. It emphasized that by promoting a greater involvement of family members, telemedicine services can improve patient outcomes and the effectiveness of daily disease management. The framework’s design, by focusing on both patients and their families, helped ensure that family members were equipped with the tools and knowledge needed to support patients in managing their conditions at home.

The SWOT analysis (Box 1, item 2.1) conducted within the framework revealed gaps in the traditional healthcare system, particularly in the support for families in managing chronic diseases. It highlighted that prior to the introduction of telemedicine services, there was often inadequate support for families, who are crucial in ensuring adherence to treatment and making necessary lifestyle changes. By identifying these gaps, the framework underscored the potential of telemedicine to directly address these issues and improve family involvement in patient care. This helped improve the daily management of patients with chronic kidney failure, type 2 diabetes, and chronic heart failure by providing families with remote monitoring, education, and guidance.

The section on care needs addressed by the telemedicine service (Box 1 item 2.2) further showed how the service was tailored to meet both the patient’s and the family’s needs. With telemedicine, families received real-time access to healthcare professionals, educational resources, and reminders about the patient’s care plan, which greatly improved their capacity to manage the disease. This functionality ensured that families could monitor patients’ health on a daily basis, reinforcing the family’s active role in disease management. By addressing these needs, the framework helped to ensure that the families were better prepared to manage their loved ones’ conditions, ultimately leading to improvements in patient outcomes.

In the functional description (Box 1 item 2.3), the framework described how the telemedicine service was designed to support both patients and their families. It provided families with easy access to remote consultations, health monitoring tools, and personalized health advice. The framework emphasized that these features were essential in helping families manage the daily care of patients, particularly those with chronic kidney failure, type 2 diabetes, and chronic heart failure. By strengthening the communication between healthcare providers and families, telemedicine services directly contributed to improving the quality of care at home and ensuring more effective disease management.

The target population and therapeutic and socio-economic indications (Box 1 item 2.4) reinforced that the telemedicine service was specifically designed for patients with chronic diseases and their families. The framework recognized that the involvement of family members was critical to achieving positive health outcomes. By targeting families of patients with chronic kidney failure, type 2 diabetes, and chronic heart failure, the telemedicine service ensured that those responsible for daily care were given the support needed to manage their loved one’s condition more effectively. This strategic focus on families helped to improve their engagement in healthcare decision-making and better adherence to care plans, resulting in better long-term health outcomes.

Finally, the section on evidence from the scientific literature (Box 1 item 2.5) highlighted the growing body of research that supports family-centered care as an essential component of managing chronic diseases. The framework incorporated evidence showing that when families are actively involved in the care process, patients experience improved adherence to treatment, better management of lifestyle changes, and reduced healthcare utilization. Telemedicine services, as outlined in the framework, directly facilitated this by providing continuous support, education, and monitoring, ensuring that families had the resources they needed to manage their loved ones’ conditions effectively.

Overall, the framework has played a pivotal role in improving the daily management of patients with chronic kidney failure, type 2 diabetes, and chronic heart failure, particularly through enhancing family involvement. By identifying and addressing the specific needs of families, the framework has helped ensure that families are empowered to manage their loved ones’ health at home. Through improved communication, education, and real-time monitoring, the telemedicine service has contributed to better disease management, improved patient outcomes, and increased satisfaction among both patients and their families. This comprehensive approach has had a lasting impact on chronic disease management, reinforcing the importance of family-centered care in improving health outcomes.

3.2. Quantitative Outcome from the Tabular Schema of KPIs and Metrics

Participants in the project were given the freedom to define specific key performance indicators (KPIs) based on the project’s objectives (WP1, WP3, WP4). This approach aligns with the literature review conducted in [35], which emphasizes the importance of tailoring KPIs to the specific goals of each project.

In addition to this, experts from the project work packages (WP1, WP3, WP4) and the technology assessment (WP2) assessed the applicability of proposed KPIs within the framework of a national guideline [36], and this is the true outcome of this study aligned with the objectives. This evaluation aimed to determine how effectively these KPIs align with and support the objectives and standards set out in the national telemedicine framework.

WP2 developed the evaluation tables (Table 1) and the metrics described below. In this process, the project units (WP1, WP3, WP4), which had established specific KPIs for their own areas according to the literature trends [35], contributed their practical experience and project-specific insights assessing the set reported in [36].

Their involvement was important for evaluating how well the proposed KPIs fit within the context of their own projects and the national guidelines.

To assess the applicability of the proposed KPIs [36] in Table 1, we decided to develop and implement specific metrics that would allow for precise and large-scale quantification. These metrics are designed to provide a standardized and objective method for evaluating the relevance, applicability, and priority of each KPI within the context of the telemedicine project, ensuring a thorough and systematic assessment process.

The following parameters reported in

Table 3. The proposed metrics for assessing the applicability and priority of KPIs.

Parameter	Description
A-match-KPI	This parameter indicates the amount of KPIs in the guidelines deemed applicable.
A-match-KPI by category	This parameter indicates the amount of applicable KPIs within each category of the guidelines.
Priority-KPI	This parameter assigns a priority score to the applicable KPIs.
Priority-KPI by category	This parameter assigns a priority score to the applicable KPIs within each category.
Distance-from-GL-KPI	Considering that the KPIs proposed in the guidelines have a conventional priority of three (medium priority), this parameter expresses the signed distance (based on the assigned priority) from the KPIs proposed in the guidelines.
Distance-from-GL-KPI by category	Considering that the KPIs proposed in the guidelines have a conventional priority of three (medium priority), this parameter expresses the signed distance (based on the assigned priority) from the KPIs proposed in the guidelines, categorized by each category.

have been defined to assess the applicability and priority of KPIs based on the guidelines [36]:

The next sections report the applicability and priority (

Table 4. Applicability and priority assigned by a single project.

Category	CODE	Indicator	Applicability Score (max = 3 When All the Projects are Applicable; min = 0—When All the Projects Are Not Applicable)	Priority Prg1	Priority Prg2	Priority Prg3
DIMENSION Volume of services provided	DIM01	Absolute size: number of users followed/12 months	2	1	4	5
	DIM02	Dynamic aspect of the size indicator: number of users followed in the last 12 months/number of users followed in the previous 12 months	1	1	5	1
	DIM03	Target coverage in %: % of users followed compared to the total number of users affected by the service’s targeted condition in the area of interest	1	1	1	5
	DIM04	Average size: average number of contacts/month	2	1	3	3
	DIM05	Average dimension: average number of contacts per month per user (applicable for telehealth and televisits in telemonitoring)	2	1	4	3
CONTINUITY IN TIME Duration and stability of the service	CON01	Duration: number of months of activity since service activation	3	3	3	4
	CON02	Stability: dispersion indices (e.g., standard deviation, range) of the average dimension indicator over a reference time period	2	3	1	4
COMPLEXITY Organizational complexity of the service	COMPL01	Qualitative indicator: type of professionals involved in service delivery (GP, specialist doctor, nurse, etc.)	3	5	2	4
	COMPL02	Quantitative indicator: number of operators involved in service delivery (person–months)/number of users	2	5	5	4
QUALITY Service response standards and performance	QUAL01	Service standard: standard response time	2	3	1	3
	QUAL02	Response performance: number of services within the standard time	2	3	1	3
EFFICIENCY Service cost	EFFI01	Total annual cost of maintaining the service (staff, equipment, etc.)/number of users followed	2	5	5	4
EFFECTIVENESS Comparison with the population of users affected by the condition targeted by the telemedicine service but followed in a conventional manner in the area of interest	EFFE01	Reduction in mortality: % of deaths in the last 12 months among users followed in telemedicine/% of deaths in the last 12 months among users followed in a conventional manner	0	1	4	1
	EFFE02	Reduction in the incidence of re-hospitalizations among users: % of re-hospitalizations in the last 12 months among users followed in telemedicine/% of re-hospitalizations in the last 12 months among users followed in a conventional manner	1	1	4	4
	EFFE03	Reduction in the number of hospital days: number of hospital days in the last 12 months per user followed in telemedicine/% of hospital days in the last 12 months per user followed in a conventional manner	0	1	4	1
	EFFE04	Reduction in the time spent by users in emergency services and number of emergency room visits: time (hours) spent in the last 12 months in emergency services per user followed in telemedicine/time (hours) spent in emergency services in the last 12 months per user followed in a conventional manner	0	1	4	1
	EFFE05	Improvement in quality of life: standard measures of quality of life, possibly modified ad hoc (e.g., SF health surveys, SF36, SF12 questionnaires)	1	1	4	5
USER SATISFACTION User satisfaction (patients and caregivers)	SAT01	Qualitative indicator: specific questionnaires administered to users (patients, caregivers)	3	1	4	5
	SAT02	Absolute quantitative indicator (drop-out): number of users who voluntarily leave the telemedicine path/12 months (drop-out)	2	5	5	5
	SAT03	Relative quantitative indicator (drop-out): number of users who voluntarily leave the telemedicine path/12 months/number of users followed (drop-out)	2	1	5	5

) assigned by each project and the outcomes (

Table 5. The number of KPIs in the guidelines deemed applicable across different projects.

Project	A-Match-KPI
Prg1	8
Prg2	9
Prg3	16

,

Table 6. The number of applicable KPIs within each category of the guidelines for each project.

Category	CODES	A-Match-KPI-by-Categ Prg1	A-match-KPI-by-Categ Prg2	A-Match-KPI-by-Categ Prg3
DIM—5 indicators	DIM01-DIM05	0/5	4/5	4/5
CONTIN_TEMP—2 indicators	CON01-CON02	2/2	1/2	2/2
COMPL—2 indicators	COMPL01-COMPL02	2/2	1/2	2/2
QUAL—2 indicators	QUAL01-02	2/2	0/2	2/2
EFFICIEN—1 indicator	EFFI01	1/1	0/1	1/1
EFFECTIVENESS—5 indicators	EFFE01-05	0/5	0/5	2/5
SATISFACTION—3 indicators	SAT01-03	1/3	3/3	3/3

,

Table 7. The priority scores assigned to the applicable KPIs for each project.

Project	Priority-KPI
Prg1	32/100
Prg2	36/100
Prg3	66/100

,

Table 8. The priority scores assigned to the applicable KPIs within each category for each project.

Category	CODES	Priority-KPI-by-Categ Prg1	Priority-KPI-by-Categ Prg2	Priority-KPI-by-Categ Prg3
DIM (max value = 5 × 5 indicators = 25)	DIM01-DIM05	NA/25	16/25	16/25
CONTIN_TEMP (max value = 5 × 2 indicators = 10)	CON01-CON02	6/10	3/10	8/10
COMPL (max value = 5 × 2 indicators = 10)	COMPL01-COMPL02	10/10	2/10	8/10
QUAL (max value = 5 × 2 indicators = 10)	QUAL01-02	6/10	NA/10	6/10
EFFICIEN (max value = 5 × 1 indicator = 5)	EFFI01	5/5	5/5	4/5
EFFICACY (max value = 5 × 5 indicators = 25)	EFFE01-05	NA/25	NA/25	9/25
SATISFACTION (max value = 5 × 3 indicators = 15)	SAT01-03	5/15	15/15	15/15

,

Table 9. The signed distance of the applied KPIs from the conventional priority (priority = 3) proposed in the guidelines.

Project	Distance-KPI (Average Value = 3 × 20 Indicators = 60)
Prg1	32–60 = −28
Prg2	36–60 = −24
Prg3	66–60 = 6

and

Table 10. The signed distance of the applied KPIs from the conventional priority (priority = 3) proposed in the guidelines, categorized by each category.

Category	CODES	Distance-KPI-by-Categ (Average Value) Prg1	Distance-KPI-by-Categ (Average Value) Prg2	Distance-KPI-by-Categ (Average Value) Prg3
DIM (average value = 3 × 5 indicators = 15)	DIM01-DIM05	NA	16–15 = 1	16–15 = 1
CONTIN_TEMP (average value = 3 × 2 indicators = 6)	CON01-CON02	6–6 = 0	3–6 = −3	8–6 = 2
COMPL (average value = 3 × 2 indicators = 6)	COMPL01-COMPL02	10–6 = 4	2–6 = −4	8–6 = 2
QUAL (average value = 3 × 2 indicators = 6)	QUAL01-02	6–6 = 0	NA	6–6 = 0
EFFICIEN (average value = 3 × 1 indicator = 3)	EFFI01	5–3 = 2	5–3 = 2	4–3 = 1
EFFICACY (average value = 3 × 5 indicators = 15)	EFFE01-05	NA	NA	9–15 = −6
SATISFACTION (average value = 3 × 3 indicators = 9)	SAT01-03	5–9 = −4	15–9 = 6	15–9 = 6

) based on the metrics assigned in Table 3. We have chosen to anonymize the projects; thus, Prg1–Prg3 do not directly correspond to WP1-3. This choice is made because there is no direct correlation between the applicability/priority of a KPI and the quality of a specific project activity, as it merely reflects the KPI’s suitability within the project context.

Figure 3 shows a representation with a radar chart associated with the priority of the KPIs. For each KPI, both the CODE and the short keys in Italian defined in Table 1 are reported.

From the table, the following considerations emerge:

All DIMENSION indicators are considered applicable, but not all have been assigned the same priority. In an observational project with few participants, it is logical that DIMENSION indicators have a low priority. Given the highly innovative and complex setting, higher priority was assigned to the COMPLEXITY of service delivery, as well as to EFFICIENCY and SATISFACTION indicators.

However, some DIMENSION indicators are not applicable in certain study contexts. For instance, the “Target coverage in %” indicator is not relevant for a pilot service available only in specific areas of the region. Similarly, the “Dynamic aspect of the dimension indicator” cannot be applied in a project with a limited duration. It was also suggested that the “average number of contacts/month” indicator, which is considered applicable, can help evaluate the resources (time, professionals, etc.) needed to support the offered service.

For TEMPORAL CONTINUITY, the most applicable indicator is the duration indicator, which received an equal priority score of four alongside another indicator.

For COMPLEXITY (organizational complexity of the service), the most applicable indicator is the qualitative one, specifying the types of professionals involved in service delivery. Both complexity indicators received the highest priority score of five.

For SERVICE QUALITY, both proposed indicators are applicable and, when applicable, are considered with medium priority (score of three). They are not applicable in real-time response scenarios.

In one project, the proposed EFFICIENCY indicator (“Total annual cost”) is not applicable because it is managed by central management control rather than the project team. When applicable, this efficiency indicator is considered to be of high priority (scores of 4–5).

EFFECTIVENESS indicators were mostly deemed not applicable across projects due to their unsuitability for feasibility studies or pilot projects with a limited number of participants. Although not applicable, these indicators received varied priorities depending on the nature of the telemedicine project, with priority 1 for feasibility studies and high priority (4–5) for studies where they are applicable with large numbers. It was noted that the “Improvement in quality of life” indicator was considered to be of high priority (4–5), emphasizing the importance of evaluating whether telemedicine use satisfies the user. If the quality of life does not improve or remain stable, even with cost savings, the service might not be a viable alternative to conventional methods.

In all three projects, a SATISFACTION indicator was considered applicable. The qualitative indicator was the most applicable and received the highest priority (score of five). It was noted that the “Qualitative indicator” provides crucial feedback for building and implementing the service in a real-world context. Meanwhile, the “Absolute quantitative indicator (Drop-Out)” is essential for ensuring the validity of study results and understanding which types of users are less inclined to use telemedicine.

Table 5 reports the number of KPIs in the guidelines deemed applicable across different projects. This table provides an overview of how well each project aligns with the KPIs outlined in the guidelines, showing the overall applicability of the KPIs in each project. The values indicate the extent to which the KPIs recommended in the guidelines are considered relevant and implementable for each project.

Table 6 reports the number of applicable KPIs within each category of the guidelines for each project. This table offers a detailed breakdown of KPI applicability by category, allowing for a more granular view of how well KPIs within specific categories align with the guidelines across different projects. It highlights which categories are better represented and which may require additional focus.

Table 7 reports the priority scores assigned to the applicable KPIs for each project. This table assigns a priority score to each KPI based on its relevance and importance for the project, reflecting how critical each KPI is considered in the context of achieving project goals. It helps in identifying which KPIs are prioritized within each project.

Table 8 reports the priority scores assigned to the applicable KPIs within each category for each project. This table provides insights into the prioritization of KPIs within specific categories, indicating how each category’s KPIs are valued in the context of each project. It highlights the relative importance of KPIs within different categories.

Table 9 reports the signed distance of the applied KPIs from the conventional priority (priority = 3) proposed in the guidelines. This parameter expresses how the priority of the KPIs used in each project deviates from the conventional medium priority (3) set in the guidelines. The signed distance indicates whether the priorities are higher or lower compared to the guideline’s standard.

Table 10 reports the signed distance of the applied KPIs from the conventional priority (priority = 3) proposed in the guidelines, categorized by each category. This table provides a category-specific view of how the priorities of KPIs in each category deviate from the conventional medium priority (3). It helps to understand deviations in priority on a more granular level, focusing on specific categories within the projects.

This structured approach helps to understand not only the applicability of KPIs but also their prioritization and how closely they align with the guidelines in different project contexts.

Specific considerations

Based on Table 5 that reports the number of KPIs from the guidelines that are deemed applicable across different projects, the following consideration emerges: specifically, Project Prg3 shows the highest applicability with 16 KPIs applicable, indicating that a greater proportion of the recommended KPIs are relevant and can be implemented in this project. In contrast, Project Prg1 shows the lowest applicability, with eight KPIs applicable, suggesting that fewer of the KPIs from the guidelines are considered relevant or feasible in this context. This variation highlights how different projects may vary in terms of the suitability of the KPIs.

By examining the number of applicable KPIs within each category (Table 6), it is possible to identify areas of strength and those requiring additional focus. For example, in Project Prg1, the DIM (dimension) category shows no applicable indicators, while the CONTIN_TEMP (temporal continuity) and COMPL (complexity) categories are better represented. In Project Prg2, the EFFICIEN (efficiency) and EFFECTIVENESS (efficacy) categories have fewer applicable indicators, whereas Project Prg3 shows higher applicability across most categories. This breakdown helps pinpoint which categories of KPIs are effectively addressed and which need further attention in each project.

Table 7 reports the priority scores assigned to the applicable KPIs for each project. This table reflects the relative importance of each KPI in the context of achieving project goals, as assessed through a priority scoring system. A higher priority score indicates that the KPIs are deemed more critical to the project’s success. For example, Project Prg3 has the highest priority score of 66 out of 100, suggesting that the KPIs selected for this project are considered highly important for achieving its objectives. Conversely, Projects Prg1 and Prg2 have lower priority scores (32 and 36, respectively), indicating a relatively lower emphasis on these proposed KPIs in those projects.

Table 8 presents the priority scores assigned to the applicable KPIs within each category for each project. This table offers insights into how KPIs are prioritized within specific categories. It shows how each category’s KPIs are valued in the context of the project, providing a clear picture of which categories are receiving more focus and which are less prioritized. For instance, Project Prg3 assigns the highest priority to the COMPL (complexity) and CONTIN_TEMP (temporal continuity) categories, while the DIM (dimension) and QUAL (quality) categories receive varying levels of attention. This categorization helps in understanding which aspects of the project are considered most crucial according to the priority scores.

Table 9 reports the signed distance of the applied KPIs from the conventional priority level set by the guidelines (priority = 3). This parameter indicates how the priority of the KPIs in each project deviates from the medium priority standard established in the guidelines. Positive distances suggest that the KPIs are prioritized higher than the conventional level, while negative distances indicate a lower prioritization. For example, Project Prg3 has a positive signed distance (+6), implying that the KPIs in this project are considered more critical than the medium priority level. In contrast, Projects Prg1 and Prg2 have negative distances (−28 and −24, respectively), reflecting a lower priority alignment compared to the standard.

Table 10 shows the signed distance of the applied KPIs from the conventional priority, categorized by each KPI category. This table provides a detailed view of how the priorities of KPIs in each category deviate from the medium priority level set by the guidelines. It helps to understand deviations in priority on a more granular level. For example, the DIM (dimension) category in Project Prg3 shows a positive distance of +1, suggesting a slight increase in priority compared to the medium level. Conversely, the EFFICACY (efficacy) category in Prg3 shows a significant negative distance (−6), indicating a much lower prioritization relative to the guidelines. This category-specific analysis allows for a deeper understanding of how different aspects of the projects align with or diverge from the guideline priorities.

Overall, this structured approach provides a comprehensive understanding of KPIs proposed in [29], encompassing applicability and prioritization across different projects. By examining the overall applicability, category-specific applicability, priority scores, and deviations from the conventional priorities, stakeholders can gain valuable insights into how well the KPIs proposed align with the guidelines and how they are prioritized within each project context. This detailed analysis helps in assessing the effectiveness of the KPI framework and identifying areas for improvement in aligning with the guidelines.

General Comment

The results presented in the tables provide valuable insights into the applicability and prioritization of KPIs across different projects. Analyzing the data reveals the following key observations:

• Applicability and alignment: The varying numbers of applicable KPIs across projects, as shown in Table 5, indicate that each project aligns differently with the guidelines. Projects like Prg3 demonstrate higher applicability, suggesting a stronger alignment with the recommended KPIs, while others, such as Prg1 and Prg2, show lower applicability. This disparity highlights the need for further tuning of the KPI selection process to ensure that the KPIs are better tailored to the specific contexts of each project.
• Category-specific insights: The detailed breakdown of KPI applicability by category in Table 6 shows that certain categories, like DIM (dimension) and QUAL (quality), are not consistently well represented across all projects. This suggests that specific KPIs within these categories may need to be adjusted or more carefully selected to improve their relevance and effectiveness in different project settings.
• Priority scores and deviations: The priority scores reported in Table 7 and Table 8 indicate how critical each KPI is considered within the projects. The discrepancies between the assigned priority scores and the conventional priority levels (as shown in Table 9 and Table 10) suggest that there are significant deviations in how KPIs are prioritized. Some projects prioritize KPIs more highly than the conventional medium level, while others fall short. This underscores the need for a reassessment of KPI priorities to ensure they accurately reflect the project’s goals and context.
• Need for tuning: Overall, the results point to a need for tuning the KPI framework to better fit the specific needs of each project. The variation in applicability and priority highlights that a one-size-fits-all approach may not be optimal. Instead, a more tailored approach that considers the unique characteristics and requirements of each project could lead to more effective KPI implementation and evaluation.

In summary, while the analysis provides a comprehensive overview of KPI applicability and prioritization, it also underscores the necessity for refining the KPI selection and prioritization processes. Adjusting and tuning specific KPIs to better align with project goals and contexts will enhance their relevance and utility, ultimately contributing to more successful project outcomes.

4. Discussion

The Telemechron project strategically targets the management of chronic kidney failure, chronic heart failure with comorbidities, and type 2 diabetes mellitus—conditions that not only pose significant health risks but are often accompanied by additional frailties, making their management even more complex. The project underscores the importance of addressing these intertwined chronic conditions and associated frailties, as they require continuous monitoring and multifaceted care strategies. Telemedicine plays a crucial role in this context by offering remote monitoring and timely interventions, which are essential for managing both the primary conditions and their complementary frailties. This approach not only enhances patient outcomes but also helps reduce the burden on healthcare systems by minimizing hospital visits and improving resource allocation. As the project nears its conclusion, the study has focused on the following key areas: sharing detailed insights into the project’s structure, analyzing and quantifying improvements in TA practices (first point of view), and examining the effectiveness and limitations of KPIs (second point of view). By addressing these complex chronic conditions and their related frailties, the Telemechron project highlights the transformative potential of telemedicine in making healthcare more efficient, accessible, and patient-centered.

The discussion is structured into the following four sections:

Section 4.1 “Framing the Contribution of This Study Within the Telemechron Project”

This section contextualizes the study’s role in advancing the Telemechron project by highlighting the integration of efforts from all operational units (OUs) and work packages (WPs). It also uses flowcharts to illustrate how the study is positioned within the project’s activities and its previous dissemination efforts.

Section 4.2 “Output from the Two Points of View”

This section examines the results derived from the study’s analysis of two critical perspectives. It discusses the outcomes from the application of the technology assessment tool and the intervention model for KPIs, providing insights into the metrics and parameters used.

Section 4.3 “Expanded Considerations on the Telemechron Framework for Telemedicine”

This section provides an expanded analysis of the Telemechron project’s multidimensional framework, highlighting its innovative approach to telemedicine integration and evaluation.

Section 4.4 “Work in Progress and Suggestions for Future Investigations”

This section reviews the status of the project and outlines areas for future research. It offers recommendations for refining KPI models, integrating advanced technologies, and conducting comparative studies to enhance the overall evaluation framework.

4.1. Framing the Contribution of This Study Within the Telemechron Project

This proposed study serves as a significant contribution from the collaborative efforts of the entire workgroup, which consists of four operational units (OUs) assigned to the four distinct work packages (WPs) of the project. It is specifically focused on technology assessment (TA) and the applicable key performance indicators (KPIs) within the Telemechron project. Highlighting the journey to this project output is crucial, as it is encapsulated in a scientific article.

This contribution was made possible by two critical ongoing review studies—one focused on TA [33] and another on KPIs [35]—both conducted by OU 2/WP2. These studies were essential in aligning with the project’s goals and in the validation of a pivotal tool, the framework [34], which was conducted by OU 2/WP2. The review study on TA [33] was fundamental in pinpointing the appropriate TA framework tool, which was subsequently validated [34] and applied in this project output article.

Similarly, the review study on KPIs was indispensable for addressing the nuances of TM metrics within the project and identifying robust metric models. The scientific article we present provides a comprehensive final assessment based on the established framework and introduces a thoroughly evaluated set of proposed KPIs. It elaborates on and applies well-defined parameters for a model of intervention, specifically focusing on the selection and definition of KPIs. This study involved all four OUs/WPs, ensuring a holistic and integrated approach to the assessment through the use of innovative and precisely defined parameters. This detailed methodology ensures that the project’s objectives are met with high accuracy and efficiency, providing in-depth insights into the effectiveness and impact of the Telemechron project.

The study presented here constitutes a project report that examines two key points of view of technology assessment within the context of telemedicine development. The first point of view revolves around the use of a framework introduced in the national Telemechron project [34]. This framework, designed to systematically evaluate telemedicine applications, is based on a comprehensive dossier provided to project partners. This dossier underwent several rounds of review and audit to ensure its robustness and relevance. The report details the final assessment of this framework with a quantitative approach, offering insights into its effectiveness in guiding the evaluation and implementation of telemedicine solutions.

The second point of view involved also presenting a set of KPIs to the partners, derived from a nationally relevant document [36]. These KPIs were organized in a tabular format to facilitate their assessment. The purpose of this exercise was to evaluate the applicability of these KPIs to the telemedicine model implemented across three major chronic conditions. This analysis seeks to understand how a generic set of KPIs aligns with or deviates from the specific requirements of telemedicine applications in chronic disease management.

This second point of view should be considered using the following three important factors:

• Framework design and KPI development: The framework required partners to create and propose specific KPIs tailored to their telemedicine applications. These KPIs were critically assessed within the technology assessment framework and received highly favorable evaluations. This indicates that the framework not only provided a structured approach for assessment but also facilitated the development of relevant and actionable KPIs.
• Scientific trends and literature review: A review of the scientific literature conducted as part of the project highlighted a trend towards the creation of specific targeted KPIs that align with the unique aspects of telemedicine and its application domains [35]. This trend underscores the need for KPIs that are closely tied to the specific contexts and objectives of telemedicine initiatives rather than relying on generic metrics.
• Diverse focuses in KPI design: An analysis of national and international reference documents [37,38,39,40,41] shows that different entities prioritize various aspects in their KPI design and proposals. For instance, international organizations such as the WHO [37] and national bodies like the NHS [40] or ATA [38] have different focal points depending on their healthcare system visions and objectives. This variation reflects the broader debate on how to best measure and assess telemedicine effectiveness in different healthcare settings and for diverse patient populations.

4.2. Output from the Two Point of Views

The added value from the first point of view of the study is a quantitative final assessment within a telemedicine project, measuring improvement from initial conditions in line with the true and current objective of the study. This includes the demonstrated utility of the proposed tool for technology assessment in the field of telemedicine. All of this has been achieved through a dedicated tool, the framework described and validated in [34].

This technology assessment tool has enabled an evaluation of telemedicine services by providing a comprehensive overview of the service, including its scope, features, and functionalities. It assesses how well the service aligns with strategic goals and integrates with existing healthcare systems. Additionally, the tool reviews the quality of the service’s design and the effectiveness of its implementation strategies. It assesses the level of acceptance and usage among patients and healthcare providers and measures performance against established metrics and standards. Overall, the tool demonstrates its effectiveness by delivering tangible improvements across these areas, offering a structured approach to enhance and optimize telemedicine services. The framework’s strengths are its flexibility and adaptability, which are crucial for assessing various telemedicine contexts. This need for a versatile model aligns with findings from multiple studies and reviews on technology assessment, including Mackintosh et al.’s emphasis on integrating telemedicine with clinical decision support systems [1], Doupi’s focus on comprehensive evaluation in health informatics [42], and Ekeland and Grøttland’s patient-centered approach using the MAST model [43]. These insights, alongside earlier studies by Giansanti et al. [44,45], underscore the importance of a flexible and systematic evaluation framework that addresses diverse needs and contexts effectively.

The results from the second point of view investigated have provided insights into the applicability and limitations of a general set of key performance indicators (KPIs) proposed based on a nationally relevant document [36].

This second point of view highlights the following three specific added values:

• Evaluation Model and Framework
  The first added value is the proposed evaluation model based on a schema (Table 1). This schema starts with indicators derived from a national Italian guideline (columns #1, #2, #3) and offers an intervention and analysis framework (columns #4 and #5). Although this schema is grounded in a national guideline, it can serve as a universal evaluation model. It is adaptable, allowing for the replacement of categories and dimensions with those from other reference documents as needed. This flexibility ensures that the model remains relevant across various contexts and settings.
• Defined Metrics for Applicability
  The second added value is the set of defined metrics (see Table 3) used to assess the applicability and applicability rate of the proposed KPI set. While these metrics are applied within a national context, they possess generalizability and can be utilized with any KPI set derived from other national or international documents. This feature enhances the model’s versatility, allowing it to be adapted for diverse evaluation needs.
• Alignment with the Literature and Specific KPI Definitions
  The third added value confirms the necessity of defining specific KPIs rather than relying on predefined sets, such as those outlined in the Italian national guidelines [36]. This highlights the importance of developing tailored KPIs that precisely address specific evaluation needs rather than using generic one-size-fits-all measures.

4.3. Expanded Considerations on the Telemechron Framework for Telemedicine

Expanding further on the analysis, it can be emphasized that the framework developed for the Telemechron project addresses multiple domains of interest related to telemedicine and its integration into the healthcare domain (also including aspects related to KPIs). This framework stands out as an innovative tool compared to other telemedicine frameworks previously encountered [33,34], which have primarily focused on metrics. The framework we are referring to here can be defined as a dossier-style framework, offering a holistic and multidimensional approach that goes beyond simple metric analysis and embraces a comprehensive evaluation of all relevant aspects of telemedicine services.

In contrast to frameworks that mainly emphasize technical metrics and service performance, the Telemechron framework incorporates a broader spectrum of dimensions, such as organizational design, strategic alignment, technological requirements, financial considerations, and service outcomes. This multifaceted approach ensures that all critical elements are thoroughly evaluated, allowing for continuous improvement and a better integration of telemedicine services into the healthcare system.

Comparing this approach to a different socially inspired framework, we can look at [46] “A Socially Inspired Framework for Human State Inference Using Expert Opinion Integration”. This framework proposes a group decision technique using ordered weighted averaging (OWA) to aggregate opinions from different classifiers, ultimately aiming to infer human states using multiple cues or signals. The study discusses biosensing, which involves two key processes, namely data acquisition and information inference. The socially inspired framework for human state inference works by integrating multiple cues from various experts. The main concept behind this approach is that conventional inference algorithms are treated as inference experts, and the framework utilizes expert opinion elicitation to improve the inference process. The key elements of the socially inspired framework include the following:

Multiple cues: the framework draws on various sources of information.

Multiple experts: different experts contribute, each with different levels of expertise.

Expertise association: experts may have specialized knowledge in different aspects of human states and cues.

Consensus procedures: different procedures are used to arrive at an agreed opinion or outcome, such as identifying the human state in this context.

The paper demonstrates [46] the framework by inferring fatigue as a human state and comparing it with previous studies, showing that this approach leads to better results in terms of accuracy.

Both frameworks—the Telemechron framework and the socially inspired framework for human state inference—share a multidimensional and expert-driven approach, albeit in different fields.

The Telemechron framework focuses on evaluating telemedicine services through a comprehensive lens, covering various aspects of healthcare delivery, from technology to organizational integration. Its use of multiple experts within makes it a collaborative and iterative process for improving telemedicine systems.

The socially inspired framework, on the other hand, is more focused on human state inference, utilizing expert opinion aggregation and various cues to infer human conditions like fatigue. This framework integrates diverse experts (e.g., those with varying expertise in biosensing technologies) to arrive at a consensus on human states.

Both frameworks use expertise and an aggregation of multiple perspectives to produce better outcomes—whether it is for improving telemedicine systems or accurately determining human states. The Telemechron framework offers a comprehensive and systematic approach by integrating multiple dimensions of healthcare, making it suitable for a wide range of telemedicine services. In contrast, the socially inspired framework focuses more specifically on health state inference through signal integration, providing a specialized solution that excels in improving accuracy in specific areas, such as fatigue inference. Both frameworks have their strengths; while Telemechron excels in a holistic approach to telemedicine, the socially inspired framework enhances precision and effectiveness in particular health-related assessments.

Thus, while the domains and specific applications differ, the underlying principles of multidimensional analysis, expert collaboration, and continuous feedback loops provide a solid foundation for innovative and adaptive approaches in both frameworks.

4.4. Work in Progress and Suggestions for Future Investigations

The project is still ongoing, and this study focuses on a portion of the technology assessment activities. Other specific contributions not related to technology assessment but concerning the design and implementation of telemedicine services come from the work packages WP1, WP3, and WP4, which focus on design and protocols, as illustrated by the study reported in [47] proposed by OU3/WP3. This study [47] focuses on evaluating a telemedicine project aimed at improving home-based management of heart failure and type 2 diabetes by promoting lifestyle changes rather than assessing the technology itself. It compares the impact of the project—providing teleconsultations, daily monitoring, and personalized support—against routine care, with the goal of improving outcomes such as physical activity and disease management.

Further developments are anticipated in both technology assessment (OU2/WP2) and the project’s specific operational units (OU1/WP1, OU3/WP3, OU4/WP4). Additionally, a detailed analysis of the content of the frameworks completed by the three operational units is planned. The comprehensive and multifaceted results from the analysis of the frameworks will be reviewed in collaboration with project partners and presented through detailed reports to the relevant oversight bodies. Following this, there will be opportunities for further dissemination of the findings at both national and international levels, ensuring that the insights gained from the technology assessment are widely shared, contributing to the broader body of knowledge and potentially influencing future developments in telemedicine.

The developed framework and KPI model have demonstrated their utility in evaluating key aspects of telemedicine, showing concrete improvements and useful metrics for KPI applicability. Future research should focus on the following:

• Enhanced utilization of metrics: exploring the application of metrics in various telemedicine contexts and internationally.
• Refinement of KPI models: continuing to develop context-specific KPIs for deeper insights into telemedicine services.
• Integration with advanced technologies: adapting the framework to include new technologies such as AI and machine learning.
• Cross-national comparisons: comparing the effectiveness of the framework and metrics across different countries and healthcare systems.
• Feedback and iteration: gathering feedback from stakeholders to refine the metrics and model.
• Long-term impact analysis: conducting longitudinal studies to assess the sustainability and effectiveness of the framework and metrics over time.

5. Conclusions

The study represents a significant contribution to the Telemechron project by integrating the collaborative efforts of all four OUs across its distinct WPs. Central to this contribution are advancements in TA and the design and application of a model for assessing a set of KPIs for telemedicine within the project.

The framework, a dedicated tool for TA, has proven effective in quantitatively assessing various aspects of telemedicine services and measuring improvements following three rounds of enhancement initiatives. Thanks to the collaborative input from all OUs, the KPI analysis model has yielded valuable insights into the applicability of the proposed KPI set. It has also provided useful and generalizable metrics and parameters applicable to both national and international telemedicine contexts.

Looking ahead, the project is set to continue evolving. Future research should focus on refining KPI models, exploring their application across different telemedicine settings, and integrating emerging technologies such as AI. Additionally, cross-national comparisons will be essential to evaluate the effectiveness of the framework and metrics across diverse healthcare systems. Collecting stakeholder feedback and conducting longitudinal studies will be crucial for further refining the framework and enhancing its applications in telemedicine.