The Use of Business Intelligence Software to Monitor Key Performance Indicators (KPIs) for the Evaluation of a Computerized Maintenance Management System (CMMS)

Abstract

The increasing use of electromedical equipment in hospital care services necessitates effective management of complex devices often unsupported by existing control systems. This paper focuses on developing a pool of evaluation indices for the Clinical Engineering Department (CED) of the ASST Grande Ospedale Metropolitano Niguarda in Milano (Italy), aiming to enhance awareness of the economic value, assess operational units, and optimize maintenance processes. Leveraging business intelligence, this study identifies 18 key performance indicators (KPIs) across logistics, technical, and equipment management categories. An interactive dashboard, implemented using Power BI, facilitates dynamic analysis and visualization of these KPIs, providing insights into the maintenance efficiency and obsolescence of medical devices. It offers a comprehensive framework for ongoing monitoring and decision-making. The results showcase the potential of the developed KPIs and dashboard to enhance operational insights and guide improvements in the healthcare facility’s maintenance processes.

1. Introduction

Currently, there is a monumental digital revolution characterized by the emergence, evolution, and widespread proliferation of digital infrastructures and tools. This transformative shift is reshaping the very paradigms in which individuals exist, operate, and engage in our daily endeavors [1]. The realm of public healthcare, once considered somewhat resistant to rapid technological change, is now fully immersed in the tide of digital transformation and technological innovation, mirroring trends seen across various sectors of society and industry [2,3,4].

The gradual integration of digital technologies into healthcare settings has become a driving force for advancing care delivery methods, fostering precision and personalization in treatment approaches. Concurrently, it has spurred the implementation of novel organizational and management strategies within clinical structures [5,6,7]. The transformative impact of digital health is clearly not limited to clinical therapies but also extends to the management of biomedical technologies and the evolution of operational management models within healthcare systems [3,8].

This convergence of digital advancements and healthcare represents a pivotal moment in the industry’s evolution, with profound implications for patient care, organizational efficiency, and overall healthcare delivery. Navigating this digital landscape reveals the increasing importance of embracing and harnessing technological innovations to drive positive change and ensure the continued advancement of healthcare services.

In this context, the role of clinical engineering is crucial in the healthcare setting. It is an area of biomedical engineering, defined as the application of engineering principles and technologies to enhance the quality of healthcare services. This includes improving organizational processes, acquiring and managing equipment, and developing or adapting hospital information systems and telemedicine networks [9].

The positive outcomes achieved in terms of economic management and safety quickly led to the widespread adoption of the Clinical Engineering Department (CED) in the United States, Canada, and major European countries. Recognizing its importance, the World Health Organization (WHO) has repeatedly advocated for the establishment of CED in both industrialized and developing countries [10].

The introduction of clinical engineering expertise has resulted in reduced risks associated with technology use in healthcare facilities and facilitated the controlled adoption of new technologies, leading to cost savings. Additionally, clinical engineering oversees maintenance processes and ensures the safe use of medical devices through risk analysis, safety plans, and incident monitoring [9].

Considering the different activities of a CED, equipment data acquire a lot of importance, and it is crucial to keep updated on the status of clinical devices to ensure a high standard of care quality, reduce expenses, and minimize the risk of adverse events for patients and users [11].

The intricacy of electromedical equipment, characterized by its quantity and diversity, presents challenges in managing these devices effectively [12]. To address this growing complexity, CEDs implement management software systems called Computerized Maintenance Management Systems (CMMS) to ensure the safe and efficient management of the hospital’s medical equipment [13].

CMMS are software packages that utilize connections to databases containing data related to a company’s device support activities to digitize documentation of all activities concerning them [14]. The main functionalities that a CMMS should support, as identified by the WHO, include the following:

• Institutionalizing and interconnecting data within a healthcare technology management program;
• Contributing to the organization and monitoring of inspections and preventive maintenance;
• Tracking repairs;
• Monitoring equipment performance indicators;
• Monitoring clinical engineering staff performance indicators;
• Generating reports for planning training programs;
• Providing libraries of regulatory requirements and safety information;
• Generating documentation suitable for accreditation by regulatory and standard organizations [15].

Utilizing a CMMS offers numerous advantages over traditional paper-based methods, including easier data storage and retrieval, streamlined maintenance management, enhanced patient safety, reduced workload and working time, decreased risk of human error, and improved tracking of medical equipment throughout its life cycle [16,17,18]. Furthermore, the adoption of a CMMS aligns with the increasing demand across all sectors to gather structured data for processing, analysis, and exchange between operational units.

By collecting and analyzing data related to maintenance activities, CEDs can identify process inefficiencies and address device issues, thereby promoting technological advancement.

In the healthcare context, the continuous growth in data collection has allowed business intelligence (BI)—defined as “the techniques, technologies, systems, practices, methodologies, and applications that analyze critical business data to help a business better understand its business and market and make timely business decisions” [19]—to emerge as an area of study for both professionals and researchers in various sectors. This reflects the breadth and impact of data-related issues faced by contemporary business organizations.

BI solutions enable the collection of information from financial and operational data to make smarter decisions, aiming to improve process efficiency and effectiveness. Successful companies recognize that leveraging business intelligence can provide a competitive advantage by converting data into information and knowledge, answering not only the question “what?” but also “why?” [20].

There are two main types of BI systems: those centered on data and those focused on processes. Data-centered perspectives use BI systems to understand organizational capabilities by collecting, transforming, and integrating data to provide complex and competitive insights, aiming to enhance the timeliness and quality of decision-making. Process-centered perspectives view organizations as sets of well-integrated processes, where BI is used to assimilate information into these processes [21].

Despite the advantages, using BI to solve business problems brings challenges such as data access, structure, and supporting hardware technology.

In the healthcare environment, the challenge lies in converting vast amounts of data into valuable insights and knowledge. BI’s ability to add value by gathering data from various sources and combining them in a common repository allows for in-depth analysis and supports decision-making processes.

Furthermore, emerging BI tools can provide answers to key questions more rapidly and potentially with higher quality using analysis and visualization tools [22].

Analysis of data allows for the examination of various situations obtaining relevant information for business support. In healthcare, the use of BI provides important information, particularly on improvement opportunities. The rigorous and systematic approach of these techniques allows for improved investment performance and consequently increases the level of care provided.

The BI operations can pose challenges for any company, but in the healthcare sector, there are additional layers of complexity, such as privacy issues related to sensitive patient data regulated by privacy laws [23].

Sabherwal and Becerra-Fernandez outline four interconnected capabilities of business intelligence: organizational memory, information integration, insight creation, and presentation. These capabilities are crucial for various sectors, with healthcare organizations benefiting significantly [24].

Also, for the CED, it is increasingly essential to have a measurable understanding of the efficiency of the healthcare company’s maintenance service.

Despite the widespread use of business intelligence tools in the healthcare sector, few studies apply these tools to the management of electromedical equipment within the context of clinical engineering. Additionally, existing studies define generic indicators that do not allow for a detailed investigation of all the processes carried out by clinical engineering departments. This study aims to define a specific set of indicators tailored to the analyzed facility.

The primary objective of the following study is to develop a pool of evaluation indices capable of allowing analysis of the entire machinery fleet present in the facility, gaining greater awareness of the economic value managed by the structure, and better understanding and evaluating the critical issues of the various existing operational units. In particular, the research was conducted at the Clinical Engineering Department of the ASST Grande Ospedale Metropolitano (GOM) Niguarda in Milano (Italy), where 10 evaluation indices developed according to the ISO 9001:2015 [25] certification system have already been used. However, these do not cover all areas of the facility’s work. Therefore, the possibility of expanding this list of indices through an in-depth study of the current standard, its structure, and its needs has been identified.

In addition, another aim of this study was to verify the feasibility of calculating the defined KPI using the data available in the management software system.

Furthermore, visualizing the trend of evaluation indices is essential for making decisions. Simple and intuitive methods allow anyone to grasp complex information and consequently decide how to behave and improve the system.

Therefore, the additional goal of implementing a dashboard to observe and monitor the trend of the detected indices has been defined. BI tools enable thorough and systemic analysis by providing the necessary information to make the best decision.

The ultimate goal is to harness the potential of BI to monitor the trend of the detected indices. These technologies, as will be demonstrated, are essential today to optimize the system to the fullest.

2. Literature Review

A literature review was undertaken to identify existing indicators in the literature and to define new indicators that are more closely aligned with the CED’s needs.

An electronic search was conducted on PubMed and Scopus databases. The following words were used to perform the research: “indicator” AND “medical equipment” AND “management”.

The following criteria for inclusion were employed in the article selection process:

• Written in the English language;
• Full articles written in English, excluding reviews, perspectives, and communications;
• Full text available;
• Published from 2014 to September 2023;
• Reporting the indicators;
• Focused on the management of electromedical equipment.

Otherwise, the following exclusion criteria were considered:

• Articles concerning the management of medical devices that were not electromedical equipment;
• Studies that define a maintenance prioritization index;
• Papers that report cases of management in critical contexts or situations.

The review’s references were checked to find relevant papers that were included in the research.

Article titles and abstracts underwent screening to assess their relevance according to the inclusion and exclusion criteria.

A total of 303 articles was obtained from the electronic databases research previously mentioned, while the number of records identified through snowballing was 19. After duplicate removal, 293 papers remained. The screening of titles and abstracts resulted in the exclusion of 272 items. Among the remaining 21 articles, 16 papers did not meet the inclusion criteria.

Table 1. Literature review’s results.

Source	Year	Indicator Used
Bhardwaj, P. et al. [26]	2022	Uptime; downtime; response time; mean time to repair.
Iadanza, E. et al. [12]	2019	Downtime; uptime; mean time to restoration; mean time between failure; class failure ration; global failure rate; age failure rate; negligent actions; “1 day” actions; scheduled maintenance (SM) with failure; scheduled maintenance coverage rate; percentage of no problems found in SM; number of devices per technician; cost of service ratio; internal maintenance cost (% respect to the total maintenance cost); SM cost (% with respect to total cost); corrective maintenance cost (% with respect to total maintenance cost); cost of spare part (% respect to the total maintenance cost).
Gonnelli, V. et al. [27]	2018	Total CED expense as a percentage of total cost of acquisition (cost of acquisition ratio); CM (and SM) expense as a percentage of total CED expense; in-house (and external contracts) expense as a percentage of total CED expense; spare part (and supplies) costs; hourly cost of technicians (internal and external); repair time; uptime; downtime; class failure rate; age failure rate; number of technicians per number of capital devices; number of SM performed per number of capital devices; percentage of SM with problems (i.e., not coded as NPF); “delinquent work-orders” (i.e., not completed within 30 days).
Camila, R. S. et al. [28]	2015	Mean time between failure; mean time to repair; availability.
Oshiyama, N. F. et al. [29]	2013	Number of corrective maintenance events; total time spent on corrective maintenance; corrective maintenance costs.

presents the five papers included in the review together with the indicators used.

3. Materials and Methods

In this section, the methods for defining and identifying KPIs are given, and the criteria and tools used to create the dashboard are shown according to Section 3.1 and Section 3.2, respectively.

3.1. Indicator Definition

The initial phase involved examining the processes of the clinical engineering structure, along with conducting a literature search [12,26,27,28,29] to identify the classes of key performance indicators.

The KPI classes identified are the following:

• Administrative management: Related to the costs of activities. Examples of these indices are the costs related to preventive and corrective maintenance;
• Logistics management: Related to CED’s activities as purchase, service, rental, loan donation, spare parts, and accessories, in and out of the warehouse, both from healthcare facilities and from external companies All the indices belonging to this category are time indices. Examples of logistics management indicators are the average arrival time of spare parts or average call closure times;
• Technical management: Related to all activities carried out by technicians on biomedical equipment. They are useful indicators to assess the level of efficiency and coordination of technical staff. Examples of such indices may be “One-Day Action” or “Mean Time To Repair (MTTR)”;
• Training: Indicators measuring the level of staff training.;
• Quality: Measures to assess that the management of biomedical equipment is appropriate, effective, safe, and economical;
• Equipment management: Related to the management of all the equipment managed or used by the CI. Examples are downtime and uptime measures.

The latter did not exist prior to the following work but was defined following the study of the facility’s needs.

The second step involved outlining the structure of the KPIs. It is crucial that a KPI is defined clearly and unambiguously to ensure it is comprehensive and leaves no room for misinterpretation [29,30]. The requirements for defining a KPI to be at least comprehensive are as follows:

• Name: Name of the KPI, using a standard naming system. The name should be self-explanatory, such as “Mean Time to Repair”;
• Number and type: Number associated with the KPI. It is also necessary to indicate to which class it belongs (administrative, logistics, technical, training, quality, equipment management). The two pieces of information can be combined to create an alphanumeric abbreviation identifying the KPI;
• Short definition: Short description of the KPI, similar to a name, e.g., “Average time to repair a device”;
• Detailed definition: A more comprehensive description of the KPI, including sources, formulas, possible limitations, and applicability in the organization. The detailed definition should also include the rationale for choosing and adopting the KPI in the decision-making and review process;
• Formula: Mathematical equation of the KPI;
• Numerator: Description of the numerator, including inclusion and exclusion criteria;
• Denominator: Description of the denominator, including inclusion and exclusion criteria. Often, the denominator is the ‘total’ (to obtain percentage KPIs);
• KPI unit: Format of the KPI result (days, months, or percentage);
• Statistical adjustments: Illustration of statistical techniques used on the dataset to reduce the presence of confounding values (such as outliers often also caused by sampling or transcription errors);
• Reference values (benchmarks): KPI values obtained from external organizations that are similar to the object of study, often indicated as a model for best hospital practice, or from other internal departments that can provide a direct comparison. These are useful for setting a baseline and an ideal target for process improvement.

Therefore, considering the criteria suggested by the standard [31], as well as the literature analyzed and the knowledge and needs in the field, a set of 18 KPIs has been designed, in addition to the 10 already implemented by the structure (Appendix A).

The KPIs identified relate exclusively to the categories of logistics, technical, and equipment management. This is because the indicators already present for the remaining categories comply with the standard and are sufficient for the objective set. Instead, it was decided to investigate technical, logistical, and equipment-related aspects that had been neglected until now in more detail.

The methodology previously reported, together with the process analysis and the literature review, leads to defining the KPI and the methodology for their calculations (

Table 2. KPI calculation.

KPI	Formula	Indicator Type
Average downtime in the warehouse	$\frac{Average idle time of the repaired device in the warehouse}{\mathrm{Total} \mathrm{number} \mathrm{of} \mathrm{devices} \mathrm{repaired} \mathrm{in} \mathrm{the} \mathrm{warehouse}}$	Logistics management type
Receiving spare parts meantime	$\frac{Total turnaround time for arrival of replacement parts}{Number of orders}$	Logistics management type
Mean arrival time	$\frac{{\mathrm{Arrival} \mathrm{in} \mathrm{the} \mathrm{warehouse}}^{\prime }\mathrm{s} \mathrm{time}-{\mathrm{opening} \mathrm{ticket}}^{\prime }\mathrm{s} \mathrm{time}}{Number of tickets received}$	Logistics management type
Percentage of external downtime	$\frac{equipment downtime due to external maintainance}{theoretical time of use}$	Logistics management type
Average time since first intervention	$\frac{time of opening ticket-time of the first intervention}{Number of close ticket}$	Technical management
Mean time to repair	$\frac{\mathrm{closure} \mathrm{request} \mathrm{time}-maintenance start time}{total number of maintenance operations}$	Technical management
Average request closing time	$\frac{\mathrm{closure} \mathrm{request} \mathrm{time}-open request time}{total number of corrective maintenance operations}$	Technical management
Supported devices for technical personnel	$\frac{number of request assigned to each technician}{total number of device}$	Technical management
One-day action	$\frac{\begin{array}{c}maintenance corrective maintenance completed within 24\\ h.\end{array}}{number of maintenance corrective maintenance}$	Technical management
Negligent actions	$\frac{Number of request open for more than X days}{total number of corrective maintenance}$	Technical management
Average failure time	$\frac{time between two failures }{total number of corrective maintenance}$	Equipment management
Uptime	$\frac{device activity time }{theoretical time of use}$	Equipment management
Uptime for life-saving equipment	$\frac{device activity tim{e}_{| life saving equipment}}{theoretical time of us{e}_{| life saving equipment}}$	Equipment management
Corrective maintenance downtime	$\frac{downtime due to corrective maintenance }{theoretical time of use}$	Equipment management
Preventive maintenance downtime	$\frac{downtime due to preventive maintenance }{theoretical time of use}$	Equipment management
Inventories that generate request	$\frac{device that generate request\left(ward, specific device type\right)}{total number of devices \left(ward, specific device type\right)}$	Equipment management
Number of requests over number of devices per hospital wards	$\frac{number of request \left(specific ward\right)}{total number of devices \left(specific ward\right)}$	Equipment management
Failure rate category	$\frac{\begin{array}{c}number of corrective maintenance for a specific class of\\ devices\end{array}}{total number of corrective maintenance}$	Equipment management

).

3.2. Dashboard

Subsequently, a dashboard was developed to visualize the defined KPIs. First, the platform to be used for development was identified.

There are various business intelligence platforms and tools on the market (Power BI, Tableau, Qlik Sense, for example), and the choice of use depends mainly on specific needs, personal preferences, and the technical characteristics sought.

After exploring several possibilities on the market, the choice for the development dashboard for our project fell on the Power BI tool. The choice of this instrument was dictated by the cost-effectiveness of the Microsoft product, the ease of data integration and ease of use, and the moderate amount of data under analysis. The considerations of this choice are reported below:

• Using the Microsoft ecosystem: If one already uses Microsoft systems, integration with Power BI may be easier. This platform, being proprietary to the US company, is seamlessly integrated with other Microsoft applications, simplifying data sharing and collaboration within the Microsoft environment itself;
• Ease of use: Power BI’s intuitive and user-friendly interface makes the platform a particularly advantageous choice for less experienced users of data analysis;
• Price: In terms of cost, the Microsoft platform offers a free plan with limited functionality. This allows anyone to enjoy the benefits of business intelligence. Other platforms allow access to the service only through subscriptions of different durations and functionalities. Power BI also has different premium plans;
• Scalability: The management of large amounts of data is allowed by all platforms. Compared to other platforms, Power BI is considered less scalable for advanced analysis and the needs of large companies;
• Customization and visualization: In some cases, visualization and customization play a key role in the creation of a dashboard or report. The possibility of having a wide range of visualization tools makes the product more dynamic and intuitive;
• Analysis capabilities: Visualizations and, above all, advanced analysis capabilities and complex and detailed analyses are essential for large companies;
• Community and support: BI platforms today have a solid base of users and online communities. The choice may depend on the availability of support resources and the possibility of finding answers to your questions.

Initiating the implementation process for a dashboard involves two primary steps: defining the requisite data and importing it into the application. Power BI offers the flexibility to leverage various data sources such as databases, Excel files, and cloud services. In our context, Excel files downloaded to the ASST GOM Niguarda’s CMMS, which is ControlASSET^® [32], were used. Although using Excel simplifies data management, it lacks real-time synchronization with the CMMS.

The available dataset comprises three main files:

• “Maintenance”: This file contains details of maintenance activities since January 1, 2022. It includes fields such as “Number”, “Description”, “Contact”, and more;
• “Equipment”: This file provides information on various equipment types and their attributes;
• “Criticality”: This file, containing ISO-related criticality assessments, offers insights into equipment descriptions, functions, damages, and more.

Before importing the data, the header rows were removed to prevent formatting errors post-import. Subsequently, the tables were edited using the “Transform Data” function, performing the following actions:

• In the “Equipment Table”, two empty columns were removed and transformed into inventory numbers integers for consistency;
• Similarly, in the “Maintenance Table”, an empty column was addressed and ensured data type consistency;
• For the “Criticality Table”, data types were verified and standardized values for clarity.

Each page of our dashboard follows a standardized process: starting with key performance indicators (KPIs), new columns were created, if necessary, primary diagrams were generated, dynamic filters were implemented, and, finally, graphical and design refinements were applied.

Detailed implementation focuses on individual pages of our dashboard include the following:

• “Average Request Closing Time”: This page calculates the average time to close maintenance requests, visualizing data through line charts and stacked column histograms with dynamic filters for enhanced usability;
• “One-Day Action”: This page identifies maintenance actions completed within a day, presenting data through pie and donut charts, accompanied by a detailed table for comprehensive analysis;
• “Mean Time To Repair”: Utilizing bar charts, this page evaluates repair times for technicians and equipment types, facilitating data interpretation through dynamic filters;
• “Supported Devices for Technical Personnel”: Visualizing maintenance assignments and device statuses, this page provides insights into technician workload and equipment conditions;
• “Negligent Action”: By categorizing open calls based on duration and status, this page highlights potential operational inefficiencies, aiding in proactive maintenance management;
• “Number of Requests vs. Installed Devices”: This page compares maintenance requests with installed devices per department, offering insights into resource allocation and operational efficiency;
• “Details of Department Requests”: Building on the previous page, this section provides detailed breakdowns of maintenance requests per department, facilitating deeper analysis;
• “Criticality”: This page focuses on equipment criticality, visualizing device distribution and criticality percentages per operating unit;
• “Obsolescence”: Utilizing device lifespan data, this section assesses equipment obsolescence and provides insights into equipment age and distribution.

Each page incorporates specific filters tailored to the data displayed, ensuring a customized and user-friendly experience.

4. Results

This section will showcase all selected KPIs (Table 2), detailing their descriptions, usage, and graphical representation. Furthermore, it will present a retrospective investigation covering the years 2022–2023, allowing numerical calculation of KPI values where sufficient data from the CMMS were available. Following this, the section will provide implementation on dashboards and graphical representation for the mentioned KPIs.

4.1. Logistics Management KPIs

4.1.1. Average Downtime in the Warehouse

The “Average downtime in the warehouse” (Appendix B) indicator evaluates the average time devices spend in a state of repair in the warehouse. Unlike other metrics where technicians return repaired devices to the department, in this case, it is handled by a department operator. Monitoring this indicator is crucial for assessing the efficiency of warehouse department coordination to ensure service continuity. For ASST GOM Niguarda, this indicator is monitored and calculated annually by the logistics area coordinator, with results expressed in days. The value of this indicator contributes to the calculation of total downtime, and any increase requires investigation into which departments are experiencing issues and the underlying causes. Keeping this value as low as possible is a goal of the CED, not just as an efficiency and coordination measure but also because prolonged device downtime causes logistical issues due to limited warehouse space. Currently, data required for calculation are missing from the ControlASSET^® management system, specifically information on device retrieval from the warehouse. Therefore, implementation of this indicator on the dashboard will be delayed until the management system is completely renewed.

4.1.2. Receiving Spare Parts Meantime

The “Receiving spare parts meantime” (Appendix B) index evaluates the average time elapsed between the order of spare parts and the arrival of the material. For ASST GOM Niguarda, the established frequency of measurement and calculation is annual and is the responsibility of the logistics area coordinator. The index is expressed in days. Through the calculation of this index, it is possible to assess the efficiency of the supply service provided by the external company. For the analysis to be meaningful, it is important to compare the obtained temporal value with a benchmark, which can be calculated considering the origin site of the materials. Currently, the calculation is not possible due to a lack of useful data in the ControlASSET^® management system: information on material arrival is missing. Therefore, the aforementioned index will not be implemented on the dashboard until the complete renewal of the management system used.

4.1.3. Mean Arrival Time

The “Mean Arrival Time” (Appendix B) index assesses the average time from reporting a malfunction until the device arrives in the warehouse for repair. It aids in evaluating warehouse department coordination efficiency for continuous service. ASST GOM Niguarda measures and calculates this annually under the logistics coordinator’s supervision, with results in hours. In this case, as well, the lower this index is, the more efficient the service will be. Identifying causes of delays through workflow analysis can optimize processes. However, the lack of necessary data in ControlASSET^® prevents current calculation, delaying dashboard implementation until the management system is updated.

4.1.4. Percentage of External Downtime

The “Percentage of external downtime” (Appendix B) indicator measures downtime attributed to external maintenance, especially for contract devices, where downtime tends to be longer due to external servicing. ASST GOM Niguarda monitors and calculates this annually under logistics supervision, with results in hours. The index helps assess external workload volumes and, with high values, prompts analysis for workflow optimization or contract adjustments. However, lacking necessary data in ControlASSET^® hinders current calculations, delaying dashboard implementation until system renewal. Keeping this value as low as possible is a CED’s goal.

4.2. Technical Management KPIs

4.2.1. Average Time since First Intervention

The “Average time since first intervention” (Appendix B) index evaluates the time between reporting a malfunction and beginning maintenance work. It is divided into two time windows: fault detection to reporting and reporting to intervention. The index helps assess technical staff efficiency and coordination, but the lack of necessary data in ControlASSET^® prevents current calculation; thus, it is not implemented in the dashboard. It is calculated in hours and should be as low as possible.

4.2.2. Mean Time to Repair (MTTR)

The KPI “Mean Time to Repair” (Appendix B) assesses the average time taken to repair a device from when the technician starts work until the device is completely restored. This measure is crucial for evaluating the efficiency of the technical department in restoring equipment functionality promptly. ASST GOM Niguarda measures MTTR annually under the supervision of the technical area coordinator, with data extracted from the ControlASSET^® management system and expressed in hours. Due to missing “start work time” data, “hours worked” data are utilized for calculation. MTTR is calculated as the ratio of total repair time to total maintenance interventions, aiming to minimize downtime and identify root causes of delays through Root Cause Analysis (RCA). MTTR helps set time objectives for repair and optimizes processes to meet those objectives. Through Power BI implementation, MTTR trends can be visualized by technicians (Figure 1) but also for devices, allowing for analysis and identification of devices exceeding repair time limits. Adjustment of the time window provides insights into MTTR trends over different periods, aiding in continuous improvement efforts within the technical department. This KPI is measured in hours, and it is possible to see that it is dependent on the technician who is in charge of the maintenance. So, this indicator should be similar to every technician and should be around 20 h.

4.2.3. Average Request Closing Time

The KPI “Average requests closing time” (Appendix B) evaluates the average time interval between reporting a fault and restoring the device. It consists of two distinct contributions: “Average time since first intervention”, which is the time between fault reporting and maintenance initiation, and “Mean Time to Repair”, which is the actual repair duration. ASST GOM Niguarda measures this annually under the supervision of the technical area coordinator, using data from the CMMS and expressing the indicator in hours. The calculation is performed as the difference between the closure time and the start time of maintenance, divided by the total number of maintenance interventions.

Monitoring the index value is crucial, as is aiming to keep it as low as possible. However, it is important to note that long call closure times do not always correspond to long maintenance interventions; calls are often closed after maintenance ends. Therefore, the dashboard implementation allows observation of the “Average requests closing time” for each technician and device type, enabling the technical area coordinator to optimize the process.

Figure 2 presents a column chart showing the monthly number of maintenance requests generated from January 2022 to 1 September 2023, with a dashed line indicating the trend of the index. Despite the relatively high average number of maintenance requests generated, the index value slightly decreased in the first 8 months of 2023 compared to 2022.

The potentials of Power BI enable data filtering by final allocation, internal/external maintenance, maintenance cause, and inventory number. For instance, users can observe the number of faults for a specific inventory within the selected time frame and the average time calls remain open.

Figure 3 provides a practical example of using the index, focusing only on ECG devices to illustrate realistic calculation. The average call closure time for ECG devices is observed to be 9.11 days, aligning with expectations. Peaks in April and May 2022 coincide with director changes during those months. While the overall value of “Average requests closing time” may be unreliable due to impure data, a detailed analysis of individual device classes can already provide valuable insights.

4.2.4. Supported Devices for Technical Personnel

The KPI “Supported devices for technical personnel” (Appendix B) determines the workload supported by specialized technical staff. ASST GOM Niguarda monitors and calculates this index daily under the supervision of the technical area coordinator, using data extracted from the ControlASSET^® management system. The index is expressed as a percentage.

Implementing this indicator on a dashboard, as shown in Figure 4, allows for various analyses. The first graph, on the left, is a stacked column chart providing detailed information. Firstly, it shows how many devices each technician is working on based on the number of assigned calls, aiding the technical area coordinator in quickly assigning new calls to technicians. Secondly, it evaluates the number of devices each technician has worked on within a predefined time window.

Overall, the indicator provides a comprehensive overview of the technical staff’s activity. In this case, there is not a specific number to reach because the number of devices repaired by the technician depends on the type of malfunction that occurs and on the maintenance time.

4.2.5. One-Day Action

The “One-Day Action” (Appendix B) index evaluates the percentage of maintenance tasks completed within 24 h of the occurrence of a fault. It is calculated based on the mean time to repair (MTTR) and is expressed as a percentage. ASST GOM Niguarda monitors and calculates this index annually, and it is managed by the technical area coordinator using data from the ControlASSET^® management system. The index aims to minimize machine downtime for service continuity, especially by providing temporary replacements for devices under contract or requiring spare parts. The corresponding dashboard (Figure 5) provides visual representations of call closure percentages. Filters are available for various maintenance parameters. Overall, the index allows for assessing the efficiency and speed of the technical department in handling maintenance tasks, particularly simple interventions completed within 24 h. The goal of clinical engineering is to keep this value high, as the higher it is, the greater the number of devices that can be repaired in a day.

4.2.6. Negligent Actions

The “Negligent Actions” (Appendix B) indicator evaluates the percentage of procedures not completed within a predetermined assignment period. ASST GOM Niguarda monitors and calculates this index semi-annually, managed by the technical area coordinator using data from the ControlASSET^® management system. The index is expressed as a percentage and is calculated by dividing the number of overdue procedures by the total number of maintenance requests within the specified assignment period. The standard assignment period is 30 days, as per the literature. The objective is to minimize this value and analyze the causes of any peaks, considering the type of maintenance tasks. Temporary replacements are provided until the device is fully restored to ensure service continuity. The dashboard (Figure 6) visually represents the percentage of closed and open calls, categorized into assigned and unassigned ones. Additionally, it displays a summary of the distribution of open practices and the days elapsed since the opening date. The analysis of these results highlights the considerable number of open tickets with high resolution times.

4.3. Equipment Management KPIs

4.3.1. Average Failure Time

The “Average Failure Time” (Appendix B) index evaluates the average time between the resolution of a failure and the occurrence of a second one. It serves as a reliability parameter to assess not only the frequency of failures on a device but also the average period of equipment availability.

For ASST GOM Niguarda, the index’s detection and calculation frequency is set annually and is managed by the quality group. Data for calculation are extracted from the ControlASSET^® management system, and the index is expressed in days. The calculation involves subtracting the date of the opening of a new maintenance request from the date of closure of the previous one and dividing it by the total number of corrective maintenance actions.

The objective is to maximize the “Average Failure Time”, meaning to extend the time between two successive failures as much as possible. Calculating and graphically representing this index is essential for monitoring trends over time and investigating the causes of any decrease in its value.

It is expected that the index will be higher for technologies with low technical complexity and lower for devices prone to more frequent failures, such as those with high technical complexity. Additionally, comparing the causes of two successive failures could be useful, especially if there are recurring reasons, allowing for efficient solutions.

Currently, the index is not implemented in the dashboard.

4.3.2. Uptime

The “Uptime” (Appendix B) index measures the actual availability time of a device, expressed as a percentage. ASST GOM Niguarda calculates it annually, managed by the quality group, using data from the ControlASSET^® management system. It is computed by dividing the device’s uptime by its theoretical usage time. The goal is to maintain a high value to reduce equipment downtime and associated costs. While not currently implemented in the dashboard, special attention is given to life-saving devices, leading to the development of a related indicator called “Uptime for life-saving equipment.”

4.3.3. Uptime for Life-Saving Equipment

Particular attention must be paid to ‘life-saving’ devices. This is why it was decided to implement an indicator that more accurately monitors this type of device.

The index developed is ‘Uptime for life-saving equipment’ (Appendix B) and bases its principle and use on the ‘Uptime’ index.

4.3.4. Corrective Maintenance Downtime

The “Corrective Maintenance Downtime” (Appendix B) index evaluates the machine downtime due to corrective maintenance, which is caused by sudden failures rather than scheduled maintenance or electrical checks. ASST GOM Niguarda calculates it annually, managed by the quality group, using data from the ControlASSET^® management system. The index is expressed as a percentage and is calculated by dividing the time of corrective maintenance downtime by the theoretical uptime. Currently, the index is not implemented in the dashboard because the classification of devices by risk class will not be available in ControlASSET^® until the management system is completely renewed. It is important to keep this KPI as low as possible in order to increment the availability of the electromedical device.

4.3.5. Preventive Maintenance Downtime

The “Preventive Maintenance Downtime” (Appendix B) index evaluates the machine downtime due to scheduled maintenance. ASST GOM Niguarda calculates it annually, managed by the quality group, using data from the ControlASSET^® management system. The index is expressed as a percentage and is calculated by dividing the time of preventive maintenance downtime by the theoretical uptime. The “time of machine downtime due to preventive maintenance” refers to the time needed to perform scheduled maintenance, while the “theoretical uptime” is assumed to be 365 days for devices with high criticality and 250 days for other devices. Similar to other indices, this index is not currently implemented in the dashboard pending the complete renewal of the management system. Also, in this case, the KPI should be as low as possible to increase the medical device availability.

4.3.6. Inventories That Generate Request

The “Inventories that generate requests” (Appendix B) index allows for accurately assessing which devices generate the highest number of requests and, therefore, experience a high number of failures. Specifically, it provides observations tailored to each department.

ASST GOM Niguarda calculates this index annually, managed by the quality group, using data extracted from the ControlASSET^® management system. The index is expressed as a percentage and is calculated by dividing the number of requests generated by inventory by the total number of devices.

Unlike the “Number of requests over number of devices per hospital wards” indicator, which will be implemented later, this index focuses on identifying which devices frequently encounter failures and how many such devices exist. Additionally, valuable insights can be derived by combining this index with an analysis of device obsolescence. This could lead to the decommissioning of obsolete devices that require frequent maintenance.

Currently, the index is not implemented in the dashboard due to a lack of necessary data for calculation.

4.3.7. Number of Requests over Number of Devices per Hospital Wards

The “Number of requests over number of devices per hospital wards” (Appendix B) index assesses the ratio of maintenance requests opened by each department to the number of devices installed in them. ASST GOM Niguarda calculates this annually, using data from ControlASSET^®, expressed as a percentage. The index aids in identifying departments with high maintenance request rates relative to installed devices. Figure 7 displays the dashboard, offering a comprehensive view of departmental data. Detailed analyses help pinpoint causes, especially concerning equipment obsolescence. This index is crucial for promptly addressing maintenance needs and optimizing equipment management.

4.3.8. Failure Rate Category

The “Failure Rate Category” (Appendix B) indicator is the index of the overall failure rate that considers the specific category of the device. ASST GOM Niguarda calculates this annually, using data from ControlASSET^®, and expresses the index as a percentage.

The index is calculated as the number of maintenance interventions per specific device class divided by the total number of maintenance interventions. Devices in a high-risk class are expected to have a higher number of interventions. This indicator should be as low as possible; otherwise, it indicates that the medical device category should be replaced by new ones. An accurate analysis of the index values would enable proper planning of preventive maintenance for high-risk devices, thereby avoiding unexpected failures and service interruptions.

The implementation of the index also allows for investigating the types of failures by device class to prevent systematic breakdowns where they occur. Currently, the index is not implemented on the dashboard due to the absence of device classification by risk class in ControlASSET^® until the complete renewal of the management system.

5. Discussion

The identification and implementation of a set of performance indicators allows the Clinical Engineering Department of ASST GOM Niguarda to have a broader operational view of the hospital, with a comprehensive and measurable understanding of the maintenance management of its technological assets, the economic value managed by the facility, and a better understanding and evaluation of the critical issues of the various existing operational units.

The existing indicators within the facility have proven to be of fundamental importance in identifying maintenance processes that have not yet been investigated. Building upon these, along with a thorough study of the existing literature, it was possible to generate a diverse set of 18 KPIs tailored to the facility’s needs, considering its size and strategic objectives.

The division of the 18 indices into KPIs for logistics management, technical management, and equipment management has allowed for an overview of processes and activities and has enabled the selection of parameters to monitor in order to identify improvement opportunities useful for achieving the facility’s objectives.

In particular, process efficiency indicators such as “One-day action”, “Negligent action”, or “Mean Time to Repair”, along with coordination indicators like those for logistics management, have highlighted the innovativeness of the proposed solution.

The definition of the structure of these indices and their calculation methods will facilitate the easy development of any other indices should new needs or further studies arise. Currently, the segment of economic indices has not been thoroughly explored due to data gaps within the ControlASSET^® management system; it is left to the facility to implement these based on the proposed ones.

Moreover, defining benchmarks would make the index monitoring system comprehensive and actionable. As these benchmarks are not present in the literature, given that the indices were adapted to the ASST GOM Niguarda facility, they can be determined through a thorough study of the temporal trends of the indices themselves.

Merely defining KPIs is not sufficient to achieve the objective. Hence, a dashboard has been implemented to allow both graphic and quantitative visualization of individual indicators. This assists the monitoring personnel in conducting an immediate and intuitive analysis of employee occupancy status, process trends, and instant identification of any critical issues. Additionally, the support of filters for maintenance type, technician, year, and department contributes to making the analysis more precise and comprehensive.

The primary constraints of this study stem from the data accessibility within CED of the ASST GOM Niguarda. Specifically, the challenges encountered pertain to the absence of certain data crucial for indicator computation. These data elements are unavailable due to the absence of corresponding fields in the CED’s CMMS. Thus, it is necessary to modify the CMMS database structure to encompass a more comprehensive perspective on operational oversight within the department.

Moreover, the existing data are frequently inputted inaccurately by users, resulting in an analysis that may not fully reflect the actual circumstances. Enhanced diligence on the part of users during data input is imperative, which is achievable through heightened awareness of operational management practices. To ensure the integrity of the data, the implementation of a semi-automated data entry system, capturing elements such as time and date, would prove beneficial.

The work carried out has thus contributed to identifying anomalies in the data and gaps in the application itself. Proposals have been provided regarding the inclusion of additional mandatory fields for technicians to fill out and changes to existing fields to enable the calculation of all proposed KPIs.

The primary challenge of data accessibility can be addressed by incorporating specific fields within the management system to collect the necessary data for indicator calculation. Additionally, to resolve this issue, it is essential to directly link the management system’s database to the Power BI dashboard. This will enable real-time visualization of the KPIs, eliminating the need to download Excel files from ControlASSET.

Another potential element to consider in future developments is to define thresholds for each indicator. This can be performed based on the analysis of previous years’ data and setting thresholds that continuously improve the performance of the service provided by the CED.

Finally, the dashboard has been implemented on Power BI rather than the ControlASSET^® application, as the latter is not owned by the facility. Implementing this interactive corporate dashboard on the new application would be the optimal solution and would avoid the need to repeatedly extract data for calculation. This way, calculations could be performed immediately whenever necessary, and they could be accessible to various stakeholders in the maintenance process.