Evaluation Methodology for Circular and Resilient Information Systems

Featured Application

The findings of this study can be directly applied in the process of evaluating circular and resilient information systems in manufacturing, providing a practical tool for practitioners and researchers.

Abstract

Digital technologies nowadays provide essential support for companies, making them a priority for businesses and a prominent area of study for researchers. In response to the increasing emphasis on sustainability and resilience, new information systems are developing to meet evolving business needs, namely circular and resilient information systems (CRISs). These systems integrate with traditional ones to optimise key performance indicators (KPIs) related to circularity and resiliency. Despite extensive methodologies for evaluating traditional information systems, systems designed for circularity and resiliency need to be assessed in parallel and in depth. Existing evaluations focus on efficiency and user satisfaction but often neglect the unique demands of circularity and resiliency. This study introduces a novel evaluation methodology for CRISs. Through a case study of an innovative system and the established literature, we address real-life needs and challenges in manufacturing. In particular, the system serves the needs of three distinct case studies: Carbon Fibre-Reinforced Polymer (CFRP) waste utilisation in drone manufacturing, recovery of magnets from Waste Electrical and Electronic Equipment (WEEE), and the repurposing of citrus processing waste into juice by-products. Our methodology is built on the 5W1H method to make our approach context-specific and aligned with each case’s unique requirements, making it also replicable for other industries. Our findings offer insights and a tool for practitioners and researchers to evaluate CRIS performance. The research highlights the importance of a two-fold evaluation approach for CRISs, evaluating both pilot-specific KPIs and the system’s technical performance. Policy implications suggest the need for regulatory frameworks and incentives to support the adoption, as well as evaluation, of CRISs and promote sustainable and resilient industrial practices.

1. Introduction

The growing importance of sustainability and resilience in today’s world has led to the design of new types of information systems to fulfil the emergent needs of companies [1]. These systems aim to integrate with traditional systems, enabling organisations to evaluate and optimise key performance indicators (KPIs) related to circularity and resiliency [2,3]. Circularity focuses on the sustainable management of resources, promoting reuse, recycling, and regeneration to minimise waste and environmental impact. Resiliency, on the other hand, is about the ability of companies to adapt, recover, and continue to function effectively in the face of disruptions and adverse conditions. Such integration is essential for sustainable development, adaptive responses to anticipated and unforeseen challenges, and emerging customer needs.

Despite the availability of numerous methodologies to evaluate traditional information systems in IS research [4], there remains a pressing research gap in methodologies tailored to assess the performance of systems specifically designed to enhance circularity and resiliency [5]. Conventional evaluation approaches exist [6], focusing on efficiency, effectiveness, and user satisfaction metrics. Nevertheless, they often need to address the unique challenges and new requirements posed by the concepts of circularity and resiliency. This gap highlights the need for a specialised evaluation framework to comprehensively assess the technical performance and sustainability impact of circular and resilient information systems (CRISs).

To address this research gap, our study is guided by the following key research questions:

• What specific evaluation criteria are necessary to assess the performance of circular and resilient information systems effectively?
• How can these criteria be integrated into a comprehensive evaluation methodology that addresses a CRIS’s technical and sustainability aspects?
• In what ways can the proposed methodology be applied across different industries to validate its versatility and effectiveness in real-world scenarios?

Our study builds on established evaluations in the literature to present a novel evaluation methodology specifically for circular and resilient information systems. Additionally, we detail the development and implementation of our evaluation methodology, illustrating its application through a case study of developing an innovative circular and resilient information system. This case study approach also allows us to consider manufacturing companies’ real-life needs and challenges. Overall, we provide a tool and insights for researchers and practitioners to analyse and measure the outcomes and performance, as well as potential challenges, of their information systems and enhance the sustainability and resilience of manufacturing companies and supply chains.

2. Background Literature

Manufacturing businesses face several challenges, such as traceability across the value chain, the integration of sustainability into business objectives, the exploitation of critical enablers such as data and advanced technologies, and overcoming various disruptions [3,5]. To this end, due to the growing demand for manufacturing systems that can simultaneously address the challenges of sustainability and operational resilience, the literature has proposed the development of a circular and resilient information system (CRIS) [1]. A CRIS integrates advanced digital technologies such as AI, Digital Twins, and optimisation into a unified framework that facilitates the management of all relevant circular and resilient processes at a value chain level. A CRIS targets a set of key circularity and resiliency performance indicators (KPIs) such as material reuse rates, waste reduction, the efficiency of recycling processes, and recovery time after disruptions. The detailed design, operational mechanisms, and overall need for such a system in advanced manufacturing are further discussed by Farmakis et al. [1]. This study extends these results, presenting the methodology for the detailed evaluation of the CRIS system and highlighting the specific metrics for assessment.

Briefly, circular information systems (CISs) are crucial in promoting sustainability. They improve resource efficiency by increasing and prolonging the utilisation of products, components, and waste materials, thereby creating circular material flows [2], enhancing the efficiency of processes, and advancing sustainability objectives and circular economy efforts [7]. Nevertheless, there are numerous challenges that must be overcome to implement such systems; also, it is observed that there is a notable absence of detailed design frameworks for them, underscoring the necessity for empirical development evidence and functionality evaluation [5].

Regarding resilient information systems (RISs) in manufacturing, Fowler et al. [3] define them as systems capable of enduring and recovering businesses from disruptions such as supply chain failures. Similarly to CISs, RISs can also present numerous challenges as well, as they must be capable of constantly adapting to changes, addressing socio-technical complexities, and identifying critical categories of potential changes [8]. Additionally, they require methodologies for the systematic design and validation (through evaluation methodologies) of their capability to enable resilience by utilising specific metrics for assessment [8].

The EU-funded project Plooto—Product Passport Through Twinning of Circular Value Chains (GA no: 101092008)—envisages the creation of a digital-based system that merges the benefits of CISs and RISs to provide their collective benefits to the manufacturing industry. In particular, it aims to design and develop a circular and resilient information system (CRIS) that offers benefits associated with both CISs (i.e., waste reduction, environmental benefits, economic cost reduction, data and insights, resource efficiency, innovation and competitive advantage, and information sharing opportunities), as presented by Fakoya and van der Poll [9], Kamble et al. [10], and Kerdlap et al. [11], as well as RISs (i.e., risk mitigation, reduced downtime, cybersecurity risk protection, production continuity, flexibility, information recovery and protection, and environmental protection), as presented by Blay et al. [12], Dubey et al. [13], Ihlenfeldt et al. [14], and Sheth and Kusiak [15].

The CRIS will utilise information from the different factories in a supply chain to enable them to collaboratively become circular in nature, thus utilising the Secondary Raw Material (SRM) across the (now) circular value chain. With the use of technologies such as digital twins, optimization, process modelling, Digital Product Passport (DPP), blockchain, and respective governance frameworks, the CRIS will enable the monitoring of the requalification of SRM and its utilisation in the next steps of different products’ creation. However, this system, which relies on emerging technologies and aims to unify two different categories of IS (i.e., CISs and RISs), also requires a new approach to its examination.

In order to enable the appropriate and rigorous evaluation of such a system, more than the current evaluation methodologies for CISs and RISs are needed as they are focused on covering the requirements of each system (CISs and RISs) in the respective evaluation process. As such, a novel and inclusive evaluation approach is sought after. The research and outputs of the process to define and develop an evaluation approach for circular and resilient information systems (motivated by the Plooto CRIS) are presented in the following chapters.

2.1. Circularity and Resiliency in Information Systems’ Evaluation

Unlike traditional linear models, CISs emphasise resource efficiency, closed-loop processes, and waste minimisation. Modgil et al. [16] refer to the use cases of real-life CIS paradigms to address circular economy challenges, such as supporting waste management infrastructure, advancing recycling technology, and more. This process uses quantitative and qualitative techniques to evaluate these systems. Moreover, according to Gregor and Hevner [17], in the context of the DSR framework, the evaluation of a system is divided into pre-evaluation (functional requirements) and post-evaluation (performance of the developed system). The normative literature contains a plethora of evaluation methodologies for information systems (ISs) across different stages.

In their seminal research, Petter et al. [18], by scrutinising the literature on IS success models, dimensions, and measurements from 1992 to 2007, reported their findings, part of which firmly match our initial research. Fitting to the CRIS’s project evaluation information system, Irani et al. [19] propose the “5M” model (Man, Machine, Method, Material, Money). In contrast, several measures have been proposed in the general context of the evaluation of manufacturing (IS). Regarding evaluating the success of an implemented IS from the scope of its usage and its benefits, the DeLone and McLean IS Success Model is an established model, having been used extensively in the literature [20,21]. This model identifies six critical dimensions of IS success, developed by DeLone and McLean in 1992 and later updated in 2003 [21]. The first of the dimensions is the “System’s Quality”, where the performance characteristics of the IS (including factors such as reliability, usability, response time, and functionality) are measured. The second dimension is the “Information Quality”. Critical attributes for evaluation relevant to information quality are accuracy, relevance, completeness, and timeliness. The third one is the “Service Quality”. This aspect evaluates the quality of the support services provided to users of the information system (user support, training, and the support team’s responsiveness). The fourth dimension is “Use”, or in some cases “intention to use”, in which actual usage can be measured (or the intention to use the IS). The fifth dimension of the success model is “User Satisfaction”, and the sixth dimension is the “Net Benefits” for evaluating the impact of the IS in the organisation.

From the scope of IS usability, Mator et al. [22] systematically presented some of the most eligible approaches for obtaining empirical measurements. The Post-Study System Usability Questionnaire, or PSSUQ, and the After-Scenario Questionnaire (ASQ), are commonly used in the literature for IS usability measurement [23,24]. Of all the proposed evaluation methodologies, “heuristic evaluation” is mentioned as the most prominent in the 1990s [25]. Nevertheless, measuring an information system’s usability directly comes with a high degree of difficulty [26]. For Cradle to Cradle (C2C) designed products, to maximise the benefits for humans and the environment, Bjørn and Hauschild [27] suggest the Life Cycle Assessment (LCA) evaluation, as it performs efficiently. C2C product creation is a philosophy focusing on the materials’ continuous reuse in a closed-loop system [28]. The role of the information system in data management, analysis, and reporting in the LCA evaluation is crucial, and the IS’s effectiveness should also be evaluated.

Regarding the evaluation of the system’s applicability, Rosemann and Vessey [29] were among the first to approach this issue systematically and noted the importance of further research and methodology development. Matching the criteria for Plooto’s project CRIS applicability, the evaluation methodology that Jang et al. [30] followed in their research, based on the pre-evaluation and post-evaluation stages (including evaluations through prototype demonstration, surveys, and expert interviews), appears to align with our objectives. Using quantitative and qualitative evaluation criteria is strongly suggested by Mokalled et al. [31] in their selection process for the most applicable Security Information and Event Management system. Specifically, they divide the evaluation methodology into two phases: (i) quantifying each requirement of the information system and (ii) “measuring the applicability of the solution using a qualitative-based method after defining a list of indicators that enables the evaluation of this applicability”. However, as they mention, an installation–testing phase should be conducted at the end to further confirm the IS’s applicability. Lastly, for a higher-level CRIS system evaluation, addressing issues such as functionality, reliability, performance, scalability, security, interoperability, maintainability, compliance, and standards is described in the ISO/IEC/IEEE 29119-1:2022(en) software testing, from the International Organisation for Standardisation and the International Electrotechnical Commission [32].

Regarding the evaluation of the information system’s overall user experience, Laugwitz et al.’s [33] seminal research can be used to record the overall experience of the factory employees during the use of the developed system. This questionnaire measures the attractiveness and the perspicuity of the IS, examines its efficiency and dependability, and examines its stimulation and novelty. Lastly, a qualitative approach is considered relevant to the evaluation methodologies for information systems because it provides in-depth information, mainly through a series of interviews or Delphi studies [34]. There is a huge body of literature concerning using qualitative methods in IS research [4]. But, to a better degree, in the case of Plooto project evaluation and the CRIS system, qualitative data analysis via semi-structured interviews will shed light on unknown phenomena and previously unidentified variables.

2.2. CRIS Evaluation Methodology Design

A practical evaluation methodology design for an ICT (Information and Communication Technology) system will ensure its functionality, efficiency, and alignment with the overall organisational goals and justify the reason for its development. A robust evaluation framework can enable the identification of its strengths and weaknesses toward guiding improvements and potential adaptations to meet the pilots’ actual needs. Furthermore, a well-structured evaluation methodology fosters accountability and transparency, providing stakeholders with clear insights into the system’s value and potential.

Towards developing the evaluation methodology of Plooto (and thus the CRIS system), the process of 5W1H was followed to act as a structured approach to developing an evaluation methodology grounded in theory. The 5W1H methodology is a systematic tool that helps break down complex problems into manageable parts by asking key questions that cover all essential aspects of a situation. This ensures that no critical factor is overlooked during the evaluation process. In particular, and for evaluating the circular and resilient information system of Plooto, the 5W1H methodology will initially be used for information gathering and problem-solving relevant to the approach followed. The 5W1H methodological tool, an enhanced version of the 5W tool, answers six specific questions (Who, What, When, Where, Why, and How) during the initial information-gathering stages. Both 5W and 5W1H have been previously used by researchers relevant to manufacturing sector studies [35,36,37]. This approach also serves as a useful tool for cross-case analysis in the case of multiple pilots [38]. Additionally, this process helps us to adapt measures in the improvement phase and thoroughly understand the current situation in detail, following the principle of 5W1H [39].

Our followed methodology will support involved stakeholders and evaluators to thoroughly understand and then evaluate the system’s requirements, functionalities, impacts, and areas for improvement by addressing issues such as (i) identifying the people involved or affected (Who), (ii) defining what the issue/s or situation is (What), (iii) specifying the time frame or when the issue occurs or occurred (the critical times when the system is most needed (When), (iv) determining the location or context of the event or problem (the physical and virtual environments in which the system operates) (Where), (v) seeking to understand the cause or reason behind the situation (problems it aims to solve, the processes it intends to improve, etc.) (Why), and (vi) describing the method, solution, or process involved in the situation or in problem-solving (How).

The existing body of literature proposes a plethora of IS evaluation methodologies regarding different aspects, stages, and angles. Most of them include quantitative and qualitative methods, considering pre-existing KPIs, examination, and information gathering. In this case, a mixed-method approach with a systematic methodological design for each dimension and stage is contemplated to evaluate the information system efficiently. Thus,

Table 1. Plooto W51H high level—methodological approach.

Value Networks	Who	What	When	Where	Why	How
VN1: CFRP waste for drones VN2: WEEE for magnets VN3: Citrus processing waste for juice by-products	Identify the stakeholders involved or affected in VN1, VN2, and VN3	Define the specific goals of the CRIS in VN1, VN2, and VN3 Identify the key features and functionalities of the CRIS in VN1, VN2, and VN3 Specify the data sources used for evaluation in VN1, VN2, VN3	Specify the critical times when the CRIS is most needed or evaluated in VN1, VN2, and VN3 Determine the duration of the evaluation period(s)	Determine the physical and virtual environments where the CRIS operates in VN1, VN2, and VN3	Understand the problems the CRIS aims to solve, as well as the processes it intends to improve in VN1, VN2, and VN3	Describe the methods and the processes involved in the implementation and evaluation of the CRIS. Define the metrics to be used to evaluate system performance in VN1, VN2, and VN3

presents the results of the W5I1 methodological approach followed to delineate the theoretical grounding of the CRIS evaluation methodology.

Having presented the overarching theoretical evaluation methodology framework, the focus shifts to selecting the most meaningful KPIs for consideration for the different value networks in order to be able to define a list of KPIs per value network that respond to a successful and effective pilot implementation. In that direction, all identified KPIs were examined for their validity, computability, and availability of data and past values, as well as computation methodology in parallel with the pilot partners.

In particular, relevant to the KPI validity, each KPI was examined for its relevance to the specific value network and pilot context for gaining additional maturity. For computability and data availability, each KPI was examined with the pilots to ensure that (a) data availability for the metrics that compose each KPI was (or will be) available and (b) the provided data and the KPI computation methodology would lead to the formation of the KPI. Similarly, for past KPIs’ value availability, the KPIs were examined for their potential to create the benchmark dataset against which the application of the new system would compare. Lastly, and relevant to the KPI computation methodology, the KPIs were examined for their ability to be measured under real past values–real new values (compared to real past–simulated new, simulated past–real new, and simulated past–simulated new options) in order to ensure an evaluation approach on actual production settings.

As the evaluation methodology developed for a CRIS system must account for the evaluation of both circularity and resilience-related KPIs, prior work on circularity and resilience KPI identification was utilised to develop a super-set of KPIs eligible for examination, as presented in Baroni et al.’s [40] project report on the Sustainability Balanced Scorecard Framework.

The following chapters present the KPIs and instruments used in the Plooto evaluation process, starting with the pilot KPIs and proceeding with the remaining system-related evaluation approach.

3. Results

As the evaluation of a CRIS relates to the system evaluation of the produced system, its application, and the examination of its benefits post-application, three different value networks were selected to account for the application domain and setting, stemming from a European-funded project. Plooto—Product Passport Through Twinning of Circular Value Chains (GA no. 101092008)—envisages the creation of a CRIS that, when applied, enables waste reduction as well as the end-to-end traceability of Secondary Raw Material (SRM). It thus offers an application setting where interconnected digital services for the real-time decision-making, monitoring, and certification of materials and products will enable factories in newly formed value chains to maximise the utilisation of SRM in a sequential and co-dependent manner; this would initially enable circularity, but even more so resilience, in their operations. Therefore, the evaluation setting for the CRIS brought forth by Plooto includes its three value chains that, although diverse in needs, combine circularity and resilience in their day-to-day operations. In particular, the three pilots included in the evaluation process and methodology are pilot 1: CFRP waste for drone manufacturing, pilot 2: WEEE for magnet production, and pilot 3: citrus processing waste for juice by-products. The pilots and the evaluation approach per pilot (in parallel to the CRIS evaluation) are presented in the following sections.

3.1. Plooto Evaluation Approach

Following the outcomes of the literature review and towards the development of the evaluation methodology, it was identified that (a) both the CRIS system and the pilots require their respective evaluation approaches, evaluation instruments, and KPIs and (b) all should include both quantitative and qualitative evaluation approaches as the developed system is novel in terms of the need for new combinations of instruments and KPIs that meet the evaluation needs for information systems as well as circular ISs and resilient ISs (the intersection where a CRIS resides). Furthermore, as the system is novel and currently undergoing design and development, a two-round application and evaluation approach was selected to be conducted. This was identified as needed in order to enable the reception of initial information on the technical side of the CRIS integration outcomes and, in turn, provide the information needed for its refinement (round 1) and then conduct the final piloting activities (round 2) with the CRIS system fully operational and able to provide the envisaged benefits to the pilots. As such, the first iteration will serve as the first round of CRIS system/pilot validation (scheduled for execution July 2024–September 2024) with the goal to have (a) the technical side of the CRIS system to be evaluated from a systems/integration perspective and (b) the pilots to experience the system, validate that their requirements are being covered, and, lastly, identify and propose any improvements needed for the final CRIS system. In the second iteration (scheduled for November 2025–December 2025), the pilots will utilise the final integrated and deployed CRIS solution for its operation and evaluation, as can be seen in Figure 1. However, it should be noted that, in order to thoroughly examine the outcomes of such (and similar) systems, continuous evaluation should take place (post-application) in order to also account for long-term effects.

The following sections present the KPIs and instruments utilised in the CRIS evaluation process, starting from the pilot KPIs that the CRIS envisages supporting and proceeding with the CRIS system-related evaluation approach.

3.1.1. Pilot 1—CFRP Waste for Drones

Carbon Fibre-Reinforced Polymer (CFRP) composites have rapidly increased their use in modern products and have become strategic materials for the European industry, with the global demand expected to reach 285 kt in 2025 [41]. Prepregs (fabric pre-impregnated with a resin system) are a type of CFRP that is commonly used and uncured prepregs (entire rolls of material that have reached the expiration date or scraps coming from the cutting operations) constitute the largest part of CFRP waste. The CRIS aims to showcase the value brought forth by utilising a system to enable the (re-)use of this Secondary Raw Material in value chains. The overall goal of this pilot with the CRIS utilisation is to design a process of reusing the carbon fibre waste generated during daily operations. On that account, the requalification for expired prepreg rolls and uncured scraps must occur to ensure efficient material processing. The main problem that arises is that this material and waste have high variability in their properties, which eventually may lead to sub-optimal processing. As such, the identified KPIs for the evaluation of the CRIS that reflect the success of this processing include prepreg shelf life, prepreg disposal, value of uncured prepreg scraps, and unused CFRP waste in the production of composite materials.

KPI1.1: Prepreg Shelf Life

This evaluation KPI relates to the shelf life of prepregs and reflects the maximum time that prepregs can be stored under specific conditions and continue to remain usable for their intended function. To support the creation of the KPI, the CRIS should monitor the same material code (in terms of lot number) used by the factory, while simultaneously tagging the material with the creation date. In this way, the CRIS can track when the factory will use the material and acquire the new expiration date in order to calculate it. The formula to calculate the KPI is as follows:

$$KPI1.1={\displaystyle \frac{\left(N_c-n\right)}{n}}\ast 100$$

(1)

where N_c is the number of days passed from the prepreg production day to the new expiration day (defined through the requalification procedure conducted in another factory of the value chain) and n is the number of days passed from prepreg production to the expiration day defined by the prepreg manufacturer and indicated in the technical data sheet (baseline shelf life is equal to 365 days for base materials, where dynamic numbers should be considered for different material(s)).

KPI1.2: Prepreg Disposal

This evaluation KPI relates to the amount of prepregs that are disposed of in landfills because they become unusable. Through the CRIS, the tracked and monitored prepregs will enable the value chain’s participating factories to monitor the overall quantity of prepregs that is disposed of in a landfill for a specific time period (e.g., monthly). Following that, and relative to the amount of time the pilot is operational in the different rounds, an extrapolation of the results can reflect the potential of the CRIS to reduce the prepreg disposal rates. The formula to calculate the KPI is as follows:

$$KPI1.2={\displaystyle \frac{N}{month}}$$

(2)

where N is the quantity of prepreg in Kg that ends up in landfills.

KPI1.3: Value of Uncured Prepreg Scraps

This evaluation KPI relates to the identified economic benefits derived from the CRIS’s application on the material constituting the focal point (prepreg) of the project. It is reflected by the value of uncured prepreg scraps that the system can calculate through tracking the respective material and transforming it into its monetary equivalent for the given time period. The formula to calculate the KPI is as follows:

$$KPI1.3={\displaystyle \frac{N}{monthly}}$$

(3)

where N is the cost of prepreg in EUR that ends up in landfills and t is the selected time unit (e.g., monthly, custom).

KPI1.4: Unused CFRP Waste in the Production of Composite Materials

This evaluation KPI relates to a value-network-wide process evaluation that the CRIS enables that involves the quantity of CFRP waste coming from the production factory, the quantities that were successfully requalified, and, lastly, the ones actually used in another factory’s production, having been successfully repurposed from being discarded in a landfill. The formula to calculate the KPI is as follows:

$$KPI1.4={\displaystyle \frac{\left(a-b\right)+(b-c)}{a}}\ast 100,$$

(4)

where a is the quantity in Kg that Factory A is sending to Factory B instead of disposing of it in landfill, b is the quantity in Kg of the CFRP that Factory B requalifies, and c is the quantity in Kg that Factory C used from the material obtained after the requalification.

3.1.2. Pilot 2—WEEE for Magnets

The rapid increase in demand for permanent magnets (PMs) has led to the need for access to enough rear earths (Res) to become a problem. One among several processes for making magnets is the injection of a raw material called neodymium iron boron (NdFeB) into magnetic plastic via a process of injection molding. As the produced magnets serve the production of various solutions, they have applications in sectors such as automotives, electronics, and motors, among others. Once these products (that have the magnets embedded) reach their end of life, they become Waste Electrical and Electronic Equipment (WEEE). The overall goal of the CRIS in this pilot is to enable the involved factories to reuse the maximum possible reclaimed magnetic material from the WEEE, reprocess it, and reuse it in the production of new magnets. The problem, in this case, is that there are complexities in the specific properties of magnets that can highly influence the final output and the to-be-designed requalification process for contaminated sintered Sr-ferrite magnets and bonded NdFeB. As such, the identified KPIs for the evaluation of the CRIS, that reflect the success of this processing, include the reduction of WEEE landfilled (for the bonded materials’ part), the usage of SRM in PM magnet pellets’ production origin, the number of validated materials, and the minimization of raw materials’ insertion PLES08.

KPI2.1: Reduction of WEEE Landfilled (for the Bonded Materials’ Part)

This evaluation KPI relates to the potential of reducing WEEE landfilled from magnets that would become unusable if not processed through a CRIS, thus showcasing the reduction in magnets that are destroyed from WEEE landfilled by the total quantity of magnets extracted. The formula to calculate the KPI is as follows:

$$KPI2.1=\mathrm{K}\mathrm{g} \mathrm{o}\mathrm{f} \mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s} \mathrm{e}\mathrm{x}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d} \mathrm{b}\mathrm{y} \mathrm{F}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{t}$$

(5)

KPI2.2: Usage of SRM (Bonded NdFeB, Sr-Ferrite) in PM Magnet Pellets’ Production Origin (%)

This evaluation KPI relates to the potential of reducing WEEE landfilled from magnets that would become unusable if not processed through CRIS, thus showcasing the reduction in magnets that are destroyed from WEEE landfilled by the total quantity of magnets extracted. The formula to calculate the KPI is as follows:

$$KPI2.2={\displaystyle \frac{\mathrm{K}\mathrm{g} \mathrm{o}\mathrm{f} \mathrm{r}\mathrm{e}\mathrm{c}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}\mathrm{d} \mathrm{p}\mathrm{e}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{s} \mathrm{i}\mathrm{n} \mathrm{F}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{y}}{\mathrm{K}\mathrm{g} \mathrm{o}\mathrm{f} \mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{p}\mathrm{e}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{s} \mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d} \mathrm{i}\mathrm{n} \mathrm{F}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{y}}}$$

(6)

KPI2.3: Number of Types of Validated Material

This evaluation KPI is a value-network-wide KPI that relates to the different materials considered for Plooto and is thus utilised successfully in CRIS. The formula to calculate the KPI is as follows:

$$KPI2.3=\mathrm{N}\mathrm{o} \mathrm{o}\mathrm{f} \mathrm{M}\mathrm{a}\mathrm{g}\mathrm{n}\mathrm{e}\mathrm{t} \mathrm{T}\mathrm{y}\mathrm{p}\mathrm{e}\mathrm{s} \mathrm{s}\mathrm{u}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{f}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{y} \mathrm{i}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{d}\mathrm{u}\mathrm{c}\mathrm{e}\mathrm{d} \mathrm{i}\mathrm{n} \mathrm{C}\mathrm{R}\mathrm{I}\mathrm{S}$$

(7)

KPI2.4: Minimisation of Raw Materials’ Insertion PLES08

This evaluation KPI relates to the identified need for recycled pellets, which might require adding raw magnetic material to obtain the desired magnetic properties in the end product. The formula to calculate the KPI is as follows:

$$KPI2.4=\mathrm{P}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{a}\mathrm{g}\mathrm{e} \mathrm{o}\mathrm{f} \mathrm{R}\mathrm{a}\mathrm{w} \mathrm{M}\mathrm{a}\mathrm{g}\mathrm{n}\mathrm{e}\mathrm{t}\mathrm{i}\mathrm{c} \mathrm{M}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{l} \mathrm{i}\mathrm{n} \mathrm{r}\mathrm{e}\mathrm{c}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}\mathrm{d} \mathrm{p}\mathrm{e}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{s}$$

(8)

3.1.3. Pilot 3—Citrus Processing Waste for Juice By-Products

Citrus pellets are produced from orange peel, specifically from citrus waste, the residue usually originating from several citrus fruit varieties, which includes the peel, pulp, and seeds. Their production process has parameters with a large range of variability that needs to be controlled to allow for the valorisation of the end products (from the processing of SRM). Therefore, the goal of the CRIS in this pilot is to enhance the by-product transformation for the cattle feed and involves molasses, d-limonene, and COD. In this pilot, the main process of transforming waste into animal feed is already active. However, the problem lies in optimising the end product and creating a proof of value of the by-products to the cattle feed industry. As such, KPIs that reflect the success of this processing have been identified to examine and evaluate the effectiveness of CRIS in enabling this new process. They include the production of animal feed, production of high-quality molasses, production of d-limonene, the volume of CPWW, the COD of CPWW, the volume of CPWW that goes to biological treatment, and revenues from animal feed.

KPI3.1: Production of Animal Feed

This evaluation KPI relates to the potential of the CRIS to enable the production of animal feed in the factory. The formula to calculate the KPI is as follows:

$$KPI 3.1=\sum number of produced animal feed in tonnes per year$$

(9)

KPI3.2: Production of High-Quality Molasses

This evaluation KPI relates to the potential of the CRIS to enable the production of high-quality molasses in the factory. The formula to calculate the KPI is as follows:

$$KPI 3.2=\sum number of produced high quality molasses in tonnes per year$$

(10)

KPI3.3: Production of d-Limonene

This evaluation KPI relates to the potential of the CRIS to enable the production of d-limonene in the factory. The formula to calculate the KPI is as follows:

$$KPI 3.3=\sum number of produced d-Limonene in tonne per year$$

(11)

KPI3.4: Volume of CPWW

This evaluation KPI relates to the potential of the CRIS to enable the reduction in the volume of the generated CPWW in the factory. The formula to calculate the KPI is as follows:

$$\mathrm{KPI}3.4={\displaystyle \frac{\mathsf{\Sigma } volume of CPWW before CRIS-\mathsf{\Sigma } Volume of CPWW after CRIS}{\mathsf{\Sigma } Volume of CPWW before CRIS}}$$

(12)

KPI3.5: COD of CPWW

This evaluation KPI relates to the potential of the CRIS to enable reduced COD in CPWW in the factory. The formula to calculate the KPI is as follows:

$$\mathrm{KPI}3.5={\displaystyle \frac{\mathsf{\Sigma } amount of COD in CPWW before CRIS-\mathsf{\Sigma } amount of COD in CPWW after CRIS}{\mathsf{\Sigma } amount of COD in CPWW before CRIS}}$$

(13)

KPI3.6: Volume of CPWW that Goes to Biological Treatment

This evaluation KPI relates to the potential of the CRIS to enable the measurement of the volume of the CPWW that goes to biological treatment in the factory. The formula to calculate the KPI is as follows:

$$\mathrm{KPI}3.6={\displaystyle \frac{\begin{array}{c}\mathsf{\Sigma } volume of CPWW to biological treatment before CRIS-\\ \mathsf{\Sigma } volume of CPWW to biological treatment after CRIS\end{array}}{\mathsf{\Sigma } volume of CPWW to biological treatment before CRIS}}$$

(14)

KPI3.7: Revenues from Animal Feed

This evaluation KPI relates to the potential of the CRIS to enable the increment in total revenues from selling animal feed produced in the factory. The formula to calculate the KPI is as follows:

$$\mathrm{KPI}3.7=\sum Revenues from animal feed, animal feed components$$

(15)

3.1.4. CRIS System Evaluation

A circular and resilient information system consists of various components and provides respective services based on several modules, which, in unison, enable the system to fully valorise its resources and provide factories with the envisaged benefits (as evaluated by the respective KPIs). As it is a novel system that supports circularity and resilience goals, it will also have to undergo evaluation in various technological and user-related aspects in order to enable (a) its refinement based on the results of the first round of pilots’ application and (b) its final performance evaluation in the second round of evaluation. As the CRIS is a software solution, several evaluation approaches can be utilised, as presented in the literature review. In summary, Hosseini et al. [6] also present six overarching categories of architectural analysis and evaluation methods for examining the quality of an IS (as can be seen in Figure 2) that can be considered for the CRIS at hand.

In order to enable a rigorous evaluation approach, the system evaluation should take place utilising both measurement-based and questionnaire-based approaches, namely, mathematical modelling, measurement, scenario-based, and questionnaire approaches. The reason behind utilising both measurement-based and question-based approaches is that they are essential to evaluate its performance using a combination of KPIs and qualitative data from interviews and questionnaires. KPIs provide quantifiable metrics that offer objective insights into specific aspects of system performance, such as availability and reliability. However, relying solely on these numerical indicators may overlook nuanced user experiences and contextual factors. Therefore, integrating qualitative data from interviews allows a deeper understanding of user perspectives, challenges, and suggestions not captured by KPIs. This triangulation of data [42]—combining quantitative KPIs with qualitative feedback—ensures a comprehensive evaluation, identifying both measurable outcomes and underlying issues, thus guiding more informed development decisions. The following sections explicate the approach for the system evaluation of the CRIS following the designed evaluation methodology.

• CRIS System evaluation of functionality performance

To evaluate the system as a whole, various qualitative and quantitative approaches will be utilised. Starting from mathematical modelling approaches, availability and reliability will be evaluated.

Reliability: This KPI relates to the likelihood that a system or system component is functional at a specific moment in time under a particular set of environmental conditions. The formula to calculate the KPI is as follows:

$$R\left(t\right)=\mathrm{P}\left(\mathrm{T}>\mathrm{t}\right)=1-F\left(t\right)$$

(16)

where R(t) is the reliability, F(t) is the failure probability, and T is the time to failure.

Availability: This KPI relates to a component’s/system’s readiness for immediate use at any given time. The formula to calculate the KPI is as follows:

$$\mathrm{A}\mathrm{v}\mathrm{a}\mathrm{i}\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}=\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}-\mathrm{D}\mathrm{o}\mathrm{w}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}/\mathrm{F}\mathrm{u}\mathrm{l}\mathrm{l} \mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\times 100\mathrm{\%}$$

(17)

Extending the overarching CRIS system-wide KPIs, each module, respectively, will be evaluated in specific metrics that reflect its scope of application and expected outcomes, as presented in

Table 2. Modules of CRIS evaluation metrics.

Component	Supporting Processes	KPIs
Dashboard/UI	Viewing of VN-specific KPIs. Adding, deleting users, etc. Execution of 3rd-party services. Monitoring of internal processes. Monitoring of value network.	Number of users designing their own Dashboard.
Sustainability Balanced Scorecard	Viewing of current status relevant to sustainability.	The number of metrics in terms of environment, society, economy, growth, governance, and pilot-specific pillars will be determined according to pilots’ preferences per use case.
Analytics Service	Pilot specific for energy consumption monitoring, environmental footprint monitoring, and anomaly detection.	Reliability of the service to perform real-time analysis when new data are incorporated into the system. Forecasting accuracy (mean absolute error, root mean squared error, bias), anomaly detection precision/recall/F1 score (perceived ability of the system to correctly identify anomalies), system performance (processing time per dataset, system throughput).
Blockchain	Pilot specific for collaboration establishment support in a trustworthy environment	Number of hashed contracts processed, transaction latency.
Digital Twins Modeler	Creation and updating of Digital Twins.	Number of Digital Twins modelled.
Value Network Modeller	Creating, viewing, and acting on the value network and the involved organisations.	Number of companies introduced in the value network; number of companies depicted in the value network.
Optimisation Service	Pilot specific for creation of optimisation of the production planning processes.	Number of production plans generated.
Guidelines for Certification	Evaluation of the CRIS systems’ compliance for certification.	Number of guidelines per value network produced.
PSM Tool	Development of the process models introduced in the CRIS.	Number of control variables for the entire value chain.

.

Extending the evaluation of the CRIS relevant to its performance and the performance of comprising modules, the evaluation methodology also includes dimensions that relate to the overall system and its evaluation from the end-user perspective, particularly the CRIS evaluation of its learnability, usability, and user experience.

Relevant to the CRIS learnability, it refers to the ease with which new users can begin to effectively interact with the system and achieve a reasonable level of proficiency. To enable the learnability evaluation, the “Software Evaluation: Criteria-based Assessment” of the Software Sustainability Institute [43] offers a dedicated section with questions relevant to the existence of a getting started guide, the provision of instructions for basic (or all) use cases, the provision of reference guides, and API documentations, all of which are relevant for the CRIS evaluation. Furthermore, as the newly developed system will be fed with live data from the shopfloor, interoperability and integration issues might arise that need to be accounted for and evaluated. Another section of the criteria-based assessment can be utilised in this direction as it offers a dedicated section on sustainability and maintainability that includes evaluation instruments and processes on the analysability, changeability, evolvability, and interoperability categories [43].

Firstly, from the end-user perspective, its overall usability was found to be important for evaluation, as well as the user experience the users will have in the process of its application. For the CRIS usability, the evaluation methodology will employ the widely utilised System Usability Scale (SUS). The SUS was developed by Brooke [44] as a “quick and dirty” survey scale that quickly and easily assesses the usability of a given product. Its ease of use and reliable results have set the SUS as an industry standard in terms of usability evaluation, as it has been reported as the prime instrument for usability evaluation in over 2300 individual surveys in over 200 studies [45].

Lastly, for the user experience evaluation part of the CRIS system, the widely utilised and validated instrument of the User Experience Questionnaire (UEQ) by Laugwitz et al. [33] will be used to record the overall experience of the factory employees using the developed system, based on its attractiveness, perspicuity, efficiency, dependability, stimulation, and overall novelty.

The aspects above of the CRIS evaluation relate to the end-users’ (factory employees’) point of view. They will be administered during the pilot rounds of evaluations and will serve as the data collection instruments for the qualitative part of the evaluation of the CRIS system.

Finally, focusing only on real-time key performance indicators is not sufficient. To this end, a periodical assessment of the systems and performance under different conditions is necessary to evaluate the long-term impact of circular and resilient information systems (CRISs). In other words, evaluations at regular intervals can allow businesses to track an CRIS’s ongoing and sustained system performance that may emerge over time, whether positive or negative, after the introduction of the CRIS in the day-to-day activities on the shop floor.

4. Discussion

This research aims to identify and form an evaluation approach for information systems based on emerging technologies, particularly circular and resilient information systems. In the course of the CRIS design and through the involvement of the pilots in the requirement elicitation, a series of evaluation criteria were identified that will enable the validation of the system post-application. Initially, it was identified that a CRIS should be evaluated in a two-fold manner, i.e., its capacity to assist the pilots in their utilisation of SRM through their collaboration in the new circular supply chain (pilot KPIs’ evaluation), as well as its capacity to constantly and consistently perform its tasks (technical evaluation).

Relevant to the evaluation of the pilots’ KPIs, a series of metrics were identified that the system should monitor, evaluate, and present to the user. These KPIs were inherently different from pilot to pilot, as the production contexts (and SRM) were different in nature. In particular, for CFRP waste, the identified KPIs related to the optimisation of the SRM (prepreg) requalification process and its subsequent utilisation will lead to fewer scraps being discarded as they are used in new composite materials. Therefore, it was identified that both in-factory as well as network-wide KPIs should be utilised by the CRIS to evaluate the overall effectiveness of its application for the pilots in the circular supply chain. For WEEE for magnets, the identification KPIs related to the optimisation of SRM (bonded NdFeB, Sr-ferrite) use in new magnet production. Similar to the previous pilot, both in-factory and network-wide KPIs should be utilised by the CRIS. Lastly, in citrus processing waste for juice by-products, the identified KPIs related to the optimisation of the process of the transformation of the SRM (e.g., molasses, d-limonene, and COD), as well as the evaluation of the subsequent business-related outcomes (e.g., revenues from produced animal feed). In this case, the circularity potential is within the factory and, as such, the CRIS should only enable monitoring of KPIs that relate to a single partner rather than network-wide KPIs. It is thus evident that the CRIS system should be evaluated in (a) its ability to monitor varied KPIs relevant to pilot-specific contexts—in order to enable the factories to operate in a resilient manner—as well as (b) its ability to monitor collaboratively produced KPIs from different factories in an interconnected factory. The latter is in order to enable the factories of the network to operate in a circular manner and close the loop of SRM utilisation.

Relevant to the technical evaluation of the CRIS system, results indicate that, as the system consists of various components and services that are naturally classified as established (e.g., simulation) as well as emerging (e.g., Digital Twins), the system should have a three-fold evaluation approach, i.e., component evaluation, system evaluation, and, lastly, user evaluation. For component evaluation, the examined components included the Dashboard, the Sustainability Balanced Scorecard, and the involved services (analytics, simulation, optimisation, and blockchain), as well as the services that enabled the formation of circular chains (value network modeller) and the twining of the individual factories (Digital Twins modeller), each of which has its own metrics that the evaluation will be based upon. For the integrated CRIS system, a number of evaluation approaches and metrics available from the established IS, RIS, and CIS evaluation literature were identified as applicable for CRIS evaluation, with the most prominent being the reliability and availability of the system. Furthermore, and in relation to the evaluation of the CRIS from the end-user perspective, results indicate that a CRIS system should be evaluated on its learnability, usability, and user experience, as the system is novel and the success of its application and day-to-day use will rely on the ability of the involved users to easily understand it and seamlessly utilise it in their decision-making process.

Furthermore, the ethical use of data and sensitive information protection, as well as transparency and regular communication for stakeholders on CRIS impact, are essential considerations for safeguarding trust among parties and avoiding any potential negative long-term impact. To achieve this, CRIS must be combined with policies and frameworks regarding data collection, storage, access, and sharing that comply with privacy regulations. For example, consent and data anonymisation techniques could mitigate risks among businesses or individuals that share data. In our study, all data collected and analysed are based on stakeholders’ consent and anonymity when deemed necessary. Additionally, the outcome of a CRIS evaluation can allow businesses to understand the positive or negative impact and decide whether they take value from sharing their data with other actors in the supply chain.

Finally, examining the impact of CRIS on various dimensions may be crucial for various industries. When our methodology is adjusted to a particular case, the potential impacts on economic growth, social welfare, and the environment should also be considered. To this end, the set of KPIs derived from the study of Comini et al. (2020) can also be taken into account. This study categorises manufacturing KPIs based on the sustainability triangle in terms of economic, social, and environmental pillars.

Overall, this research presents an evaluation methodology for novel circular and resilient information systems (CRISs) that utilise established and emergent technologies to enable both circularity and resilience in manufacturing. It sets the groundwork for a two-fold evaluation approach (pilot-specific and technology-specific) to account for all needed aspects that will enable a rigorous evaluation approach. Its findings included 15 pilot-specific KPIs for three different circular value networks, 11 technology-specific KPIs for the system and sub-component, and three dimensions for end-user evaluation. Its outputs can thus be utilised by relevant information systems that are developed to enable manufacturing to introduce parallel circularity and resilience in their day-to-day operations.

5. Conclusions

This study makes significant contributions to the fields of information systems and sustainability by introducing a novel evaluation methodology for circular and resilient information systems (CRISs). It addresses the critical need to assess these systems beyond traditional metrics like efficiency and user satisfaction. The methodology provides a replicable tool for practitioners to evaluate CRIS across diverse industries, as demonstrated in the aforementioned case studies. Furthermore, this research supports companies in enhancing circularity and resilience and has policy implications, advocating for regulatory frameworks to support the adoption and evaluation of CRIS in manufacturing systems. Thus, this study effectively bridges the gap between theory and practice, advancing both academic research and practical implementations in circular and resilient information systems.

Focusing on circularity and resilience, this study addresses gaps in traditional assessment methods and bridges the gap between theory and practice. Additionally, its application to diverse case studies and consideration of policy implications further solidify its practical relevance and potential impact on shaping sustainable industrial practices. Our approach can be utilised as guidance and can be replicated in other manufacturing industries following the practical example of the CRIS system implemented in the context of the Plooto EU project, offering a tool for practitioners to access relevant, innovative information systems and technologies.

This research also has limitations that highlight an avenue for further research. Initially, as the system is based on emerging technologies that are, in turn, utilised in a novel manner (a combination of a CIS and RIS) to enable parallel benefits, the evaluation of such a system is also novel. As such, further research is required on new CRIS systems. Another limitation stems from the selected cases and their potentially limiting scope imposed on the system. Although the cases are from diverse manufacturing contexts and the SRM included in this research is of high importance in the European economic landscape, the case remains that different cases would require different pilot KPIs and, as such, a CRIS should account for (and be evaluated on) them as well. As such, further studies could investigate the integration of more advanced KPIs, such as those related to social sustainability, to create an even more holistic evaluation framework. In this direction, further research on different SRM and manufacturing sectors, as well as different geographic regions, will enable the CRIS evaluation methodology to be validated and extended, respectively, in further diverse contexts. These future directions would help refine and expand the applicability of our methodology, ensuring it remains relevant as technologies and sustainability practices evolve.