After the platform presentation, in this section, the architecture is explained, and two use cases that validate the aforementioned approach are presented.
3.1. Architecture
The proposed architecture consists in five main elements that will communicate with each other, namely the cloud middleware, the supply chain, the smart factory, the customer, and the product, as shown in
Figure 10. The middleware hosted in the cloud will be the mutual point of interaction. Thus, all the other elements need to reach the cloud to share and exchange data, information, orders, and so on.
In this way, each component needs to expose its functionalities to the others to be reachable. Thus, this architecture will follow a service-based approach, which means that all the components of the architecture will expose its functionalities as services, making them available for the other components, and then be able to establish communication with the different components and systems. These services will be exposed through standard interfaces that are the point of interconnection between two systems or sub-systems in a transparent way.
The main components of the system, as well as their inner components, will interact with each other through the system’s cloud middleware (
Figure 11), where all the data and information will be stored, available, and updated continuously or periodically, depending on the type of data. The elements will not interact directly with each other, rather they will interact with the common middleware to reach the other elements. The cloud system will ensure that even if an element is down, its latest information will still be available to the remaining elements. Both vertical, i.e., integration and interoperation of all software tools and hardware inside the factory, and horizontal, i.e., capacity to interoperate all the actors that contribute to the value chain, such as suppliers, factories, and the product itself, communication should be supported by the cloud middleware.
The middleware works as a system or software component that is used to connect different applications or systems. The objective is to create communication between applications without them having to know about each other’s state and with minimum knowledge about the network connection [
39]. Middleware systems have been more adopted lately since vast quantities of systems with different responsibilities are involved and must work cooperatively to keep the production in the good direction, from sensors and actuators on the shop floors to management systems for ERP.
This architecture presents two middleware systems, the one in the cloud and the one in the factory, that will provide a service registry, Yellow Page structure, that stores the diverse services available in the system.
The distributed and cloud approach that handles the interconnection of the elements by following service-oriented principles will avoid limitations for scalability, as well as problems related to single points of failure.
Both middleware systems, the one on the cloud and the one internal to the factory, will work with a publish-subscribe model. This model enables an application to announce events to multiple consumers by asynchronously broadcasting messages to different parts of the system.
Our approach considers that each component that sends data to the cloud or to be consumed by other entities is a publisher. A component that reads messages from a specific topic is a subscriber. The same entity or software can be a publisher and a subscriber at the same time. Typically, a publisher translates changes in the system or action into event messages, using a generic message format. Those messages are then published into specific topics in the middleware through a standard interface. The subscribers are notified that a new message is available on the topics that they are subscribed to.
Figure 12 exemplifies this pattern.
A publish/subscribe model is an effective way to decouple sub-systems and manage communications between them, independently. Thus, this model avoids blocking sub-systems to wait for responses and quickly return to their main processing responsibilities, increasing the scalability and responsiveness of the overall system.
Moreover, this model offers improved reliability under increased loads, handles recurrent failures more effectively, and provides a straightforward integration between systems using different platforms, programming languages, or communication protocols, as well as between factory systems and cloud applications.
By design, publish/subscribe methods can guarantee some security features such as only delivering messages to registered and authenticated clients and can also retain some messages in case of any failure of the clients. The resilience of the communication channel can also be ensured with load distribution between the server nodes and high availability methods implemented by the publish/subscribe mechanism that ensures, besides availability, scalability.
The publish/subscribe methods that the middleware is implemented with are agnostic to the programming languages that send or receive the messages. This enables interoperability between stack holders by design.
In our work, we present the use of two middleware systems, independent from each other because, at the shop floor, it is usually common to have legacy elements such as machines, working stations, and some sensors that may not be able to communicate directly to the cloud, either by their constraints or because of internal policies from the manufacturing companies. Human operators can input data to the cloud using a Human Machine Interface (HMI) that can write to this middleware, and then the message is translated into the cloud middleware. Machines can use their actual MES system with their actual metrics or use a specific gateway to translate the messages to the middleware on the shop floor, and the messages are forwarded to the cloud gateway.
Figure 13 shows these approaches.
Both middleware systems, the one in the cloud and the one in the factory, will then communicate through a standard communication protocol to update and access each other’s information.
In the same way that some shop-floor machines can require a gateway to translate messages to the shop-floor middleware, other stakeholders may also need a gateway to translate their information to the cloud middleware, to send and receive messages that will be used in a standard way.
Figure 14 shows an example of a supply chain element that uses this approach. “Smart Transport” and “Legacy Transport” can be routed from raw material suppliers or routes of sub-assemblies between factories or even final products.
This example from
Figure 14 can also be applied to other elements such as products. In this example only standard communication is used. A legacy product can need a gateway to convert the legacy data to the standard communication from this platform. A legacy product is a product that was not designed to use these functionalities and/or does not have ways to communicate in a standard way. For instance, a brand-new car can be considered legacy if its configuration does not have a module to automatically connect to the platform. If the final user wants to use the functionalities from the platform, they can afterward acquire a gateway to send and receive the information from and to the platform.
3.2. Data Models
One of the biggest challenges of I4.0 is interoperability in real industrial systems. There are still a lot of difficulties in dealing with the representation and exchange of data coming from different entities and from different operating levels [
41]. Our goal is to have full interaction of hardware resources, such as working stations and robots, and software applications, such as ERP, MES, or databases, from end to end in a bi-directional way. To have an architecture accepted by the industry, it is crucial to connect and integrate legacy systems.
Following this line of thought, the adoption of standard interfaces as the core drivers for pluggability and interoperability will contribute to enabling the interaction between hardware devices and software applications transparently. The standard interfaces should support devices, tools, and applications in order to fully expose and define their services in an exclusive, standardized, and clear way to increase the whole interoperability. These standard interfaces should provide a set of common functionalities:
The definition of the list of services to be implemented by the interface.
The description of each service (name, input, and output parameters).
The definition of the data model handled by the services.
From the point of view of the overall system, the standard interface abstracts the fundamental functions, making transparent the way the different components interact and operate. Additionally, a key requirement for the development of standard interfaces is the usage of service orientation to expose the corresponding devices and applications’ functionalities as services.
Nevertheless, to integrate legacy devices and systems, with their data models and specific requirements, it is necessary to allow the translation of legacy data into a uniform representation. To do so, it is necessary to add gateways as a middleman that will translate the legacy data and allow those devices and systems to have additional intelligence and be integrated into the CPS paradigm.
3.3. Technologies and Implementation
The architecture proposed needed to be validated, and, to do so, a set of technologies and some implementation details were chosen.
3.3.1. Communication at the Integration Layer
The primary objective of the integration layer is to facilitate the trade and flow of information from one system to another system that benefits from the data. This process can be, for example, a machine sending its data to an optimization software that generates new knowledge and new inputs that then flow in the opposite direction to the machinery, optimizing the process. To control the flow of information of different machinery and software mentioned, a publish/subscribe approach was chosen based on a message broker. To implement this approach, a study was performed to test different known software. Several message brokers were analyzed and compared in key points relevant to the Integration process that is conducted.
Using a publish/subscribe model that uses a topic approach allows the publishing of data by, for example, the MES system, in well-defined topics. This means that different topics are used to write different types of information, for example, distinct topics for sending energy consumption information and receiving production orders. This can be done by communicating directly with the hardware/software tools, using an auxiliary program, or through a service, depending on the configuration of each tool. The publisher can write on several different topics different types of information. These topics can then be subscribed to by multiple subscribers that are notified by the message broker as soon as new information is available. The message broker will then send the new information to all the subscribers of the topic in question. The message broker can communicate using different kinds of data-interchange formats, mainly Avro, JSON, and Protobuf, being able to interchange and auto-format different types of data as needed. This functionality can be particularly useful when setting up streams or communicating with a different machine that possesses different data formats.
The topic approach lets the machine, or the software, send data when it is more convenient, without the necessity of waiting for synchronized communication with other entities or causing a bottleneck in the flow of information [
40]. These topics can then be subscribed to by any tool (for instance, software of logistics, simulation, energy consumption, or any other) that wants to receive the published data and has permission to do so. This approach also allows easy integration for more machines or optimization software that need to receive or publish data. New tools only need to communicate with the message broker and have knowledge of the structure of the topics and expected data format.
3.3.2. Enterprise Resource Planning
A commonly used definition of an ERP system is a set or collection of all-inclusive software used to manage business operations and information, particularly in the manufacturing sector. All the business operations inside a company may be managed and integrated using an ERP system. In a typical ERP operation, two or more agents are involved in the completion of the business process. For example, a customer ordering a part. The request is served from an external source such as a web application in the ERP. This request is converted into a sales order and afterward converted into a purchase order, and when the order is completely processed, the user receives the product delivery information. In all these instances, the transaction passes through different actors, where it can be approved or rejected.
Figure 15 shows the class diagram of ERP and web applications, and the relations between them. The web application data model includes users of the systems and purchase orders they can make. The ERP data model shows the relations between sale orders, products, and production orders and how information moves from the initial user pre-order to the production of the product.
The User class (
Figure 16) represents users in the web application. The Partner class (
Figure 17) is related to the User class in the ERP system with a relation of one-to-one. To facilitate our integration with the blockchain (for traceability and trustability), the User class contains information about blockchain authToken and publicKey. The Partner class (
Figure 17) is mostly used to identify the customers.
Users can create PurchaseOrders (
Figure 18a), which are related to the Product class (
Figure 18b) and ERP SaleOrder class (
Figure 18c). PurchaseOrder contains necessary information about the sale to the cloud. The Product class contains the selected configuration of the product to be produced. SaleOrder is the class that contains the ERP sale information and has relation to the Partner class, which is the customer of the sale. Each customer can have multiple SaleOrders. Products ordered are contained in SaleOrderLine (
Figure 18d) class, and SaleOrder can have multiple SaleOrderLines.
The Product class is related to the ProductTemplate (
Figure 19a) and Production (
Figure 19b) classes. ProductTemplate is the functional description of a kind of product, it includes the bill of materials (BoM) identification, category, and company, among other details.
For instance, ERP might have a basic template for the Basic product (on ProductTemplate). The Product class describes it more in-depth, with other details, for example, the color of the product. The Production class describes the manufacturing details of the ordered product and sub-product/parts. Production has a relation with the BillOfMaterial class (
Figure 19c) which contains the list of parts needed to manufacture the product. BillOfMaterial can contain multiple BillOfMaterialLines that represent the parts needed to manufacture the final product to be delivered to the customer.
3.3.3. Manufacturing Execution Systems
MESs are tools designed to help on the shop floor and to follow the processes from the raw material until the finished product (or sub-product). These tools enable the decision makers to act in real time to improve processes. MESs are nowadays supported by IIoT where all kinds of sensors (and actuators) can be connected and interacted with from high- to low-level systems. In this way, there is a holistic approach from tools like ERP and LCA or PLCs and workers.
The use of MES can reduce time and costs on the shop floor if the information flows properly and if both the machines and processes from the factory are fully digitalized.
The MES architecture proposed is presented in
Figure 20. MES is a type of software that can be used by a manager and has two interfaces: one with REST API, so machines and shop-floor workers can interact with the system (local), and another based on publish/subscribe methods, so external software can interact with the system (cloud-enabled). The local REST API can be seen as a gateway that was directly built in MES to simplify the operations on the shop floor. This MES software is also supported by a database that records all the actions relevant to the production environment.
With the publish/subscribe interface, MES can read or write on external software without changing their internal state and, if needed, can postpone this task giving priority to shop-floor control and management through the REST API (both communications are bi-directional as can be seen in
Figure 21).
The REST API of MES enables the shop-floor machines to interact with the system but also enables the HMI to be connected, and a worker can add more information manually about the production processes. This API features the following services: Get Production, Add Production, Get total Production, Get Events, Set Events, Alter Event, Get Worker Registers, Set Worker Registers, Alter Worker Registers, Get Production Unit, Set Production Unit, and Alter Production Unit (as depicted in
Figure 22). With this data, MES can create and show graphics and metrics available to managers to help with decision support.
3.3.4. Traceability/Trustability
In terms of traceability and trustability, the proposals stemming from the early investigation into the appropriate architecture to support a more generic framework of component interaction resulted in the use of a blockchain substrate. The ecosystem under evaluation required the integration of several key components including (a) an ERP to provide a customer interface and order placement, subsequently passed on to (b) the MES component to provide manufacturing control and (c) provide a programmatic allocation of hardware resources. The implementation was performed using Hyperledger Sawtooth (
https://www.hyperledger.org/use/sawtooth (accessed on 7 September 2023)) since it was the one that after brief research seemed more appropriate. This technology has a loose coupling architecture, is more energy-efficient (than other blockchain implementations), has a Byzantine-fault tolerance, and implements parallel scheduling, the smart contracts can be developed in several programming languages like Python, C++, and JavaScript, among others, and it has two modes of operation, Permissioned and Permissionless.
Since blockchain is a decentralized technology to be running in the cloud, four nodes were implemented inclusive of the required services to support the architecture integration requirements and support any future integrations, with real-world manufacturing environments. The following figure (
Figure 23) outlines the network and exposes the services implemented.
A transaction family under the Hyperledger Sawtooth blockchain framework consists of application-specific business logic. A transaction family smart contract aimed to alter the data stored in addresses under the platform namespace. All data under a namespace prefix follow a consistent address and data encoding/serialization scheme that is determined by the transaction family, which in turn defines the namespace.
Hyperledger Sawtooth represents the state of all transaction families in a single instance of a Merkle–Radix tree on each Sawtooth validator. Data are stored in leaf nodes, and each node is accessed using an addressing scheme represented as a 70-hex character combination. The address format contains a 6-hex character prefix defining the namespace and 64-hex characters describing schemes for subdividing further and distinguishing entities (agents, records, etc.).
3.4. Use Cases
To validate the architecture, two use cases were developed. Since the main idea is to validate the architecture and its flexibility, some components were shared between both scenarios (such as the ERP), and others were completely different, suitable, and tailored for each one of the cases. For instance, one of the MES used follows the proposal from
Section 3.3.3, and the other follows a more conventional approach trying to demonstrate that the architecture and this platform are compatible with both approaches. This section is not meant to describe to the reader how each component works but that each component is compliant with the architecture interfaces.
The ERP, located in the cloud, sends a production order to MES. In the MES, a manager from the factory will check if the order can be produced and accept or reject the order, accordingly. If the order is accepted, the MES sends the production order to the machine, within the factory, so it can start production. The machine gives the status of production to the MES as well as a confirmation when the production is finished. The transactions of this process are registered in the blockchain, and thus it can be reliably accessed, for example, by the ERP. This process is demonstrated in
Figure 24.
In this scenario, energy consumption was one type of information agreed on by all stakeholders as vital data that should be exchanged in a trustable manner. With this information, the client can map and be informed of the impact that the production of his/her product has on the environment.
3.4.1. Application of 3D Printer for Additive Manufacturing
This use case is an end-to-end demonstration that can be applied to a network of micro-factories that can produce products with additive technologies such as 3D printers.
Using the same sequence diagram from
Figure 24, a new product is bought by the client, which places a new order (
Figure 25 and
Figure 26 depict the user interface to add an order).
Conventionally, the term “client” often evokes the notion of the end consumer; however, within the realm of micro-factories, the client paradigm expands to encompass entities such as Original Equipment Manufacturers (OEM).
Figure 25 shows a listed price value. This listed price varies in conformance with several factors, such as the factory chosen proximity, production scheduling, premium freight, or electricity cost. The price has a fluctuation calculated by the ERP using these metrics that can be shared using the same mechanism of authenticity as the energy consumption that we focus on here. In
Figure 26, we illustrate the option to customize a component by sending a valid file (such as an *.stl file) through the ERP to the manufacturing site. This file, too, undergoes authentication and gains traceability through the blockchain component.
The order execution is validated by the ERP which sends an invoice to the client (
Figure 27) and, through the cloud middleware, the information to the factory (
Figure 28).
As can be seen in
Figure 28, the ERP interface displays solely the generated hashes. However, the information exchanged—readily available for auditing—comprises comprehensive details and specifications. In this instance, these details encompass factors such as filament type, color, density, and notably, the file housing the design of the part intended for printing.
With this information, the factory can accept or refuse the order (
Figure 29). An acknowledgment is sent through the cloud middleware, and the manufacturing process (if accepted) starts (
Figure 30).
In
Figure 29, it is possible to see that the MES system is completely different from the ERP. They are instantiated in two different countries—ERP in Finland, MES in Portugal, and blockchain nodes in Greece—and rely on the cloud to exchange information. Time performance evaluations were not conducted during these tests, as they fell outside the scope of our objectives. The primary focus was on information exchange, a process consistently achieved within a delay of less than five seconds, and the authenticity of the data that could not be accessible from third parties.
At the end of the production (
Figure 31), the information about energy consumption is sent to the cloud middleware to be available to all tools that need it (
Figure 32).
Figure 32 shows the output of the analyses that can be performed, by an authorized stakeholder, in which the progress and all the commitments done for the blockchain can be tracked. Stakeholders requiring access to energy consumption data, such as the ERP for conducting price analyses in this scenario, must be authorized at the blockchain level. The MES system internally stores these data within its own database; however, upon committing to the blockchain, it cannot read it from there due to a lack of reading authorization. An authorized entity has access to the information such as the one in
Figure 32 where the “Consumption” can be seen, and the “flow” can be crosschecked in
Figure 28. Looking at this use case, we can say that it was possible to use the architecture and the implemented platform in a heterogeneous environment since we demonstrated the use of several actors with different capabilities to achieve a common goal. This is an example of the use of cloud computing in the middleware, where horizontal integration, ERP, MES, and the blockchain are all different stakeholders, and the use of IoT in the sensors that measure the power consumption. This use case could be expanded using tools that make big data analyses or that contribute actively to developing other additive manufacturing techniques.
3.4.2. Welding Stations for the Automotive Industry
Just like the previous use case, in this scenario, the idea is to demonstrate an end-to-end interaction between all the actors, using the same architecture but highlighting the flexibility to have more stakeholders. For simplicity, we will maintain the ERP software (demonstrated by
Figure 25,
Figure 26,
Figure 27 and
Figure 28), which will send the information to another partner to produce a different part. This use-case manufacturer is represented by two different industrial robotic cells, designed to perform spot welding processes (the two robotic cells can be seen on
Figure 33). This scenario is implemented in an R&D facility and respects all current industry standards, in terms of safety, hardware, and ISOs; however, they are not included in any active production system or value chain. This means that these cells can provide industrial representative cells, without the major drawbacks of using cells integrated with active production systems.
This use case uses a more traditional approach in which the machines and MES are connected to an “Enterprise Information BUS”. The topology is depicted in
Figure 34 and allows one, with few modifications, to use any cloud system on top of it.
This system uses OPC UA as a medium of communication between the components, since it is the most widely accepted communication protocol in most industries, especially when considering the realities of the automotive sector and all the safety and internal policies of the different constructors.
The “Station Server” is responsible for providing communication middleware between hardware devices and the system in a bi-directional way. The “Device Adapter” is responsible for collecting the raw sensor data provided in the station server and processing it into relevant KPIs for the production process being performed and deconstructs complex orders to be produced by the unit. The “Enterprise Information Bus” is mainly responsible for creating bi-directional communications between components and providing features, such as the following: (i) allowing connection to potential cloud services, for which it provides an easy-to-use and integrated exporter that can be adapted to the most commonly used communication mediums seen in cloud services such as Kafka Apache and REST Servers; (ii) collecting production orders from one or multiple ERPs, in order to create instances of products that need to be produced; and (iii) managing the production and distribution of products according to the stations available in the network.
In this use case, MES has the task of assigning products (to be produced) to the available stations. To do so, data from various factors need to be validated such as production deadlines, sustainability, product quality requirements, and station usage, among many others. It is important to mention that the MES can access and use information from other components in the network and external cloud systems, like the orders from the ERP allowing for a more data-driven decision production optimization in the MES process.
This use case highlights the flexibility of the architecture and platform since the only requirement was the inclusion of the publish/subscribe mechanism and the agreed information to exchange, namely the spots and materials to weld as inputs and the energy consumption as an output. The architecture and implemented platform are agnostic to the actors that connect to it. The I4.0 paradigms represented here are the use of cloud infrastructure, horizontal integration, and IoT. In this scenario, we could also easily connect tools that enable augmented reality on the shop floor (and enhance the vertical integration inside the company) and simulation tools that could enhance the performance of the welding robots.