Next Article in Journal
Transforming Industrial Manipulators via Kinesthetic Guidance for Automated Inspection of Complex Geometries
Previous Article in Journal
Development of a Target-to-Sensor Mode Multispectral Imaging Device for High-Throughput and High-Precision Touch-Based Leaf-Scale Soybean Phenotyping
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 23
Issue 7
10.3390/s23073755
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity

by
Dominik Łuczak
*,
Stefan Brock
and
Krzysztof Siembab
Faculty of Automatic Control, Robotics and Electrical Engineering, Poznan University of Technology, 60-965 Poznań, Poland
*
Author to whom correspondence should be addressed.
Sensors 2023, 23(7), 3755; https://doi.org/10.3390/s23073755
Submission received: 5 March 2023 / Revised: 30 March 2023 / Accepted: 31 March 2023 / Published: 5 April 2023
(This article belongs to the Topic AI and Data-Driven Advancements in Industry 4.0)
Browse Figures
Versions Notes

Abstract

:
The human-centric and resilient European industry called Industry 5.0 requires a long lifetime of machines to reduce electronic waste. The appropriate way to handle this problem is to apply a diagnostic system capable of remotely detecting, isolating, and identifying faults. The authors present usage of HTTP/1.1 protocol for batch processing as a fault diagnosis server. Data are sent by microcontroller HTTP client in JSON format to the diagnosis server. Moreover, the MQTT protocol was used for stream (micro batch) processing from microcontroller client to two fault diagnosis clients. The first fault diagnosis MQTT client uses only frequency data for evaluation. The authors’ enhancement to standard fast Fourier transform (FFT) was their usage of sliding discrete Fourier transform (rSDFT, mSDFT, gSDFT, and oSDFT) which allows recursively updating the spectrum based on a new sample in the time domain and previous results in the frequency domain. This approach allows to reduce the computational cost. The second approach of the MQTT client for fault diagnosis uses short-time Fourier transform (STFT) to transform IMU 6 DOF sensor data into six spectrograms that are combined into an RGB image. All three-axis accelerometer and three-axis gyroscope data are used to obtain a time-frequency RGB image. The diagnosis of the machine is performed by a trained convolutional neural network suitable for RGB image recognition. Prediction result is returned as a JSON object with predicted state and probability of each state. For HTTP, the fault diagnosis result is sent in response, and for MQTT, it is send to prediction topic. Both protocols and both proposed approaches are suitable for fault diagnosis based on the mechanical vibration of the rotary machine and were tested in demonstration.

1. Introduction

Modern factories, cites, and households are equipped with an increasing number of electromechanical systems, which consume energy and have limited lifetimes. The appropriate maintenance of these devices can increase their useful life, which is cost-effective and environmentally friendly, reducing the number of electronic waste. In the literature, the term “electronic trash” or “trash” can also be found [1,2,3], but more often it is “electronic waste”, “e-waste” [4,5] or WEEE (Waste Electrical and Electronic Equipment) [6,7,8]. Maintenance of industrial machines is more demanding due to the complexity of electrical and mechanical parts. Perspective maintenance allows to avoid the unwanted stoppage of the production cycle and prevents unwanted damages of equipment. Deep understanding of available tools and the application of fault diagnosis is more demanding with the number of scientific articles already published in the field. Under keyword “fault diagnosis”, Google Scholar lists nearly 1.6 million articles. Limitation of results to “industrial machines” with operator AND gives 1.6 thousand articles. Another aspect of modern factories, cities, and households is increasing capability of exchange data through the global Internet network, which is named IoT (Internet of Things) or more narrowly, IIoT (Industrial Internet of Things) for industrial interconnections. Another term is Industry 4.0 which “describes the organization of production processes based on technology and devices autonomously communicating with each other along the value chain” [9,10]. However, the new term “Industry 5.0” was introduced towards a sustainable, human-centric and resilient European industry [8]. The publication database with the keyword “fault diagnosis” and “IoT” gives almost 12.3 thousand publications in the field. Therefore, both areas of fault diagnosis and IoT are popular and a growing field in the research area. The review of the patent database gives a new view on IoT and fault diagnosis. Since January 2020, the IPC (International Patent Classification) has been extended with the subclass G16Y titled “information and communication technology specially adapted for the internet of things”. Additionally, subgroup G16Y40/00 “IoT characterized by the purpose of the information processing” is dedicated to maintenance and management. Further detailed classification symbols are G16Y40/10 for detection; monitoring, G16Y40/20 analytics; diagnosis and G16Y40/40 maintenance of things. The Espacenet patent search gives nearly 2.8 thousand patents in a search by classification symbols G16Y40/10, G16Y40/20, and G16Y40/40. In the following sections, the author makes efforts to present a brief review of fault diagnosis and IoT, however, due to the large number of articles and patents, the limited pages of articles, and the limited time resources of the author, some aspects were omitted.
The manuscript is organized as follows: In chapter 2, general structure of the data collection system with local and Internet connectivity is described, which gives a view of the communication aspects of the fault diagnosis system with IoT for batch and stream data processing; in chapter 3, in detail are described IoT protocols such as HTTP (Hypertext Transfer Protocol) and MQTT (Message Queuing Telemetry Transport) for batch and stream data processing and their usage for data ingestion and data routing in the fault diagnosis system; in chapter 4, the structure of the fault diagnosis system from the signal processing point of view is presented; in chapter 5, different feature extraction methods that can be used for diagnosis in frequency domain such as FFT (fast Fourier transform) algorithms (radix-2 and radix-4), STFT (short-time Fourier transform), and group of algorithms of SDFT (sliding discrete Fourier transform) are explained; in chapter 6, a demonstration rig with MQTT connectivity is presented with dataset description in the time domain, frequency domain, and time–frequency domain. Based on preliminary selected single-axis data (gyroscope Z axis) in frequency domain features calculated by SDFT, a classifier was trained and tested which is an enhancement and reduction of computation requirements in comparison to features calculated by STFT; in Chapter 7, a proposed concept of usage 3-axis acceleration and 3-axis gyroscope sensor to create RGB (red, green, and blue) images from six time–frequency domain features which can be recognized by CNN (convolutional neural network) is presented and verified. The proposed approach proves that CNN can be successfully used in multispectrogram recognition which are organized into RGB images for fault diagnosis without the need for preliminary selection of the vibration axis. The proposed recognition of the RGB six-spectrogram and enhanced usage of SDFT instead of STFT was compared with other available methods and the results are shown in Table 1. CNN is used in vision-based recognition and vision applications [11,12,13,14]. Therefore, CNN can be applied to recognize specially prepared time–frequency images, as shown in the proposed method.

2. General Structures of Data Collection and Processing for Fault Diagnosis

The IoT maintenance system is based on measurements delivered in time series to the cloud. The most important part is to prepare data pipelines for fast transmission from sensors to the fault diagnosis server. At low level, analogue sensors are handled by ADC (analogue-to-digital converter). However, modern sensors are equipped with ADC and communication interfaces. Therefore, the sensor is connected to thing (microcontroller or microcomputer) by communication interface like SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit), UART (universal asynchronous receiver-transmitter) or another device-specific communication interface. In the first stage of buffer, the classical producer–consumer application is applied between the sensor and the thing (microcontroller) where the sensors are equipped with internal buffers of several samples. The sensor is a producer of data and the thing is a consumer. The next stage is the use of data by microcontroller internal algorithms. The next stage of buffering is a producer–consumer application where the microcontroller is a data producer and the consumer is a cloud (server)-based fault diagnosis system. In each stage of data processing, the producer–consumer uses buffer for data. The general structure of the buffer-connected data pipelines is presented in Figure 1. Modern buffers offer routing to selected destinations which lead to processing in chain by micro services which are easy to modify compared to change in one large service. The collected data can be processed by the fault diagnosis system in two general ways, (a) batch processing or (b) stream processing. Batch processing is performed with data collected in a time window, but not often. Data stream processing is similar to batch processing but is performed repetitively in micro batch and can use the result of the previous micro batch result in the current micro batch processing.
The general structure of data pipelines presented does not explain which protocol is used by the thing to ingest data to the cloud. Data collected in the thing network called OT (Operational Technology) are ingested into the cloud in the IT (Information Technology) network by the bridge system (Figure 2). OT part of the network is very wide. In IIoT, fieldbus interfaces can be CAN (Controller Area Network), RS232 (Recommended Standard 232), RS485 (Recommended Standard 485), industrial Ethernet, HART (Highway Addressable Remote Transducer), and other physical layers. The main fieldbus technologies used in industry are presented in the IEC (International Electrotechnical Commission) 61784-1:2019 standard [19]. The 2019 standard groups communication technologies into 19 CPF (Communication Profile Families). The CPFs are divided into 24 profiles: Foundation™ Fieldbus H1 (31.25 kbit/s), HSE (High-speed Ethernet); ControlNet, Ethernet/IP™, DeviceNet®, Profibus DP (Decentralized Peripherals) and PA (Process Automation), P-NET®, WorldFIP® which has 3 profiles, Interbus® which has 3 profiles, CC-Link (Control and Communication Link) which has 3 profiles, HART®, WirelessHART®, SERCOS (acronym for SErial Realtime COmmunications System) I and II, and Mechatrolink-II and M-III. Each CPF profile has described the physical layer, the data link layer, and the application layer of the OSI-ISO (Open System Interconnection—Open Source Initiative) model (Figure 3). The IT network has adopted well the network and transport layer by IPv4 (Internet Protocol version 4) [20], IPv6 (Internet Protocol version 6) [21] and connectionless UDP (User Datagram Protocol) [22], and connection-oriented TCP (Transmission Control Protocol) [23], respectively. Layers above the transport layer are described in the next section. The link layer used by physical and data is Ethernet defined by standard 802.3-2022 [24] approved on 13 May 2022. The physical layer allows for the use of operation over coaxial, twisted pair, or fiber optic cables, or electrical backplanes. The Ethernet standard covers speeds of operation from 1 Mb/s to 400 Gb/s.

3. Internet of Things Protocols for Batch and Stream Processing of Fault Diagnosis

The protocols used in the presentation layer in OSI-ISO model (Figure 3) are generally responsible for ensuring the security of transmission in three main aspects: authentication, confidentiality, and integrity of data. The fault diagnosis server side of the channel is always authenticated, and the client (microcontroller) side is optionally authenticated. Confidentiality means that the data sent over the channel after establishment is only visible to the endpoints. Integrity means that data sent over the channel after establishment cannot be modified by attackers without detection. The latest version of TLS (Transport Layer Security) is 1.3 [25] presented in 2018. The layer allows us to use asymmetric cryptography or symmetric cryptography with a pre-shared key. The highest level in the reference model is the application layer, where the HTTP (Hypertext Transfer Protocol) and MQTT (Message Queuing Telemetry Transport) protocols are widely used in IoT applications. The text-based HTTP protocol was widely used in version 1.1 [26] developed in 1997 [27]. HTTP/1.1 can be upgraded to the WebSocket protocol [28], which is useful for established connections to exchange data streams in a bidirectional manner. The next version 2.0 of HTTP was shown in 2015 [29], where the protocol was changed to binary encoding instead of text. The newest version 3.0 of HTTP was introduced in June 2022 [30]. HTTP/3 uses a new transport layer protocol QUIC (Quick UDP Internet Connections) [31] standardized in 2021 instead of TCP. The QUIC transport protocol uses UDP and TLS to provide a stream of binary frames between client and server. Streams can be intended by the client or server as unidirectional or bidirectional. The HTTP/1.1 protocol can be adopted in the batch processing of faults as the request–response principle (Figure 1). The connection is closed by the fault diagnosis server after the response has been sent to the client request. HTTP/1.1 upgraded to WebSocket, HTTP/2, and HTTP/3 are suitable for the stream processing principle (Figure 1) because the connection is not closed after the server sends the response. The HTTP server can be implemented on the microcontroller (thing) or on the remote computer (cloud), see Figure 4.
Fault diagnosis as batch processing with HTTP/1.1 request–response header is shown in Figure 5. The microcontroller acts as an HTTP client which sends sensor-collected data in JSON (JavaScript Object Notation) format of 4294 character length by the POST method. The server path is /…/predict.php in the request line. The HTTP server responds 200 OK in the status line, which means that the data were processed correctly and machine state prediction is given in the response message body in JSON format. Presented response contains predicted class “prediction”:“idle” and probability “score” of each class as array in order: normal, idle, and fault.
Data from the application layer are encapsulated by lower layer headers. Figure 6 presents the encapsulation of HTTP/1.1 response from server to client. Data from HTTP/1.1 layer are put into the field payload of TCP and the header of TCP needs to be filled. The default TCP port is 80 for the HTTP/1.1 server.
The data segment is then a payload for IPv4, and the IPv4 header needs to be filled. The IPv4 header field upper-layer protocol is set to 6. The destination address and the source address must be known. The last encapsulation is into an Ethernet frame where the data packet is a payload of the Ethernet frame, and the header of the Ethernet needs to field. The hardware MAC (media access control address) destination address and source address need to be known. If only the IPv4 destination address is known before sending the Ethernet frame, the MAC destination address must be resolved by the ARP (Address Resolution Protocol) [32].
The other application layer protocol adopted for IoT applications is MQTT in version 3.1.1 [33] published in 2015 and version 5.0 [34] updated in 2019. MQTT provides client–server publish/subscribe messaging (Figure 7). The client connects to the server (broker) and subscribes to the topic. Messages published by other clients to the same topic will be pushed to all clients subscribed to this topic by the broker (server). MQTT use TCP at transport layer which establishes a connection as long as the client requires. The default destination TCP port of the broker is 1883.
The three step TCP handshake from the 52432 client port to the 1883 server port is presented in Figure 8 to establish the connection. After successful subscription, the MQTT microcontroller client send messages to other clients through the broker. The topic can be treated as a buffer name in the producer–consumer architecture. The designed fault diagnosis system consists of two MQTT clients subscribed to topics: sensor/IMU6DOF/raw and sensor/IMU6DOF/predict. The microcontroller MQTT client writes sensor data to the topic sensor/IMU6DOF/raw, and the MATLAB MQTT client retrieves those samples in JSON format. Received samples are transformed into features by digital signal processing methods, which are then evaluated by the classification algorithm. The prediction of classification algorithm is transformed into JSON (e.g., {“prediction”:“idle”,“score”:[0,0.9999999955,4.5E-9]}) and sent to sensor/IMU6DOF/predict topic.
MQTT can be used in either stream and batch data processing (Figure 1) of a fault diagnosis system. It is important to note that all messages are sent throughout the broker, which leads to central-oriented messaging with routing possibilities. Many clients who subscribed to the same topic received the same data. Therefore, clients can perform different micro services on the same data stream like feature extraction, data storage in a database, or visualization of data stream. This leads to transmission of one-to-many (one-point to multiple points) messages. In contrast to MQTT, WebSocket allows only point-to-point data streaming without the built-in message routing mechanism.
Another protocol used in IoT applications is CoAP (Constrained Application Protocol), defined in 2014 [35]. The protocol is dedicated to 8-bit microcontrollers with limited RAM (random access memory) and ROM (read-only memory). CoAP typical applications include smart energy and building automation. The direct quote from study [35] follows: “The goal of CoAP is not to blindly compress, but rather to realize a subset of REST (representational state transfer) common with HTTP but optimized for M2M (machine-to-machine) applications.” Many similarities to HTTP can be found. Similarly to HTTP, the CoAP uses request/response model with GET, PUT, POST, and DELETE methods. In contrast to HTTP, the CoAP uses the UDP protocol in the transport layer (see Figure 3). The use of UDP protocol in transport layer is unreliable which means that it does not handle missing packets, duplicate packets or packets retrieved in a different order caused by a different path of packets in the Internet. Therefore, in 2018 [36], CoAP was designed to use TCP, TLS, and WebSocket transports.
According to the Content-Type (see Section 3.1.1.5 in [26]) and Content-Encoding (see Section 3.1.2.2 in [26]) part in HTTP 1.1, many different data types are allowed in the message body part. Commonly adopted content types for the IoT application are JSON (JavaScript Object Notation) [37,38], XML (Extensible Markup Language) [39] or raw bytes. The full list of the allowed media types [40] is published by IANA (Internet Assigned Numbers Authority) in [41]. More than 1500 available formats are in the single application section of Media Types published on 21 November 2022. All allowed formats are grouped into sections, which are as follows: application, audio, font, example, image, message, model, multipart, text, video. Up-to-date XML media types “application/xml” or “text/xml” were published in 2014 [42] and the up-to-date JSON media type “application/json” was published in 2017 [43].

4. General Structure of Fault Diagnosis and Perspective Maintenance

The fault diagnosis system (Figure 9) detects faults based on changes in features over time. The fault diagnosis system works with the client–server architecture in the IT network [44,45,46,47,48,49,50]. The features are extracted from raw measurements and data or pre-processed data. Fault detection means recognition in the change of machine state caused by one or more faults. It can be treated as anomaly state detection, so any state different from normal behavior can be detected. Fault isolation is the recognition of which parts of the machine have faults. Finally, fault identification answers how wide the damages are. The information about the fault can be used by the maintenance team to repair the system. However, if repair is not possible immediately and the system cannot be stopped safely, then FTC (fault tolerant control) is a key component. The system can have hardware redundancy, which will be used if failure occurs, however, it is expensive to duplicate parts and not always technically possible to apply. FTC allows to continue the system usage of the system with performance constrains [51,52,53,54,55,56].
Fault diagnosis can be made only on data, which is called a data-driven approach. Another option is to prepare a mathematical model of the diagnosed system and analyze the residual data between the model and the system. Today, the investigated system model is called a digital twin [57,58,59,60,61,62,63,64]. Modeling is a very wide area in which static behavior [65,66,67], dynamic changes and continuous system [68,69], and discrete states [70] of systems can be modeled. Contrary to the analysis of model and real system residuum is the analysis of estimated parameters of system. The fault diagnosis system compares the estimated parameters in a predefined bounded region. The fault is announced if one of the parameters is outside the defined boundaries longer than the chosen amount of time.
The sensor presented in Figure 9 can be any kind. It can be an additional sensor only for fault diagnosis or a sensor already present in the system used by the control algorithms. The electromechanical machine or power system can be investigated by many different sensors and signals: current [71,72] and voltage [73,74], torque [75,76], angular velocity/position [77,78], linear 3-axis acceleration/speed/position [16,17], Doppler laser vibrometer [79], transmission coefficient and reflexion coefficient of omnidirectional antenna [80], strain/tension [81,82,83,84], power consumption [85,86,87,88], internal/external temperature at selected points [89,90] or surface temperature by thermal camera [91,92], depending on frequency range: displacement [93], vibrations [15,18,94,95,96], sound [97,98,99], sound from several microphones [100] or ultrasound [101,102], vibro-acoustic [103], chemical analysis of lubrication [104,105], chemical analysis by spectral imaging [106,107,108,109], camera imaging in human colour spectrum [110,111,112,113], and signals to virtual image [114,115,116,117,118].

5. Feature Extraction Methods

The decision made by the fault diagnosis system can be taken based on data in the time domain [119,120] or other domains, for example, the frequency domain [121,122,123,124], the time–frequency domain [125,126,127,128,129,130,131] or the time-scale domain [132,133,134,135].
Analysis with frequency domain is achieved by using one of the FFT (fast Fourier transform) algorithms. These algorithms decompose the analyzed signal into sinusoidal/cosinusoidal components at frequency from 0 Hz up to half of the sampling frequency. The use of the fast Fourier transform algorithm radix-2 [136,137] requires a number of samples equal N = 2 k , k ∈ N . However, the radix-4 algorithm requires N = 4 k [138,139]. The authors in their publications mostly inform the reader about the use of FFT without mentioning the algorithm used [140,141,142]. Frequency analysis can be applied to stationary signals in one time window. On the other hand, if knowledge about the sinusoidal/cosinusoidal components and time of their duration in the analyzed signal is required, then time–frequency analysis needs to be applied.
The use of FFT on short time-shifted windows leads to the STFT (short-time Fourier transform) method. STFT changes a one-dimensional time series signal into a two-dimensional time and frequency [143,144,145]. The STFT parameters are as follows: time window shape (e.g., rectangle, Hamming [146,147], Hanning [148,149]), step size or overlap of next time window that influence time resolution, time window length that influence frequency resolution f r e s = f s / N , where N —signal length in samples and f s —sampling frequency. Selection of time window length is a trade-off between good time localization or good frequency localization of the sinusoidal/cosinusoidal components. STFT requires an FFT calculation on each shifted time window. If the step size is equal to 1, it means that FFT will be calculated on N − 1 old and 1 new samples, which is not computationally effective. The problem of spectrum update at each new sample was solved using the SDFT method (sliding discrete Fourier transform).
SDFT was presented in 1997 [150] and well described in 2003 (rSDFT, 2003) [151] with an update in 2004 (rSDFT, 2004) [152]. Using study [152], the single spectrum component S k at frequency k is calculated by S k n = e j 2 π k / N ( S k n − 1 + x n − x n − N ) , where N —signal length in samples, n —time index, x —signal value. The discrete transfer function of study [152] is H r S D F T , 2004 z = e j 2 π k / N 1 − z − N / ( 1 − e j 2 π k / N z − 1 ) , which allows for stability analysis of the algorithm. In 2010, mSDFT (modulated SDFT) was introduced [153]. In study [153], SDFT 2003 with the 2004 update was named rSDFT (recursive SDFT). The mSDFT compared to rSDFT-2004 has reduced the accumulated error and potential instabilities. The stability of algorithms was analyzed and a new guaranteed stable SDFT algorithm (gSDFT) was published in 2015 [154]. The gSDFT algorithm has slightly smaller accumulated errors and better performance compared to mSDFT. Other improvements of mSDFT, called cascade integrator comb (CIC)—SDFT, which allow the embedded usage of the B-spline window function, were presented in 2017 [155]. In the same year, optimal sliding DFT (oSDFT) was presented [156], which reduces the number of multiplications by 73.44%, 64.58%, 82.30%, and 29.17% as compared to FFT, rSDFT, mSDFT, and gSDFT, respectively. The oSDFT accelerates the sliding transform process by 57.02%, 7.51%, 39.77%, and 9.38% compared to the FFT, rSDFT, mSDFT, and gSDFT algorithms, respectively. Misleading may be the name oSDFT (observer-based SDFT) introduced in 2018 [157], which uses the state observer method for digital signal processing purposes.

6. Demonstration of Fault Diagnosis with MQTT Communication

The demonstration shows classification of computer fan work into classes: idle, normal, or fault. The failure is caused by a paper clip added to the fan blade. The prepared demonstration setup (shown in Figure 10) consists of a NUCLEO board with STM32F746ZG microcontroller that handles the IMU-6 DOF (inertial measurement unit with 6 degrees of freedom) MPU6050 sensor. Data are collected synchronously with constant sampling time equal to 5 ms (sampling frequency 200 Hz). The vector of 128 samples from the 3-axis accelerometer and 3-axis gyroscope is then transformed into JSON (JavaScript Object Notation) as the object {“accelerometer”:{“x”:[],“y”:[],“z”:[]},”gyroscope”:{“x”:[],“y”:[],“z”:[]}}, where the array “[]” contain samples. Data are sent on the topic “sensor/IMU6DOF/raw” by the MQTT client implemented in the microcontroller to the MQTT broker hosted on a personal computer. Eclipse Mosquitto™ was used as an MQTT broker [158]. The data stream was processed in real time by MATLAB R2022b with the MQTT client [159].
The first stage is data collection at predefined class states for several seconds. Data collected in one micro batch in time domain are shown in Figure 11. Data were collected for three states of the fan: (1) idle class—fan power is switched off, (2) normal class—fan power without paper clip is switched on, and (3) fault class—fan power with paper clip is switched on. The time domain signal for the idle class has a constant signal with noise, the signal in the normal class has small oscillations, and the signal in the fault class has significant oscillations compared to previous classes. Data were transformed to the frequency domain by the FFT function in MATLAB, resulting in a complex number vector. The absolute value of the complex numbers is shown in Figure 12. Each class has different frequency features in the range of 0 Hz to 100 Hz. The most significant change in frequency occurs on the Z axis of the gyroscope.
The second stage of preparation was the extraction of features using mSDFT [153]. Each step of mSDFT returns 33 complex numbers at frequencies in the range from 0 Hz to 100 Hz. The algorithm was implemented in MATLAB and the absolute values of the complex number were stored in the files. Each row of the file contains 33 points that were used as a feature vector. The sliding discrete Fourier transform result can be gathered after each new sample. The multiple spectrums calculated at different times are shown in Figure 13 and Figure 14. The time–frequency analysis showed that for each class, the dominant frequency components are constant and do not vary with time.
The third stage of the demonstration requires the preparation of a classification. The features saved in the files for each class were read and loaded into MATLAB Classification Learner [160]. Several classifiers were trained, and the best was used for real-time verification. The dataset has 4200 observation sets randomly divided into 80% training and 20% test dataset. The training process was carried out on 1120 sets of fault class, 1120 sets of idle class, and 1120 sets of normal class. One set consists of 33 frequency features calculated by mSDFT from the gyroscope Z axis signal. The confusion matrix of the trained classifier is shown in Figure 15. The classifier can be used by applying new calculated features to predictFcn in MATLAB. The prediction result is sent to the MQTT sensor/IMU6DOF/predict topic in the data pipeline (see Figure 1).

7. Recognition of a Time–Frequency RGB Image of Vibration

Data from each axis of IMU 6-DOF were transformed into the time–frequency domain by applying STFT (short-time Fourier transform) with 32 samples of window length and 31 samples of overlap (see Figure 16, Figure 17, Figure 18, Figure 19 and Figure 20). As a result of STFT at 128 time domain samples of single axis, there was a two-dimensional signal of 65 frequencies at 97 time moments. The process was repeated for each axis of accelerometer and gyroscope. Finally, six images for 128 × 6 samples of IMU 6-DOF were obtained in the time–frequency domain for each class: idle in Figure 17, normal in Figure 18, and fault in Figure 19. These six images are combined into one RGB (red, green, and blue) image of size 130 × 97 × 3 in Figure 16; representative RGB images for each class are shown in Figure 20.
Data collected in the time domain for each class were converted into time–frequency RGB images. Together, the dataset has 2670 RGB images divided into classes: fault with 890 RGB images, idle with 890 RGB images, and normal with 890 RGB images. All images, which were divided into training and validation datasets. From the dataset, 80% is the training set (2136 RGB images) and 20% is the testing set (534 RGB images) which were randomly selected. CNN (convolutional neural network) training was performed in MATLAB Deep Learning Toolbox [161] with the support of NVIDIA GPU (graphics processing unit) with CUDA® (Compute Unified Device Architecture). Validation of the trained convolution neural network confirms good classification, shown as a confusion matrix in Figure 21.

8. Discussion

The cloud-based fault diagnosis system is suitable for human-centric and resilient European Industry 5.0 that requires a long lifetime of machines to reduce the amount of e-waste. A large number of industrial protocols used, of which only part are grouped in Communication Profile Families, underline the problem of data collection and ingestion into the cloud system. However, Internet protocols are developed separately without considering the needs of industrial processes. Therefore, the adoption of existing and new Internet protocols for industrial processes is a challenge. Batch and stream processing requires different handling. Batch processing can be applied with the request–response HTTP/1.1 protocol with a closing connection. However, stream processing requires an established connection for a long time, which can be achieved by WebSocket, HTTP/2.0, HTTP/3.0 or MQTT v3.1 and v5.0. It is worth considering the data exchange problem as a producer–consumer problem with a buffer in the middle. The producer can collect data from many different sensor technologies to investigate the normal and abnormal behavior of an industrial system. An interesting concept is the usage-only energy meter with the energy consumption profile for fault diagnosis. The decision about fault detection is made on the selected signal characteristics. A wide range of diagnostic applications, especially in rotating machines, use frequency analysis or time–frequency analysis. Therefore, short-time Fourier transform (STFT) and sliding discrete Fourier transform algorithms (rSDFT, mSDFT, gSDFT, and oSDFT) were underlined in the article. In batch processing, FFT (radix-2 or radix-4) or STFT can be applied. Real-time stream processing in the frequency domain can be performed using one of the SDFT algorithms.
The manuscript contains a demonstration of the fan fault diagnosis. Data were collected synchronously using a 3-axis accelerometer and a 3-axis gyroscope and microcontroller. The data collected were transformed into a JSON text structure. Those data in JSON are sent to an HTTP server for batch processing and as a result, fault diagnosis server returns a prediction in JSON format. Another investigation was conducted by sending data collection in JSON from the microcontroller client to the MQTT broker in real time as a stream (micro batch) on the sensor/IMU6DOF/raw topic for fault prediction. This approach allowed us to use two fault diagnosis predictors (two MQTT clients subscribed to sensor/IMU6DOF/raw topic) based on mSDFT and the second based on recognition of the time–frequency RGB image of vibration. For the first predictor, the time series data were transformed into the frequency domain by the mSDFT and evaluated by classification algorithm. The second predictor calculates six STFT and combines the result into an RGB image which is evaluated by the convolutional neural network. The result of both predictors is sent to sensor/IMU6DOF/predict topic in JSON format which contains the predicted state and probability for each class. This approach allowed us to classify the state of machine by two different algorithms and can be further extended by other algorithms to ensure voting of prediction from different algorithms to increase software algorithm redundancy.

9. Conclusions

The presented two communication methods allow for rapid prototyping of the fault diagnosis system with cloud connectivity. The HTTP server allows for one-to-one fault diagnosis, where data from one microcontroller are evaluated by one fault diagnosis server endpoint. On the other hand, MQTT allowed for one-to-many fault diagnoses when the same data were evaluated by two fault diagnosis clients with different feature extraction and different classification algorithms. HTTP and MQTT use TCP, however, due to connection closing in HTTP and connection remaining in MQTT, their work differently and both are suitable for fault diagnosis. In the authors’ opinion, it is more convenient to use MQTT for rapid prototyping and modularity of a variety of environments of the fault diagnosis system.
The authors have shown two approaches for fault diagnosis. One is based on frequency analysis which was enhanced in comparison to classical ones by usage of sliding Fourier transform (SDFT) instead of fast Fourier transform (FFT). This modification allows us to update the spectrum based on a new sample in time domain and spectrum in the previous step, which work as an IIR (Infinite Impulse Response) filter where filter input is one time domain sample and filter output is one frequency domain sample (complex number). The second proposed approach transforms data from IMU 6-DOF sensor into six spectrograms by STFT, which are used to create an RGB image. This time–frequency RGB image is evaluated by a convolutional neural network for fault diagnosis. The proposed approach proves that CNN can be successfully used in multispectrogram recognition which are organized into RGB images for fault diagnosis without the need for preliminary selection of a vibration axis.
Future research will be focused on increasing the TRL (technology readiness level). Research was carried out at TRL 1 to validate the proof of principles and concept. Further work will be conducted to increase the TRL to higher levels to validate in the laboratory environment the rotary electric machine (electric drive) with fault diagnosis system. At the next TRL, more faults of electric drive will be considered with different natures (electrical and mechanical), e.g., fault in inverter topology, fault in bearing, and shaft cracks.

Author Contributions

Internet of Things, D.Ł.; SDFT and time–frequency RGB image feature extraction, D.Ł.; demonstration of fault diagnosis system, D.Ł.; fan fault selection, D.Ł. and S.B.; electromechanical faults review, D.Ł., S.B. and K.S. All authors have read and agreed to the published version of the manuscript.

Funding

This work has been funded by the National Science Centre, Poland, grant number 2015/17/N/ST7/03796.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Kasper, W.C. Power from Trash. Environ. Sci. Policy Sustain. Dev. 1974, 16, 34–38. [Google Scholar] [CrossRef]
  2. Sullivan, J. Trash or Treasure: Global Trade and the Accumulation of E-Waste in Lagos, Nigeria. Afr. Today 2014, 61, 89–112. [Google Scholar] [CrossRef]
  3. Kan, Y.; Liu, H.; Yang, Y.; Wei, Y.; Yu, Y.; Qiu, R.; Ouyang, Y. Two Birds with One Stone: The Route from Waste Printed Circuit Board Electronic Trash to Multifunctional Biomimetic Slippery Liquid-Infused Coating. J. Ind. Eng. Chem. 2022, 114, 233–241. [Google Scholar] [CrossRef]
  4. Ji, X.; Yang, M.; Wan, A.; Yu, S.; Yao, Z. Bioleaching of Typical Electronic Waste—Printed Circuit Boards (WPCBs): A Short Review. Int. J. Environ. Res. Public Health 2022, 19, 7508. [Google Scholar] [CrossRef] [PubMed]
  5. Rene, E.R.; Sethurajan, M.; Kumar Ponnusamy, V.; Kumar, G.; Bao Dung, T.N.; Brindhadevi, K.; Pugazhendhi, A. Electronic Waste Generation, Recycling and Resource Recovery: Technological Perspectives and Trends. J. Hazard. Mater. 2021, 416, 125664. [Google Scholar] [CrossRef] [PubMed]
  6. Pan, X.; Wong, C.W.Y.; Li, C. Circular Economy Practices in the Waste Electrical and Electronic Equipment (WEEE) Industry: A Systematic Review and Future Research Agendas. J. Clean. Prod. 2022, 365, 132671. [Google Scholar] [CrossRef]
  7. Marinello, S.; Gamberini, R. Multi-Criteria Decision Making Approaches Applied to Waste Electrical and Electronic Equipment (WEEE): A Comprehensive Literature Review. Toxics 2021, 9, 13. [Google Scholar] [CrossRef]
  8. Breque, M.; De Nul, L.; Petridis, A. Directorate-General for Research and Innovation (European Commission). In Industry 5.0: Towards a Sustainable, Human Centric and Resilient European Industry; Publications Office of the European Union: Luxembourg, 2021; ISBN 978-92-76-25308-2. [Google Scholar]
  9. Smit, J.; Kreutzer, S.; Moeller, C.; Carlberg, C. Industry 4.0, European Parliament, Policy Department A: Economic and Scientific Policy; European Parliament: Strasbourg, France, 2016. [Google Scholar] [CrossRef]
  10. Teixeira, J.E.; Tavares-Lehmann, A.T.C.P. Industry 4.0 in the European Union: Policies and National Strategies. Technol. Forecast. Soc. Chang. 2022, 180, 121664. [Google Scholar] [CrossRef]
  11. Piechocki, M.; Kraft, M.; Pajchrowski, T.; Aszkowski, P.; Pieczynski, D. Efficient People Counting in Thermal Images: The Benchmark of Resource-Constrained Hardware. IEEE Access 2022, 10, 124835–124847. [Google Scholar] [CrossRef]
  12. Kraft, M.; Aszkowski, P.; Pieczyński, D.; Fularz, M. Low-Cost Thermal Camera-Based Counting Occupancy Meter Facilitating Energy Saving in Smart Buildings. Energies 2021, 14, 4542. [Google Scholar] [CrossRef]
  13. Ptak, B.; Pieczyński, D.; Piechocki, M.; Kraft, M. On-Board Crowd Counting and Density Estimation Using Low Altitude Unmanned Aerial Vehicles—Looking beyond Beating the Benchmark. Remote Sens. 2022, 14, 2288. [Google Scholar] [CrossRef]
  14. Yu, Y.; Samali, B.; Rashidi, M.; Mohammadi, M.; Nguyen, T.N.; Zhang, G. Vision-Based Concrete Crack Detection Using a Hybrid Framework Considering Noise Effect. J. Build. Eng. 2022, 61, 105246. [Google Scholar] [CrossRef]
  15. Chen, H.-Y.; Lee, C.-H. Vibration Signals Analysis by Explainable Artificial Intelligence (XAI) Approach: Application on Bearing Faults Diagnosis. IEEE Access 2020, 8, 134246–134256. [Google Scholar] [CrossRef]
  16. Kim, M.S.; Yun, J.P.; Park, P. Deep Learning-Based Explainable Fault Diagnosis Model With an Individually Grouped 1-D Convolution for Three-Axis Vibration Signals. IEEE Trans. Ind. Inform. 2022, 18, 8807–8817. [Google Scholar] [CrossRef]
  17. Zhang, X.; Zhao, Z.; Wang, Z.; Wang, X. Fault Detection and Identification Method for Quadcopter Based on Airframe Vibration Signals. Sensors 2021, 21, 581. [Google Scholar] [CrossRef]
  18. Wang, Y.; Yang, M.; Li, Y.; Xu, Z.; Wang, J.; Fang, X. A Multi-Input and Multi-Task Convolutional Neural Network for Fault Diagnosis Based on Bearing Vibration Signal. IEEE Sens. J. 2021, 21, 10946–10956. [Google Scholar] [CrossRef]
  19. IEC 61784-1:2019 Industrial Communication Networks—Profiles Part 1: Fieldbus Profiles. Available online: https://webstore.iec.ch/publication/59887#additionalinfo (accessed on 28 October 2022).
  20. RFC791; Internet Protocol. Internet Engineering Task Force: Fremont, CA, USA, 1981.
  21. RFC2460; Internet Protocol, Version 6 (IPv6) Specification. Internet Engineering Task Force: Fremont, CA, USA, 1998.
  22. RFC768; User Datagram Protocol. Internet Engineering Task Force: Fremont, CA, USA, 1980.
  23. RFC793; Transmission Control Protocol. Internet Engineering Task Force: Fremont, CA, USA, 1981.
  24. IEEE Std 802.3-2022 (Revision of IEEE Std 802.3-2018); IEEE Standard for Ethernet. IEEE: Piscataway, NJ, USA, 2022. [CrossRef]
  25. RFC8446; The Transport Layer Security (TLS) Protocol Version 1.3. Internet Engineering Task Force: Fremont, CA, USA, 2018.
  26. RFC2616; Hypertext Transfer Protocol—HTTP/1.1. Internet Engineering Task Force: Fremont, CA, USA, 1999.
  27. RFC2068; Hypertext Transfer Protocol—HTTP/1.1. Internet Engineering Task Force: Fremont, CA, USA, 1997.
  28. RFC6455; The WebSocket Protocol. Internet Engineering Task Force: Fremont, CA, USA, 2011.
  29. RFC7540; Hypertext Transfer Protocol Version 2 (HTTP/2). Internet Engineering Task Force: Fremont, CA, USA, 2015.
  30. RFC9114; HTTP/3. Internet Engineering Task Force: Fremont, CA, USA, 2022.
  31. RFC9000; QUIC: A UDP-Based Multiplexed and Secure Transport. Internet Engineering Task Force: Fremont, CA, USA, 2021.
  32. RFC826; An Ethernet Address Resolution Protocol: Or Converting Network Protocol Addresses to 48.Bit Ethernet Address for Transmission on Ethernet Hardware. Internet Engineering Task Force: Fremont, CA, USA, 1982.
  33. MQTT Version 3.1.1. Available online: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/errata01/os/mqtt-v3.1.1-errata01-os-complete.html (accessed on 31 October 2022).
  34. MQTT Version 5.0. Available online: http://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html (accessed on 31 October 2022).
  35. RFC7252; The Constrained Application Protocol (CoAP). Internet Engineering Task Force: Fremont, CA, USA, 2014.
  36. RFC8323; CoAP (Constrained Application Protocol) over TCP, TLS, and WebSockets. Internet Engineering Task Force: Fremont, CA, USA, 2018.
  37. JSON. Available online: https://www.json.org/json-en.html (accessed on 23 November 2022).
  38. ECMA-404; The JSON Data Interchange Syntax; Ecma International, 2nd edition, Geneva, Switzerland, December 2017. Available online: https://www.ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf (accessed on 30 February 2023).
  39. Extensible Markup Language (XML). Available online: https://www.w3.org/XML/ (accessed on 23 November 2022).
  40. RFC6838; Media Type Specifications and Registration Procedures. Internet Engineering Task Force: Fremont, CA, USA, 2013.
  41. Media Types. Available online: https://www.iana.org/assignments/media-types/media-types.xhtml (accessed on 23 November 2022).
  42. RFC7303; XML Media Types. Internet Engineering Task Force: Fremont, CA, USA, 2014.
  43. RFC8259; The JavaScript Object Notation (JSON) Data Interchange Format. Internet Engineering Task Force: Fremont, CA, USA, 2017.
  44. Mendes, M.J.G.C.; Neto, Í.M.M.S.; Calado, J.M.F. Fault Diagnosis System via Internet Applied to a Gantry Robot—A Proposal for Industry 4.0. In Proceedings of the 2018 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), Torres Vedras, Portugal, 25–27 April 2018; pp. 160–166. [Google Scholar]
  45. Jagatheesaperumal, S.K.; Rahouti, M.; Ahmad, K.; Al-Fuqaha, A.; Guizani, M. The Duo of Artificial Intelligence and Big Data for Industry 4.0: Applications, Techniques, Challenges, and Future Research Directions. IEEE Internet Things J. 2022, 9, 12861–12885. [Google Scholar] [CrossRef]
  46. Natesha, B.V.; Guddeti, R.M.R. Fog-Based Intelligent Machine Malfunction Monitoring System for Industry 4.0. IEEE Trans. Ind. Inform. 2021, 17, 7923–7932. [Google Scholar] [CrossRef]
  47. Rastogi, V.; Srivastava, S.; Mishra, M.; Thukral, R. Predictive Maintenance for SME in Industry 4.0. In Proceedings of the 2020 Global Smart Industry Conference (GloSIC), Chelyabinsk, Russia, 17–19 November 2020; pp. 382–390. [Google Scholar]
  48. Anil Kumar, T.C.; Bhusal Sharma, N.; Mishra, A.K.; Patil, P.; Sarveswara Reddy, S.; Bhardwaj, R. Fault Identification Model Using IIoT for Industrial Application. Meas. Sens. 2022, 24, 100526. [Google Scholar] [CrossRef]
  49. Radonjić, M.; Vujnović, S.; Krstić, A.; Zečević, Ž. IoT System for Detecting the Condition of Rotating Machines Based on Acoustic Signals. Appl. Sci. 2022, 12, 4385. [Google Scholar] [CrossRef]
  50. Behera, S.; Misra, R. Generative Adversarial Networks Based Remaining Useful Life Estimation for IIoT. Comput. Electr. Eng. 2021, 92, 107195. [Google Scholar] [CrossRef]
  51. Casado-Vara, R.; Vale, Z.; Prieto, J.; Corchado, J.M. Fault-Tolerant Temperature Control Algorithm for IoT Networks in Smart Buildings. Energies 2018, 11, 3430. [Google Scholar] [CrossRef] [Green Version]
  52. Akay, A.; Lefley, P. Open-Circuit Fault-Tolerant Control of Multi-Phase PM Machines by Compensating the d-q Axes Currents. Energies 2021, 14, 192. [Google Scholar] [CrossRef]
  53. Patel, H.R.; Shah, V.A. Stable Fault Tolerant Controller Design for Takagi–Sugeno Fuzzy Model-Based Control Systems via Linear Matrix Inequalities: Three Conical Tank Case Study. Energies 2019, 12, 2221. [Google Scholar] [CrossRef] [Green Version]
  54. Nguyen, N.P.; Hong, S.K. Fault Diagnosis and Fault-Tolerant Control Scheme for Quadcopter UAVs with a Total Loss of Actuator. Energies 2019, 12, 1139. [Google Scholar] [CrossRef] [Green Version]
  55. Siembab, K.; Zawirski, K. Modified Space Vector Modulation for Fault Tolerant Control of PMSM Drive. In Proceedings of the 2016 IEEE International Power Electronics and Motion Control Conference (PEMC), Varna, Bulgaria, 25–28 September 2016; pp. 1064–1071. [Google Scholar]
  56. Łuczak, D.; Siembab, K. Comparison of Fault Tolerant Control Algorithm Using Space Vector Modulation of PMSM Drive. In Proceedings of the 16th International Conference on Mechatronics—Mechatronika 2014, Brno, Czech Republic, 3–5 December 2014; pp. 24–31. [Google Scholar]
  57. Moghadam, F.K.; Nejad, A.R. Online Condition Monitoring of Floating Wind Turbines Drivetrain by Means of Digital Twin. Mech. Syst. Signal Process. 2022, 162, 108087. [Google Scholar] [CrossRef]
  58. Leng, J.; Wang, D.; Shen, W.; Li, X.; Liu, Q.; Chen, X. Digital Twins-Based Smart Manufacturing System Design in Industry 4.0: A Review. J. Manuf. Syst. 2021, 60, 119–137. [Google Scholar] [CrossRef]
  59. Peng, F.; Zheng, L.; Peng, Y.; Fang, C.; Meng, X. Digital Twin for Rolling Bearings: A Review of Current Simulation and PHM Techniques. Measurement 2022, 201, 111728. [Google Scholar] [CrossRef]
  60. Wang, T.; Cheng, J.; Yang, Y.; Esposito, C.; Snoussi, H.; Tao, F. Adaptive Optimization Method in Digital Twin Conveyor Systems via Range-Inspection Control. IEEE Trans. Autom. Sci. Eng. 2022, 19, 1296–1304. [Google Scholar] [CrossRef]
  61. Nguyen, T.; Duong, Q.H.; Nguyen, T.V.; Zhu, Y.; Zhou, L. Knowledge Mapping of Digital Twin and Physical Internet in Supply Chain Management: A Systematic Literature Review. Int. J. Prod. Econ. 2022, 244, 108381. [Google Scholar] [CrossRef]
  62. Teng, S.Y.; Touš, M.; Leong, W.D.; How, B.S.; Lam, H.L.; Máša, V. Recent Advances on Industrial Data-Driven Energy Savings: Digital Twins and Infrastructures. Renew. Sustain. Energy Rev. 2021, 135, 110208. [Google Scholar] [CrossRef]
  63. Unal, P.; Albayrak, Ö.; Jomâa, M.; Berre, A.J. Data-Driven Artificial Intelligence and Predictive Analytics for the Maintenance of Industrial Machinery with Hybrid and Cognitive Digital Twins. In Technologies and Applications for Big Data Value; Curry, E., Auer, S., Berre, A.J., Metzger, A., Perez, M.S., Zillner, S., Eds.; Springer International Publishing: Cham, Switzerland, 2022; pp. 299–319. ISBN 978-3-030-78307-5. [Google Scholar]
  64. Cohen, Y.; Singer, G. A Smart Process Controller Framework for Industry 4.0 Settings. J. Intell. Manuf. 2021, 32, 1975–1995. [Google Scholar] [CrossRef]
  65. Zhu, G.; Li, Z.; Wu, N. Model-Based Fault Identification of Discrete Event Systems Using Partially Observed Petri Nets. Automatica 2018, 96, 201–212. [Google Scholar] [CrossRef]
  66. Xu, B.; Yin, X.; Yin, X.; Wang, Y.; Pang, S. Fault Diagnosis of Power Systems Based on Temporal Constrained Fuzzy Petri Nets. IEEE Access 2019, 7, 101895–101904. [Google Scholar] [CrossRef]
  67. Wang, H.; Wu, Y.; Min, G.; Miao, W. A Graph Neural Network-Based Digital Twin for Network Slicing Management. IEEE Trans. Ind. Inform. 2022, 18, 1367–1376. [Google Scholar] [CrossRef]
  68. Łuczak, D. Nonlinear Identification with Constraints in Frequency Domain of Electric Direct Drive with Multi-Resonant Mechanical Part. Energies 2021, 14, 7190. [Google Scholar] [CrossRef]
  69. Luczak, D. Mathematical Model of Multi-Mass Electric Drive System with Flexible Connection. In Proceedings of the 2014 19th International Conference on Methods and Models in Automation and Robotics (MMAR), Miedzyzdroje, Poland, 2–5 September 2014; pp. 590–595. [Google Scholar]
  70. Luczak, D.; Nowopolski, K. Identification of Multi-Mass Mechanical Systems in Electrical Drives. In Proceedings of the 2014 16th International Conference on Mechatronics—Mechatronika (ME), Brno, Czech Republic, 3–5 December 2014; pp. 275–282. [Google Scholar]
  71. Huang, W.; Du, J.; Hua, W.; Lu, W.; Bi, K.; Zhu, Y.; Fan, Q. Current-Based Open-Circuit Fault Diagnosis for PMSM Drives With Model Predictive Control. IEEE Trans. Power Electron. 2021, 36, 10695–10704. [Google Scholar] [CrossRef]
  72. Wu, C.; Guo, C.; Xie, Z.; Ni, F.; Liu, H. A Signal-Based Fault Detection and Tolerance Control Method of Current Sensor for PMSM Drive. IEEE Trans. Ind. Electron. 2018, 65, 9646–9657. [Google Scholar] [CrossRef]
  73. Jiang, L.; Deng, Z.; Tang, X.; Hu, L.; Lin, X.; Hu, X. Data-Driven Fault Diagnosis and Thermal Runaway Warning for Battery Packs Using Real-World Vehicle Data. Energy 2021, 234, 121266. [Google Scholar] [CrossRef]
  74. Chang, C.; Zhou, X.; Jiang, J.; Gao, Y.; Jiang, Y.; Wu, T. Electric Vehicle Battery Pack Micro-Short Circuit Fault Diagnosis Based on Charging Voltage Ranking Evolution. J. Power Sources 2022, 542, 231733. [Google Scholar] [CrossRef]
  75. Gao, S.; Xu, L.; Zhang, Y.; Pei, Z. Rolling Bearing Fault Diagnosis Based on SSA Optimized Self-Adaptive DBN. ISA Trans. 2022, 128, 485–502. [Google Scholar] [CrossRef]
  76. Wang, C.-S.; Kao, I.-H.; Perng, J.-W. Fault Diagnosis and Fault Frequency Determination of Permanent Magnet Synchronous Motor Based on Deep Learning. Sensors 2021, 21, 3608. [Google Scholar] [CrossRef]
  77. Feng, Z.; Gao, A.; Li, K.; Ma, H. Planetary Gearbox Fault Diagnosis via Rotary Encoder Signal Analysis. Mech. Syst. Signal Process. 2021, 149, 107325. [Google Scholar] [CrossRef]
  78. Ma, J.; Li, C.; Zhang, G. Rolling Bearing Fault Diagnosis Based on Deep Learning and Autoencoder Information Fusion. Symmetry 2022, 14, 13. [Google Scholar] [CrossRef]
  79. Abbas, S.H.; Jang, J.-K.; Kim, D.-H.; Lee, J.-R. Underwater Vibration Analysis Method for Rotating Propeller Blades Using Laser Doppler Vibrometer. Opt. Lasers Eng. 2020, 132, 106133. [Google Scholar] [CrossRef]
  80. Dutta, S.; Basu, B.; Talukdar, F.A. Classification of Motor Faults Based on Transmission Coefficient and Reflection Coefficient of Omni-Directional Antenna Using DCNN. Expert Syst. Appl. 2022, 198, 116832. [Google Scholar] [CrossRef]
  81. Zhang, X.; Niu, H.; Hou, C.; Di, F. An Edge-Filter FBG Interrogation Approach Based on Tunable Fabry-Perot Filter for Strain Measurement of Planetary Gearbox. Opt. Fiber Technol. 2020, 60, 102379. [Google Scholar] [CrossRef]
  82. Zhang, P.; Lu, D. A Survey of Condition Monitoring and Fault Diagnosis toward Integrated O&M for Wind Turbines. Energies 2019, 12, 2801. [Google Scholar] [CrossRef] [Green Version]
  83. Wu, J.; Yang, Y.; Wang, P.; Wang, J.; Cheng, J. A Novel Method for Gear Crack Fault Diagnosis Using Improved Analytical-FE and Strain Measurement. Measurement 2020, 163, 107936. [Google Scholar] [CrossRef]
  84. Fedorko, G.; Molnár, V.; Vasiľ, M.; Salai, R. Proposal of Digital Twin for Testing and Measuring of Transport Belts for Pipe Conveyors within the Concept Industry 4.0. Measurement 2021, 174, 108978. [Google Scholar] [CrossRef]
  85. Pu, H.; He, L.; Zhao, C.; Yau, D.K.Y.; Cheng, P.; Chen, J. Fingerprinting Movements of Industrial Robots for Replay Attack Detection. IEEE Trans. Mob. Comput. 2022, 21, 3629–3643. [Google Scholar] [CrossRef]
  86. Rafati, A.; Shaker, H.R.; Ghahghahzadeh, S. Fault Detection and Efficiency Assessment for HVAC Systems Using Non-Intrusive Load Monitoring: A Review. Energies 2022, 15, 341. [Google Scholar] [CrossRef]
  87. Sabry, A.H.; Nordin, F.H.; Sabry, A.H.; Abidin Ab Kadir, M.Z. Fault Detection and Diagnosis of Industrial Robot Based on Power Consumption Modeling. IEEE Trans. Ind. Electron. 2020, 67, 7929–7940. [Google Scholar] [CrossRef]
  88. Sánchez-Sutil, F.; Cano-Ortega, A.; Hernández, J.C. Design and Implementation of a Smart Energy Meter Using a LoRa Network in Real Time. Electronics 2021, 10, 3152. [Google Scholar] [CrossRef]
  89. Wang, Z.; Tian, B.; Qiao, W.; Qu, L. Real-Time Aging Monitoring for IGBT Modules Using Case Temperature. IEEE Trans. Ind. Electron. 2016, 63, 1168–1178. [Google Scholar] [CrossRef]
  90. Dhiman, H.S.; Deb, D.; Muyeen, S.M.; Kamwa, I. Wind Turbine Gearbox Anomaly Detection Based on Adaptive Threshold and Twin Support Vector Machines. IEEE Trans. Energy Convers. 2021, 36, 3462–3469. [Google Scholar] [CrossRef]
  91. Glowacz, A. Fault Diagnosis of Electric Impact Drills Using Thermal Imaging. Measurement 2021, 171, 108815. [Google Scholar] [CrossRef]
  92. Al-Musawi, A.K.; Anayi, F.; Packianather, M. Three-Phase Induction Motor Fault Detection Based on Thermal Image Segmentation. Infrared Phys. Technol. 2020, 104, 103140. [Google Scholar] [CrossRef]
  93. Li, Z.; Zhang, Y.; Abu-Siada, A.; Chen, X.; Li, Z.; Xu, Y.; Zhang, L.; Tong, Y. Fault Diagnosis of Transformer Windings Based on Decision Tree and Fully Connected Neural Network. Energies 2021, 14, 1531. [Google Scholar] [CrossRef]
  94. Rauber, T.W.; da Silva Loca, A.L.; de Assis Boldt, F.; Rodrigues, A.L.; Varejão, F.M. An Experimental Methodology to Evaluate Machine Learning Methods for Fault Diagnosis Based on Vibration Signals. Expert Syst. Appl. 2021, 167, 114022. [Google Scholar] [CrossRef]
  95. Meyer, A. Vibration Fault Diagnosis in Wind Turbines Based on Automated Feature Learning. Energies 2022, 15, 1514. [Google Scholar] [CrossRef]
  96. Lee, J.-H.; Pack, J.-H.; Lee, I.-S. Fault Diagnosis of Induction Motor Using Convolutional Neural Network. Appl. Sci. 2019, 9, 2950. [Google Scholar] [CrossRef] [Green Version]
  97. Cao, Y.; Sun, Y.; Xie, G.; Li, P. A Sound-Based Fault Diagnosis Method for Railway Point Machines Based on Two-Stage Feature Selection Strategy and Ensemble Classifier. IEEE Trans. Intell. Transp. Syst. 2022, 23, 12074–12083. [Google Scholar] [CrossRef]
  98. Shiri, H.; Wodecki, J.; Ziętek, B.; Zimroz, R. Inspection Robotic UGV Platform and the Procedure for an Acoustic Signal-Based Fault Detection in Belt Conveyor Idler. Energies 2021, 14, 7646. [Google Scholar] [CrossRef]
  99. Karabacak, Y.E.; Gürsel Özmen, N.; Gümüşel, L. Intelligent Worm Gearbox Fault Diagnosis under Various Working Conditions Using Vibration, Sound and Thermal Features. Appl. Acoust. 2022, 186, 108463. [Google Scholar] [CrossRef]
  100. Yao, Y.; Wang, H.; Li, S.; Liu, Z.; Gui, G.; Dan, Y.; Hu, J. End-To-End Convolutional Neural Network Model for Gear Fault Diagnosis Based on Sound Signals. Appl. Sci. 2018, 8, 1584. [Google Scholar] [CrossRef] [Green Version]
  101. Zhang, Z.; Li, J.; Song, Y.; Sun, Y.; Zhang, X.; Hu, Y.; Guo, R.; Han, X. A Novel Ultrasound-Vibration Composite Sensor for Defects Detection of Electrical Equipment. IEEE Trans. Power Deliv. 2022, 37, 4477–4480. [Google Scholar] [CrossRef]
  102. Wang, W.; Xue, Y.; He, C.; Zhao, Y. Review of the Typical Damage and Damage-Detection Methods of Large Wind Turbine Blades. Energies 2022, 15, 5672. [Google Scholar] [CrossRef]
  103. Wang, X.; Mao, D.; Li, X. Bearing Fault Diagnosis Based on Vibro-Acoustic Data Fusion and 1D-CNN Network. Measurement 2021, 173, 108518. [Google Scholar] [CrossRef]
  104. Maruyama, T.; Maeda, M.; Nakano, K. Lubrication Condition Monitoring of Practical Ball Bearings by Electrical Impedance Method. Tribol. Online 2019, 14, 327–338. [Google Scholar] [CrossRef] [Green Version]
  105. Wakiru, J.M.; Pintelon, L.; Muchiri, P.N.; Chemweno, P.K. A Review on Lubricant Condition Monitoring Information Analysis for Maintenance Decision Support. Mech. Syst. Signal Process. 2019, 118, 108–132. [Google Scholar] [CrossRef]
  106. Rizk, P.; Younes, R.; Ilinca, A.; Khoder, J. Wind Turbine Ice Detection Using Hyperspectral Imaging. Remote Sens. Appl. Soc. Environ. 2022, 26, 100711. [Google Scholar] [CrossRef]
  107. Rizk, P.; Younes, R.; Ilinca, A.; Khoder, J. Wind Turbine Blade Defect Detection Using Hyperspectral Imaging. Remote Sens. Appl. Soc. Environ. 2021, 22, 100522. [Google Scholar] [CrossRef]
  108. Meribout, M. Gas Leak-Detection and Measurement Systems: Prospects and Future Trends. IEEE Trans. Instrum. Meas. 2021, 70. [Google Scholar] [CrossRef]
  109. Li, Y.; Yu, Q.; Xie, M.; Zhang, Z.; Ma, Z.; Cao, K. Identifying Oil Spill Types Based on Remotely Sensed Reflectance Spectra and Multiple Machine Learning Algorithms. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 9071–9078. [Google Scholar] [CrossRef]
  110. Zhou, Q.; Chen, R.; Huang, B.; Liu, C.; Yu, J.; Yu, X. An Automatic Surface Defect Inspection System for Automobiles Using Machine Vision Methods. Sensors 2019, 19, 644. [Google Scholar] [CrossRef] [Green Version]
  111. Yang, L.; Fan, J.; Liu, Y.; Li, E.; Peng, J.; Liang, Z. A Review on State-of-the-Art Power Line Inspection Techniques. IEEE Trans. Instrum. Meas. 2020, 69, 9350–9365. [Google Scholar] [CrossRef]
  112. Davari, N.; Akbarizadeh, G.; Mashhour, E. Intelligent Diagnosis of Incipient Fault in Power Distribution Lines Based on Corona Detection in UV-Visible Videos. IEEE Trans. Power Deliv. 2021, 36, 3640–3648. [Google Scholar] [CrossRef]
  113. Kim, S.; Kim, D.; Jeong, S.; Ham, J.-W.; Lee, J.-K.; Oh, K.-Y. Fault Diagnosis of Power Transmission Lines Using a UAV-Mounted Smart Inspection System. IEEE Access 2020, 8, 149999–150009. [Google Scholar] [CrossRef]
  114. Ullah, Z.; Lodhi, B.A.; Hur, J. Detection and Identification of Demagnetization and Bearing Faults in PMSM Using Transfer Learning-Based VGG. Energies 2020, 13, 3834. [Google Scholar] [CrossRef]
  115. Long, H.; Xu, S.; Gu, W. An Abnormal Wind Turbine Data Cleaning Algorithm Based on Color Space Conversion and Image Feature Detection. Appl. Energy 2022, 311, 118594. [Google Scholar] [CrossRef]
  116. Kreutz, M.; Alla, A.A.; Eisenstadt, A.; Freitag, M.; Thoben, K.-D. Ice Detection on Rotor Blades of Wind Turbines Using RGB Images and Convolutional Neural Networks. Procedia CIRP 2020, 93, 1292–1297. [Google Scholar] [CrossRef]
  117. Xie, T.; Huang, X.; Choi, S.-K. Intelligent Mechanical Fault Diagnosis Using Multisensor Fusion and Convolution Neural Network. IEEE Trans. Ind. Inform. 2022, 18, 3213–3223. [Google Scholar] [CrossRef]
  118. Łuczak, D.; Brock, S.; Siembab, K. Fault Detection and Localisation of a Three-Phase Inverter with Permanent Magnet Synchronous Motor Load Using a Convolutional Neural Network. Actuators 2023, 12, 125. [Google Scholar] [CrossRef]
  119. Liang, M.; Cao, P.; Tang, J. Rolling Bearing Fault Diagnosis Based on Feature Fusion with Parallel Convolutional Neural Network. Int. J. Adv. Manuf. Technol. 2021, 112, 819–831. [Google Scholar] [CrossRef]
  120. Toma, R.N.; Kim, J.-M. Induction Motor Bearing Fault Diagnosis Using Statistical Time Domain Features and Hypertuning of Classifiers. In Advances in Computer Science and Ubiquitous Computing; Park, J.J., Fong, S.J., Pan, Y., Sung, Y., Eds.; Springer: Singapore, 2021; pp. 259–265. [Google Scholar]
  121. Bi, X.; Cao, S.; Zhang, D. Diesel Engine Valve Clearance Fault Diagnosis Based on Improved Variational Mode Decomposition and Bispectrum. Energies 2019, 12, 661. [Google Scholar] [CrossRef] [Green Version]
  122. Wang, J.; Li, S.; Xin, Y.; An, Z. Gear Fault Intelligent Diagnosis Based on Frequency-Domain Feature Extraction. J. Vib. Eng. Technol. 2019, 7, 159–166. [Google Scholar] [CrossRef]
  123. Li, H.; Feng, G.; Zhen, D.; Gu, F.; Ball, A.D. A Normalized Frequency-Domain Energy Operator for Broken Rotor Bar Fault Diagnosis. IEEE Trans. Instrum. Meas. 2021, 70, 3009011. [Google Scholar] [CrossRef]
  124. Luczak, D. Frequency Analysis of Mechanical Resonance in Direct Drive. In Proceedings of the 2012 12th IEEE International Workshop on Advanced Motion Control (AMC), Sarajevo, Bosnia and Herzegovina, 25–27 March 2012; pp. 1–5. [Google Scholar]
  125. Ramteke, S.M.; Chelladurai, H.; Amarnath, M. Diagnosis and Classification of Diesel Engine Components Faults Using Time–Frequency and Machine Learning Approach. J. Vib. Eng. Technol. 2022, 10, 175–192. [Google Scholar] [CrossRef]
  126. Lim, H.; Kwon, G.-Y.; Shin, Y.-J. Fault Detection and Localization of Shielded Cable via Optimal Detection of Time–Frequency-Domain Reflectometry. IEEE Trans. Instrum. Meas. 2021, 70, 3092514. [Google Scholar] [CrossRef]
  127. Iglesias-Martínez, M.E.; Antonino-Daviu, J.A.; Fernández de Córdoba, P.; Conejero, J.A. Rotor Fault Detection in Induction Motors Based on Time-Frequency Analysis Using the Bispectrum and the Autocovariance of Stray Flux Signals. Energies 2019, 12, 597. [Google Scholar] [CrossRef] [Green Version]
  128. Luczak, D. Spectral Analysis of Digital Filter Tuned for Mechanical Resonant Frequency Reduction in Multi-Mass Mechanical Systems in Electrical Direct Drive. Int. J. Simul. Syst. Sci. Technol. 2020, 17, 11.1–11.8. [Google Scholar] [CrossRef]
  129. Strack, J.L.; Carugati, I.; Orallo, C.M.; Maestri, S.O.; Donato, P.G.; Funes, M.A. Three-Phase Voltage Events Classification Algorithm Based on an Adaptive Threshold. Electr. Power Syst. Res. 2019, 172, 167–176. [Google Scholar] [CrossRef]
  130. Yoon, Y.; Brahma, A. Air–Fuel Ratio Imbalance Diagnostic of Spark-Ignited Engines With Modulated Sliding Discrete Fourier Transform. J. Dyn. Syst. Meas. Control 2020, 142, 081003. [Google Scholar] [CrossRef]
  131. Peña-Alzola, R.; Sztykiel, M.; Jones, C.E.; Norman, P.J.; Moore, G.; Pou, J.; Burt, G.M. First-Fault Detection in DC Distribution With IT Grounding Based on Sliding Discrete Fourier-Transform. IEEE Trans. Power Electron. 2021, 36, 3649–3654. [Google Scholar] [CrossRef]
  132. Li, X.; Bi, F.; Zhang, L.; Yang, X.; Zhang, G. An Engine Fault Detection Method Based on the Deep Echo State Network and Improved Multi-Verse Optimizer. Energies 2022, 15, 1205. [Google Scholar] [CrossRef]
  133. Gu, J.; Peng, Y.; Lu, H.; Chang, X.; Chen, G. A Novel Fault Diagnosis Method of Rotating Machinery via VMD, CWT and Improved CNN. Measurement 2022, 200, 111635. [Google Scholar] [CrossRef]
  134. Łuczak, D. Mechanical Resonance Frequensies Identyfication of Direct Drive Using Wavelet Analysis. In Proceedings of the 2012 17th International Conference on Methods & Models in Automation & Robotics (MMAR), Miedzyzdroje, Poland, 27–30 August 2012; pp. 29–32. [Google Scholar]
  135. Łuczak, D. Mechanical Vibrations Analysis in Direct Drive Using CWT with Complex Morlet Wavelet. Power Electron. Drives 2023, 8, 65–73. [Google Scholar] [CrossRef]
  136. Singleton, R. A Method for Computing the Fast Fourier Transform with Auxiliary Memory and Limited High-Speed Storage. IEEE Trans. Audio Electroacoust. 1967, 15, 91–98. [Google Scholar] [CrossRef]
  137. Cooley, J.W.; Tukey, J.W. An Algorithm for the Machine Calculation of Complex Fourier Series. Math. Comput. 1965, 19, 297–301. [Google Scholar] [CrossRef]
  138. Corinthios, M.J.; Smith, K.C.; Yen, J.L. A Parallel Radix-4 Fast Fourier Transform Computer. IEEE Trans. Comput. 1975, C-24, 80–92. [Google Scholar] [CrossRef]
  139. Corinthios, M.J. A Fast Fourier Transform for High-Speed Signal Processing. IEEE Trans. Comput. 1971, C-20, 843–846. [Google Scholar] [CrossRef]
  140. Soares, M.N.; Mollet, Y.; Kinnaert, M.; Gyselinck, J.; Helsen, J. Multiphysical Time- and Frequency-Domain Fault Detection and Isolation Technique for Power-Electronic Converters in DFIG Wind Turbines. IEEE Trans. Power Electron. 2021, 36, 3793–3802. [Google Scholar] [CrossRef]
  141. Han, T.; Ding, L.; Qi, D.; Li, C.; Fu, Z.; Chen, W. Compound Faults Diagnosis Method for Wind Turbine Mainshaft Bearing with Teager and Second-Order Stochastic Resonance. Measurement 2022, 202, 111931. [Google Scholar] [CrossRef]
  142. Li, H.; Fan, B.; Jia, R.; Zhai, F.; Bai, L.; Luo, X. Research on Multi-Domain Fault Diagnosis of Gearbox of Wind Turbine Based on Adaptive Variational Mode Decomposition and Extreme Learning Machine Algorithms. Energies 2020, 13, 1375. [Google Scholar] [CrossRef] [Green Version]
  143. Liu, D.; Cheng, W.; Wen, W. Rolling Bearing Fault Diagnosis via STFT and Improved Instantaneous Frequency Estimation Method. Procedia Manuf. 2020, 49, 166–172. [Google Scholar] [CrossRef]
  144. Wang, Z.; Yang, J.; Li, H.; Zhen, D.; Xu, Y.; Gu, F. Fault Identification of Broken Rotor Bars in Induction Motors Using an Improved Cyclic Modulation Spectral Analysis. Energies 2019, 12, 3279. [Google Scholar] [CrossRef] [Green Version]
  145. Grcić, I.; Pandžić, H.; Novosel, D. Fault Detection in DC Microgrids Using Short-Time Fourier Transform. Energies 2021, 14, 277. [Google Scholar] [CrossRef]
  146. Fu, X.; Yang, K.; Liu, M.; Xing, T.; Wu, C. LightFD: Real-Time Fault Diagnosis with Edge Intelligence for Power Transformers. Sensors 2022, 22, 5296. [Google Scholar] [CrossRef]
  147. Liu, J.; Hao, R.; Zhang, T.; Wang, X. Vibration Fault Diagnosis Based on Stochastic Configuration Neural Networks. Neurocomputing 2021, 434, 98–125. [Google Scholar] [CrossRef]
  148. Miao, Y.; Zhang, B.; Li, C.; Lin, J.; Zhang, D. Feature Mode Decomposition: New Decomposition Theory for Rotating Machinery Fault Diagnosis. IEEE Trans. Ind. Electron. 2023, 70, 1949–1960. [Google Scholar] [CrossRef]
  149. Shi, H.; Li, Y.; Bai, X.; Zhang, K. Sound-Aided Fault Feature Extraction Method for Rolling Bearings Based on Stochastic Resonance and Time-Domain Index Fusion. Appl. Acoust. 2022, 189, 108611. [Google Scholar] [CrossRef]
  150. Douglas, S.C.; Soh, J.K. A Numerically-Stable Sliding-Window Estimator and Its Application to Adaptive Filters. In Proceedings of the Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136), Pacific Grove, CA, USA,, 2–5 November 1997; Volume 1, pp. 111–115. [Google Scholar]
  151. Jacobsen, E.; Lyons, R. The Sliding DFT. IEEE Signal Process. Mag. 2003, 20, 74–80. [Google Scholar] [CrossRef] [Green Version]
  152. Jacobsen, E.; Lyons, R. An Update to the Sliding DFT. IEEE Signal Process. Mag. 2004, 21, 110–111. [Google Scholar] [CrossRef]
  153. Duda, K. Accurate, Guaranteed Stable, Sliding Discrete Fourier Transform [DSP Tips & Tricks]. IEEE Signal Process. Mag. 2010, 27, 124–127. [Google Scholar] [CrossRef]
  154. Park, C.-S. Fast, Accurate, and Guaranteed Stable Sliding Discrete Fourier Transform [Sp Tips&Tricks]. IEEE Signal Process. Mag. 2015, 32, 145–156. [Google Scholar] [CrossRef]
  155. Gudovskiy, D.A.; Chu, L. An Accurate and Stable Sliding DFT Computed by a Modified CIC Filter [Tips & Tricks]. IEEE Signal Process. Mag. 2017, 34, 89–93. [Google Scholar] [CrossRef]
  156. Park, C.-S. Guaranteed-Stable Sliding DFT Algorithm With Minimal Computational Requirements. IEEE Trans. Signal Process. 2017, 65, 5281–5288. [Google Scholar] [CrossRef]
  157. Kollar, Z.; Plesznik, F.; Trumpf, S. Observer-Based Recursive Sliding Discrete Fourier Transform [Tips & Tricks]. IEEE Signal Process. Mag. 2018, 35, 100–106. [Google Scholar] [CrossRef]
  158. Eclipse Mosquitto. Available online: https://mosquitto.org/ (accessed on 22 November 2022).
  159. Create MQTT Client Connected to Broker—MATLAB. Available online: https://www.mathworks.com/help/icomm/ug/icomm.mqtt.client.html (accessed on 22 November 2022).
  160. Train Models to Classify Data Using Supervised Machine Learning—MATLAB. Available online: https://www.mathworks.com/help/stats/classificationlearner-app.html (accessed on 23 November 2022).
  161. Deep Learning Toolbox. Available online: https://www.mathworks.com/products/deep-learning.html (accessed on 7 February 2023).
Sensors 23 03755 g001 550
Figure 1. General structure of data pipelines in fault diagnosis.
Figure 1. General structure of data pipelines in fault diagnosis.
Sensors 23 03755 g001
Sensors 23 03755 g002 550
Figure 2. General network structure with network bridge.
Figure 2. General network structure with network bridge.
Sensors 23 03755 g002
Sensors 23 03755 g003 550
Figure 3. Model OSI-ISO Open System Interconnection—Open Source Initiative.
Figure 3. Model OSI-ISO Open System Interconnection—Open Source Initiative.
Sensors 23 03755 g003
Sensors 23 03755 g004 550
Figure 4. Client–server architecture with HTTP/1.1.
Figure 4. Client–server architecture with HTTP/1.1.
Sensors 23 03755 g004
Sensors 23 03755 g005 550
Figure 5. HTTP/1.1 request–response as batch processing of machine state prediction.
Figure 5. HTTP/1.1 request–response as batch processing of machine state prediction.
Sensors 23 03755 g005
Sensors 23 03755 g006 550
Figure 6. Response from the fault prediction server with HTTP/1.1 encapsulated in the lower layers of the OSI-ISO model.
Figure 6. Response from the fault prediction server with HTTP/1.1 encapsulated in the lower layers of the OSI-ISO model.
Sensors 23 03755 g006
Sensors 23 03755 g007 550
Figure 7. MQTT 3.1 client–server architecture.
Figure 7. MQTT 3.1 client–server architecture.
Sensors 23 03755 g007
Sensors 23 03755 g008 550
Figure 8. Captured MQTT v3.1 packets captured with TCP handshake for microcontroller MQTT client write sensor data to the topic sensor/IMU6DOF/raw as stream processing (micro batch) for machine state prediction.
Figure 8. Captured MQTT v3.1 packets captured with TCP handshake for microcontroller MQTT client write sensor data to the topic sensor/IMU6DOF/raw as stream processing (micro batch) for machine state prediction.
Sensors 23 03755 g008
Sensors 23 03755 g009 550
Figure 9. General structure of the fault diagnosis system.
Figure 9. General structure of the fault diagnosis system.
Sensors 23 03755 g009
Sensors 23 03755 g010 550
Figure 10. Demonstration system for fan fault diagnosis.
Figure 10. Demonstration system for fan fault diagnosis.
Sensors 23 03755 g010
Sensors 23 03755 g011 550
Figure 11. Accelerometer and gyroscope time domain data in 3-axis: red—X, green—Y, and blue—Z.
Figure 11. Accelerometer and gyroscope time domain data in 3-axis: red—X, green—Y, and blue—Z.
Sensors 23 03755 g011
Sensors 23 03755 g012 550
Figure 12. Accelerometer and gyroscope frequency domain data in 3-axis: red—X, green—Y, and blue—Z.
Figure 12. Accelerometer and gyroscope frequency domain data in 3-axis: red—X, green—Y, and blue—Z.
Sensors 23 03755 g012
Sensors 23 03755 g013 550
Figure 13. Accelerometer data in the time–frequency domain using mSDFT in the 3-axis: top row, Z, middle row, Y, bottom row—X.
Figure 13. Accelerometer data in the time–frequency domain using mSDFT in the 3-axis: top row, Z, middle row, Y, bottom row—X.
Sensors 23 03755 g013
Sensors 23 03755 g014 550
Figure 14. Gyroscope data in the time–frequency domain using mSDFT in 3-axis: top row—Z, middle row—Y, bottom row—X.
Figure 14. Gyroscope data in the time–frequency domain using mSDFT in 3-axis: top row—Z, middle row—Y, bottom row—X.
Sensors 23 03755 g014
Sensors 23 03755 g015 550
Figure 15. Confusion matrix of the trained classifier (train—left; test—right).
Figure 15. Confusion matrix of the trained classifier (train—left; test—right).
Sensors 23 03755 g015
Sensors 23 03755 g016 550
Figure 16. STFT RGB image creation and convolutional neural network architecture.
Figure 16. STFT RGB image creation and convolutional neural network architecture.
Sensors 23 03755 g016
Sensors 23 03755 g017 550
Figure 17. STFT RGB image of idle class for accelerometer x, y, and z axis (top) and gyroscope x, y, and z (bottom).
Figure 17. STFT RGB image of idle class for accelerometer x, y, and z axis (top) and gyroscope x, y, and z (bottom).
Sensors 23 03755 g017
Sensors 23 03755 g018 550
Figure 18. STFT RGB image of normal class for accelerometer x, y, and z axis (top) and gyroscope x, y, and z (bottom).
Figure 18. STFT RGB image of normal class for accelerometer x, y, and z axis (top) and gyroscope x, y, and z (bottom).
Sensors 23 03755 g018
Sensors 23 03755 g019 550
Figure 19. STFT RGB image of the fault class for the accelerometer x, y, and the z axis (top) and gyroscope x, y, and z (bottom).
Figure 19. STFT RGB image of the fault class for the accelerometer x, y, and the z axis (top) and gyroscope x, y, and z (bottom).
Sensors 23 03755 g019
Sensors 23 03755 g020 550
Figure 20. Time–frequency RGB images for class: idle (left), normal (middle), fault (right).
Figure 20. Time–frequency RGB images for class: idle (left), normal (middle), fault (right).
Sensors 23 03755 g020
Sensors 23 03755 g021 550
Figure 21. Confusion matrix of the trained (left) and tested (right) CNN classifier.
Figure 21. Confusion matrix of the trained (left) and tested (right) CNN classifier.
Sensors 23 03755 g021
Table
Table 1. Comparison of proposed and enhanced methods.
Table 1. Comparison of proposed and enhanced methods.
IoT
Connectivity		Types of Fault and LabelsSignal or SensorFeatures
Extraction Method		FeaturesClassificatorArticle
MQTT and HTTPdemonstration of fan blades’ imbalance (normal, fan off, fan with fault)3-axis accelerometer and 3-axis gyroscopeSDFT or STFT at six axisRGB image made of six time–frequency domain dataCNNProposed
MQTT and HTTPdemonstration of fan blades’ imbalance (normal, fan off, fan with fault)preliminary selected one axis of 3-axis accelerometer and 3-axis gyroscopeSDFT at one axis enhanced to STFTfrequency domain dataclassical
classifier		Enhanced
Not specifiedbearing (normal, inner ring, outer ring, ball)vibration one-axisSTFTcolor spectrogram of one signalCNN[15]
Not specifiedbearing normal and four faulty states (ball, inner ring, outer ring, inner + outer)3-axis accelerometerfrequency transform with weight mapfrequency domain for each axisCNN[16]
Not specifiedblades non-damaged and two fault (5% and 15% broken blades)from one axis to 3-axis of angular velocityWPT (wavelet packet transform)—wavelet name not specifiedthird level of WPT decompositionLSTM (long and short-term memory)[17]
Not specifiedbearing (normal, outer, ball, inner)raw data is one-dimensional signal;
sensor is not specified		CWT (continuous wavelet transform), STFTCWT, time domain, and frequency domain features aggregationMIMTNet (multiple—input, multiple—task CNN)[18]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Łuczak, D.; Brock, S.; Siembab, K. Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity. Sensors 2023, 23, 3755. https://doi.org/10.3390/s23073755

AMA Style

Łuczak D, Brock S, Siembab K. Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity. Sensors. 2023; 23(7):3755. https://doi.org/10.3390/s23073755

Chicago/Turabian Style

Łuczak, Dominik, Stefan Brock, and Krzysztof Siembab. 2023. "Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity" Sensors 23, no. 7: 3755. https://doi.org/10.3390/s23073755

APA Style

Łuczak, D., Brock, S., & Siembab, K. (2023). Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity. Sensors, 23(7), 3755. https://doi.org/10.3390/s23073755

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop