Correlation Data Augmentation-Based YOLO-Integrated Object Detection of Thermal-Equalization Video Using Line Scanning Inductive Thermography

Abstract

Active infrared thermography (IRT) in non-destructive testing is an attractive technique used to detect wide areas in real-time on site. Most of the objects inspected on site generally have rough surfaces and foreign substances, which significantly affects their detectability. To solve this problem, in this study, line scanning (LS)-based induction thermography was used to acquire thermal image data of a specimen containing foreign substances. The heat distribution caused by foreign substances was removed using the Gaussian filtering-based Fast Fourier Transform (FFT) algorithm. After that, the data augmentation was performed by analyzing the correlation, and crack detection for the images was performed using you only look once (YOLO) deep learning. This study presents a method for removing non-uniform heat sources using the FFT algorithm, securing virtual data augmentation, and a detection mechanism for moving inspection objects using AI deep learning.

1. Introduction

In industrial sites, when defects occur during the manufacturing process or operation, inspection is performed using the non-destructive testing (NDT) techniques [1]. There are various types of the NDT techniques [2,3,4]; representative examples include visual testing (VT), acoustic emission (AE), penetrant testing (PT), magnetic particle testing (MPT), infrared thermography (IRT), ultrasonic testing (UT), radiographic testing (RT), and acoustic emission (AE). In the recent era of 4.0 NDT, the aforementioned technologies are essential elements of artificial intelligence (AI), and they determine the presence or absence of abnormalities in the inspection object in real time. Each technology can be appropriately selected depending on the inspection environment, shape of the object, inspection conditions, and cost.

The characteristics of each NDT technology are as follows: VT technology is suitable for the monitoring and surface inspection of inspection targets in the production process. However, it is only capable of detecting surface defects and is highly dependent on the experience of the inspector. PT technology is suitable for the surface inspection of mass-produced products. However, it is limited to surface defects and is subject to many environmental variables. MPT technology can detect surface defects of ferromagnetic targets. However, it is limited to the surface and is only applicable to ferromagnetic materials. In addition, the surface of the inspection target must be flat and coatings must be removed. UT technology can inspect the surface or interior. However, it is mostly performed manually and relies on the skill and experience of the inspector, and the equipment is expensive. RT technology can inspect the surface or interior. However, it is expensive and time-consuming. In addition, it is hazardous to the body and causes waste problems due to radiation. AE technology allows for surface and internal defect inspection using information on abnormal propagation. However, the structure under inspection attenuates the stress waves.

Table 1. Comparison of characteristics for each NDT technology [5,6,7].

NDT Technique	Capability	Limitation	Cost
VT	Simple surface monitoring	Limited to surface defects; experience dependent	Low
PT	Suitable for mass production	Sensitive to environment; surface only	Low to medium
MPT	Effective for ferromagnetic parts	Surface only; flat, uncoated surface required	Medium
IRT	Non-contact, fast surface inspection	Limited depth detection; sensitive to environmental conditions	High
UT	Surface and internal inspection	Manual; skill dependent; expensive equipment	High
RT	Surface and internal inspection	Expensive; time consuming; hazardous	Very high
AE	Surface and internal analysis	Stress wave attenuation issues	High

shows the comparison for each NDT technology.

Among the NDTs, active IRT is a technique that efficiently inspects a wide area in real time using a non-contact technique [8,9,10]. It is a highly reliable technology and has been widely applied in recent years. The IRT combines the advantages of various technologies. Active IRT has various types of heat sources, and the eddy current is popular among them. It is called eddy current thermography (ECT) or inductive thermography (IT) [11]. As the name of the technology suggests, it utilizes an eddy current as a heat source. An eddy current is induced in the inspection target by a coil through which an alternating current flows and the temperature of the surface of the heated inspection target is measured by the infrared (IR) camera. Considering the specimens of this study for the NDT technology presented in Table 1, it can be determined that, among IRT technologies, IT technology is suitable.

IT technology can be classified into stationary and motional. Stationary IT technology has fixed inspection objects and IR cameras, while motional IT technology involves relative movement. Stationary IT technology is not suitable for wide-range inspection due to its limited measurement range. On the other hand, high-frequency IR cameras can inspect moving or rotating objects and have a very wide inspection range. However, various post-processing techniques are essential for the automatic defect detection of moving inspection objects.

In general, the surface of the inspection object in industrial sites is often covered with foreign substances and is not smooth. In addition, emissivity adjustment is required for accurate temperature measurement of the IR camera, but it is impossible to attach black tape or paint all surfaces. When measuring a surface with non-uniform emissivity, the accuracy of defect detection based on deep learning may decrease. Therefore, the non-uniform temperature distribution was solved by applying the Fast Fourier Transform (FFT) algorithm based on Gaussian filtering [12].

For automated defect detection of moving inspection objects, the you only look once (YOLO) technique is the most efficient. YOLOv4 can classify and detect in real time and can predict at a faster speed than Faster RCNN. Big data is required to improve the accuracy of detecting cracks in thermal images. Therefore, data augmentation is required to secure big data.

In the era of the fourth industrial revolution, AI is an essential technology, and various AI deep learning technologies are being applied for image processing. The possibility of applying various AI technologies (not just YOLO) to the IRT field is being reviewed. Wu. Z et al. [13] presented an improved lightweight MobileNetV2_DeepLabV3+ algorithm for dam crack detection, utilizing a dataset of 1560 images annotated with the “LabelMe” tool and depth cameras. Enhancements to the ASPP module and a multifeature fusion structure improved segmentation accuracy, achieving better performance compared to existing methods like U-Net and PSPNet while significantly reducing computational costs and training time, making it suitable for drone-based automatic crack detection. Li. H et al. [14] proposed an improved lightweight YOLOv5s model for real-time detection of pitaya fruits in field environments, integrating ShuffleNetV2, BiFPN, SE attention modules, and simOTA for enhanced feature extraction, efficient multistage fusion, and improved occlusion handling, achieving a compact model with high precision. Jumaah H.J. et al. [15] presented detailed analyses of Kirkuk City using Sentinel-2 data, GIS, and Envi-based techniques, creating true color, false color, and highlight-optimized maps to visualize natural landscapes, vegetation health, land cover, and urban areas.

The core contributions of this study are as follows. First, the non-uniform heat distribution caused by foreign substances was solved using the Gaussian-filtering-based FFT algorithm. Second, in order to secure big data for AI deep learning, a correlation equation based on regression analysis was derived to obtain virtual data. Finally, a process for detecting cracks on moving objects using YOLOv4 was presented.

2. Theory of LS Inductive Thermography

2.1. Mathematical Methods

When aluminum metal is excited by a coil through which a high-frequency alternating current flows, eddy currents are induced in the inspection object as a surrounding magnetic field is formed [11,16,17]. The penetration depth of the eddy current can be calculated by the skin depth equation and is determined by the material properties of the object and the frequency of the alternating current. The skin depth equation is as follows:

$$\delta ={\displaystyle \frac{1}{\sqrt{\pi \sigma \mu f}}}$$

(1)

where $\delta$ is the penetration depth, $\sigma$ is the electrical conductivity, $\mu$ is the permeability, and $f$ is the frequency of the alternating current. The penetration depth is inversely proportional to frequency and permeability, so appropriate settings are required.

When an eddy current is induced in an inspection object with an internal defect by a coil, the local current density increases and heat is generated. This is called Joule’s heating. In the case of magnetic materials, the eddy current corresponds to the heat source, and the equation is as follows.

$$q_s={\left|J\right|}^2/\sigma$$

(2)

where $q_s$ is the intensity of the heat source and $J$ is the density of the current. Among the factors that increase temperature by Joule’s heating, the effect of σ is very small and can be ignored. Therefore, σ can be considered as a constant.

The heat source inside the moving inspection object diffuses isotropically, and the process for this is as follows.

$$\rho c_p{\displaystyle \frac{\partial T}{\partial t}}+\rho c_{p}v·\nabla T=\nabla ·\left(k\nabla T\right)+q_s$$

(3)

where $\rho$ is the density of the metal, $c_p$ is the specific heat, $T$ is the temperature, $t$ is the time, $v$ is the moving speed of the metal, and $k$ is the heat transfer coefficient of the metal.

2.2. LS-Based Inductive Thermography

Figure 1 shows the principle of induction thermography based on line scanning (LS). There is relative movement between the IR camera and the inspection object. If there is no relative movement, the shape of coil is recorded together with the thermal image as in [18,19]. The thermal image detects defects due to relative temperature scale, and if the temperature of the coil is measured, the detectability may be affected. The coil area can be removed by post-processing, but this takes a lot of time.

When the inspection object passes through the coil, eddy currents are induced, generating heat. The defective area generates a higher temperature than other areas, which is measured by the IR camera. The measured data are transmitted to the PC. As shown in Figure 2, the thermal image is generated as a single image by accumulating the scanning line area for all frames. The reason for performing this process is to acquire the maximum thermal contrast data of the defective area and sound area after the inspection object passes through the coil. In addition, since the data for the fixed scanning line area is accumulated, a uniform temperature distribution image can be acquired. For example, the number of images acquired at a moving speed of 5 mm/s is 2934, and the image of the sequence is a combination of 2934 scanning lines based on the process shown in Figure 2.

3. Experimental Setup

3.1. Device Setup

Figure 3 shows the experimental setup of the LS-based induction thermography system. The induction thermography system consists of the current excitation device, cooling device, copper coil, sliding guide rail, and PC. The IR camera was FLIR’s SC645 model (FLIR Systems Inc., Wilsonville, OR, USA; un-cooled, 640 × 480 pixels, 7.5~13 μm, 30 Hz). The distance between the IR camera and the specimen was set to 700 mm. The frequency of the excitation device is set to 27 kHz, and the intensity of the current is set to 135 A. The cooling device cools the heat generated by the coil. The coil is made of copper and is manufactured in a “U” shape. The movement range of the specimen is 1000 mm by the sliding guide rail, and it moves along the guide rail. The images acquired by the IR camera are transmitted to the PC, converted to Excel using the FLIR S/W (R&D suite 3.1), and then post-processed.

3.2. Aluminum Specimen

In Figure 4, (a) shows the specimen made of aluminum and (b) shows the dimensions. There are a total of six artificial crack defects. Line 1 has the same width but a different depth, and Line 2 has the same depth but a different width. The total thickness of the aluminum specimen was 10 mm, and it was of a size of 150 × 120 mm. The respective depths of cracks A, B, and C in Line 1 are 7 mm, 5 mm, and 3 mm. The depths of defects A, B, and C are all 5 mm. In particular, there are naturally occurring foreign substances on the surface around the cracks in Line 1.

Table 2. Material properties of the aluminum specimen.

Density	$2710 \mathrm{k}\mathrm{g}/{\mathrm{m}}^3$
Thermal Conductivity	$327 \mathrm{W}/\mathrm{m}·\mathrm{K}$
Heat Capacity	$900 \mathrm{J}/\mathrm{k}\mathrm{g}·\mathrm{℃}$
Electrical Conductivity	$37.7\times {10}^6$$\mathrm{S}/\mathrm{m}$
Relative Permeability	9.21

shows the material properties of the aluminum specimen.

4. Results

4.1. Data Acquisition of Thermal Image

Temperature data for each moving speed were acquired using LS-based induction thermography.

Table 3. The number of frames for each moving speed.

Moving Speed	Frame
5 mm/s	2934
7 mm/s	2323
9 mm/s	1652
11 mm/s	1480
13 mm/s	1301
15 mm/s	1051

shows the number of frames for each moving speed. The number of frames for each moving speed of 5, 7, 9, 11, 13, and 15 mm/s was calculated to be 2934, 2323, 1652, 1480, 1301, and 1051, respectively.

When capturing images at a specific moment while the specimen is moving, the temperature of the specimen only increases as it gets closer to the induction coil. Therefore, by applying the process shown in Figure 2, all frame images were accumulated based on a single y-axis line to obtain a single image. Figure 5 shows the accumulated single image for each moving speed. The imcrop function was applied to the image so that only the shape of the specimen is visible.

In the thermal image, thermal noise occurs at the upper part due to foreign substances on the specimen surface, making the crack shape of the first row unclear. The foreign substance on the surface reduces the reflectivity and relatively increases the emissivity, resulting in a relatively high temperature distribution. Therefore, the Gaussian-filtering-based FFT algorithm was applied to remove the distribution caused by the foreign matter and perform thermal equalization.

4.2. Thermal Equalization with FFT Algorithm

The Gaussian-filtering-based the FFT algorithm is a method to reduce the influence of high frequencies in the time or space domain. Here, high frequency refers to an area where the temperature rises, such as the defective area, due to foreign substances on the surface of the specimen. In addition, the current density locally increases at the edges of the specimen due to Joule’s heating. Therefore, these areas can also be considered as high frequency, and applying the FFT algorithm can reduce the influence of crack recognition when YOLOv4 is applied. As a result, the detectability and accuracy of cracks can be improved. For thermal equalization in Figure 5, it is necessary to move to the frequency domain, and the equation is as follows [12,20,21].

$$H\left(u,v\right)=e^{-{\displaystyle \frac{1}{2}}({\displaystyle \frac{{\left(u-u_0\right)}^2}{{\sigma }_u^2}}+{\displaystyle \frac{{\left(v-v_0\right)}^2}{{\sigma }_v^2}})}$$

(4)

where $H\left(u,v\right)$ is the frequency domain of Gaussian filtering, $(u,v)$ is coordinate of frequency domain, $u_0$ and $v_0$ are center position of Gaussian filtering, and ${\sigma }_u$ and ${\sigma }_v$ are standard deviation of Gaussian distribution.

$$G\left(u,v\right)=F(u,v)·H(u,v)$$

(5)

where $F\left(u,v\right)$ is the image with the FFT algorithm applied to thermal image $F\left(x,y\right)$ and $G\left(u,v\right)$ is the image of frequency domain. And then, if the inverse transform is performed on $G\left(u,v\right)$ again, it is as follows.

$$G\left(x,y\right)=IFFT\left\{G\left(u,v\right)\right\}$$

(6)

Figure 6 shows an image that applies the Gaussian-filtering-based the FFT algorithm to each moving speed. The Gaussian standard deviation is set to 200. The degree to which the foreign substance distribution has been removed can be confirmed, compared to Figure 5. However, it is difficult to qualitatively detect them because it is affected by the crack in the first row. Therefore, setting the ROI labeling of the crack area is very important.

For each moving speed, the FFT algorithm was applied to the images of all frames to perform thermal equalization, and the images were integrated into one video to obtain six data for each moving speed. After that, it was applied to YOLOv4 to perform AI-based automatic crack detection.

4.3. Data Augmentation

Before performing the YOLOv4 algorithm, data augmentation was performed to secure big data for transfer learning [22,23,24]. Virtual data for the moving speeds of the specimen were acquired at 3 and 17 mm/s. Regression analysis was performed based on the maximum temperature value for each image in Figure 5.

Table 4. Maximum temperature and residual values for each moving speed.

Moving Speed	Max. Temperature	Residual Values
5 mm/s	30.286 °C	0.358
7 mm/s	29.527 °C	−0.079
9 mm/s	28.905 °C	−0.379
11 mm/s	28.771 °C	−0.192
13 mm/s	28.692 °C	0.05
15 mm/s	28.564 °C	0.242

shows the maximum temperature values and residual values for each moving speed. Figure 7 shows a graph analyzing the residual plot by performing regression analysis.

In Figure 7a, to obtain the data for 3 and 17 mm/s, the correlation equation was obtained as follows.

$$y=-1.559\ln \left(x\right)+32.612$$

(7)

When 3 and 17 mm/s were substituted into Equation (7), the values were calculated as 30.485 °C and 27.394 °C, respectively. In order to obtain reference data for substituting the predicted values that came out later, the residuals for each moving speed were analyzed. The smallest residual value of 0.05 °C was confirmed at 13 mm/s, and the differences with 3 and 17 mm/s were calculated as 2.209 °C and −0.495 °C, respectively. After that, 2.209 °C and −0.495 °C were input to all frames of 13 mm/s to obtain virtual big data for 3 and 17 mm/s.

The videoLabeler application of MATLAB S/W (R2023a) was used for labeling the augmented data. Since labeling 2602 images took a lot of time, the temporal interpolator algorithm was used. The temporal interpolator algorithm interpolates the ROI of the crack for each frame to predict the axis-aligned or projected ROI between frames, and the process is shown in Figure 8. Assuming that the total length of the video is 80 s, cracks are labeled for images corresponding to sections 30, 40, 50, and 60 s. After that, the ROI points set at 30, 40, 50, and 60 s are interpolated. For example, the ROI is automatically interpolated between 30 and 40 s, so the ROI set on the crack moves accordingly. In this way, the ROI area between 30 and 60 s is labeled along the crack. However, since the size of the labeling may be enlarged or reduced in a specific section, refinement is required.

In this study, when conducting experiments to acquire data, the algorithm is highly optimized because the specimen moves at a constant speed for the LS, enabling efficient labeling in a very short time. The data with applied labeling were saved in the form of groundTruth to prepare for transfer learning [25].

4.4. Automatic Detection Using YOLOv4

Crack detection on the surface of the aluminum specimen was performed based on the detector of YOLOv4 that completed training, and the YOLOv4 version was applied [14,26,27]. In order to improve the crack detectability, the Gaussian-filtering-based FFT algorithm was applied to the images for each moving speed. After extracting the images of all frames for each moving speed, the FFT algorithm was applied and then merged into one image again.

Figure 9 shows the crack detection images for a moving speed of 5 mm/s using YOLOv4. Figure 9a,b are the points at which the specimen passes through the coil. In Figure 9c, all six cracks passed through the coil, and only two cracks were detected in the first row. Figure 9d is the point at which the specimen completely passed through the coil, and all cracks were detected; however, it can be confirmed that other foreign substances were also recognized as cracks and detected. Afterward, Figure 9e,f completely passed through the coil, and none of the cracks in the first row were detected. Through this, it can be confirmed that the crack detectability is high because the thermal contrast between the defective area and the sound area is large when the specimen passes through the coil.

Based on the results in Figure 9, the detection of each moving speed was confirmed for the moment when the aluminum specimen with the most cracks passed through the coil. Figure 10 shows the crack detection images for moving speeds of 7, 9, 11, 13, and 15 mm/s. At 7 mm/s, all six cracks were detected, and no objects other than cracks were detected. At 9 and 11 mm/s, one crack was detected in the first row and one crack shape was incorrectly detected in the foreign substance area. At 13 and 15 mm/s, no cracks were detected in the first row and one crack shape was incorrectly detected in the foreign substance area. In particular, it can be confirmed that the image shows that no cracks were detected in the second row at 15 mm/s.

Figure 9 and Figure 10 show that the highest detectability is achieved at 7 mm/s and that as the moving speed of the inspection object increases, the thermal contrast between the defective and sound area decreases, which tends to decrease the detectability.

5. Conclusions

This study performed crack detection on aluminum specimens with foreign substances on the surface using the LS-based induction thermography technique. Foreign substances on the surface were removed using the Gaussian-filtering-based FFT algorithm. When setting the Gaussian standard deviation to remove foreign substances, the shape of cracks may be removed. Therefore, the value of the Gaussian standard deviation was set to 20 to preserve the shape of the cracks. Virtual big data was secured through augmentation, and transfer learning was performed after applying ROI labeling using the temporal interpolator algorithm. After that, crack detection for each moving speed was performed using YOLOv4 deep learning, and the highest detectability was confirmed at 7 mm/s. If the moving speed of the inspection object is too fast or slow, the shape of the defect may not be clear in the thermal image due to low thermal contrast. Therefore, it is very important to set an appropriate speed when inspecting a moving object using induction thermography.

The limitations of this study are that the detectability significantly decreases when a certain speed range is exceeded and only aluminum materials were considered. In addition, the environmental conditions of the lab are very good compared to the actual field. The objects inspected in the actual industrial field have rough surfaces and poor environmental conditions. The experiments conducted in this study are on a lab scale, which is different from the actual industrial field. Therefore, future research is essential to improve the detection speed, apply various materials, and consider the environment. In this environment, future research will be conducted to optimize algorithms that can improve detectability.