System Invariant Method for Ultrasonic Flaw Classification in Weldments Using Residual Neural Network

Abstract

The industrial use of ultrasonic flaw classification using neural networks in weldments must overcome many challenges. A major constraint is the use of numerous systems, including a combination of transducers and equipment. This causes high complexity in the datasets used in the training of neural networks, which decreases performance. In this study, the performance of a neural network was enhanced using signal processing on an ultrasonic weldment flaw dataset to achieve system invariance. The dataset contained 5839 ultrasonic flaw signals collected by various types of transducers connected to KrautKramer USN60. Every signal in the dataset was from 45 FlawTech/Sonaspection weldment specimens with five types of flaw: crack, lack of fusion, slag inclusion, porosity, and incomplete penetration. The neural network used in this study is a residual neural network with 19 layers. The performance evaluation of the same network structure showed that the original database can achieve 62.17% ± 4.13% accuracy, and that the invariant database using the system invariant method can achieve 91.45% ± 1.77% accuracy. The results demonstrate that using a system invariant method for ultrasonic flaw classification in weldments can improve the performance of a neural network with a highly complex dataset.

1. Introduction

Ultrasonic testing is a non-destructive testing technique that is commonly used in many industries. Because most defects occur in weldments, ultrasonic inspections are usually focused on weldments, and thus detecting and classifying ultrasonic flaw signals is a long-standing task in the field of ultrasonic testing. Determining flaws requires numerous experiences, and it is difficult to perform non-destructive testing efficiently because there is the possibility of human misjudgment. To solve this problem, many studies on ultrasonic testing to determine the presence, type, shape, and size of flaws using artificial intelligence are being conducted.

Song et al. [1] used probabilistic neural networks on ultrasonic weldment defect signals. Masnata et al. [2] used a neural network and Fisher discriminant analysis to classify cracks and volumetric defects. Polikar et al. [3] applied a neural network to develop a frequency-invariant weldment defect classification system for pipes. Margrave et al. [4] used neural networks to detect flaws in steel pipes. Song et al. [5] developed an intelligent ultrasonic evaluation system (IUES) for real-time weldment flaw classification. Simas Filho et al. [6] developed a flexible decision support system based upon neural network and principal component analysis to evaluate fiber-metal laminates. Cruz et al. [7] also developed a decision support system that selects parameters for the neural network to detect flaws in steel-welded joints. Meng et al. [8]. used a combination of wavelet transform and a convolutional neural network to classify void and delamination defects in CERP specimens. With the exception of [4], all these methods are limited due to their complexity [9,10,11].

A large amount of data must be collected to ensure the performance of artificial intelligence (AI). It is highly difficult to collect a large amount of real inspection data in the ultrasonic testing research field. Many studies collect data from specimens to solve this problem; however, there is a limit on the amount of data that one research group can collect. This works well for systems with little variation, such as those that use an ultrasonic transducer from the same manufacturer. However, inspectors use various types of transducer and equipment for industrial ultrasonic testing. The limitation of dataset size can be a problem in this scenario for the following reasons. First, the AI may not sufficiently train signals from each system. Second, flaw signals from various systems may interfere with the training process, finding global minimum. To solve this problem, this study proposes a system invariant technique to reduce the complexity of ultrasonic weldment flaw databases and compares the performance of two different databases, one without a system invariant method and the other with a system invariant method, using the same neural network architecture.

2. Ultrasonic Weldments Flaw Databases

To properly train neural networks, a database of well-inspected ultrasonic weldments flaw signals was developed. The signals were collected from 45 FlawTech/Sonaspection weldment specimens, including five types of flaw: crack, lack of fusion, slag inclusion, porosity, and incomplete penetration. The signals were collected with 21 different ultrasonic transducers to make the system complex. This group of transducers included 2, 4, and 5 MHz central frequencies, 45, 60, and 70 degree inspection angles, and four different manufacturers (KrautKramer, GE, Olympus, and TKS). All the inspections were performed using KrautKramer and USN-60 flaw detector as the ultrasonic pulser-receiver instrument and a 50 MHz sampling frequency. The list of transducers is presented in

Table 1. List of transducers.

Manufacturer	KrautKramer			GE		Olympus	TKS
	MWB 45-2	WB 45-2	WB 45-4	MWB 45-4	SWB 45-5	A430S (45$°$, 2 MHz)	4C14 X 14A45
	MWB 60-2	WB 60-2	WB 60-4	MWB 60-4	SWB 60-5	A430S (60$°$, 2 MHz)	4C14 X 14A60
	MWB 70-2	WB 70-2	WB 70-4	MWB 70-4	SWB 70-5	A430S (70$°$, 2 MHz)	4C14 X 14A70

.

To construct the database, 5839 signals were acquired for each flaw and, subsequently, 2048 points were extracted with the flaw signal (Figure 1). The number of each weldment flaw is presented in

Table 2. Ultrasonic weldment flaw database.

Title 1	Flaws	No. of Signals
1	Crack	2899
2	Lack of Fusion	1196
3	Slag Inclusion	634
4	Porosity	493
5	Incomplete Penetration	617
	Total	5839

.

To check the performance of the neural networks for training, the database was separated. To balance the learning between each flaw, 40% of the crack signal and 10% of other flaw signals were randomly divided into the test signal. The compositions of the training and testing signals are described in

Table 3. Ultrasonic weldment flaw database.

Weldment Flaw Database
Training Database		Testing Database
Crack	1740	Crack	1159
Lack of Fusion	1076	Lack of Fusion	120
Slag Inclusion	571	Slag Inclusion	63
Porosity	444	Porosity	49
Incomplete Penetration	555	Incomplete Penetration	62
Total	4386	Total	1453

.

3. Database Augmentation

Although 4386 signals were obtained, considering the complexity of the system, the amount of data was not sufficient. Therefore, the number of training data was increased through augmentation. In terms of data augmentation, noise addition and time shifting were effective in various studies [3,10].

Because ultrasonic testing is usually performed in harsh environments, the possibility of the testing results being corrupted by electric noise, presented as white Gaussian noise (WGN), exists. Additive WGN was used to initiate this. The mathematical details of WGN are described in [12,13]. The WGN is added with two different signal-to-noise ratios (SNR) levels i.e., SNR 20 and SNR 15 (Figure 2). The number of testing signals increased threefold from 4386 to 13,158, including the original signal.

Another augmentation technique was adjusted after developing the augmentation database with WGN. In terms of ultrasonic testing signals, time shifting is equivalent to changing the distance between the transducer and the flaw location [10]. Therefore, new signals were created that shifted 20 and 10 points backward, and 20 and 10 points forward (Figure 3). Because the system features a sampling frequency of 50 MHz, one point on the time axis is 0.02 μs. Because of time shifting, including the original signal, the testing signal number increased fivefold, from 13,158 to 65,790. This augmented dataset is referred to as the original database and is described in

Table 4. Augmented database (original database).

Original Database
Training Database		Testing Database
Crack	26,100	Crack	1159
Lack of Fusion	16,140	Lack of Fusion	120
Slag Inclusion	8565	Slag Inclusion	63
Porosity	6660	Porosity	49
Incomplete Penetration	8325	Incomplete Penetration	62
Total	65,790	Total	1453

.

4. Artificial Neural Networks

Artificial neural networks (ANNs) are computational neural models inspired by a sequence of biological neurons. ANNs operate on the basis of the backpropagation algorithm proposed by Rumelhart et al. [14]. In the early stage of ANNs, a significantly simplified form of neural network, known as a multilayer perceptron (MLP), was developed, which led to fully connected neural networks (FCNNs) [15]. FCNNs are composed of input, hidden, and output layers. Each layer contains nodes with weights, and each node in the layer is connected to every node in the next layer. Deep neural networks are networks with more than two hidden layers.

The process of training a neural network involves adjusting the weight and connections between the layers. First, the data enter the input layer and pass through the hidden layers, while the dot product between the input data and weight matrix, whose size depends on the shape of the hidden layers, is computed. While computing, the activation function is used for tuning the training direction. After calculating the output, a loss function is computed, which gives the error between the true label and the predicted output. The error is then inverted using a backpropagation algorithm, and the weights are slightly updated to reduce this error. This process can be repeated until the error becomes sufficiently small or saturated [16,17,18,19].

4.1. Convolutional Neural Network

Convolutional neural networks (CNNs) were first proposed by Lecun et al. [20]. They later gained popularity in the computer vision community [21]. CNNs feature a convolutional layer that is connected to a specific spatial region in the previous layer and is based on the size of a defined filter, in contrast to FCNNs. This architecture allows the network to focus on lower-level features before assembling them into higher-level features. CNNs also include pooling layers, which help to reduce the computational load by downsampling the input data, but increase the complexity of the network [22].

4.2. Residual Neural Network

Residual neural networks (ResNets), first proposed by Kaiming He et al. [23], are based on CNN but include shortcuts between the input and output of the unit layer. If the input data are referred to as x, the output of ResNets is described by Equation (1):

F(x) + x

(1)

where F is the weight of computation. The calculational direction of ResNets minimizes the value of F(x) + x, which means F(x) becomes 0 since x is a fixed (input) value here. Conventional CNNs feature a weak spot in significantly deep networks, as their performance degrades after a certain depth. However, significantly deep networks are required to train with large amounts of data. ResNets solve this problem through their distinct network architecture as previously described, obtaining the best results by training 152 layers of ResNet using the CIFAR-10 dataset [24,25,26,27].

4.3. Residual Neural Network Architecture

The ResNet used in this study was designed in Keras, based on Tensorflow (Google open-source software for deep learning). The algorithm included an input layer, nineteen convolutional layers, and three stages of ResNet blocks. Each stage features the same architecture, but a different number of feature maps. The detailed architecture of the stage with feature map FM is described in

Table 5. Detailed architecture of unit stage of ResNet.

Adopted ResNet Stage
	Layer Type	Kernel Size/Stride	Feature Maps	Activation Function	Description
1	Input layer	-	-	-	x
2	Conv 1	$(1\times$$3)/(1\times 1$)	FM	Relu	y = Conv 1 (x)
3	Dropout	0.5	-	-	y = Dropout(y)
4	Conv 2	$(1\times$$3)/(1\times 1$)	FM	Relu	y = Conv 2 (y)
5	Dropout	0.5	-	-	y = Dropout (y)
6	Shortcut	-	-	-	x = x
7	Conv s	$(1\times 1$$)/(1\times 1$)	FM	Relu	x = Conv s (x)
8	Add shortcut	-	-	Relu	x = x + y
9	Conv 3	$(1\times$$3)/(1\times 1$)	FM	Relu	y = Conv 3 (x)
10	Dropout	0.5	-	-	y = Dropout (y)
11	Conv 4	$(1\times$$3)/(1\times 1$)	FM	Relu	y = Conv 4 (y)
12	Dropout	0.5	-	-	y = Dropout (y)
13	Shortcut	-	-	-	x = x
14	Add shortcut	-	-	Relu	x = x + y
15	Conv 5	$(1\times$$3)/(1\times 1$)	FM	Relu	y = Conv 5 (x)
16	Dropout	0.5	-	-	y = Dropout (y)
17	Conv 6	$(1\times$$3)/(1\times 1$)	FM	Relu	y = Conv 6 (y)
18	Dropout	0.5	-	-	y = Dropout (y)
19	Shortcut	-	-	-	x = x
20	Add shortcut	-	-	Relu	x = x + y

. The final ResNet architecture used when these unit stages were combined is shown in

Table 6. Architecture of ResNet.

Adopted ResNet Architecture
	Layer Type	Kernel Size/Stride	Feature Maps	Output Size
1	Input Layer	-	-	2048
2	Conv 1	$($$32)/(1\times 8$)	64	$(256\times 64$)
3	Dropout	0.5	-	$(256\times 64$)
4	Activation	Relu	-	$(256\times 64$)
5	Max Pool	$($$3)/(1\times 2$)	-	$(128\times 64$)
6	Stage 1	-	64	$(128\times 64$)
7	Max Pool	$($$3)/(1\times 2$)	-	$(64\times 64$)
8	Stage 2	-	128	$(64\times 128$)
9	Max Pool	$($$3)/(1\times 2$)	-	$(32\times 128$)
10	Stage 3	-	256	$(32\times 256$)
11	Dense Layer	300	-	300
12	Dropout	0.5	-	-
13	Output Layer	5	-	5

.

The number of nodes in the input layer was maintained at the same level as the sampling point of the input signal, which was 2048. The first convolutional layer contained 64 kernels of size 1 × 32, which remained large because of their performance for ultrasonic and vibration signals. [10,11,28]. The pooling layer reduces the number of layers but does not affect performance [22]. After the data have passed through stage 3, there is one fully connected (dense) layer and an output layer. The number of nodes in the fully connected layer was chosen as the best performance over multiple trials. The number of nodes in the output layer was kept equal to the number of flaws for classification. The activation function used in the layers was rectified linear unit (Relu), described by Equation (2)

$$Relu=\left\{\begin{array}{c}\left(x<0\right) f\left(x\right)=0\\ \left(x\ge 0\right) f\left(x\right)=x\end{array}\right.$$

(2)

because of its state-of-the-art performance in deep learning [29,30,31]. The loss function used is sparse categorical cross-entropy, described by Equation (3):

$$-{\displaystyle \sum }_{i=1}^M{y}_i\log \left(p_i\right)$$

(3)

where M is the number of data, y is a vector of the true label, and p is the predicted label. Sparse categorical cross-entropy gives the result as integer form, and it is widely used for multiclass classification problems. The activation function applied to the output layer is softmax, described by Equation (4):

$$\sigma {\left(y\right)}_i=\frac{e^{y_i}}{{{\displaystyle \sum }}_{k=1}^K{e}^{y_k}}$$

(4)

where y is the input vector to the output layer, and i indexes the output nodes, i = 1, 2, K. In this study, the number of classes is 5, so K is 5. Additionally, dropout normalization was used to avoid overfitting. The network is then trained to minimize cross-entropy loss.

ResNet is trained with optimizer adaptive moment estimation (Adam) for 500 epochs. Adam is an algorithm for the first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The hyper-parameters feature intuitive interpretations and typically require little tuning. The method is suitable for the classification of flaw signals in complex systems since it is appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients [32].

4.4. ResNet Performance Evaluation

The performance of ResNet was then evaluated on the original database (Figure 4). The results show that highly complex datasets with a comparably low amount of data can be a significant penalty when training a neural network. In epoch 461, the training accuracy increased to 92.63%, whereas the testing accuracy reached 70.75% in epoch 241. Here, the meaning of testing accuracy is simply how much data were predicted to be the same as the true label. For a more detailed analysis of the testing accuracy, the average accuracy after epoch 100 was 62.17%, with a standard deviation of 4.13%. This means that training ResNet with the original database can result in a network performance of 62.17% ± 4.13%.

Prior research demonstrates that for simpler databases acquired with KrautKramer transducers, the testing accuracy exceeds 90% [10,11]. This low testing accuracy has two possible explanations. One is that the network is overfitted, and the other is that the complexity of the database is too high. However, since this network already includes many regularization techniques that are widely known to solve overfitting, it is difficult to regard it as a problem of overfitting alone. Or, at least, it is difficult to solve this problem with only the regularization technique. Therefore, reducing the complexity of the database down can be a possible solution for this problem.

5. System Invariant Method

Because the database was obtained from 21 different transducers, the neural network may be confused by other types of signals because of the unique characteristics of each transducer. There are several approaches to making ultrasonic signals invariant to system or frequency to improve the performance of the neural network training. Polikar et al. created a frequency invariant system to enhance the performance of a neural network for ultrasonic signals by normalizing the frequency [3]. Song et al. developed an intelligent system for ultrasonic flaw classification in weldments [5]. During the process, the system was normalized using a reference signal in the frequency domain.

5.1. Principle of System Invariant Method

Weldment flaws signal $V_f$ include the system, flaw scattering pattern, and specimen as variables. Among these, systems and specimens increase the complexity of the database, which can inhibit performance in classifying neural network flaws, so the database must be refined by extracting only the flaw scattering patterns. The schematic experiment setup is shown in Figure 5.

Because the ultrasound system is linear time-invariant (LTI), $V_f$ can be expressed in a time domain with convolution between system variables, as in Equation (5) [33]:

$$V_f\left(t\right)=\beta \left(t\right)*C_1\left(t\right)*T_{12}\left(t\right)*C_2\left(t\right)*A\left(t\right)*C_2\left(t\right)*T_{21}\left(t\right)*C_1\left(t\right)$$

(5)

where $\beta$ is system factor, $C_1$ is the diffraction correction in the wedge, $T_{12}$ is the transmission coefficient of the wedge to the specimen, $C_2$ is the diffraction correction in the specimen, $A$ is the scattering pattern from the flaw, and $T_{21}$ is the transmission coefficient of the specimen to the wedge.

To eliminate system and specimen parameters, signals were acquired from a reference specimen using the same system used for the weldment signal acquisition. A reference signal was acquired using a circular reference block to obtain the same ultrasound path for every inspection angle for each transducer–system combination. Here, 21 reference signals by 21 different types of transducers were obtained using the same ultrasonic pulser-receiver device. The schematic experimental setup for the reference signal is shown in Figure 6.

Furthermore, the reference signal $V_R$ can be described in a time domain as a convolution between system factors, as in Equation (6):

$$V_R\left(t\right)=\beta \left(t\right)*C_1\left(t\right)*T_{12}\left(t\right)*{C^{\prime }}_2\left(t\right)*R\left(t\right)*{C^{\prime }}_2\left(t\right)*T_{21}\left(t\right)*C_1\left(t\right)$$

(6)

where ${C^{\prime }}_2$ is diffraction correction in the reference specimen and $R$ is the reflection coefficient from the round surface. In the frequency domain, the signals $V_f$ and $V_R$ can be combined by simple multiplication, not convolution [34,35]. Using fast Fourier transforms (FFT), $V_f$ and $V_R$ are described in the frequency domain as Equations (7) and (8):

$$V_f\left(\omega \right)=\beta \left(\omega \right)C_1\left(\omega \right)T_{12}\left(\omega \right)C_2\left(\omega \right)A\left(\omega \right)C_2\left(\omega \right)T_{21}\left(\omega \right)C_1\left(\omega \right)$$

(7)

$$V_R\left(\omega \right)=\beta \left(\omega \right)C_1\left(\omega \right)T_{12}\left(\omega \right){C^{\prime }}_2\left(\omega \right)R\left(\omega \right){C^{\prime }}_2\left(\omega \right)T_{21}\left(\omega \right)C_1\left(\omega \right)$$

(8)

Divide these two terms, flaw scattering pattern $A$ can be extracted as Equation (9):

$$\frac{V_f\left(\omega \right)}{V_R\left(\omega \right)}={\left(\frac{C_2\left(\omega \right)}{{C^{\prime }}_2\left(\omega \right)}\right)}^2\frac{A\left(\omega \right)}{R\left(\omega \right)}$$

(9)

The reflection $R$ is affected only by the mechanical property of the two mediums and the incidence angle. Here, the round surface of the reference block fixes the incidence angle as normal when the transducer angle varies. Therefore, $R_i$ is constant, while i represents each transducer with a different angle. Furthermore, the reflection coefficient for steel to air is significantly close to −1 in this case. This means that Equation (9) is described as Equation (10):

$$\frac{V_f\left(\omega \right)}{V_R\left(\omega \right)}\cong -{\left(\frac{C_2\left(\omega \right)}{{C^{\prime }}_2\left(\omega \right)}\right)}^{2}A\left(\omega \right)$$

(10)

Diffraction correction $C\left(\omega \right)$ in far-field is represented as Equation (11):

$$C\left(\omega \right)=\frac{-ik{a}^2}{2z}$$

(11)

where $k$ is the wavenumber, $a$ is the radius of the transducer and $z$ is the flight distance. $C_2$ and ${C^{\prime }}_2$ feature the same frequency and radius as the transducer, except for distance. Thus, Equation (10) reduces to:

$$\frac{V_f\left(\omega \right)}{V_R\left(\omega \right)}\cong -{\left(\frac{z’}{z}\right)}^{2}A\left(\omega \right)$$

(12)

where $z’$ is the distance between the transducer and the flaw of the weldment specimen, while $z$ is the distance between the transducer and the round surface of the reference specimen. With constant $a=-{\left(z’/z\right)}^2$, $A\left(\omega \right)$ can be represented as Equation (13):

$$aA\left(\omega \right)\cong \frac{V_f\left(\omega \right)}{V_R\left(\omega \right)}$$

(13)

Applying inverse fast Fourier transform to $A\left(\omega \right)$ can restore the time domain signal of the flaw scattering pattern. When training a neural network, each signal becomes normalized for increasing performance; the constant $a$ disappears. The flow diagram of this process is shown in Figure 7.

5.2. Applying System Invariant Method to Original Database

To reduce complexity, the system invariant method should be used to reconstruct the ultrasonic weldment signals in the original database. Examples of crack signals before and after using the system invariant method are shown in Figure 8. The system factor was removed from the original signals, leaving only the scattering pattern from flaws. The database that uses the system invariant method is referred to as the invariant database.

5.3. ResNet Performance Evaluation with Invariant Database

Because the original database and the invariant database featured the same number and composition, training the same architecture of ResNet with the original database and invariant database is suitable for verifying the effect of the system invariant method.

The performance of ResNet was then evaluated on the invariant database, as shown in Figure 9. The results show that the system invariant method can reduce database complexity and enhance network performance. The training accuracy increased to 92.15% in epoch 446, which is comparable to the original database, whereas the testing accuracy was higher than 95.04% in epoch 491. For a more detailed analysis of the testing accuracy, the average accuracy after epoch 100 was 91.45%, with a standard deviation is 1.77%. This means that training ResNet with the invariant database can obtain a network performance of 91.45% ± 1.77%.

A further reason why the invariant method can improve network performance is that the test curve behaves similarly to the training curve, with only a minor shift up and down. This implies that the system invariant method can confuse the network into believing that training and testing data with the same flaw type are similar signals with which to classify.

5.4. Performance Comparison

Table 7. ResNet performance comparison between the original and the invariant database.

	Original Database		Invariant Database
(Epoch > 100)	Training	Testing	Training	Testing
Average Accuracy (%)	92.63	62.17	92.15	91.45
Standard Deviation (%)	0.64	4.13	0.43	1.77

and Figure 10 show a performance comparison. Comparing the training accuracies, the invariant database featured lower average accuracy, but a smaller standard deviation than the original database, making it difficult to determine which is better. It is more rational to consider this as an error that can occur naturally during the training process of a neural network, rather than a performance deviation.

Comparing the test accuracies, there was a significant improvement in accuracy for the invariant database. The average accuracy improved by 29.28 pp from 62.17% to 91.45%, making it possible to classify ultrasonic weldment flaw signals more precisely; furthermore, the standard deviation improved by 2.36 pp from 4.13% to 1.77%, making the network performance more stable. Furthermore, the average training and test accuracies of the invariant database were almost similar, suggesting that the system invariant technique can effectively secure the system invariant.

6. Summary

In this study, a system invariant method was developed and tested using a complex database obtained from 21 different transducers. The flaw signal database contained 65,790 training data and 1453 test data, referred to as the original database. By developing a system invariant method in the frequency domain and using it in the original database, an invariant database was constructed. Consequently, comparing performance using a residual neural network, the invariant database showed a significant improvement in accuracy compared to original database. The proposed system invariant method showed meaningful performance improvements against complex ultrasonic equipment. In future studies, practical research is needed in order to overcome difficulties that may arise when the system invariant technique is applied to the actual ultrasonic testing site.