Extracting Weld Bead Shapes from Radiographic Testing Images with U-Net

Abstract

Metals created by melting basic metal and welding rods in welding operations are referred to as weld beads. The weld bead shape allows the observation of pores and defects such as cracks in the weld zone. Radiographic testing images are used to determine the quality of the weld zone. The extraction of only the weld bead to determine the generative pattern of the bead can help efficiently locate defects in the weld zone. However, manual extraction of the weld bead from weld images is not time and cost-effective. Efficient and rapid welding quality inspection can be conducted by automating weld bead extraction through deep learning. As a result, objectivity can be secured in the quality inspection and determination of the weld zone in the shipbuilding and offshore plant industry. This study presents a method for detecting the weld bead shape and location from the weld zone image using image preprocessing and deep learning models, and extracting the weld bead through image post-processing. In addition, to diversify the data and improve the deep learning performance, data augmentation was performed to artificially expand the image data. Contrast limited adaptive histogram equalization (CLAHE) is used as an image preprocessing method, and the bead is extracted using U-Net, a pixel-based deep learning model. Consequently, the mean intersection over union (mIoU) values are found to be 90.58% and 85.44% in the train and test experiments, respectively. Successful extraction of the bead from the radiographic testing image through post-processing is achieved.

1. Introduction

1.1. Research Background and Necessity

Welding is an essential technology in industry and provides metals for applications in the building of ships, aircrafts, and automobiles. In the shipbuilding and offshore plant industry, welding is a major part of the production process, and it is important to inspect the condition of the welds because the weld zone affects the strength and durability of the structures. Non-destructive testing technologies are primarily used to determine the quality of the weld zone. Representative non-destructive testing technologies include radio-graph testing, ultrasonic testing, magnetic particles testing, liquid penetrant testing, eddy current testing, leak testing, and visual testing [1]. Radiographic testing and ultrasonic testing are especially used in the shipbuilding field [2]. Radiographic testing is the most preferred method by ship owners as the resulting images can be stored permanently, and the interior of the weld zone can be examined visually.

Currently, to collect information on welding in several shipyards, people are directly checking welding inspection information on shipbuilding and offshore plant structures of as many as 500 blocks or more. To inspect and determine the quality of the weld zone, experts as well as considerable time, expenditure, and manpower are required to manually examine the weld images. Additionally, the checking of the weld zone and information, is subjective and made through experience, resulting in inconsistent and irrational results. If deep learning could be used to automate the extraction of the weld bead shape, it would save considerable time and money, and efficient quality inspections could be conducted. As a result, objectivity can be secured in the quality inspection and determination of the weld zone in the shipbuilding and offshore plant industry. Depending on the bead and base metal generated in the welded area during a welding operation, areas with the same shape may be classified as different types of defects. Therefore, when determining the weld quality, it is necessary to efficiently perform inspections and determine whether the weld zone has defects or inadequate quality by extracting only the bead from the weld image and identifying the bead’s generation pattern.

Algorithms that extract only weld beads from the radiographic testing image have been used to perform weld detection via image preprocessing; however, they have been inefficient owing to noise outside beads that occurs via thresholding, which requires the algorithms to perform the additional task of removing the noise [3,4]. Additionally, image preprocessing methods including noise reduction, contrast enhancement, thresholding, and labeling, were performed to segment only bead from the radiographic testing image [5]. In the image where thresholding was performed, the boundary of the bead area was defined and the edge was extracted to segment only the bead area [6,7]. Attempts to extract only beads from the radiographic testing image are continuously being made in various ways. However, since several tasks are performed manually, efficient tasks are not performed. It is difficult to say that the weld bead was extracted automatically. Therefore, it takes a lot of time and money to extract the weld bead and detect weld defects. This study aims to automatically segment the shape and location of the weld bead from the radiographic testing image through deep learning and the extracting of only the weld bead. To accurately detect weld defects, the weld bead must be extracted from the radiographic testing image, and this task can be performed efficiently if the weld bead extraction is automated.

1.2. Research Methods

This study used the deep learning-based U-Net [8] algorithm to automatically examine weld bead shapes in radiographic testing images and determine the shapes and locations of the beads for detailed defect classification. For increased accuracy of the bead detection, image preprocessing was performed via histogram equalization [9] to improve the contrast of the images before the model training. Furthermore, to diversify the data and improve the deep learning performance, data augmentation was performed to artificially expand the image data [10]. It was checked whether the performance of deep learning was improved through training results before and after image preprocessing and data augmentation were applied. The weld images were captured using radiographic testing, and the radiographic images were marked with information to classify and distinguish them. For training, mask images were created by performing a labeling process that removed areas other than the bead area from the weld images. The weld images and the data with labeled bead areas were combined to create the training dataset to train the bead areas, such that the bead shape extraction could be automated. The images that constituted the dataset were divided into training and testing images to determine the training accuracy. The images had a variety of pixel sizes of approximately 1030 × 300 or greater. It was found that high-quality images reduced the learning speed. To efficiently train the images, the image pixel sizes were reduced to 256 × 256, then training was performed [11]. After completing the training, the shapes and locations of the predicted weld beads were compared to those of the actual beads. Image post-processing was performed from the predicted bead shape and location image to extract only the weld bead.

2. Methods

Given the properties of radiographic testing images, the current conditions during the process of converting analog images to digital images via scanning may result in varying image quality. Additionally, it is necessary to have a method that can extract various weld bead shapes from limited data as it is difficult to collect a large amount of data owing to security concerns. To resolve these two complications, this study utilized a preprocessing method that changed various types of images into standardized images, and a data augmentation method that artificially expanded the data to avoid the problem of overfitting that can occur while using a small dataset during deep learning.

2.1. Contrast Limited Adaptive Histogram Equalization

In weld images that have been captured by radiographic testing, there are cases where the weld zone cannot be seen clearly as the image is too dark, and it is difficult to extract the weld bead shape. Therefore, this study used a histogram equalization (HE) method as a preprocessing technique to improve the contrast of the image by changing the image’s pixel distribution. HE adjusts the image’s histogram so that the pixel intensities that are concentrated within a certain range are distributed evenly. However, because HE redistributes the pixels of the image using a single histogram, it can only be performed if the pixel distribution is the same throughout the entire image. Furthermore, because various bright and dark areas exist in the image, the image may become overly bright or distorted when HE is performed. To compensate for this, the contrast-limited adaptive histogram equalization (CLAHE) [12] was developed. CLAHE is a method that divides the image into several blocks of a fixed size and performs HE on each block to compensate for the challenges of the normal HE. As shown in Figure 1, it undergoes a process of redistributing the pixel intensities that are above a certain height in the histogram. A value that limits a certain height in the histogram is called clip limit. Clip limit depends on the normalization of the histogram and thereby on the size of the neighborhood region, and is usually set between 3 and 4.

2.2. Data Augmentation

Data augmentation is a technique used during deep learning that artificially expands the dataset when there is a small amount of data available for training. Using an insufficient amount of data for deep learning can cause overfitting, which may reduce the deep learning performance. Moreover, for a deep learning model to generalize the test data efficiently, more data must be used for training. Therefore, it is necessary to expand and diversify the data based on the dataset that has already been procured to improve the learning performance of deep learning. Standard data augmentation methods include various techniques such as rotating, cropping, flipping, and adding noise to images. The data conversion efficiency of these methods has already been verified in several studies [13].

2.3. U-Net

In recent years, several studies have been conducted on image segmentation techniques owing to the increasing interest in self-driving vehicles and object recognition [14]. Deep learning models are often used for video and image segmentation. U-Net is an end-to-end fully convolutional network-based model [15] designed for image segmentation, and has been used for image segmentation in applications such as X-ray and magnetic resonance imaging (MRI) [16].

U-net has a U-shaped network configuration, as shown in Figure 2. As it uses a convolutional neural network (CNN) [17] structure, it contains several layers and extracts features from images. After extracting the features, transpose convolution is used to begin the image segmentation based on these features. The U-shaped structure can be broadly divided into three parts. The first is the contracting path, which examines a broad range of image pixels and extracts context information. The second is the expanding path, which combines context information with pixel location information (localization) to distinguish the object to which each pixel belongs. The third is the bottleneck, which converts the contracting path to the expanding path.

In the contracting path, a downsampling process is repeated to generate a feature map. The bottleneck is the segment that converts the contracting path to an expanding path. In the expanding path, the feature map is upsampled. Additionally, the expanding path performs the role of combining the contextual information that was generated in the contracting path with the location information using skip connections. If the size of an image is reduced and increased again, detailed pixel information is lost, which is a major challenge in segmentation requiring dense pixel-by-pixel predictions. Clearer image results are obtained from the decoder part through the skip connections, which directly pass important information from the encoder (contracting path) to the decoder (expanding path), allowing for more accurate predictions.

3. Data and Training

3.1. Data Configuration

In this study, 500 weld images were obtained for use in the experiments, and the CLAHE image preprocessing technique was used to improve the contrast of the weld images. The 500 weld images were from three in three different shipyards and each shipyard had own radiographic testing environment such as X-ray energy. Figure 3a,b show the original image and the image after CLAHE was performed, respectively. In the images presented in this paper, there is a white rectangular box, which covers the information about the radiographic image for security. Figure 4a,b show the histograms of the original image and the image after CLAHE was performed, respectively. Furthermore, among data augmentation techniques, rotating, cropping, flipping, and brightness adjustment were used to expand the weld images to a total of 2500 images.

This study also used mask images as training data for the U-Net model. Mask images are created by performing a labeling process. These mask images only contained pixel intensities of 0 or 1, where 0 indicates the area outside bead and 1 indicates the bead area. Figure 5 shows a binary mask image in which labeling is performed on the image presented in Figure 3b.

When image data are read, the images are saved in PNG and JPEG formats, and the label information is saved separately. Additionally, when images are read in the PNG or JPEG format and encoded and decoded several times, significant performance reductions occur in the training stage, which become an inefficient part of the process.

To perform the training efficiently and improve the speed, the weld images were combined with the mask images that had gone through the labeling process and contained only the 0 and 1 values, and the Tfrecord files were created. A Tfrecord file is a binary data format for storing TensorFlow training data. Tfrecord files allow the management of the training data and the label information in one file. Tasks such as encoding and decoding can be batch processed in a single file, eliminating the need for additional work and increasing the training efficiency and speed [18]. The weld images and mask images are encoded into a Tfrecord file, and the width and height of the image are additionally saved. Additionally, the encoding format is set to bytes.

The most common method for evaluating deep learning models is to divide and use the collected data [19]. The 500 raw images and 2500 images after data augmentation was performed were divided and specified as the training and test dataset: 350 and 1750 images were used for training, and the remaining 150 and 750 images were used for testing. Of the collected data, the training data were used in the model training process. The test data were used to deduce the model’s performance; that is, how well the model learned using the training data.

3.2. Deep Learning Model Training

U-Net was used to extract the features from the weld images and predict the shape and location of the weld beads. For training, U-Net received weld images and mask images that would become the training target images. The original weld images and mask images that would become the training target images were combined into the Tfrecord files, and these were used as the training dataset. High-quality images reduce the training speed, making it difficult to achieve good performance as there are many parameters that must be learned. Therefore, to perform training efficiently, the image pixel sizes were reduced to 256 × 256 before training was performed. Additionally, the input images were normalized to (0,1), and the activation function of each layer was set up so that ReLU [20] was used to apply the normalized values. The training was performed with a batch size of 1 and a learning rate of 0.00001; 18,000 iterations were performed. Binary cross entropy [21] was used as the loss function, and Adam [22] was used as the optimization function. Algorithm 1 shows a detailed algorithm.

Table 1. Experimental environment.

	Value
OS	Window 10 64 bit
CPU	Intel(R) Core(TM) [email protected] GHz
RAN	32 GB
GPU	NVIDIA TITAN Xp

presents hardware and software for the experimental environment. Depending on the hyperparameters and the experimental environment, the time required for the experiment varies. The model was trained using only the training dataset, and the trained model was tested using the test dataset. Figure 6 shows the results of testing the model that trained 2500 images in which CLAHE and data augmentation are performed: (a) is the actual weld image used as the input, (b) is the actual mask image, and (c) is the mask image that was predicted via the training process. Figure 7 shows the loss functions of the training and test data. Figure 7a shows the loss function when training 500 raw images, Figure 7b shows the loss function when training 500 images in which CLAHE is performed, Figure 7c shows the loss function training 2500 images in which CLAHE and data augmentation are performed. From the loss function, it can be determined that overfitting did not occur in the model.

Algorithm 1. Learning for image segmentation.

Input: Image pixel feature $\left(X_i,y_i\right)$, $i=1, \dots , N$, and training step $t$.

Output: U-net model parameters $W$, Predicted values $\widehat{y}$.

Loss function:

$BCE=-\frac{1}{N}{\displaystyle \sum }_{i=1}^N{y}_i\times log\left(p\left(y_i\right)\right)+\left(1-y_i\right)\times log\left(1-p\left(y_i\right)\right)$

Optimization: Adam

Require: $\alpha$: Step size

Require: ${\beta }_1,{\beta }_2 \in \left[0,1\right]$

Require: Model parameters $W$

$m_0=0$ (Initializer initial $1^{\mathrm{st}}$ moment vector)

$v_0=0$ (Initializer initial $2^{\mathrm{nd}}$ moment vector)

$t=0$ (Initializer timestep)

While $W_t$ not converged do

$t=t+1$

$g_t={\nabla }_t{f}_t\left(W_{t-1}\right)$

$l{r}_t=l{r}_{t-1}\times \sqrt{1-{\beta }_2^t}/\left(1-{\beta }_1^t\right)$

$m_t={\beta }_1\times m_{t-1}+\left(1-{\beta }_1\right)\times g_t$

$v_t={\beta }_2\times v_{t-1}+\left(1-{\beta }_2\right)\times g_t^2$

$W_t=W_{t-1}-l{r}_t\times m_t/\left(\sqrt{v_t}+ϵ\right)$

End While

Return $W_t$ (Resulting parameters)

4. Results

4.1. Experimental Results

To evaluate the performance of the image segmentation model, this study used the intersection over union (IoU), mean intersection over union (mIoU), and pixel accuracy [23], which are evaluation methods from the PASCAL visual object classes (PASCAL VOC) [24]. The values of each of these parameters were obtained for each pixel.

Table 2. Confusion matrix.

	Predicted Positive	Predicted Negative
Actually Positive	True positive (TP)	False negative (FN)
Actually Negative	False positive (FP)	True negative (TN)

depicts a comparison of the actual and predicted values to measure the performance of the trained model. The IoU is an evaluation index that measures how well the predicted bead shape and location in the weld image match the actual bead shape and location. Specifically, the IoU measures how well the areas with a pixel intensity of 1 match when comparing the mask image and the predicted image, as shown in Equation (1). The mIoU is the average of how well each of the bead areas and base metal areas match, as shown in Equation (2). The pixel accuracy is a number that expresses the accuracy of the prediction with respect to the entire base metal in the weld image, as shown in Equation (3).

Table 3. Results of experiment: (a) the result of 500 raw images; (b) the result of 500 images in which CLAHE is performed; (c) the result of 2500 images in which CLAHE and data augmentation are performed.

		IoU	mIoU	Accuracy
(a)	Train	83.00%	89.10%	97.48%
	Test	73.58%	83.47%	95.53%
(b)	Train	83.24%	89.93%	97.58%
	Test	75.01%	84.88%	95.61%
(c)	Train	83.78%	90.58%	98.01%
	Test	76.01%	85.44%	95.90%

presents the results for the training and test data. In Table 3, (a) is the result of training with 500 raw images, (b) is the result of training with 500 images in which CLAEH is performed, and (c) is the result of training with 2500 images in which CLAHE and data augmentation are performed.

$$IoU=\frac{TP}{TP+FP+FN}$$

(1)

$$mIoU=\left(\frac{TP}{TP+FP+FN}+\frac{TN}{TN+FP+FN}\right)/2$$

(2)

$$Pixel accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(3)

Figure 8 shows the predicted mask image results when only the weld images were input to the trained model without the mask images. Figure 8a shows the image that was input to the trained model, and Figure 8b shows the predicted shape and location of the bead in the input image.

To extract only the weld bead from the predicted bead shape image as a post-processing operation, only the area with a pixel intensity of one was used; however, when detecting the bead, problems can occur owing to noise in the image. To resolve these problems, the algorithm shown in Algorithm 2 was used to find the largest area among the areas with a pixel intensity of one and perform a bounding box labeling operation on the weld image, as shown in Figure 9. The coordinates of the four edges of the bounding box were determined, and the image was cropped to extract only the weld bead, as shown in Figure 10.

Algorithm 2. Algorithm for cropping image.

Input: Raw image, mask area coordinates:

$M=\left(x_i^1,y_i^1,x_i^2,y_i^2\right)$

Output: bead image

Calculate areas of $M$: ${\widehat{x}}_i^1$: min$\left(x_i^1,x_i^2\right)$ ${\widehat{x}}_i^2$: max$\left(x_i^1,x_i^2\right)$ ${\widehat{y}}_i^1$: min$\left(y_i^1,y_i^2\right)$ $y_i^2$: max$\left(y_i^1,y_i^2\right)$ $a_{i }=\left({\widehat{y}}_i^2-{\widehat{y}}_i^1\right)\times \left({\widehat{x}}_i^2-{\widehat{x}}_i^1\right)$ $A=\left\{a_1,a_2,\cdots ,a_{i }\right\}$

2 Extract the largest area coordinates: $A_{max}=\max \left(A\right)$ $M_{max}=\left(x_m^1,y_m^1,x_m^2,y_m^2\right)$

3 Draw bounding box in mask image: Rectangle$\left(\left(x_m^1,y_m^1\right),\left(x_m^2,y_m^2\right)\right)$

4 Cropping image: Bead image = Raw image $\left[y_m^2:y_m^1, x_m^2:x_m^1\right]$

4.2. Results of Image Segmentation with Other Algorithms

The image segmentation method presented in this study was compared with other algorithms. Two algorithms were used for comparison. The first method is k-means clustering [25]. The k-means algorithm is a clustering algorithm that divides given data into k regions. In the field of image segmentation, it refers to a process of dividing an image into several parts; that is, a set of pixels. The second method is image thresholding [26], which is often used in image processing. Image thresholding is one of the simple methods of image segmentation, which distinguishes objects and backgrounds in an image and resets the pixel intensity with only two intensities. Two methods were performed on the same weld image as in Figure 9. Figure 11 shows an image in which k-means algorithm is performed. Figure 12 shows an image in which image thresholding is performed. When image segmentation was performed in two methods, it was difficult to extract only beads from the image. In the radiograph image, information of the image was also indicated in text form. The k-means and image thresholding methods determined the bead and text as the same area and segmented the image. Additionally, the border of the image was determined to be the same area as the bead. Accurate image segmentation was impossible because the contrast of the image varies depending on the light transmittance and a specific area may become too bright. Figure 13a,b shows the binary images of Figure 11 and Figure 12, respectively. To evaluate the performance of the algorithms, binary images were compared with the true mask image set to obtain IoU, mIoU, and pixel accuracy.

Table 4. Results of comparing with the true mask image set: (a) the result of images in which k-means is performed; (b) the result of images in which thresholding is performed.

	IoU	mIoU	Accuracy
(a)	20.06%	40.22%	44.18%
(b)	18.34%	29.93%	38.72%

presents the performance evaluation results as average values for the image set. In Table 4, (a) is the result of images in which k-means is performed, and (b) is the result of images in which thresholding is performed.

5. Conclusions

To efficiently inspect and determine defects and inadequate of the weld zone, a method of extracting only the weld bead from radiographic testing image was conducted.

The contrast of the weld image was improved using CLAHE as an image preprocessing operation, so that the weld zone became more prominent. Additionally, by performing data augmentation, 500 weld images were expanded to a total of 2500 images. The U-Net deep learning algorithm was used as an algorithm for extracting weld beads. After the training, the performance of the three models was compared.

The mIoU values for training and test data derived from model trained with images applied with CLAHE and data augmentation were 90.58% and 85.44%, respectively, and those for training and test data derived from model trained with images applied with only CLAHE were 89.93% and 84.88%, respectively. On the other hand, the mIoU values for training and test data derived from model with raw images were 89.10% and 83.47%, showing the lowest values. The deep learning performance was improved by applying CLAHE and data augmentation.

In the radiographic testing images, information about the image as well as weld bead was marked. Additionally, although the proportion of weld bead in the image is not large, the weld bead could be effectively extracted as a result of image post-processing based on the trained results.

The method presented in this study was compared with the results of general image segmentation methods. Since general image segmentation methods determined that beads and text were the same area in the image, it was difficult to extract only beads. In contrast, it was confirmed that U-Net could extract only beads from images and produced better results than other image segmentation methods.

Five hundred radiographic testing images were used in the experiment, and a higher degree of accuracy for a wider variety of weld image types can be achieved by procuring additional images in the future. Additionally, it will be possible to reduce the time and cost of the weld quality inspections by automating the extraction of the weld bead shapes.