Quality Assessment of Tire Shearography Images via Ensemble Hybrid Faster Region-Based ConvNets

Abstract

In recent times, the application of enabling technologies such as digital shearography combined with deep learning approaches in the smart quality assessment of tires, which leads to intelligent tire manufacturing practices with automated defects detection. Digital shearography is a prominent approach that can be employed for identifying the defects in tires, usually not visible to human eyes. In this research, the bubble defects in tire shearography images are detected using a unique ensemble hybrid amalgamation of the convolutional neural networks/ConvNets with high-performance Faster Region-based convolutional neural networks. It can be noticed that the routine of region-proposal generation along with object detection is accomplished using the ConvNets. Primarily, the sliding window based ConvNets are utilized in the proposed model for dividing the input shearography images into regions, in order to identify the bubble defects. Subsequently, this is followed by implementing the Faster Region-based ConvNets for identifying the bubble defects in the tire shearography images and further, it also helps to minimize the false-positive ratio (sometimes referred to as the false alarm ratio). Moreover, it is evident from the experimental results that the proposed hybrid model offers a cent percent detection of bubble defects in the tire shearography images. Also, it can be witnessed that the false-positive ratio gets minimized to 18 percent.

1. Introduction

Industry 4.0 is the novel digital technology meant for industries, and this paradigm enables the communication, collection, and analysis of data through machines, thereby allowing quicker, more agile, and efficient processes for making superior quality goods with minimal expenditure. Moreover, this digital industrial technology will assist in enhancing productivity, enabling industrial development, and revamping the profile of the personnel involved, thereby nurturing changes in the competence of business organizations and states. Further, this paradigm will foster superior efficiencies and will modify the conventional production associations between the suppliers, manufacturers, and clients and also the communication amongst humans and machines. Besides, due to the phenomenal growth in technology and agile expansion of Industry 4.0, several manufacturing firms have embraced automation, thereby replacing labor-intensive tasks in conventional production units [1]. Also, it can be observed that enabling technologies for smart tire quality assessment for realizing intelligent tire manufacturing has been gaining prominence to address challenges such as automated tire defects detection.

Generally, it can be witnessed that in the modern-day scenario, there is a humongous and swift growth in the gadgets, equipment, and devices connected to the internet. These devices, gadgets, and equipment have profound computing characteristics, and at the same time, they are exceedingly performance-oriented [2]. Due to all these facts, the concept of deep learning has evolved into a newer dimension, and it plays a significant part in processing and recognizing images, speech, and video, and so on. Furthermore, the implementation of a deep learning paradigm for automation in conventional manufacturing units significantly minimizes the usage of physical labor, and it also enhances the overall competence and efficacy of these units. Digital shearography is a laser-based measuring approach that relies on the processing of digital data, interferometry, and phase-shifting paradigm [3,4,5].

A shearography system was developed by applying a spatial light modulator for controlling the amount of shearing and the direction of the phase light automatically and accurately. The system eliminates the nonlinear random error and enhances the efficiency of testing [6]. A system was developed using shearography to examine the exterior heatproof covering of a cylinder, and defect detection was done using an artificial intelligence-based recognition algorithm for deep learning, namely Faster R-CNN, which detects the bounding defects [7]. A model was developed using a deep convolutional neural network for the detection of defects captured in the X-ray images. Moreover, the fully convolutional network was selected for the pixel-wise prediction of defect location and segmentation [8]. A binary classification model was established using a convolutional neural network to classify the defects in pipes into two classes, such as minor defects and major defects, and the case study was conducted from 256 shearography images [9]. An algorithm was developed based on deep learning for the classification of defects in tires using the proposed multi-column convolutional neural network (CNN) by integrating several CNNs [10].

Our research primarily focuses on the deep learning-based bubble defects detection in tire shearography images, which plays a significant role in the automation of the tire manufacturing industries. Figure 1 portrays the digital shearography set-up for capturing tire shearography images utilizing a digital computer.

The tire manufacturing process includes five substantial stages compounding, mixing, shaping, and vulcanizing, testing. Further, in the course of tire production, there is a chance for the bubble defects to appear in the tires, as the air might not be entirely removed from the tires.

Consequently, when the car is driven at high speed, such defective tires seem to suffer a greater chance of bursting, hence this scenario might place the human lives at risk. Therefore, in order to overcome this issue, the tire manufacturing units ensure that the tire is tested successfully by using several quality control mechanisms, prior to its dispatch and delivery. During the testing process, detecting the bubble defects that are present internally within the tires, turns out to be a substantial task. Moreover, the bubble defects in the four different shearography images are portrayed in Figure 2. The operator identifies the bubble detects in the shearography images and assesses whether the size of the bubble is passable. Nevertheless, the quality of inspection significantly relies on the experience and expertise of the operator. As a result, the bubble defect detection process might need substantial human expertise. Besides, the lassitude of the operator might lead to poor judgment and discrepancies in detecting the defects.

Since 2012, there has been rapid progress in the convolutional neural networks-based research on image, visual, and computer-vision-based tasks [11,12]. The approach presented in the preliminary study [13] establishes two CNN architectures for tire bubble-defects diagnosis. Although, the scheme presented in [13] provides a precise identification of the bubble defects, however, the false alarm ratio, also known as the false positive ratio—more than twenty percent—is significant. The Faster R-CNN comprises two networks, primarily for generating the region proposals it makes use of a region proposal network (RPN) and secondly, a network that utilizes these region proposals for detecting the bubble defects [14].

The key contributions of this work are summarized as follows:

• The substantial contribution of this work lies in improving the architecture established earlier in [13] for effectively realizing intelligent tire manufacturing with automated defects detection.
• A Faster Region-based convolutional neural networks (R-CNN) is combined along with the architecture described in [13] for minimizing the false positive ratio.
• Further, this significant modification to the CNN architecture helps in minimizing the labor cost involved in the tire manufacturing industry.
• The results of the proposed hybrid model indicate that this approach asserts a hundred percent detection of bubble defects in the tire shearography images.
• From the results, it can be perceived that the false alarm ratio can be minimized to 18 percent.

2. Materials and Methods

A two-stage hybrid model for detecting bubble defects in tires is proposed in this work. The primary stage includes a CNN architecture for diagnosing tire bubble-defects, and the second stage makes use of a Faster Region-based ConvNets architecture for minimizing the false positive or the false alarm ratio (FAR). A flow diagram of the proposed two-stage ensemble hybrid model is portrayed in Figure 3.

2.1. Faster Region-Based Convolutional Neural Networks

A model referred to as Regions with CNN features (R-CNN), which is a scalable object detection approach that enhances the mean average precision, was established by the research in [15]. In the research desscribed in [16], another improved version of the R-CNN model known as the Fast R-CNN was deployed with various novelties for enhancing the training and testing speed at the same time augmenting the accuracy of detection. Further, the work presented in [17] established a Faster R-CNN that introduced an RPN which shares the convolutional features of the full image with the network responsible for detection; hence, this ensures that the region proposals are achieved at a low cost. It can be observed that the RPN approach is deployed instead of the Selective Search (SS) technique [18] in the case of Faster R-CNN/ Faster Region-based ConvNets. Further, this method considerably reduces the time-period necessary for extracting the candidate regions and also for increasing the overall efficiency. Figure 4 illustrates the architectural model of the Faster R-CNN network.

The Faster R-CNN network with a ZF-net exhibits the detection results with an accuracy of 59.9% for the PASCAL VOC 2007 test set [19,20,21]. Besides, for the same test set, the Faster R-CNN network with VGG16 architecture accomplishes the detection results with 73.2% accuracy [19,20,21]. Henceforth, it can be observed that the Faster R-CNN with VGG16 architecture achieves superior accuracy, which makes it the most sought after approach. Moreover, this technique is utilized in this research to enhance the detection accuracy of the tire bubble defects. In Figure 5, the architectural model of the fully convolutional region proposal network [22] is depicted.

Figure 5 portrays the fact that the fully convolutional region proposal network applies a 3 × 3 window over the feature maps received from the ConvNets. Subsequently, for assessing the candidate regions, we make use of the anchors with various areas and ratios. Additionally, the chosen candidate expanses are placed into the 256-dimensional trajectory, and further, they are passed on as the inputs to the box regression layer (reg) and a box-class layer (cls). For each proposal, the outcome of the box-class layer approximates the target object or the non-target object probabilities. Consequently, a positive label will be allocated for an anchor with an Intersection-over Union (IoU) overlay proportion more significant than the value 0.7 in comparison to some ground truth box. Besides, the negative label will be allocated for the non-positive anchor with an Intersection-over Union proportion lesser than 0.3 for the remaining ground truth boxes. It can be clearly noted that the anchors which are neither positive nor negative have no role in the training for accomplishing the target. In the box regression layer, the positive sample co-ordinates achieved by the box-class layer are modified to suit the ground truth’s bounding box aptly.

2.2. Image Enhancement

The classification capability and competence of the convolutional neural networks rely heavily on the two vital parameters, namely, the quality and quantity of the training samples. Nevertheless, the arduous task for this research is the identification of speckle patterns encompassing the bubble defects. In order to overcome this issue; hence, the blocks from the speckle patters encompassing the bubble defects were randomly chosen. Also, the chosen data were rotated horizontally and vertically, and then the resultant dataset helps in achieving the essential dataset required for the training process. The imperfect bubble blocks were detached physically. In this way, this research could achieve about ten times the training data. Hence, this approach makes sure that the patterns of the tire bubble defects were adequate for the training process.

2.3. Classification of the Bubble Defects in Tires

It can be noticed from [13] that two convolutional neural network architectures were established for diagnosing the bubble-defects available in the treads and sidewalls of the tires. Though this approach accurately classifies the tire bubble-defects, nevertheless, the FAR seems marginally more significant than 20 percentage. Thus, our work enhances the approach in [13] by incorporating a Faster-RCNN network for reducing the false alarm ratio. The modified hybrid Faster Region-Based Convolutional Neural Networks architecture is illustrated in Figure 6. The various components of the proposed hybrid model are presented in

Table 1. The various components of the proposed hybrid model.

Convolutional Neural Networks
(a) Tread of Tires	(b) Sidewall of Tires
ConvNet3-16 ConvNet3-16	ConvNet3-16 ConvNet3-16
Max-pooling process
ConvNet3-32 ConvNet3-32	ConvNet3-32 ConvNet3-32
Max-pooling process
ConvNet3-64 ConvNet3-64	ConvNet3-64 ConvNet3-64 ConvNet3-64
Max-pooling process
ConvNet3-128 ConvNet3-128 ConvNet3-128	ConvNet3-128 ConvNet3-128 ConvNet3-128
Max-pooling process
Fully Connected-1000
Fully Connected-1000
Fully Connected-2
Softargmax function
Faster Region-based Convolutional Neural Networks
ConvNet3-64 ConvNet3-64
Max-pooling Process
ConvNet3-128 ConvNet3-128
Max-pooling Process
ConvNet3-256 ConvNet3-256 ConvNet3-256
Max-pooling Process
ConvNet3-256 ConvNet3-256 ConvNet3-256
ConvNet3-512 ConvNet3-512 ConvNet3-512
ConvNet3-512
Reshape process
Soft-max function
Reshape process
Proposal
ROI pooling layer
Full-connection
Bbox_pred	Softmax function
	Cls_prob

. In the CNN, the hyper-parameters settings for tire tread are learning rate = 0.01, epoch = 30,000, batch size = 40, gamma = 0.001, power = 0.75, and momentum = 0.9. The hyper-parameters settings for tire sidewall are learning rate = 0.00001, epoch = 18,000, batch size = 18, gamma = 0.001, power = 0.75, and momentum = 0.9. In the Faster-RCNN, the learning rate, step size, and momentum are set as 0.00001, 50,000, and 0.9, respectively, for both tire tread and tire sidewall.

2.4. The Sliding Window Phase

In this work, the original shearography image had a size of 1360 × 1024 pixels. In order to facilitate bubble defect detection, the shearography tire images are fragmented into a variety of blocks via the sliding window phase. Subsequently, it is evident that the location of the tire bubble defects is not known; consecutive sliding windows with 50% overlapping regions for the extraction of speckle patterns are used to avoid fragmenting the bubble defects and causing erroneous results.

The overlapping threshold has been selected to poise the time-period required for processing and also for the efficient detection of bubble defects. The abstract depiction of the sliding window indicating the overlap is presented in Figure 7.

Further, this research establishes a classifier that performs the process of detection of bubble defects in treads and sidewalls of tires, which is presented in Section 2.3. Moreover, the sliding window described in Section 2.4 is used to check the segmented block images sequentially for bubble defects. If the classifier classifies a block as containing bubbles, the Faster R-CNN is used to determine if the result is a false positive. If the resultant image is not false positive, then in the original image, the respective location of the block is encircled. As a result, this image is passed on to the professional operators for assessing the quality of the tires and also for removing the defective piece. Furthermore, this devised semi-automated assessment process offers cost-leadership when compared with the traditional manual inspection and also improves the reliability of the inspection process.

3. Results

In this work, the diagnosis of bubble defects in tires established in [13] and the Faster Region-based Convolutional Neural Networks approach is amalgamated for obtaining 100% detection of defects and also aiding in reducing the false alarm ratio. The evaluation metrics, such as the accuracy, sensitivity, and specificity, are used for assessing the performance of the proposed hybrid model. These metrics are computed using the following expressions:

$$Accuracy=\left(TP+TN\right)/\left(P+N\right)$$

(1)

$$Sensitivity=TP/P$$

(2)

$$Specificity=TN/N$$

(3)

where $TP$ stands for true positive, it represents the amount of diagnosed bubble patterns, which really possesses the bubble defects, and $FP$ stands for false-positive and, it indicates the amount of not bubble patterns, which are wrongly diagnosed as bubble defects. True negative $\left(TN\right)$ illustrates the amount of not bubble patterns, which are diagnosed as not bubble defects. Positives $\left(P\right)$ represents the real bubble defects and negatives $\left(N\right)$ denotes the not bubble defects. Among the evaluation metrics, sensitivity is the most necessary measure for achieving the complete detection of bubble defects.

Moreover, a tire company provided the shearography images deployed in this research. Usually, the tire bubble defects were physically delineated with the assistance of experienced professionals. The amount of training images and blocks are clearly organized in

Table 2. Training Dataset Details.

	Tire Treads		Tire Sidewalls
	No. of Images	No. of Blocks	No. of Images	No. of Blocks
Shearography without bubble	1409	8811	1545	10514
Shearography with bubbles	223	8596	102	5052

. Further, it is evident that for the training process, the tire manufacturer supplied the 325 tire shearography images with bubble defects. Subsequently, the image enhancement approach is deployed for imitating 8596 and 5052 blocks from 223 tire tread images and 102 tire sidewall images containing bubble defects. Additionally,

Table 3. Testing Dataset Details.

	Tire Treads	Tire Sidewalls
	No. of Images	No. of Images
Shearography without bubble	262	279
Shearography with bubbles	136	120

indicates the test dataset, it comprises of 541 tire shearography images deprived of bubble defects and 256 tire shearography images having bubble defects.

An area with bubble defects is expected to be smaller than the area of the default anchor of the Faster R-CNN. Therefore, in this work, the anchor’s ratio and scale are adjusted according to the area of the bubble defects.

Table 4. Anchors Configuration.

	Original	Proposed Ensemble Hybrid Model
Ratios	[0.5,1,2]	[0.3,0.4,0.5,0.75]
Scale	[8,16,32]	[8,16,32]

shows the ratio and scale adjustment of the anchors. Twelve anchor configurations are used for candidate regions in the marking of bubbles.

The proposed hybrid model has been compared with various classifiers including the Support Vector Machine (SVM) [23], Random Forest Model [24], Haar-like AdaBoost Method [25], Chang’s method [13], and the integrated model comprising of SVM, Random Forest Model, AdaBoost method. Besides, the proposed model was compared with these methods for verifying its performance.

Table 5. Diagnosis of Bubble-defects in Treads of Tire Shearography Images.

Measurement Methods	Accuracy (%)	Sensitivity (%)	Specificity (%)
Support Vector Machine [23]	55.53	92.65	36.26
Random-Forest Model [24]	59.3	96.32	40.08
Haar-like Ada-Boost Method [25]	62.81	97.06	45.04
Integrated Model comprising of Support Vector Machine, Random-Forest Model, Ada-Boost Method	79.15	96.32	70.23
Chang’s method [13]	87.94	100	81.68
Proposed Hybrid Faster Region-based Convolutional Neural Networks Model	89.16	100	83.09

illustrates the diagnosis of bubble-defects in treads of tire shearography images for several existing methods in comparison with the proposed ensemble hybrid model in terms of the evaluation metrics such as accuracy, sensitivity, and specificity. Additionally,

Table 6. Diagnosis of Bubble-defects in Sidewalls of Tire Shearography Images.

Measurement Methods	Accuracy (%)	Sensitivity (%)	Specificity (%)
Support Vector Machine [23]	50.13	81.67	36.56
Random-Forest Model [24]	44.61	85.83	26.88
Haar-like Ada-Boost Method [25]	46.37	82.5	30.82
Integrated Model comprising of Support Vector Machine, Random-Forest Model, Ada-Boost Method	61.9	85	51.97
Chang’s method [13]	85.46	100	79.21
Proposed Hybrid Faster Region-based Convolutional Neural Networks Model	86.87	100	80.15

depicts the diagnosis of bubble-defects in sidewalls of tire shearography images for numerous prevailing approaches in comparison with the proposed ensemble hybrid model in terms of the assessment metrics such as accuracy, sensitivity, and specificity. Further, it can be witnessed from these tables that the work in [13] and the presented ensemble hybrid approach achieve 100 percent sensitivity, by successfully identifying each and every bubble-defect. Also, it can be observed that the presented ensemble hybrid approach surpasses all other existing approaches in terms of specificity. However, the presented ensemble hybrid approach requires a processing time of approximately 7 seconds/image, whereas the approach established in [13] takes only a processing-time of roughly 6 seconds/image. Nevertheless, the presented ensemble hybrid model is superior in other means and also in terms of specificity, when compared with the other existing approaches.

Figure 8a–d illustrate the shearography images or the speckle patterns acquired using digital shearography, and Figure 8e–h depict the detection of bubble defects in tires using the proposed hybrid Faster Region-based convolutional neural networks model. Figure 8e–h indicate the fact that all bubble defects in tires have been detected successfully. Figure 9a–d depict the false positive or the false alarm inspection results in [13], where the shearography images do not have bubble defects; however, they get misrepresented as possessing the bubble defects. Figure 9e–h illustrate the assessment results of the hybrid Faster Region-based convolutional neural networks model using the same set of input images. It can be witnessed in Figure 9e–h that the shearography images have no bubble defects. Besides, it reveals the fact that the proposed hybrid Faster Region-based convolutional neural networks model effectively reduces the false-positive ratio or the false alarm rate.

4. Conclusions

In the tire manufacturing process, the diagnosis of bubble-defects in the treads and sidewalls of shearography tire images represents a significant task. Therefore, enabling smart tire quality assessment seems to be an essential way of realizing intelligent tire manufacturing practices that can ensure automated detection of defects. Further, an ensemble hybrid combination of the CNN with a high-performance Faster Region-based ConvNets for classifying and diagnosing the bubble-defects present in the tire shearography images. The proposed hybrid Faster Region-based convolutional neural networks model reduces misjudgments caused by human errors and achieves high consistency in the quality of bubble-defect detection. It is clearly evident from the results that in addition to thoroughly diagnosing the bubble-defects in tires, the hybrid Faster Region-based convolutional neural networks model decreases the false alarm ratio of not-bubble defects in tires from 20% to a rate of 18%. Also, it has to be noted that this hybrid system model was deployed in a tire manufacturing unit, and it produced efficient results in automatically diagnosing the bubble-defects in treads and sidewalls of tires. In the future work, more advanced CNN enabled approaches can be implemented for automated detection of defects [26,27,28,29,30], thus ensuring and realizing a sustainable tire manufacturing process.