Apple Surface Defect Detection Based on Gray Level Co-Occurrence Matrix and Retinex Image Enhancement

Abstract

Aiming at the problems of uneven light reflectivity on the spherical surface and high similarity between the stems/calyxes and scars that exist in the detection of surface defects in apples, this paper proposed a defect detection method based on image segmentation and stem/calyx recognition to realize the detection and recognition of surface defects in apples. Preliminary defect segmentation results were obtained by eliminating the interference of light reflection inhomogeneity through adaptive bilateral filtering-based single-scale Retinex (SSR) luminance correction and using adaptive gamma correction to enhance the Retinex reflective layer, and later segmenting the Retinex reflective layer by using a region-growing algorithm. The texture features of apple surface defects under different image processing methods were analyzed based on the gray level co-occurrence matrix, and a support vector machine was introduced for binary classification to differentiate between stems/calyxes and scars. Deploying the proposed defect detection method into the embedded device OpenMV4H7Plus, the accuracy of stem/calyx recognition reached 93.7%, and the accuracy of scar detection reached 94.2%. It has conclusively been shown that the proposed defect detection method can effectively detect apple surface defects in the presence of uneven light reflectivity and stem/calyx interference.

1. Introduction

In China, there are about six main types of apples grown, including Red Fuji, Gala, and Yellow Marshal, to name a few. The skin color is mostly red and greenish-yellow. China’s total apple production reached 45.973 million tons in 2021 [1]. This amount of production causes difficulties for traditional manual defect detection methods. Surface defects are also an entry point and breeding ground for disease, which can cause severe economic losses during the production and storage of apples [2].

Much of the current literature on defect detection pays particular attention to machine vision and image processing techniques. Azgomi et al., utilized a multilayer perceptual neural network to detect apple surface defects based on color and texture features [3]. Lu et al. utilized DC and AC images and their combined machine learning classification methods for enhanced detection of surface and subsurface defects in apples [4]. Zhang et al., extracted color features in HSV color space and texture features in RGB color space, respectively, to establish a deep multiscale dual-channel convolutional neural network for detecting surface defects of apples [5]. Kahraman et al. analyzed the advantages and disadvantages of deep learning methods for fabric defect detection [6]. Zhang et al. utilized a complete convolutional network based on deep learning to detect internal damage in blueberries based on hyperspectral transmittance images [7]. Ismail et al., detected fruit freshness by deep learning and stacking ensemble methods [8]. Xie et al., proposed a lightweight model based on machine vision combined with DCNN for detecting defective carrots [9]. Hu et al. optimized the YOLO V5 model to detect citrus surface defects by integrating an attention mechanism and modifying the loss function [10]. Tian et al. utilized hyperspectral imaging combined with an improved watershed algorithm to detect early citrus rot [11]. Yang et al. utilized the YOLO V3-tiny detection model to detect multispectral images covering 25 bands for potato defects [12]. Rama et al., used a UV-A light source to detect mango anthracnose by early semantic segmentation of healthy and diseased regions of mango images [13].

However, the spherical surface of the apple causes a change in the intensity of light reflection from the surface, the extent of which is determined by the distance of the region from the center region. This is reflected in the image as the grayscale intensity of the edge region of the apple is generally lower than that of the middle part. Therefore, the defects present different grayscale intensities at different locations, which easily interferes with the correct detection of the defects. In addition, the models obtained from the training of the above image segmentation methods are relatively large, which are not favorable to be deployed on embedded real-time inspection devices. The more lightweight FOMO target detection model, however, can only output the defect location and cannot obtain the defect size information. There is, therefore, a definite need for defect detection, which requires an adjustment of the image light reflectivity, while the defect detection method has to consider the possibilities in practical applications.

Most researchers investigating classification issues have utilized artificial neural networks, support vector machines, decision tree classification, etc. Ge et al., classified apple tree organs using a support vector machine based on multi-feature fusion of color and shape [14]. Caceres-Hernandez et al. investigated feature extraction and classification of watermelons based on image, acoustics, and spectroscopy using machine learning methods [15]. Bird et al. proposed a machine learning approach that combines fine-tuning, transfer learning, and generative model-based training data augmentation to improve fruit quality image classification [16]. Moch et al. used a plain Bayesian approach to classify apple defects based on apple texture features [17]. Wang et al. used a deep learning- based YOLO V5 model to recognize the calyx of apple stems using a transfer learning training method [18].

In addition to classification methods, the key research question of stem/calyx recognition is how to extract features to maximize differences between stems/calyxes and scars. Numerous studies have attempted to highlight the defect features by special imaging modalities and use machine vision techniques for defect features. Zhang et al. recognized the calyx of apple stems based on near-infrared linear-array structured lighting and 3D reconstruction techniques [19]. Zhang et al. obtained the basis for distinguishing the stems/calyxes from the scar from near-infrared spectra and utilized machine vision techniques to accomplish apple stem/calyx recognition [20]. Yuan et al. used NIR camera imaging to access the YOLO V5s model to extract defect features [21]. Lin et al. introduced cold excitation to improve the sensitivity of bruise detection, then constructed a simple long-wavelength infrared range (7.5–13 μm) TI system to obtain the thermal image features of bruised apples, and then performed the feature extraction of apple surface defects based on the YOLO V5s network [22]. Li et al. investigated an emerging structured light reflectance imaging technique to extract early decay defect characteristics of navel oranges [23]. Matsui et al. proposed an X-ray based machine vision classification method for detecting avocado stem end rot defects [24]. Zhou et al. utilized a mask region-based convolutional neural network method to detect bruise scar features in strawberry images imaged under incandescent and ultraviolet irradiation [25]. Zheng et al. proposed an Attention Feature Fusion Network based on the U-Net architecture with the addition of a mixing loss and residual refinement module (RRM) to extract the cracking features of jujube [26]. Tian et al. proposed a diameter correction method to reduce the effect of fruit size on the projected spectra and then used a one-dimensional convolutional neural network to detect citrus frostbite features [27]. Min et al. described hyperspectral imaging for defective feature detection and extraction in other berries, such as citrus, apple, and peach, in the early stages of fungal infection [28]. Mahanti et al. presented the advantages and limitations of NDT techniques, including bio-scattering, X-ray imaging, hyperspectral imaging, and thermal imaging for defect feature extraction [29].

Special imaging modalities can be beneficial in highlighting defect features. However, the main weakness of the method is that hardware imaging systems are not easily optimized once they are in place, and imaging conditions are costly to maintain. Therefore, a method for highlighting defect features under simple imaging conditions is needed to enhance the applicability of the inspection system.

Based on this, this paper proposed a defect detection method that relies on image segmentation and stem/calyx identification. The proposed method should not only solve the defect detection problem under uneven light reflectivity but also eliminate the interference of the peduncle calyx on the final results.

The main contributions of this paper are as follows: (1) An apple image surface defected segmentation method based on improved SSR image enhancement and region- growing algorithm was proposed. The method could better cope with the uneven light reflectivity due to the spherical surface of apples and improve the segmentation accuracy. (2) A stem/calyx recognition method based on improved SSR image enhancement and gray level co-occurrence matrix was proposed. The method highlighted the stem/calyx texture through improved SSR image enhancement, then collected the texture featured through a gray level co-occurrence matrix, and later classified the obtained feature data through a support vector machine to achieve the purpose of stem/calyx recognition. Experimentally, it was proved that improved SSR image enhancement could amplify the texture difference between the stem/calyx and the scar and improve the classification accuracy.

The rest of the paper is organized as follows. The second part describes the materials used in this paper and the methods proposed in this paper. Experimental results with specific analysis are given in the third part. Finally, conclusions are drawn in the fourth part.

2. Materials and Methods

2.1. Materials and Instruments

The research sample was drawn from Wuhan’s local fruit market. The types of defects to be measured included apple stems, calyxes, lightly browned scars, and severely browned scars in four categories, as shown in Figure 1. Defects vary in size and color from a lighter brown browning to a darker black browning, with some heavily decayed scars producing white mold.

The apple surface defect detection system is shown in Figure 2. The system consisted of four sections: computer (ASUSAsus Computer Inc. of Taiwan, China, AMD Ryzen 7 5800H CPU @ 3.2 GHz, RAM 16.0 GB), machine vision module OpenMV4H7Plus (Guangzhou Sing Town Information Technology Co. (Guangzhou, China), STM32H743II, 480 MHz, RAM 32 MB, flash 32 MB), OV5640 CMOS (Guangzhou Sing Town Information Technology Co.) image sensor, and led light source.

2.2. Sample Preparation

In this study, apples were placed on a white background plate, and images of apples were captured at different imaging angles, which were used to restore various possible lighting situations. The image was background-segmented according to the histogram bimodal method, and only the apple part of the image was kept as the experimental image.

According to the bimodal nature of the histogram, an iterative method was utilized to automatically find the trough point for the segmentation threshold to enhance its adaptability. However, the lighting situation affects the effect of segmentation of specific components of a given color space. For LAB-B color space for the darker places (apple stemmed, edges) and RGB-B color space for the brighter areas (light produced by the bright spots), there was a certain misjudgment rate. So, the background segmentation results obtained from the two color spaces or operation (math.) to make up for their respective shortcomings were stable to obtain the background segmentation image, as shown in Figure 3.

2.3. Proposed Method

The framework of the algorithm in this paper is shown in Figure 4. First, the R-component map of the background segmentation image was taken to reduce the interference of the apple surface color and magnify the difference between the normal surface and the scar. Secondly, the image gradient was calculated and the weighting coefficients of the bilateral filter were adaptively adjusted according to the local gradient of the filter and the distance of the filter center from the image center for adaptive filtering. Illumination layer separation was performed by Retinex to obtain the reflection layer. Adaptive gamma correction was performed on the reflective layer to improve the internal contrast of the scar. The scar was segmented, the ROI region marked by the region-growing algorithm, and features extracted according to the ROI region and the internal texture of the scar and sent to the support vector machine for classification. According to the classification result, the calyx region was eliminated and only the scar region was retained as the final defect detection result.

2.3.1. SSR Image Enhancement Based on Adaptive Bilateral Filtering

Awareness of Retinex is not recent, having first been described in 1964 by Land et al. [30]. The theory is based on the physiological process by which the human eye perceives the color and luminance of objects. Although the amount of visible light reaching the eye depends on the product of the object’s reflectivity and illuminance, human beings distinguishing the color and brightness of the object is still mainly dependent on the result of the object’s reflection of the visible light. The schematic principle of Retinex is shown in Figure 5.

So, Retinex theory views an image as the multiplication of the amount of reflection and the amount of illumination, i.e.,

$$I\left(x,y\right)=R\left(x,y\right)\times L\left(x,y\right)$$

(1)

where $\left(x,y\right)$ is the two-dimensional coordinates of the image pixel; $I$ is the observed image; $R$ is the reflection component, reflecting the color and luminance information of the object; and $L$ is the illuminance component, reflecting the overall brightness of the environment.

The Retinex theory aims to model the ambient illuminance component image to separate the reflective component image, which represents the object’s own characteristics, from the observed image for the purpose of image enhancement [31]. Logarithmic domain operations not only convert multiplication to addition to speed up the algorithm operation, but also conform to the physiological property of the human eye to perceive the brightness of the environment in a nonlinear manner. So, the reflection component can be derived in the following way:

$$\mathrm{l}og R\left(x,y\right)=\log I\left(x,y\right)-\log L\left(x,y\right)$$

(2)

A specific filter size can only obtain the illuminance and reflection components of a particular feature, which is apparently lacking. Smaller filters retain more information about the details of the image itself, while larger filters can better simulate the real illumination component. Therefore, it is necessary to combine the combined information of various scales for correction. This extends the multi-scale Retinex (MSR) brightness correction method [32], which is calculated as follows:

$$\log R\left(x,y\right)={\displaystyle \sum _{n=1}^N{w}_n\left\{\log I\left(x,y\right)-\log \left[F_n\left(x,y\right)\ast I\left(x,y\right)\right]\right\}}$$

(3)

where $R$ is the output of MSR; $w_n$ is the corresponding weight value of each scale; $I$ is the original image input; $N$ is the number of scales; $F_n$ is the filter function at each scale. The MSR algorithm includes features from multiple scales simultaneously, enabling a more extensive range of dynamic compression [33].

The halo phenomenon, also known as the artifact effect at the edges, arises due to the Retinex theory on the assumptions about the illuminance components. The theory suggests that the light situation changes spatially smoothly, but in fact, at the edges or the junction of light and dark, the light situation changes abruptly [34]. A halo phenomenon is created in the reflection component after further separation of the illumination and reflection components, as shown in Figure 6.

Bilateral filtering is used as the center-surround function of the Retinex algorithm. Compared to Gaussian filtering, bilateral filtering considers both spatial distance differences and color value differences in denoising, thus retaining more details while denoising. In contrast, Gaussian filtering only considers the spatial distance difference, which would lose some details and textures. Therefore, a bilateral filter was used instead of a Gaussian filter as the center-surround function to more accurately restore the spatial illumination variations.

For the Retinex theory, the degree of spatial light reproduction determines how realistic the reflected component is. For scarred areas, we need such enhancement. Smooth areas, on the other hand, contain less useful information, and over-enhancement would be redundant. Secondly, during imaging, the spherical fruiting body of the apple causes the light reflectance to change too quickly at shorter distances. The Retinex theory does not cope well with this situation and some compensation is needed. Based on this, the bilateral filter function was improved. The smoothness of the region was reflected by the mean value of the gradient in the filter neighborhood, and the grayscale domain weighting coefficients were adaptively adjusted according to the smoothness, then adaptively adjusted the spatial domain weighting coefficient according to the distance of the filter center from the image center to cope with the above problems. The specific calculation process is as follows:

Let any point on the image be $P\left(x,y\right)$ and the center of the filter be $Q\left(x_c,y_c\right)$; then the bilateral filter function is calculated as follows:

$$f_{fittered}\left(x,y\right)=\frac{1}{w_p}{\displaystyle \sum }_{Q\subset \omega }{I}_{Q}f\left(‖P-Q‖,{\delta }_s\right)g\left(\left|I_P-I_Q\right|,{\delta }_r\right)$$

(4)

$$f\left(‖P-Q‖,{\delta }_s\right)=e^{-\frac{{\left(x-x_c\right)}^2+{\left(y-y_c\right)}^2}{2{\delta }_s{}^2}}$$

(5)

$$g\left(\left|I_P-I_Q\right|,{\delta }_r\right)=e^{-\frac{{\left(I_P-I_Q\right)}^2}{2{\delta }_r{}^2}}$$

(6)

where $f_{fittered}\left(x,y\right)$ represents the output value of pixel $P$; $I_Q$ denotes the original value of pixel point $Q$; $I_P$ denotes the original value of pixel point $P$; $\omega$ is the neighborhood region of the pixel; $f\left(‖P-Q‖,{\delta }_s\right)$ denotes the spatial distance weight function, which is usually weighted using a Gaussian function; $g\left(\left|I_P-I_Q\right|,{\delta }_r\right)$ denotes the pixel variance weight function, which is also weighted using a Gaussian function; ${\delta }_s$ and ${\delta }_r$ denote the weighting coefficients in the spatial and gray domains, respectively; and $w_p$ is the normalization coefficient, which is used to make sure that the output pixel value is in the range of 0 to 255.

Let the image size be $mn$ and the filter neighborhood size be $(2a+1)(2a+1)$. The gradient $P_{grad}(x,y)$ of the experimental image is calculated using the Sobel operator with the following expression:

$$P_{grad}(x,y)=P(x,y)\ast Sobe{l}_H+P(x,y)\ast Sobe{l}_V$$

(7)

$$Sobe{l}_H=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]$$

(8)

$$Sobe{l}_V=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right]$$

(9)

Then the full graph gradient mean $mea{n}_{all}$ and the filter neighborhood gradient mean $mea{n}_{bf}$ are calculated as follows:

$$mea{n}_{all}=\frac{1}{mn}{\displaystyle \sum _{x=0}^{m-1}{\displaystyle \sum _{y=0}^{n-1}{P}_{grad}(x,y)}}$$

(10)

$$mea{n}_{bf}=\frac{1}{(2a+1)(2a+1)}{\displaystyle \sum _{x=x_c-a}^{x_c+a}{\displaystyle \sum _{y=y_c-a}^{y_c+a}{P}_{grad}(x,y)}}$$

(11)

Let the gray-scale standard deviation of the filter neighborhood be $st{d}_{bf}$; then the gray-scale domain weighting coefficients are adjusted as

$${\delta }_r=st{d}_{bf}\times \frac{\alpha \times mea{n}_{all}}{mea{n}_{bf}}$$

(12)

Here, $\alpha$ is the weighting constant for obtaining the weighting coefficients in the gray domain. The smaller the value, the smaller the filter intensity adjustment range. On the contrary, the larger $\alpha$ is, the larger the filter intensity adjustment range and the larger the difference between the filter intensity in the smoothing and fringe regions.

In the smoothing area, $mea{n}_{bf}<mea{n}_{all}$, ${\delta }_r$ becomes larger, and the degree of bilateral filtering becomes larger, which can better restore the spatial illumination situation; in the edge area, $mea{n}_{bf}>mea{n}_{all}$, ${\delta }_r$ becomes smaller, the weight of the pixel points with the larger gray difference in the filtering neighborhood becomes smaller, which has less influence on the filtering result, and the edge can be better preserved.

For the filter compensation at the edge of the image, the intensity of the bilateral filter should be reduced as the distance of the filter from the center of the image increases so as to prevent over-enhancement of the Retinex reflective layer at the edge of the image.

So, the spatial domain weighting coefficient ${\delta }_s$ is adjusted to

$${\delta }_s=a(1-\beta \frac{\sqrt{{(x_c-\frac{m-1}{2})}^2+{(y_c-\frac{n-1}{2})}^2}}{\sqrt{{(\frac{m}{2})}^2+{(\frac{n}{2})}^2}})$$

(13)

Here, $\beta$ is the weighting constant to obtain the spatial domain weighting coefficients. The larger the value, the greater the difference between the filtered intensity at the edges and the filtered intensity at the center.

The adaptive adjustment of the filter function gave the SSR algorithm the advantage over the MSR algorithm to consider the features of multiple scales at the same time, and the computational complexity was greatly simplified. At the same time, it compensated for the rapid change of light reflectance on the spherical surface, which made the calculation results of the Retinex reflective layer more reasonable.

2.3.2. Reflected Component Adaptive Gamma Correction

Gamma correction enhances the luminance and contrast of an image to better match the physiological properties of the human eye, which perceives brightness non-linearly [35]. The formula is as follows:

$$i_{out}=i_{in}{}^{\gamma }$$

(14)

where $i_{in}$ is the input image luminance value, $i_{out}$ is the output image luminance value, and $\gamma$ is the gamma value, which usually takes a value between 0.5 and 2.5.

The luminance and contrast of an image can be adjusted by controlling the gamma value. However, the optimal gamma value is different for different images under different lighting conditions. Therefore, it is necessary to adaptively adjust the gamma value according to the characteristics of the image itself.

The histogram reflects the characteristics of the image well, the position of the main peaks reflects the brightness of the image, and the degree of concentration of the peaks reflects the range of variation of the brightness of the image, which is the conclusion from visual observation. Measured from the data, the histogram mean corresponds to the brightness of the image, and the standard deviation corresponds to the range of variation in the brightness of the image. Based on the above principle, adaptive adjustment of gamma value was achieved by the following equation, i.e.,

$$\gamma =\frac{\log \left(\frac{\mu }{255}+0.01\right)}{\log \left(\frac{\sigma }{255}+0.01\right)}$$

(15)

where $\gamma$ is the gamma value, $\mu$ is the mean value of the image histogram, $\sigma$ is the standard deviation of the image histogram, and the 0.01 constant is added to avoid the situation of the denominator being 0 in practical applications.

The adaptive adjustment method was negatively correlated with the image brightness and positively correlated with the change in image brightness. The above gamma correction facilitated the subsequent extraction of texture features.

2.3.3. Defect Segmentation Based on Region-Growing Algorithm

Region growing is a process of aggregating pixels or sub-regions into larger regions based on pre-defined similarity criteria. The basic method is to start by selecting one or more pixels as seed points. Then, according to some similarity criterion, neighboring pixels or regions with similar properties are grouped together to grow the region until there are no more pixels or other small regions to be grouped together [36]. In practical applications, the key issues are the manner of seed point picking and the establishment of similarity criteria [37].

Traditional defect detection content based on region-growing algorithms generally involves segmenting the target region in the interference region, which has relatively complex features, while the target region generally has similar features. After improved SSR image enhancement, the defective areas were affected by the halo phenomenon and the degree of decay itself with a different gray scale starting points and trends. However, the normal surface area showed greater similarity. Thus, the effect of segmenting defects could be achieved by segmenting the normal surface area.

The normal surface area features were obvious, the gray intensity was above 250, and the distinction with defects was obvious. However, in order to prevent errors caused by large spatial distances, the image needed to be partitioned by a certain size, and a seed point was selected by gray intensity within each region.

The similarity criteria was adaptive thresholding based on the standard deviation and mean of the gray scale of the growing region. Let $R$ be the grown area; then the mean gray value $m$ and standard deviation $\sigma$ within the grown area are calculated as follows:

$$m=\frac{1}{n}{\displaystyle \sum _{(x,y)\in R}f(x,y)}$$

(16)

$$\sigma =\sqrt{\frac{1}{n}{\displaystyle \sum _{(x,y)\in R}{\left|f(x,y)-m\right|}^2}}$$

(17)

where $f(x,y)$ is the gray value of the pixel at the point; $n$ is the number of pixels in the grown area.

Let the initial threshold be $T$; then the growth region changes to generate a new threshold $T^{\ast }$, which is calculated as

$$T^{\ast }=T(1-\frac{\sigma }{m})$$

(18)

The initial threshold is determined based on the grayscale histogram of the reflected component. Let $T=(255-\mathrm{g}ra{y}_m)/2$, where $\mathrm{g}ra{y}_m$ is the grayscale histogram grayscale mean value.

2.3.4. Support Vector Machine Classification Based on Texture Features

Gray level co-occurrence matrix (GLCM) is an image analysis method based on statistical methods for reflecting image texture features that carry spatial relationships between pixels. The method describes the probability of occurrence of a pair of image elements with gray levels $i$ and $j$, respectively, separated by a distance of d image elements in the direction [38,39]. The calculation is as follows:

$$P(i,j,d,\theta )=\{((x,y),(x+D_x,y+D_y)|f(x,y)=i;|f(x+D_x,y+D_y)=j)\}$$

(19)

where $(x,y)$ is the pixel coordinate of any point on the image; $i$ is the pixel gray level of the point; $(x+D_x,y+D_y)$ is the pixel distance D from the point in the direction $\theta$ of the deviation point; $j$ is the pixel gray level of the deviation point; $\theta$ is the direction that can be selected, generally $0^{\circ }$, ${45}^{\circ }$, ${90}^{\circ }$ and ${135}^{\circ }$; $d$ is the computation step length of the gray level coevolution matrix.

The gray level co-occurrence matrix does not provide a direct reference for texture feature extraction, while the secondary statistics derived from it provide an important basis for measuring texture features. The secondary statistics used include the following:

(1)

Angular Second Moment (ASM), which reflects the degree of uniformity of the image grayscale distribution and texture coarseness and fineness, is calculated as follows:

$$ASM={\displaystyle \sum _i{\displaystyle \sum _{j}P{(i,j)}^2}}$$

(20)

(2)

Entropy (ENT), which reflects the randomness of the amount of information contained in the image, is calculated as follows:

$$ENT=-{\displaystyle \sum _i{\displaystyle \sum _{j}P(i,j)}}\times \log [P(i,j)]$$

(21)

(3)

Contrast (CON), reflecting the sharpness of the image and the depth of the texture grooves, is calculated as follows:

$$CON={\displaystyle \sum _i{\displaystyle \sum _j{(i-j)}^2}}\times P(i,j)$$

(22)

(4)

Inverse Differential Moment (IDM), reflecting the clarity and regularity of the texture, is calculated as follows:

$$IDM={\displaystyle \sum _i{\displaystyle \sum _{j}P(i,j)/[1+{(i-j)}^2]}}$$

(23)

(5)

Correlation (COR), reflecting the degree of similarity of spatial gray scale covariance matrix elements in the row or column direction, is calculated as follows:

$$COR={\displaystyle \sum _i{\displaystyle \sum _j(i-mean)\times (j-mean)\times P{(i,j)}^2}}/\mathrm{var}$$

(24)

where $P(i,j)$ is the number of occurrences of a pair of image elements with gray level $i$ and $j$, respectively; $mean$ is the mean of the gray level covariance matrix; and $\mathrm{var}$ is the standard deviation.

The support vector machine is a method based on statistical learning theory, which can apply undifferentiated learning ability to seek the best mathematical model in limited and complex sample information [40,41].

A major advantage of the support vector machine is that if the appropriate decision hyperplane cannot be solved under the original dimension, the feature data in the low-latitude space can be mapped to the high-dimensional space using a nonlinear mapping function. The appropriate decision hyperplane can be researched in the high-dimensional space to achieve the purpose of classification [42].

Take two-dimensional space as an example; let $x_i$ be the eigenvalue and $y_i$ be the data category the positive hyperplane is 1, and the negative hyperplane is −1. As shown in Figure 7, the optimal solution solved by the support vector machine is to find the decision hyperplane having the maximum soft interval with proper allowance for outliers. The larger the soft interval, the better the classification results.

Mathematically, the original problem solved by the support vector machine is the minimization problem $f(w)$ under constraint $g_i(w,b)$. The formulas for $f(w)$ and $g_i(w,b)$ are as follows:

$$f(w)=\frac{{‖\overrightarrow{w}‖}^2}{2}$$

(25)

$$g_i(w,b)=y_i(\overrightarrow{w}\cdot \overrightarrow{x_i}+b)-1\ge 0,i=1,2,3,\cdots ,s$$

(26)

where $w$ is the decision hyperplane parameter, $b$ is the intercept, and $s$ is the full sample size.

Since the constraints are inequalities and cannot be simplified by introducing Lagrange multipliers, the non-negative variable $p_i{}^2$ is introduced for the time being. According to the Lagrange multiplier method, we can turn the above convex optimization problem under the affine function constraint into a problem of finding the extreme value of the Lagrange expression. Let ${\lambda }_i$ be the penalty coefficient for violating the constraints, and the Lagrange expression is as follows:

$$L(w,b,{\lambda }_i,p_i)=\frac{{‖\overrightarrow{w}‖}^2}{2}-{\displaystyle \sum _{i=1}^s{\lambda }_i(y_i(\overrightarrow{w}\cdot \overrightarrow{x_i}+b)-1-p_i{}^2)}$$

(27)

By analyzing the results brought about by the violation of the constraints, it is clear that ${\lambda }_i\ge 0$ and $p_i{}^2=0$. Then the full KKT condition can be obtained from the first-order derivative zeros of the Lagrangian. According to the Lagrange multiplier method and the KKT conditions, the dual of the original problem, i.e., the problem of maximizing $q({\lambda }_i)$ under the constraints ${\lambda }_i\ge 0$, can be found, and $q({\lambda }_i)$ is calculated as follows:

$$q({\lambda }_i)={\displaystyle \sum _{i=1}^s{\lambda }_i}-\frac{1}{2}{\displaystyle \sum _{i=1}^s{\displaystyle \sum _{j=1}^s{\lambda }_i{\lambda }_j}}{y}_i{y}_j\overrightarrow{x_i}\cdot \overrightarrow{x_j}$$

(28)

From the above equation, the optimum solution of the dyadic question is determined only by the result of the dot products of the support vectors (the spatial similarity of the support vectors), and the spatial similarity can be obtained directly from the kernel function. Therefore, kernel functions such as linear kernel $\mathrm{k}(x,x_i)=x·x_i$, polynomial kernel $\mathrm{k}(x,x_i)={((x·x_i)+1)}^d$, and Gaussian kernel $\mathrm{k}(x,x_i)=\exp (-\frac{{‖x-x_i‖}^2}{2{\sigma }^2})$ can be introduced to replace the nonlinear mapping function in the dimensional transformation, and the computational complexity can be greatly simplified.

The above five texture features were fed into the support vector machine for classification model training. In the experiment, the order of magnitude difference between the computed results of the five texture features was too large. Thus, they were first standardized and then trained and predicted according to the ratio of training set to test set, which was 7:3.

The size of the penalty factor balances the algorithmic complexity and stability of the support vector machine. Thus, for the penalty factor, kernel function, and gamma value (some kernel functions used gamma value), the best combination of parameters can be found by grid search.

Image texture data from each stage of image processing were acquired separately for classification model training to verify the effectiveness of Retinex image enhancement and gamma correction in highlighting the differences between scars and stems/calyxes.

3. Results

3.1. Experimental Setting

The size of the experimentally captured original image was about 100 × 100, and in order to evaluate the performance of the proposed method, existing image segmentation techniques and texture feature extraction methods were compared.

First, the weighting constant $\alpha$ of the gray-domain weighting coefficients used in the SSR luminance decomposition process is based on adaptive bilateral filtering, which controlled the adjustment range of the filtering intensity. During the experiment, the range of $\frac{mea{n}_{all}}{mea{n}_{bf}}$ was between 0.5 and 2.3. During the transition from the smooth area to the edge area, the process of filter intensity reduction was too fast. When the filter size became larger, the range of $\frac{mea{n}_{all}}{mea{n}_{bf}}$ became smaller accordingly, which made the process of filter intensity change more reasonable but caused the loss of detailed information on the reflection layer. This experiment used a filter size of 40 × 40, in which the weighting constant $\alpha =0.85$ was taken, the adjustment process of filter intensity was more reasonable, and the reflection layer separation effect was better.

The weighting constant $\beta$ of the gray-domain weighting coefficients was used in the SSR luminance decomposition process of adaptive bilateral filtering by which the degree of filtering compensation at the edges was adjusted. This degree of compensation could not be too large. Otherwise, it would be in the reflection layer at the edge of the image directly producing shadow interference. After experimental testing and verification, from the center of the image to the edge, the filter strength was reduced to two-thirds of the original, which was more reasonable. So, by taking $\beta =\frac{1}{3}$, the filtering could be compensated at the edges of the image without shadow interference from the reflective layer.

Regarding the parameter determination of the support vector machine, among the kernel functions (linear kernel, polynomial kernel, Gaussian kernel), the penalty factor size (0.01, 0.1, 1, 10), and the gamma value (0.01, 0.1, 1, 10), the optimal combination of parameters determined by means of grid search was to use the linear kernel function with the penalty factor size of 1 and no gamma value (the linear kernel function had no gamma value).

3.2. Experimental Results and Comparisons

3.2.1. SSR Luminance Decomposition Results Based on Adaptive Bilateral Filtering

The RGB decomposition of the obtained background segmentation image was per-formed first, as shown in Figure 8. It could be seen that the defects had the greatest contrast with the normal surface in the R-component map, so the R-component map was enhanced as the original image for SSR image enhancement.

The brightness decomposition of SSR (filter size 40 × 40), MSR (filter size 20 × 20, 30 × 30, 40 × 40) based on bilateral filtering, and SSR (filter size base size 40 × 40) based on adaptive bilateral filtering were carried out for Figure 8a, respectively, and the results for the reflection layer obtained are shown in Figure 9. As could be seen from the figure, compared with the SSR algorithm, the decomposition results of the improved SSR algorithm in the normal surface region were basically the same as the SSR decomposition results, and the smoothing region did not cause much change in the filter strength. Compared to MSR, the decomposition results of the improved SSR algorithm in the scar region were equally good in preserving the inner edges of the scars with a clearly visible texture. Meanwhile, the scar information at the edge was also recovered by appropriately reducing the intensity of bilateral filtering close to the image edge. As could be seen from Figure 9c, when the scar appeared at the image edge, a slight interference region appeared at the image edge scar. This may be caused by the fact that the bilateral filter adaptive adjustment method made both spatial and gray domain weight coefficients appear to be adaptively reduced, and the adjustment effect of both was superimposed. Further discussion on this issue is required in the follow-up.

3.2.2. Reflected Component Adaptive Gamma Correction Results

The effect of adaptive gamma correction is shown in Figure 10. As could be seen from the reflected component maps, the internal contrast of the corrected defective area was significantly increased, and the texture features were clearer. From the observation of the gray intensity surface map (the gray intensity of the image was inverted, and then the gray intensity surface map was produced), the corrected defective area was richer in each segment of the internal gray intensity, and the enhancement effect was obvious.

3.2.3. Defect Segmentation Results of Improved SSR Image Enhancement and Region- Growing Algorithm

In order to verify the effectiveness and practicality of the defect detection method in this paper, four representative apple images were selected as experimental images, which were compared and analyzed with the image segmentation method based on the genetic algorithm (with OSTU algorithm as the fitness function and roulette algorithm as the selection algorithm) and the segmentation method based on the canny operator for edge detection built into the machine vision module OpenMV.

Figure 11a is characterized by a natural depression at the apple fruit stalk, which results in a certain degree of abrupt change in light reflectance near this region. The detection results of each method for this image are shown in Figure 11. The genetic algorithm-based segmentation method detected a large area of the fruit stalk, and the boundary between the normal surface and the fruit stalk was not obvious. The edge detection algorithm only had some detection ability for the deepest gray level zero part of the fruit stalk. Comparatively, the proposed segmentation method could detect the whole depressed area where the light reflectivity had changed abruptly, and the boundary was more reasonable, providing a more reliable texture basis for the subsequent identification of the peduncle calyx.

Figure 12a is characterized by the same natural depression at the calyx of the apple and a significant uneven light reflectivity at the right edge of the apple. The detection effect of each method on this image is shown in Figure 12. In the extraction of calyx defects, the segmentation method based on the genetic algorithm had a better detection effect at the calyx, and the boundary between the calyx and the normal surface was more reasonable, but it was easily disturbed by the uneven light reflectivity phenomenon in the light-dark area. The segmentation method based on the edge detection algorithm had some over-segmentation phenomena for the calyx with complex textures. Comparatively, the proposed segmentation method still had good segmentation ability at the calyx defects while resisting the uneven interference of light reflectivity.

Figure 13a is characterized by defects located within a dark zone caused by uneven light reflectivity, highlighting the disturbance caused by uneven light reflectivity. The detection effect of each method on this image is shown in Figure 13. The segmentation method based on the genetic algorithm had a significantly lower edge detection accuracy in the light-dark region than in the light-bright region. The segmentation method based on the edge detection algorithm had a significant difference in the detection effect in the bright and dark zones caused by the uneven light reflectivity. In the bright area, the contrast between the gray intensity of the pixels at the edge of the defect and the edge of the normal surface was obvious, and the segmentation effect was more satisfactory. In the dark area, the original strong gray contrast at the edge became flat, and the detection effect of the edge detection algorithm was almost completely lost. Comparatively, the proposed segmentation method was not affected by the light and dark areas caused by uneven light reflectivity and had the same detection accuracy in the dark area as in the light area.

Figure 14a is characterized by a more complex defect edge contour. The detection effect of each method on this image is shown in Figure 14. The segmentation method based on the genetic algorithm could roughly recover the edge contour, but the recovery ability was low at the edge and was too complex, which may lead to the destruction of the edge continuity. Segmentation methods based on edge detection algorithms could detect the general contour of defects, but the connectivity was poor, and simple image morphological processing could not recover the defect edges, which needed to be specifically edge connections. In fact, edge connectivity is more difficult than edge detection. In contrast, the proposed segmentation method eroded from the normal surface to the inside of the defect and stopped growing after obtaining the defect edge. The defect edge information could be better restored.

3.2.4. Support Vector Machine Classification Results Based on Texture Features

As shown in Figure 15, after SSR enhancement, the gray levels inside the scars were haphazard, which was related to the degree of decay inside the scars, and different scars presented different gray intensity characteristics. In contrast to the scars, the stem/calyx area had a distinct regularity to it. Both the apple stem and the calyx showed a gray intensity characteristic of lower gray intensity internally and then gradually recovered to the same gray intensity as the normal surface area as the degree of indentation diminished.

In order to investigate the effect of improved SSR image enhancement and adaptive gamma-corrected image processing methods on texture features, the texture features of the three image processing stages of extracting the R component, improved SSR image enhancement, and adaptive gamma correction were collected as sample datasets. A total of five texture features, angular second moment, contrast, entropy, inverse differential moment, and correlation, were collected for each sample, and the three-stage stem/calyx and scar texture features are shown in Figure 16. It can be seen that the distribution of data for each textural feature of the stems/calyxes has a certain regularity compared to the scars. Further analysis showed that compared to the R-component maps, the randomness of the amount of information became larger after SSR image enhancement, and the clarity and regularity of the texture were significantly improved (the value of the inverse differential moment reflected this information). The texture contrast was further increased after further adaptive gamma correction compared to the SSR image enhancement map.

The texture features of the defective samples obtained from the images of the three image processing stages were fed into the support vector machine for binary classification, respectively. The results, as shown in

Table 1. Stem/calyx recognition accuracy.

Image Processing Stage	Number of Stem/ Calyx Textures	Number of Scar Textures	Total	Accuracy (%)
R-component	200	230	430	84.4
SSR	200	230	430	93.0
Gamma	200	230	430	93.7

, indicate that SSR image enhancement and adaptive gamma correction were effective in highlighting differences in textural features between stems/calyxes and scars.

The decision hyperplane expression was obtained by training the texture data obtained by the full image processing process as follows:

$$\begin{array}{l}\mathrm{y}=0.00473\times ASM+0.05776\times CON+0.27029\times IDM\\ +0.03863\times ENT+0.01327\times COR-3.60491\end{array}$$

(29)

It is worth noting that IDM accounted for a relatively large proportion of the overall classification judgments, reflecting the fact that the clarity and regularity of the texture were quite important for stem/calyx identification, as predicted early in this study.

3.2.5. The Results of the Proposed Defect Detection Method

The practical application effects of the two sub-methods of the proposed defect detection method (image segmentation method and stem/calyx recognition method) are presented above, respectively. Finally, the defect detection method was deployed to the embedded image processing device OpenMV for experiments to verify the effectiveness and practicality of the method. Firstly, after all the defective areas were segmented by the proposed image segmentation algorithm, the built-in “find_blobs color block” function of OpenMV was used to mark the defective areas with ROI regions of interest, extract the texture featured, and then send them to the support vector machine for binary classification. According to the classification result, the stem/calyx was eliminated and only the scar area was retained as the execution result of the proposed method. In this way, on the one hand, there was no need to open a separate storage space to set labels for the classification results of the defective areas, which reduced redundant operations. On the other hand, the scar detection results could be directly displayed graphically, which was more intuitive than the form of tables, convenient to further summarize some qualities of the scars, and conducive to judging the health of the apples. Some of the test results are shown in Figure 17.

Table 2. Detection accuracy of two types of defects under the proposed algorithm.

Defect Type	Sample Size	Number of Correct Disease Condition Judgments	Accuracy (%)
Normal-stem	70	67	95.7
Normal-calyx	70	65	92.8
Scar-light browning	70	64	91.4
Scar-heavy browning	70	68	97.1
Total	280	264	94.2

shows the defect detection results for 280 pairs of apple images. Using the criteria that scars were not missed and the stem/calyx was not misclassified as a scar, healthy and diseased fruits were distinguished by whether the number of scars detected was zero or not. For the browning heavier defect type, although the detection effect was better, it was found in practical application that when the disease rot spread over a large area, the internal rotted degree would appear obvious grading or even white mold, resulting in the brightness decomposition results being affected, which led to over-segmentation. In addition, in this paper, only scars with an area larger than 20 pixels (the resolution of the experimental image was 160 × 120) were counted as the detection results, and the surface spots on the apples themselves (usually with an area of less than five pixels) were ignored.

4. Conclusions

In this paper, an image segmentation algorithm based on adaptive bilateral filtering for SSR image enhancement and a region-growing algorithm were proposed for filtering out the interference of uneven light reflectivity and segmenting the defects. A limitation of this method was that it could not differentiate between stems/calyxes and scars. To solve this problem, an algorithm for stem/calyx recognition based on texture features was also introduced. Finally, a defect detection algorithm was developed from the two methods. The algorithm could take only the segmentation results of real scars as the final results. Deploying the proposed defect detection algorithm into the embedded image processing module OpenMV4H7Plus, the accuracy of stem/calyx recognition was 93.7%, and the accuracy of scar detection was 94.2%. The results show that the proposed defect detection algorithm can complete the extraction of apple surface defects and exclude the interference of stems/calyxes in the case of uneven light reflectivity. The proposed defect detection algorithm could be used in automated apple defect detection systems to increase productivity and reduce labor costs. This may have wide applications in orchards and food processing plants.

In addition, an issue that was not addressed in this study is whether the segmentation method suffers from over-segmentation for scars with large areas of obvious decay grading. Although it does not affect the overall segmentation and extraction of scars and the judgment of the health of the apple, it causes an additional increase in the number of scars counted. Further research is needed to follow up on this issue.