Classification for GM and Non-GM Maize Kernels Based on NIR Spectra and Deep Learning

Abstract

The classification of GM and non-GM maize kernels is fundamental for further analysis of the gene action in maize. Therefore, a complete and novel detection scheme based on near-infrared spectra was designed to distinguish GM and non-GM maize kernels. Hyperspectral images (935–1720 nm) of 777 maize kernels from 3 kinds were captured, and the average spectra of the maize kernels were extracted for modeling analysis. The classical modeling methods based on feature engineering were first studied, and the backpropagation neural network–genetic algorithm model showed the best performance with a prediction accuracy of 0.861. Then, novel modeling methods based on deep learning were developed. To dig out the interactive information between different bands and match the application scenarios, the original spectra were transformed into two-dimensional matrices before establishing the deep learning models. A modified convolution neural network (i.e., VGG net) with dilated convolution was finally constructed to classify the maize kernels, and the prediction accuracy reached 0.961. This research provides a referential and novel way to detect GM maize kernels. Future research will improve the detection scheme for monitoring unauthorized GM organisms by introducing the visualization technology of deep learning.

1. Introduction

Maize, one of the three staple grains, plays an important role in food, forage, and some industrial production support. Nowadays, the requirements for maize have become diversified. For example, high-latitude areas hope to strengthen the cold resistance of maize [1], the brewing industry hopes to increase the starch content of maize [2], and the livestock and poultry industries hope to improve the protein constitution of maize [3]. Genetic modification (GM) technology provides an effective way to optimize the appointed properties of maize to satisfy the requirements of specific industries [4]. GM kernels need strict management as GM technology brings potential risks to ecological and food safety, but some individuals or companies illegally grow and sell unauthorized GM maize for profit [5]. Therefore, distinguishing GM and non-GM maize, identifying the specific category of genetic modification, and figuring out the interaction of the genes are vital to perfecting GM technology [6]. Polymerase chain reaction (PCR) or protein detection is the standard method to detect GM maize [7]. The standard method can ensure detection accuracy, but it is reagent-consuming, destructive, and laborious. Furthermore, before PCR detection, the DNA sequences of maize with different categories or genetic modifications should be clear. Otherwise, the gene comparisons will be performed unintelligibly. For protein detection, the chemical operation may cause protein denaturation and influence the detection result.

Spectral analysis technology, integrated with spectral measurement, chemometrics, and machine learning, has been widely used in the fields of agricultural engineering [8], the chemical industry, and bioengineering [9]. Due to the fact that the near-infrared (NIR) spectrum inherently reflects the vibrations of C-H, N-H, and O-H groups in carbohydrates [10], fats [11], and proteins [12], it is one of the most commonly used spectral types in the internal quality detection of grains or seeds [13]. Although the spectral detection method is rapid, nondestructive, flexible, and low-cost, it is hard to achieve the accuracy that is required as a standard method for GM detection. So, a great number of novel spectral analysis technologies were developed to pursue the high accuracy and stability of detection in some specific fields. Wang et al. used a Vis/NIR hyperspectral imaging system (400–1000 nm) to detect the aflatoxin B-1 on the maize kernel surface, and the results indicated that the traditional classification method principal components analysis (PCA)—factorial discriminant analysis was practical to detect and differentiate different levels of aflatoxin B-1 with an accuracy of 98% [14]. He et al. took advantage of NIR hyperspectral imaging together with chemometric technology to identify GM maize kernels, and an excellent prediction accuracy of almost 100% was finally achieved by the support vector machine (SVM)—competitive adaptive reweighted sampling method after multiple feature extraction and selection method tests; the research provides a promising technique to identify GM maize kernels [15]. Benes et al. employed detrended fluctuation analysis (DFA) and yield stability index (YSI) methods on NIR spectra to detect coffee seeds, and the results showed that DFA successfully differentiated the roasting levels on samples with 100% accurate agglomerative hierarchical clustering, and YSI successfully identified that a light roast is the most stable among all roasting levels [16]. The above pieces of research directly or indirectly illustrate the feasibility of NIR spectral analysis in the detection of maize kernels, but the exploration and utilization of NIR spectral features linked with GM are inadequate. According to the literature [17], the NIR spectra unfold the holistic characteristics of the object to be detected. Namely, the internal relations between different components of the object are covered under the waveforms of spectra. Moreover, compared with the non-GM parents, the GM organisms (GMOs) in maize kernels change synthetically [18]. So, the data mining methods of spectra need to be reviewed and compared to develop a scheme more suitable for GM maize kernel detection.

The traditional data mining technology of spectra can be divided into feature selection and feature extraction. Essentially, these two methods both belong to feature engineering [19]. The key point of feature selection is to pick the representative bands from the full-range spectra by projection calculation or heuristic algorithms [20]. So far, many feature selection methods have been proposed and applied [21]. The core idea of feature extraction is to transform the spectra matrix and pick out the eigenvectors with the biggest eigenvalues or information weights [22]. Similarly to feature selection, massive feature extraction methods have been developed and utilized too [23]. Although quite a lot of successful applications based on feature engineering have been reported, the disability in high-level semantic feature acquisition is still a vital shortage [24]. Deep learning technology adopts end-to-end learning mechanisms [25], and the original information will be transformed and abstracted in a task-oriented manner by the deep neural network automatically [26]. For convolution neural networks (CNNs), the features with different size receptive fields are extracted layer by layer through convolution and pooling operations [27]. Therefore, deep learning technology can overcome the shortage of feature engineering methods and has been widely used in the spectral detection of various agricultural products [28]. Shen et al. propose a method to rapidly and effectively detect impurities contained in wheat based on a combination of terahertz (THz) spectral imaging and a CNN, i.e., ResNet; the images used for input variables of CNN were generated according to the principle of maximum frequency-domain imaging, and the designed Wheat-V2 model achieved an average Fl-score of 97.83%; this research provides a nondestructive testing method for the recognition of impurities in wheat and other grains [29]. Onmankhong et al. attempted to resolve the real issue of detecting genuine high-quality Thai rice in milled and brown conditions using long-wave NIR hyperspectral imaging coupled with machine learning and deep learning approaches. For the milled rice, the CNN model shows the best performance with a classification accuracy of 95.2%. For brown rice, the SVM model achieves the best classification accuracy of 95.4%. This research provides a reference for the execution of traditional machine learning and deep learning from different views in hyperspectral information exploitation [30]. However, most of the recent works about spectral detection based on deep learning immediately perform the convolution operation on a three-dimensional hyperspectral image or perform the convolution operation on a two-dimensional (2D) image transformed from the original hyperspectral image. In fact, the average spectra of the samples in the hyperspectral image carry more stable and comprehensive information, which can better reflect the internal features of the samples. Unfortunately, the average spectra have not been well utilized in the scene of deep learning, such as the correlation and variation in different spectral domains are easily ignored.

According to the above discussion, this research will make a full investigation of the spectral detection of GM and non-GM maize kernels by traditional machine learning and deep learning methods. The traditional machine learning method based on feature engineering will be researched first to verify the feasibility of the GM and non-GM maize kernel classifications based on the spectra in the normal type. Then, the spectra will be transformed into 2D correlation matrices, and the deep learning models established based on the 2D-transformed spectral matrices will be studied to reveal the synergism of different spectral wavelengths on GM and non-GM maize kernel classifications.

2. Materials and Methods

2.1. Sample Preparation and Data Acquisition

Three categories of maize kernels were gathered for this research. One is a non-GM parent, named CK, and the other two are GM categories, named GM1 and GM2. For CK, GM1, and GM2, a total of 231, 282, and 264 full shape seeds without disease and insect damage were prepared, respectively. All the maize were planted in the experimental field of Zijingang Campus, Zhejiang University (latitude and longitude: 30.301° N, 120.095° E) in mid-May 2022, and the maize kernels were harvested on 18 August 2022. After harvest, the maize kernels were delivered to the laboratory for hyperspectral data acquisition.

A hyperspectral image acquisition system was employed to capture hyperspectral images of the maize kernels. The acquisition system (FX17, SPECIM Co., Ltd., Oulu, Finland) works in a linear-scanning way. The linear array contains 640 pixels, and each pixel captures 224 bands within 935–1720 nm. Before the acquisition, the maize kernels were placed on the platform at intervals. During the acquisition, the frame rate of the camera was set to 60 Hz, the exposure time was set to 8 ms, the scanning speed was set to 24.7 mm·s⁻¹, and the detection distance was set to 395 mm. After the acquisition, the raw hyperspectral images were calibrated using Formula (1) to decrease or eliminate the influence of dark current and illumination. The hyperspectral data acquisition was performed at room temperature.

$$R_C=\frac{R_c-B}{W-B}$$

(1)

In this formula, $R_c$ denotes the image after calibration, $R$ denotes the raw image, $W$ denotes the calibration vector of the standard white board, $B$ denotes the calibration vector of dark current.

2.2. Feature Extraction and Selection

The original spectra with high dimensions carry much redundant information, which not only makes no contribution to category detection but also increases the model complexity [31]. Therefore, it is necessary to refine the original spectra. Feature extraction and selection are the common and effective methods used for dimensionality reduction in spectra. For feature extraction, PCA, one of the most classical feature extraction methods, was employed to reduce spectral dimension [32]. For the specific implementation scheme of PCA in this research, the original spectra are transformed into a new coordinate space by linear transformation [33]. The first coordinate axis is constructed following the direction of the largest variance in the original spectra, the second coordinate axis is determined by the direction with the largest variance in the plane, which is orthogonal to the first coordinate axis, and the third coordinate axis is the one with the largest variance in the plane, which is orthogonal to the first and second coordinate axes. By analogy, the first several coordinate axes, namely, PCs, which contribute the most to the original spectral information, are finally obtained.

Genetic algorithm (GA) and successive projection algorithm (SPA) were adopted to select feature bands from the original spectra. GA is the most basic swarm intelligence algorithm. All other swarm intelligence algorithms refer to the mechanism of GA [34]. For the operation of GA in this research, each spectrum sample is treated as a chromosome encoded in a binary sequence, and each band in the spectrum is treated as a gene that takes the value of either 0 or 1 [35]. Fifty chromosomes are selected to construct the initial population. Then, evaluation, crossover, mutation, and selection are circularly executed until the population is satisfied or the circular times run out. As each gene corresponds to a band, the occurrence frequency of each gene in the final population is counted to figure out the band’s importance. According to the bands’ importance, the feature bands are finally identified. In this research, a ratio of 1 in initial chromosome sequences was set to 0.4, the crossover rate was set to 0.5, the mutation rate was set to 0.1, and the number of iterations was set to 200. Quite different from GA, SPA generally begins with identifying an initial feature band randomly, then the projections from the rest bands to the initial feature band are calculated [36]. The band with the largest projection value is selected as the second feature band. Then, the projection to the selected bands will be circularly calculated until enough feature bands are screened out. In this research, the initial band of SPA was identified by the Pearson correlation coefficients with category labels.

2.3. Classification Model Establishment and Evaluation

Three classification methods—SVM, decision tree (DT), and backpropagation neural network (BPNN)—were employed to distinguish the maize categories.

SVM is commonly used for binary classification problems by constructing a hyperplane in the transformed feature space [37]. For the case of the three categories in this research, the strategy taken partitions the mission into two one-vs-rest tasks. A DT classifier can quickly obtain results in the case of high-dimensional samples with good robustness [38]. The main idea of this method is to generate multiple subsets with distinct properties from all the samples. Additionally, each subset is trained to form a decision tree. During the training, the nodes will be selected randomly to split into two parts. The above steps are performed circularly to generate proper decision trees. For the unknown spectral sample, its category is determined by the largest number of votes from all decision trees. To improve the reliability of voting, the number of decision trees was set to 9 in this research. BPNN is a typical neural network with a global response, and the input variables are mapped with the activation function in the neural units of hidden and output layers [39]. The structure of the BPNN used in this research was designed with three layers, namely, one input layer, one hidden layer, and one output layer. For the input layer, the neuron number equals the dimension of the spectra or input variables. For the hidden layer, the neuron number is determined by the experimental formula, and the mapping function is set to the Sigmoid function. For the output layer, the neuron number is the same as the category number. The maximum number of training times was set to 1000, the goal of the training accuracy was set to 1, the number of hidden neurons was set according to the experimental formula and incremental attempt, and the learning rate is set dynamically according to the training accuracy [40].

After classification, the total classification accuracy was calculated to evaluate the model performance roughly. Furthermore, the confusion matrix was adopted to analyze the classification result, which is accurate for every category.

2.4. Spectral 2D Transformation and Deep Learning

CNN is generally operated with 2D image information. However, the original spectrum used as the input variable is a one-dimensional vector. Therefore, the original spectral information (1 row × $m$ columns) was transformed into a 2D matrix $S$ ($m$ rows × $m$ columns) for band interaction and high-level semantic information extraction according to the following formula. After transformation, the matrix size was 224 (bands) × 224 (bands).

$$S_{i,j}=\left\{\begin{array}{l}\frac{B_i+B_j}{4B_i{B}_j}, if i>j\hfill \\ \frac{B_i}{B_j}, else if i<j\hfill \\ B_i, else\hfill \end{array}\right.$$

(2)

In this formula, $i$ and $j$ range from 1 to m. $S_{i,j}$ denotes the element at $i$th row and $j$th column of matrix $S$, $B_i$ denotes the reflectance value at $i$th band, and $B_j$ denotes the reflectance value at $j$th band.

A common CNN, VGG net with a few modifications, was employed for maize classification based on 2D spectral information, as the 2D-transformed matrices match the structure of the VGG net properly [41]. The VGG net used an architecture with small convolution filters to increase the depth, and a significant improvement on the prior-art configurations was achieved by pushing the depth to 16–19. In this research, the structure and learning strategy of the network borrowed certain ideas from the template [42]. Specifically, the network contains 13 convolution layers, 5 pooling layers, and 3 full connection layers. The kernel size of the convolution filter was set to 3 × 3 or 1 × 1, the kernel size of the max-pooling filter was set to 2 × 2, the step size and padding size were both set to 1, and the activation function was set to ReLU. To sharpen the edge, mean centering was carried out on the input 2D matrix. The original softmax classifier was replaced with a three-layer BPNN. Moreover, the dilated convolution filter with a dilation rate of 2 was introduced to explore the interaction of non-adjacent bands [43]. As the dilated convolution spontaneously enlarges the receptive field, the network in this case contains 8 convolution layers. Other configurations in dilated convolution are the same as those in solid convolution.

3. Results and Discussion

3.1. Overview of Spectra

In Figure 1, the hyperspectral images are displayed in pseudo color (R: 1520 nm, G: 1109 nm, and B: 1609 nm). Through comparing and analyzing the spectral characteristics of the maize kernels and background, a reflectance threshold of 0.36 at 1106 nm and a slope threshold of −5 between 1312 nm and 1442 nm were identified to generate the binary image for target segmentation. Every connected region in the binary image after morphological processing was marked and counted [44], and the average spectrum of each kernel was subsequently obtained by calculating the average of the values in the pixels belonging to the same connected region. The average spectra with standard deviation (STD) of the samples belonging to different categories are shown in Figure 2a. It can be found that the average spectra belonging to different categories have similar waveforms, while they are obviously different in reflectance intensity. As the spectral waveform is determined by the inclusion of the maize kernel, it can be inferred that the ingredients of the maize kernels belonging to different categories are basically consistent, but the concentration of the ingredients is different. Moreover, the reflectance intensity differences between different categories change with the wavelength (Figure 2b). As the reflectance value at the specific wavelength represents the concentrations of some chemical groups [45,46], the variations in different ingredients in maize kernels exhibit a diversity of characteristics for different maize categories. In fact, the above phenomenon of spectral waveforms can also be explained by gene modification. Specifically, GM1 and GM2 maize were modified to produce more amylopectin compared with CK maize, GM1 maize was modified to produce more soluble sugar compared with CK and GM2 maize, and GM2 maize was modified to produce more protein compared with CK and GM1. In other words, the spectra have the potential to express the change in ingredients in GM and non-GM maize kernels from the overview of spectral waveforms.

3.2. Classification Based on Feature Engineering

To observe the distribution of all the samples and estimate the feasibility of classification based on feature extraction, the first three PCs were extracted according to the description in Section 2.2 to form a three-dimensional (3D) space (Figure 3). In the space, most samples belong to the same category cluster together, which means the classification of the maize category based on feature extraction is feasible, but how to convert the PCs to the right category label needs further exploration as different modeling methods have their own advantages [47]. Therefore, three classical classification models—SVM, DT, and BPNN—were established based on the first seven PCs, which account for 99.62% of the total principal component variances. The establishment of the three classification models was performed according to the description in Section 2.3, and all the samples were divided into a training set and a prediction set in a ratio of 2:1 before modeling. From the results in

Table 1. Performance of SVM, DT, and BPNN models.

Model	Accuracy-Train	Accuracy-Predict
SVM	0.749	0.710
DT	0.710	0.637
BPNN	0.766	0.733

, it can be found that the BPNN model shows the best performance, while the DT model is the worst. The advantage of BPNN mainly comes from the nonlinear mapping ability and the flexibility in use [48]. Specifically, the hidden layer with the Sigmoid function contributes to the nonlinear relationship mining, and the flexible design in the BPNN structure and training strategy helps explore the upper limit of modeling performance. The classification result for each category was also calculated and displayed in a confusion matrix (Figure 4). For both the training set (Figure 4a) and the prediction set (Figure 4b), the classification accuracy of GM2 is the highest, and the classification accuracy of GM1 is essentially the same as CK. This result is consistent with Figure 2b, namely, CK and GM1 have the least difference in spectral intensity.

In view of the considerable samples of CK and GM1 overlapping in the 3D space (Figure 3), GA and SPA were performed to select feature bands to try to establish the BPNN model with higher classification accuracy. As a result, 23 feature bands were identified by GA, and 15 feature bands were identified by SPA (Figure 5). Based on the feature bands selected by GA, the classification accuracy of the prediction set achieved 0.861. Based on the feature bands selected by SPA, the classification accuracy of the prediction set achieved 0.826 (

Table 2. Performance of BPNN models established by feature bands.

Model	Accuracy-Train	Accuracy-Predict
BPNN-GA	0.874	0.861
BPNN-SPA	0.841	0.826

). So, the modeling performance based on feature bands improved obviously compared with the model established by PCs (Table 1). The progress in classification accuracy mainly results from that the GA and SPA take completely different strategies from PCA. In GA, the label information is also taken into account to construct a fitness function; hence, the selected feature bands really contribute to the classification. In SPA, the feature bands are selected according to the projection distance between different bands, so the selected feature bands bring the principal information in a concise mathematical form instead of fusing all the bands. Additionally, the BPNN-GA model is better than the BPNN-SPA model at classification accuracy. This is principally because of the use of the label information and fitness function. Moreover, similar locations (within 5 nm) of most feature bands selected by SPA can be found in the feature bands selected by GA. Therefore, as a swarm intelligence algorithm, GA is reasonable in mathematical interpretation too. Compared with the confusion matrices in Figure 4, the classification accuracy of CK and GM1 obtained by the BPNN-GA model (Figure 6) improved greatly, but the model is not robust enough as the classification accuracy of the training set is even less than that of the prediction set for GM2. As the reflectance value of spectra is sensitive to the environment, the simple utilization of feature bands’ reflectance easily limits the model performance in robustness. Therefore, it is sensible to research the modeling methods based on the interactive relationship of spectral bands at different wavelengths.

3.3. Classification Based on Deep Learning

The original spectra (Figure 7a) were transformed into 2D matrices (Figure 7b) according to Formula (2) to exploit the synergistic relationship of different bands. As the relative relationship is more stable to environmental interference, for example, the slope between different spectral bands is not so sensitive to the shift of illumination intensity, and the 2D matrices can improve the robustness of the classification model in theory. Furthermore, the 2D matrices bring extra information besides the original reflectance values retained in the back diagonal according to the transform formula. To verify the effect of 2D transformation, all the matrices of the training set were employed to train the VGG net constructed according to the description in Section 2.4. From

Table 3. Performance of VGG net models established by 2D spectral matrices.

Model	Accuracy-Train	Accuracy-Predict
VGG-normal	0.949	0.934
VGG-dilated	0.971	0.961

, it can be found that the modeling performance of the VGG net is much higher than that of the models based on feature engineering, and the classification accuracy of the prediction set achieved 0.961 for the VGG-dilated model. This demonstrates that the CNN model established based on 2D spectral transformation is effective at distinguishing the GM and non-GM maize kernels. As for the classification accuracy of each category (Figure 8), CK is the best, and only one CK maize kernel was misclassified as GM1 in the training set. However, the distinguishment between GM1 and GM2 was not so satisfactory, as seven maize kernels were misclassified for both GM1 and GM2 in the training set, three GM1 maize kernels were misclassified as GM2, and six GM2 maize kernels were misclassified as GM1 in the prediction set. In fact, the result in Figure 4 shows that it is easier to obtain an interpretation from the intuitive spectral differences (Figure 2b), while the result in Figure 8 is closer to the actual situation as CK is the original category, and some genes in GM1 and GM2 were modified in the same way. So, compared with the classification models based on feature engineering, the classifiers established based on deep learning are more powerful for digging out the essential information to express the differences between different maize kernels.

In addition, the VGG net with dilated convolution is better than that with normal convolution (Table 3). As the dilated convolution filter expands the receptive field on the 2D matrices (Figure 7b), there is a synergistic effect between the long-distance bands, which contributes to the acquisition of robust semantic features [49].

4. Conclusions

A complete and feasible scheme based on spectral information was creatively proposed to distinguish the GM and non-GM maize kernels, and the classification models with high accuracy and good robustness were finally obtained. The classification methods based on traditional machine learning and novel deep learning were fully studied, and the deep learning models established based on 2D-transformed spectral matrices showed better performance. However, we are unable to declare that the traditional machine learning methods are completely useless because the classification models established based on feature engineering have the advantages of less calculation load, mathematical transparency, and ease of updating. Therefore, for classification tasks similar to this research, it is better to begin with traditional machine learning. If the model based on feature engineering could not meet the requirements of accuracy and robustness, the deep learning method was then executed. This research provides a novel way to utilize spectra in CNN model establishment, namely, the 2D transformation of original spectra and the dilated convolution of 2D matrices. Concerning the inherent advantages of spectral detection, the scheme developed in this research can be applied to a situation wherein there is a need for rapid and nondestructive detection of large amounts of GM maize kernels with a low operating cost. In addition, there is still some work to be performed to improve and perfect this research, for example, the detection of unauthorized GMOs. In fact, the classification of GM and non-GM maize kernels can be treated as a foreshadowing for GMO detection because the 2D-transformed spectral matrices bring information on the relative relationships between different ingredients in the maize kernels. So, the interpretability of the CNN model will be studied in the future to reveal the response relationship from the 2D matrices to the final category result. According to the response relationship, a novel, rapid, and nondestructive scheme for unauthorized GMO detection in maize kernels is expected to be developed.