A CNN-Based Method for Counting Grains within a Panicle

Abstract

The number of grains within a panicle is an important index for rice breeding. Counting manually is laborious and time-consuming and hardly meets the requirement of rapid breeding. It is necessary to develop an image-based method for automatic counting. However, general image processing methods cannot effectively extract the features of grains within a panicle, resulting in a large deviation. The convolutional neural network (CNN) is a powerful tool to analyze complex images and has been applied to many image-related problems in recent years. In order to count the number of grains in images both efficiently and accurately, this paper applied a CNN-based method to detecting grains. Then, the grains can be easily counted by locating the connected domains. The final error is within 5%, which confirms the feasibility of CNN-based method for counting grains within a panicle.

1. Introduction

With the global population growth, the demand for high-yield crop is increasing [1]. Rice, as the most common food, has become the research object of many researchers. Generally, genotype and phenotype are combined to study rice cultivation [2,3], and how to calculate the yield is one of the core questions. Phenotypic studies usually include traits such as leaves, grains, spikes, etc. [4] Grains’ number within a panicle can directly reflect the yield. Manual phenotypic analysis is a time-consuming and laborious task. Studies show that introducing computer vision technology to the field of phenotypic analysis can greatly increase plant productivity [5]. To meet the needs of automation in phenotypic research [6,7,8], methods of image pattern recognition, such as image segmentation based on the OTSU algorithm and GA (Genetic Algorithm) [9], is applied to the grain counting, which can also be used to get the number of internodes and primary branches. However, the result is not accurate, due to the excessive image patterns of the grains.

In recent years, many intelligent algorithms have started to be applied in agriculture [10,11,12]. The approach that has achieved many remarkable results in the field of agricultural image processing is deep learning [13]. The convolutional neural network (CNN) is one of the main methods of deep learning in the field of computer vision [14]. CNN is widely used for tasks such as image recognition [15], target detection [16], image segmentation [17], and image reconstruction [18], and performs better than traditional methods or even humans in many fields [19]. This is mainly because CNN can effectively extract the features of images [20,21], and some improved CNN structures perform well in target detecting [22,23,24]. A deep neural network has better capability of mining the features of an image with respect to the general neural network. It is suitable for establishing nonlinear and complex mapping relationship. However, the disadvantage is the explosive growth of parameters, especially in image processing. CNN is such a kind of deep neural network designed to reduce the number of parameters, but it maintains the ability to extract complex features of images. A typical CNN structure is (LeNet5) [25].

CNN mainly consists of three kinds of layers: the convolution layer, the pooling layer, and the fully connected layer. The convolution layer is the core part of CNN, using the convolution kernel to share parameters [26]. The size of a convolution kernel is generally no larger than 7 × 7, and the number of convolution kernels depends on that of channels of input and output in each layer. The pooling layer further reduces the number of parameters [27]. Generally, the maximum or average pooling method is adopted. The fully connected layer is the same as that in traditional neural network, which connects each “pixel” in the input map with that in output map, playing a role of integration.

Because CNN has a strong feature extraction capability, it is possible to use CNN for grain counting. There are precedents for using CNN for the segmentation and counting of granular objects. In 2015, Olaf Ronneberger used U-Net to segment a variety of biological cell microscopic images, and the IOU of the segmentation results were all above 70%, which was significantly improved compared to the suboptimal algorithm [28]. Reza Moradi Rad used U-Net to count and centroid the human embryonic cells (blastomeres), which has an average accuracy of 88.2% for embryos of –one to five cells [29]. Jian Chen used U-Net to segment and count aphids on plant leaves, achieving an accuracy of 95.63% and a recall rate of 96.50% [30]. Thorsten Falk developed a suite of software for cell detection, segmentation, counting, and morphometry using U-Net and 3D U-Net for easy use by more researchers [31]. J. Luo proposed an Expectation Maximization (EM)-like self-training method that makes it possible to train U-Net cell counting networks with a small number of samples [32]. Yue Guo combines U-Net and the Self-Attention module to propose a new network structure SAU-Net, which reduces the generalization gap for small datasets [33]. Sambuddha Ghosal designed a deep learning framework for sorghum head detection and counting, which is based on weakly supervised method [34]. However, until now, no scholar has applied U-Net to count grains within a panicle.

This article is aimed at the grain count problem. The method of sample labeling was designed. The software of sample labeling was developed. The appropriate sample expansion method was proposed. Based on numerous panicle image samples, a method of labeling and a fully convolutional network [35] based on U-Net were designed to achieve the goal of detecting grains within a panicle. The appropriate loss function was selected to train the model. Combining with the method of detecting the connected domain, the counting of grains within a panicle was finally realized. Finally, the counting effect was evaluated. The results show that the error of this algorithm was within 5%.

2. Samples Generation

2.1. Image Acquisition Instrument

In order to perform genotypic–phenotypic analysis on rice, our laboratory has designed a seed and panicle phenotyping instrument, as shown in Figure 1. The main modules of this platform include a tablet computer, electronic weighing module, embedded Bluetooth control module, top light board, backlight board, transparent tray, box, etc. This instrument can perform wind selection, weighing, and imaging on seeds and panicles. The panicle samples are taken from the rice RIL (recombinant inbred lines) population in this study, which was derived by single-seed descents from a cross between Oryza sativa ssp. Indica variety JP69 and Oryza sativa ssp. japonica variety Jiaoyuan5A, which show high yield heterosis in hybridized combination and are widely planted in Shanghai, China. All plants were planted in a paddy field in Minhang (31.03° N, 121.45° E), Shanghai, during the summer seasons in 2017–2020. The experimental design was a randomized complete block design (including one row of each inbred line and eight plants in each row).

The main function of the phenotyping instrument in this article is to collect images. The related internal structure is shown in Figure 2. When collecting images, the whole earhead is fixed by several magnetic pressers, and the camera of the tablet computer is facing the center hole of the top light board to complete the image acquisition.

2.2. Image Annotation and Preprocessing

As the number of grains within a panicle is large, the accuracy will be low if the grains’ number is directly selected as the output of the network. A better way is to transform this counting problem into semantic segmentation and connected region counting, realized by CNN, of which the input is the original image and the output is a binary image. The pixels of grains are marked as 1, and the remaining pixels are marked as 0. It is difficult to separate some connected regions when two or more grains are stuck together if the outline of the grain is marked. Therefore, this paper used spots to mark the grains’ locations, and this process is supported by a specific software developed in this paper. It is important that, if a grain is severely blocked, it will not be labeled. The diameter of the spot is 16 pixels.

Figure 3 shows the interface of the marking program. The left button can be clicked to mark the location of a grain, and the right button can be clicked to delete the mark. The final result is as shown in Figure 4.

The size of the original panicle image is 3264 × 2448, and it is reduced by a ratio of 0.77. Then, it is cut into many 160 × 160 (pixels) blocks. Generally, a block contains 10 or fewer grains. Random rotation, flip, and scaling are applied to the original samples to enrich the samples. The range of random rotating angle is from 0 to 360 degrees, and the proportion of random scaling is from 0.8 to 1.2.

More than 80% of the region in a panicle image is background. A distribution unbalance of effective and ineffective samples will arise if training samples are directly generated by a sliding window, which affects the speed and accuracy of the training process. Thus, the ratio of the number of images containing grains to that of pure background is controlled to about 5:1, as shown in Figure 5. The sixth image contains no grain, marked as pure background.

This paper has taken 120 pictures of panicles, of which 80 are chosen as the parents of the training set. After processing, more than 40,000 samples were generated.

3. Design of Neural Network

The input of the network is a RGB image, and the output is a probability image of single channel. The value of each pixel represents the probability of whether the location belongs to the center of the grain. The size of output image is in accordance with that of the input image. The overall structure is shown in Figure 6.

This network consists of the input, nine convolution cells (from C1 to C9), and the output. Each rectangle represents a set of feature maps. The height of the rectangle stands for the size of feature maps. The width stands for the number of channels. The input is a RGB image, having three channels. The output is an image of probability, only having one channel. C1, C2, …and C9 are complex convolution cells, each containing two convolution layers, and the sizes of convolution kernels are all 3 × 3. The number of convolution kernels depends on the channels of feature maps. The blue arrows represent the pooling layers, of which the sizes are all 2 × 2, and the number of channels remains the same after the pooling process. The green arrows represent the up-sampling layers, using the interpolation algorithm to double the size of feature maps. The grey arrows represent the copying and merging process, which reduces the loss of original information caused by up-sampling. The activation functions are all Rectified Linear Units (ReLUs), except the last layer, of which the activation function is the sigmoid function. The size of the input image is marked as H × H, where H should be greater than 48 and divided by 16. The number of channels in C1 is K, and the remaining parameters are as shown in

Table 1. Feature map dimensions of convolutional neural networks.

Index	Size	Activation Function
Input	H, H, 3	-
C1	H, H, K	ReLU
C2	H/2, H/2, 2 K	ReLU
C3	H/4, H/4, 4 K	ReLU
C4	H/8, H/8, 8 K	ReLU
C5	H/16, H/16, 16 K	ReLU
C6	H/8, H/8, 8 K	ReLU
C7	H/4, H/4, 4 K	ReLU
C8	H/2, H/2, 2 K	ReLU
C9	H, H, K	ReLU
Output	H, H, 1	Sigmoid

.

The loss function is similar to the crossing entropy loss function. For each pixel p,

$$l(y_p^{*},y_p,\lambda )=-[\lambda y_p^{*}\log (y_p)+(1-y_p^{*})\log (1-y_p)]$$

(1)

where $y_p^{*}$ represents the label (1 or 0) and $y_p^{}$ represents the output of network (floating-point number, from 0 to 1). In this paper, the positive samples refer to the center region of the grains, which means the positive samples are much fewer than the negative samples. To reduce the influence of this unbalance, a coefficient $\lambda$ is necessary. In this paper, $\lambda$ is taken as 5, which is equivalent to supplementing the positive samples.

The overall loss function, L, is the sum of the loss of each pixel (Euclidean distance) in each batch of images. N represents the number of a batch of images. M represents the number of pixels in an image.

$$L=\frac{1}{NM}{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{p\in Y_i}l(y_p^{*},y_p,\lambda )}}$$

(2)

In this paper, the Jaccard Similarity Coefficient (IoU) is used to evaluate the convergence of the network.

$$IoU=\frac{{\displaystyle \sum (Y^{*}\otimes Y_t)}}{{\displaystyle \sum Y^{*}}+{\displaystyle \sum Y_t}-{\displaystyle \sum (Y^{*}\otimes Y_t)}+\delta }$$

(3)

where $Y^{*}$ represents the labeling image, and $Y_t$ represents the output image processed by a threshold of 0.5. The operator $\otimes$ means the multiplication of pixels at the same location. To avoid the denominator being 0, $\delta ={10}^{-5}$.

4. Training Process

To compare the influence of different sizes of input image and number of channels on the prediction accuracy and testing speed, four groups of parameters are selected to train the networks. (H, K) are, respectively, selected as (48, 10), (80, 16), (112, 24), and (160, 32). A total of 36,000 samples were selected as the training set and the remaining 4000 as the validation set. The number of training samples per batch was 32. The total number of training iterations was 10,000.

As shown in Figure 7, at the beginning, especially the first 2000 steps, the IoU rises faster and then converges. The network reaches the optimal convergence state. In addition, the larger H of the input image, the higher the obtained IoU.

Table 2. Network model size with different parameters.

Index	Parameters	Model Size (MB)
1	H = 48, K = 10	8.8
2	H = 80, K = 16	22.5
3	H = 112, K = 24	50.5
4	H = 160, K = 32	89.8

compares the occupied size of the models with different training parameters.

5. Result

A sliding window is used to intercept the image and evaluate it. The actual input of the network is a sub region of the original image. The boundary error may rise due to the lack of information from neighboring pixel. In this paper, image splicing is carried out by the way of boundary overlap. In the stitching process, the sliding step is slightly smaller than the length of the clipping image.

The boundary trimming offset is set to u pixels, and the width of overlapping region is set to v pixels. The sliding step(Sub) in sampling process is

$$Sub=H-2u$$

(4)

The sliding step(Step) in stitching process is

$$Step=Sub-v$$

(5)

Four groups of cutting amount (u, v) are applied to the predicting process as shown in

Table 3. Parameters in four groups.

Index	Parameters	Cutting Amount	Sub	Step
1	H = 48, K = 10	u, v = 4, 6	40	34
2	H = 80, K = 16	u, v = 6, 8	68	60
3	H = 112, K = 24	u, v = 8, 12	96	84
4	H = 160, K = 32	u, v = 10, 16	140	124

.

Most pixels in one image belong to the background. If the sliding window skips the region that does not contain grains, it will reduce much computation. In this paper, the color space was conversed from RGB to the HSV, followed by segmentation in the S channel. Then, the small debris was removed by operating morphological opening. The background was filtered based on the obtained foreground. As shown in Figure 8, the foreground was clipped. This method reduces the amount of calculation by 70%.

The result of the grain counting algorithm can be visualized by Figure 9.

6. Analysis

In this paper, four networks with different parameters are tested, and three evaluation criteria are compared.

Mean absolute error (MAE) is a conventional measure of absolute error, which is robust to outliers, as shown in Equation (6):

$$MAE=\frac{1}{N}{\displaystyle \sum _{i=1}^N|z_i-z_i^{\prime }|}$$

(6)

Mean squared error (MSE) is an important indicator for regression problems, which is more sensitive to outliers than MAE:

$$MSE=\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N{(z_i-z_i^{\prime })}^2}}$$

(7)

Mean relative error (MRE) is dimensionless and usually used to measure the relative error

$$MRE=\frac{1}{N}{\displaystyle \sum _{i=1}^N\frac{|z_i-z_i^{\prime }|}{z^{\prime }}\times 100\%}$$

(8)

where $z_i$ represents the predicted grain number, $z^{\prime }$ represents the actual grain number, and $N$ represents the number of test samples.

Figure 10 shows the result under different parameters. The blue points represent the actual value, and the red points represent the predicted value. The abscissa represents the image index, and the ordinate represents the grain number.

The performance comparison of models with different parameters is shown in Rows 1 to 4 of

Table 4. Grain number prediction results of different models with difference processing time.

Index	Model Description	MAE	MSE	MRE	Processing Time (ms)
1	H = 48, K = 10	5.00	8.41	4.05	6.40
2	H = 80, K = 16	4.75	7.78	4.05	27.30
3	H = 112, K = 24	4.38	7.31	3.66	88.90
4	H = 160, K = 32	3.75	4.63	3.47	264.90
5	Wavelet Model [36]	5.68	6.96	6.08	140.40
6	P-trap [37]	13.76	17.68	14.07	~2000.00 *

* P-trap is not an open source software, and the public users are denied from the runtime part for precisely evaluating the algorithmic computational time. Hence, the time is an estimated one that includes the display time and the computing time.

. Generally speaking, the larger H and K, the higher the accuracy. However, all of these four networks obtain high accuracies. All the MAEs are lower than 5.0%, which meets most requirements. The fourth network can be applied to a device with high performance, and the first network can be applied to a mobile device. Model 1 is chosen to be deployed on the Xiaomi Tablet 2 of our phenotype instrument, which took 4.6 s to process an entire image of rice panicle.

The performance comparison between the proposed model and classical algorithms such as wavelet [36] and P-trap [37] is shown in Rows 5 to 6 of Table 4. From the results, it can be seen that the method proposed in this paper shows lower MAE and MRE than the wavelet method, and the model with parameters H = 160 and K = 32 outperforms the wavelet method in terms of MSE metrics. The model we proposed also outperforms the P-trap method with respect to MAE, MSE, and MRE. Meanwhile, the computational time is logged to show the real-time performance of different algorithms. Typically, the processing times of different algorithms are evaluated on the platform of 2.0 GHz Intel Core Duo CPU, 2.0 G RAM, MATLAB environment, and WIN10 OS, which shows the computational workloads of the deep neural-network-based models are comparable to those of analytic models in their real-time performance. Since all the CNN-based models are fast enough, even for large size batch samples, the precision factor dominates the final practice of applications, and hereby, Model 4 with 89.8 M memory occupation and 264.90 ms computation time is the optional pre-installed software for the instrument. In conclusion, the method proposed in this paper has better performance than the classical method.

7. Conclusions

In this paper, a marking method for the grain counting within a panicle is designed based on the features of a panicle image, and a convolution neural network is designed and trained. In the prediction stage, the acceleration technique based on the foreground extraction is proposed, and different parameters are compared. The final MAEs are all lower than 5.0%, overcoming the difficulty of counting grains within a panicle.