MMPCANet: An Improved PCANet for Occluded Face Recognition

Abstract

Principal Component Analysis Network (PCANet) is a lightweight deep learning network, which is fast and effective in face recognition. However, the accuracy of faces with occlusion does not meet the optimal requirement for two reasons: 1. PCANet needs to stretch the two-dimensional images into column vectors, which causes the loss of essential image spatial information; 2. When the training samples are few, the recognition accuracy of PCANet is low. To solve the above problems, this paper proposes a multi-scale and multi-layer feature fusion-based PCANet (MMPCANet) for occluded face recognition. Firstly, a channel-wise concatenation of the original image features and the output features of the first layer is conducted, and then the concatenated result is used as the input of the second layer; therefore, more image feature information is used. In addition, to avoid the loss of image spatial information, a spatial pyramid is used as the feature pooling layer of the network. Finally, the feature vector is sent to the random forest classifier for classification. The proposed algorithm is tested on several widely used facial image databases and compared with other similar algorithms. Our experimental results show that the proposed algorithm effectively improves the efficiency of the network training and the recognition accuracy of occluded faces under the same training and testing datasets. The average accuracies are 98.78% on CelebA, 97.58% on AR, and 97.15% on FERET.

1. Introduction

During the new coronavirus pneumonia epidemic, almost everyone wore masks, which invalidated face recognition technology as an important means of authentication. As a result, occluded face recognition is an urgent problem. Facial occlusion is the obstruction of facial information due to the influence of external factors. Complete facial information is often the key to normal recognition of faces by the security system; otherwise, it creates loopholes for criminals. Occluded faces are of different types, among which include complex illuminated occlusion, objects in front of faces, self-occlusion, etc. These circumstances may result in the distribution of the distorted images between the training and testing images [1]. In general, face recognition mainly includes three steps: face detection [2], feature extraction [3], and face classification [4]. Face detection is the segmentation of the face region from the whole image during subsequent feature extraction. Feature extraction is the extraction of the most discriminative features from each face image. Face classification is based on the extracted features to determine whether the unknown face image belongs to someone. At present, the key to solving the occlusion face recognition problem is still the extraction of effective features [5,6,7]. Our proposed method is centered between feature extraction and classification.

In recent years, researchers have investigated on a series of occlusion face recognition, which is mainly divided into deep learning and traditional. For example, Wright et al. [8] proposed sparse representation classification (SRC) for partially occluded face recognition. On the basis of SRC, Deng et al. [9] proposed an extended SRC (ESRC), whose basic idea is that intraclass variations of the same person can be shared by others. In order to solve the problem of weak discrimination and a large-scale occlusion dictionary, Du et al. [10] proposed kernel norm based adaptive occlusion dictionary learning (NNAODL). Gao et al. [11] proposed an approach named multilayer locality-constrained structural orthogonal Procrustes regression (MLCSOPR). One of the advantages of these traditional occlusion face recognition technologies is that they can achieve high recognition accuracy even in the case of few training images [12,13]. Methods based on deep learning have been the main research in recent years, but deep learning often incurs some challenges, including a large number of training parameters and a complex neural network structure. To improve this problem, PCANet was proposed by Chan et al. [14] at the end of 2015, and its most important contribution is to introduce subspace learning into deep learning. In other words, deep learning is connected with the traditional feature extraction method, which provides a new idea for the learning of a convolution kernel in a convolutional neural network (CNN) [15]. Compared with CNN, PCANet has a simple structure and does not need to adjust parameters by a random gradient descent algorithm in the learning process of the filter. To some extent, it avoids the overfitting problem caused by too few training data sets, and it also eliminates the tedious parameter adjustment process. At present, PCANet has been widely used in face recognition and texture recognition [16,17]. A large number of experiments have proven that PCANet still has very good performance in many different types of face databases [18,19], and they have also proven that it can extract deep features that are very suitable for classification [20,21].

It should be noted that, for occluded faces, PCANet can easily cause the loss of image information in the double PCA decomposition process. In addition, compared with the nonlinear mapping function of the activation function in CNN, PCANet has poor nonlinear fitting ability and cannot guarantee that the original data set is linearly separable [22]. Experiments by Alahmadi et al. [23] show that PCANet features are still greatly affected by occlusion. At the same time, the recognition accuracy of PCANet is low when the training samples are few. At present, the commonly used neural networks include DeepID2 [24], DeepID2+ [25], PCANet, FaceNet [26], and FROM [5]. Although they are inspired by the principles of human vision, the difference between the occlusion problem and the human eye is that the neural network needs a lot of training to obtain the occlusion face image information, whereas the human eye only needs to observe a small number of pictures. Therefore, it is still necessary to train a neural network for occlusion face recognition based on small samples. In order to solve the above problems, this paper proposes a multi-scale and multi-layer feature fusion based PCANet (MMPCANet) for occluded face recognition.

Table 1. Key differences between a number of previous methods and proposed methods.

Methods	Advantages	Disadvantages
ESRC [9]	Intraclass variations of the same person can be shared by others	Weak discrimination and large scale of occlusion dictionary
NNAODL [10]	Integrates the error image with training samples to construct the dictionary	In the case of extreme dark illuminations or unconstrained settings, the accuracy is low
PCANet [14]	Can be extremely easily and efficiently designed and learned	Easy to cause the loss of image information
L1-2D2PCANet [19]	Uses the L1-norm-based 2DPCA for filter learning	The training time is long
Methods based on deep learning [5,24,25,26]	Good performance and strong learning ability	Takes a lot of training samples and time
MMPCANet	Makes full use of the locality and continuity of occlusion space to obtain better spatial information	High storage overhead

shows key differences between a number of previous methods and the proposed methods. Our contributions are as follows:

(1) A multi-scale multi-layer feature fusion PCANet is proposed. Firstly, the channel concatenates the original image features and the output features of the first layer, and then it uses the concatenated result as the input of the second layer to regain better spatial information. In addition, in order to avoid the loss of spatial information of the image, a spatial pyramid is used as the feature pooling layer of the network.

(2) The results of the original image extracted by MMPCANet are often high-dimensional data. Random forest not only processes high-dimensional data efficiently, but it also obtains an unbiased estimation of the internal generation error. Using it instead of a support vector machine (SVM) [27] classifier can avoid many parameter adjustment problems, and it has very high accuracy.

(3) Experimental results show that our proposed MMPCANet outperforms PCANet on multiple benchmark datasets.

The organizational structure of this paper is as follows. In Section 2, the process of MMPCANet is described in detail. In Section 3, the proposed algorithm is analyzed and compared with PCANet. In Section 4, comparative experiments are conducted with other algorithms on a large number of databases. Section 5 summarizes the findings of this study.

2. Proposed Method

2.1. Multi-Layer Feature Fusion PCANet

Inspired by DenseNet [28], PCANet (MPCANet) is proposed to fuse different levels of feature information. MPCANet first concatenates the original image features and the output features of the first layer of the network at the channel level (channel concatenation), and then it takes the concatenated results as the input of the second layer of the network; therefore, MPCANet uses more feature information than PCANet. MPCANet includes the input layer, convolution layer, output layer, and classification layer. Its specific design process is as follows:

(1) Input layer of MPCANet: First, block sampling is performed on each input training image. Assuming that there are N training images of $m\times n$ size, the filter size is $k_1\times k_2$. Taking each pixel of the image $I_i^1$ as the center, l $k_1\times k_2$ blocks are obtained and vectorized. This allows the input image to build a matrix shaped like $X_i$.

$$X_i=\left[x_{i,1},x_{i,2},\cdots ,x_{i,\tilde{m}\tilde{n}}\right]ϵR^{k_1{k}_2\times \tilde{m}\tilde{n}}$$

(1)

where $\tilde{m}=m-k_1+1, \tilde{n}=n-k_2+1$. Then, the average value of the column is subtracted from each column to obtain the demeaned matrix ${\overline{X}}_i=\left[{\overline{x}}_{i,1},{\overline{x}}_{i,2},\cdots ,{\overline{x}}_{i,\tilde{m}\tilde{n}}\right]$. This is repeated for all N training images, and all ${\overline{X}}_i$ are put together to form a large matrix X.

$$X=\left[{\overline{X}}_1,{\overline{X}}_2,\cdots ,{\overline{X}}_N\right]ϵR^{k_1{k}_2\times N\tilde{m}\tilde{n}}$$

(2)

$X$ is the input of MPCANet.

(2) The first layer convolution: Let the number of filters be $L_1$. Firstly, $X{X}^{\mathrm{T}}$ is decomposed by SVD, and the feature vectors $V_l^1=\left\{V_1^1,V_2^1,\cdots ,V_{L_1}^1\right\}$ corresponding to the first $L_1$ main eigenvalues of $X{X}^{\mathrm{T}}$ are obtained. Subsequently, each feature vector is represented as a matrix with the size of $k_1\times k_2$. Each matrix is regarded as the first layer of MPCANet. In other words, the filter of first layer convolution can be expressed as follows:

$$W_l^1={\mathrm{mat}}_{k_1{k}_2}\left(V_l^1\right), W_l^1ϵR^{k_1\times k_2}$$

(3)

In the expression, $\mathrm{mat}$ is a function that maps the l-th feature vector $V_l^1$ to matrix $W_l^1$. Therefore, for image $I_i^1$, the output after the first layer convolution can be obtained by the following expression:

$$I_{i,l}^1=I_i^1*W_l^1, i=1,2,\cdots ,N; l=1,2,\cdots ,L_1$$

(4)

Mark $*$ denotes two-dimensional convolution [14]. For each input image, $L_1$ output maps can be obtained after the first layer convolution.

(3) The second layer convolution: Unlike the original PCANet, MPCANet concatenates the original input image with the output of the first convolution layer along the channel, and then it concatenates the result as the input of the second convolution layer; i.e., the input of the second convolution layer of MPCANet is $I^2$.

$$I^2=\left[I_1^1,I_2^1,\cdots ,I_N^1,I_{1,1}^1,\cdots ,I_{1,L_1}^1,\cdots ,I_{N,1}^1,\cdots ,I_{N,L_1}^1\right]\in R^{k_1{k}_2\times \left(L_1+1\right)Nmn}$$

(5)

$L_2$ is the number of filters in the second convolution layer [14]. $I_i^1$ and $I_{i,l}^1$ represent the output of the i-th original image and its first layer convolution, respectively. Enter $I^2$ and repeat almost the same operation as the first layer to learn the filter kernel of the second layer, denoted as $W_j^2=\left\{W_1^2,W_2^2,\cdots ,W_{L_2}^2\right\}$. Therefore, for each $I^2$ there are $L_2$ outputs corresponding to:

$$U_i^l={\left\{I_i^2*W_j^2\right\}}_{j=1}^{L_2},i=1,2,\cdots ,\left(L_1+1\right)N;l=1,2,\cdots ,L_1+1$$

(6)

(4) Output layer: The output layer includes nonlinear processing, block histograms, and feature concatenation. After the second layer convolution, each original input image $I_i^1$ has a $L_1+1$ set of output maps, and each group contains $L_2$ feature maps; i.e., a total of $\left(L_1+1\right)L_2$ feature maps are generated. For each set, the Heaviside step function H is used to binarize each output map, and then the binary hash code is used to convert each $L_2$ output map into an output matrix D with an element value of $0~2^{L_2}-1$.

$$D_i^l={\displaystyle \sum }_{j=1}^{L_2}{2}^{j-1}H\left(U_i^l\right)$$

(7)

A total of $L_1+1$ output matrices $D_i^l\left(l=1,2,\cdots ,L_1+1\right)$ can be obtained. Finally, the block-based histogram [29,30] is used to obtain the feature of each image. $D_i^l$ is divided into B blocks, and the histogram information of each block is counted. Then, the B histogram features are serially represented as $Bhist\left(D_i^l\right)$. Finally, all $Bhist\left(D_i^l\right)$ are connected into a matrix named the extended histogram feature vector, which is regarded as the feature representation $F_i$ of input image $I_i^1$.

$$F_i={\left[Bhist\left(D_i^1\right),Bhist\left(D_i^2\right),\cdots ,Bhist\left(D_i^{L_1+1}\right)\right]}^{\mathrm{T}}, F_i\in R^{2^{L_2}\left(L_1+1\right)B}$$

(8)

(5) Classification layer: the feature vector $F_i$ obtained by the network output is used as the input of the random forest classifier, and the classifier is trained to obtain the classification results [14]. Kremic and Subasi [27] used two methods to prove that the random forest algorithm is effective for face data.

In addition, although MPCANet is only a two-layer network model, it is as easy to extend to three or more layers as PCANet. Three-layer MPCANet (named MPCANet3) is also available. On the basis of retaining the first two layers of MPCANet structure, we cascade the output of the first layer and the second layer, and then we use the cascade results as the input of the third layer. Although we only extend to three layers, it can be extended to multiple layers in theory. A simple comparison of the processes for PCANet, MPCANet, and MPCANet3 is shown in Figure 1.

2.2. Multi-Scale PCANet

Section 2.1 establishes the mapping of input data from low-level features to high-level features using the convolution layer, nonlinear processing layer, and a block histogram, which extract the high-level semantic features of input samples. However, only extracting high-level semantic features as the final feature vector ignores the low-level features of the data, which contain more local details. The authors of [31] show that the multi-scale feature vector representation of samples combining the local detail information of low-level features and the abstract semantic information of high-level features can effectively reduce the information loss caused by multiple convolution and pooling operations. At the same time, it extracts the original redundant features by the spatial pyramid maximum pooling method, aggregates the extracted feature vectors, and retains the important information in the image after discarding irrelevant information; therefore the network obtains better noise resistance and robustness [32].

Based on the above analysis, a multi-scale and multi-layer feature fusion PCANet (MM-PCANet) is proposed. The feature pooling layer is added to MPCANet, and its structure is shown in Figure 2. $I_i$ is the input sample; $W_{L_1}^1$ and $W_{L_2}^1$ are the convolution kernels of the first layer and the second layer in the convolution layer, respectively; and $I_{i,l}^1$ and $U_i^l$ are the convolution outputs of the first layer and the second layer in the convolution layer, respectively. In the nonlinear processing layer, $D_i^1\left(L_1\right)$ and $D_i^2$ are obtained by nonlinear processing of $I_{i,l}^1$ and $U_i^l$. The specific improvement steps are as follows:

(1) The spatial pyramid pooling strategy is introduced into the feature pooling layer to construct the feature descriptor of the image layer by layer. $F_{i,1}$ and $F_{i,2}$ are the feature vectors of each layer $D_i^1\left(L_1\right)$ and $D_i^2$ obtained by spatial pyramid pooling. The feature vectors of different scales obtained by spatial pyramid pooling are cascaded to obtain multi-scale features of the image. $F_i=\left[F_{i,1},F_{i,2}\right]$ is the feature vector obtained by the concatenation of $F_{i,1}$ and $F_{i,2}$.

(2) $F_i$ is sent to the random forest algorithm for image classification. Experiments have proven the effectiveness of the random forest algorithm on face data. Therefore, the random forest algorithm, which is better at processing high-dimensional data, is used to replace SVM [33] for classification training of fused features, which has achieved good classification results. The random forest algorithm used has been described in detail in [34].

The multi-scale strategy is added to extract the low-level feature $F_{i,2}$ of the input sample by binary weighting and by the spatial pyramid pooling operation on the first layer output in the convolution layer. Concatenating the underlying feature $F_{i,2}$ and the high-level feature $F_{i,1}$, MMPCANet can extract both the high-level semantic information and the underlying detail information, thus obtaining the feature representation of multi-scale and multi-layer feature fusion of input samples.

The multi-scale features described in this paper represent the multi-scale relationship of input samples from low-level features to high-level features, and the image pyramid represents the multi-level relationship of the spatial pixel level, which is shown in Figure 3. U is a set of feature vectors extracted from local histograms, $l\in L$ denotes the partition of the l-th level of the space, L is an integer greater than or equal to zero, and each level has $2^l$ subsets; i.e., $2^l$ grids are divided in the horizontal and ordinate directions, and the number of image blocks is $2^l\times 2^l=4^l$. The pooling algorithm is applied to the feature vector in the L level, and the pooling function is expressed as

$$Z=F\left(U\right)$$

(9)

Since mean pooling easily loses the details of the target object in the image, this paper uses the maximum function as the spatial function. The maximum value of the dimension corresponding to the feature vector extracted from the image represents the corresponding response degree of the image. The formula for calculating the maximum value of local image block coding is as follows:

$$z_j=\max \left\{\left|u_{1j}\right|,\left|u_{2j}\right|,\cdots ,\left|u_{mj}\right|\right\}$$

(10)

where $z_j$ is the j-th element of $Z$, $u_{mj}$ is the j-th column element of the m-th row of matrix $U$, and m is the number of feature vectors. The maximum pooling function is used for the characteristics of each region at each level of the pyramid, and, after the pooling of all levels, the vectors are combined to complete the representation of high-level local characteristics. The advantage of introducing a spatial pyramid based on a local histogram is that the spatial pyramid uses the maximum pooling function to have strong robustness to local noise. At the same time, the image feature vector is represented by the image of the spatial pyramid in the pooling operation, which can give position information to the characteristics of the image and make the classification more accurate. With the increase in the level of division, the density of the region will also increase, and the corresponding vector dimension will increase, resulting in an increase in computational complexity. The experimental results show that, when $L \ge 3$, the computational complexity is large, and the accuracy is not significantly improved. Therefore, this paper uses the spatial pyramid model with $L=2$; i.e., the image is divided into three levels ($l=0, 1, 2$), and, after the pooling of the three levels, the feature vector is spliced as the feature vector of the high-level feature to represent $f_2$.

3. The Analysis of the Proposed Algorithm

This section mainly analyzes the differences between PCANet and MMPCANet.

The original PCANet only considers the feature vectors corresponding to k maximum eigenvalues, and it discards the remaining feature vectors that also contain useful identification information, which leads to the loss of this part of useful identification information. A simple way to avoid information loss is to use feature vectors in PCA as much as possible to construct convolution kernels. However, such operations lead to a large increase in computational complexity and the final feature dimension. Therefore, random forests that are good at dealing with high-dimensional data to improve classification efficiency are used. The input of each layer of MMPCANet comes from the output of all previous layers and realizes feature reuse through the parallel connection of features, thus achieving better performance with fewer parameters. In other words, combining the feature information of each layer improves the ability of the features extracted from the network, and thus the transfer of features is strengthened.

It is found that even if there are many occlusions in the image, taking full advantage of the non-occluded features can still increase the performance of face recognition [25]. The low-level feature of the neural network is the structure information of the image, which is also known as the edge texture information. High-level features are abstract semantic information of images. The fusion of the two features is proven to be more effective than the simple deep feature expression. Our experimental results also show that the robust feature extraction of occluded faces can reduce the impact of occlusion on face recognition and improve the robustness and accuracy of occluded face recognition.

Different from the traditional convolution neural network, in the convolution layer of the MMPCANet model, the convolution kernel is obtained by the PCA algorithm from the training concentration rather than by the Stochastic Gradient Descent (SGD) algorithm for adjustment. Compared with the existing stochastic gradient descent, our model can effectively avoid the shortcomings of excessive parameters, long training time, and the need of experience for initializing parameters and fine tuning.

In order to test the effect of the random forest algorithm in face recognition, this paper compares the differences by accuracy and training time between the random forest and the SVM in CelebA datasets [35], and it defines the factor α to represent the average training benefit of the network to the sample.

$$\alpha =\frac{p}{c}$$

(11)

where $c$ represents the time spent training a sample and $p$ represents the recognition rate of the network. Figure 4 shows the curve of the average factor between the random forest and the SVM in CelebA datasets (the same number of training samples and test samples). It can be seen from Figure 4 that the average factor of random forest is always greater than that of SVM when the number of training samples is different. When the number of training samples is more than 500, the convergence speed of the average factor of random forest is significantly faster than that of SVM. It is proven that random forest has better performance than SVM on small data sets. In the case of the shortage of occlusion face data sets, random forest is more practical than SVM, and the average training time of random forest is 0.15 s faster than SVM.

4. Experiments and Results

In this section, the proposed algorithm is evaluated on three public face databases. These face databases are the CelebA dataset, the AR face database [36] and the FERET database [29]. The detailed comparison table of datasets is shown in

Table 2. Detailed comparison of datasets.

Datasets	Subject	Images	Description
CelebA [35]	10,177	202,599	Includes faces without occlusion in front and occluded faces, such as wearing sunglasses and smiling expressions.
AR [36]	126	3276	Includes different expressions, illumination, and occlusion changes.
FERET [29]	1199	14,126	The images of the same person have different expressions, illumination, posture, and age changes.

.

In order to illustrate the performance of MMPCANet in tasks, the algorithm is compared with classical non-deep learning methods (ESRC, NNAODL), PCANet, 2DPCANet [30], and L1-2D²PCANet [19]. For comparative analysis, PCANet, 2DPCANet, L1-2D²PCANet, and the new network proposed in this paper are set with the same network parameters. The number of filters in the first and second stages is $L=8$, the filter size is set to $5\times 5$, the local histogram block size of the output layer is $7\times 7$, and the block area overlap ratio is zero. Each experiment was run 20 times, and the average was computed as the result. The results show that, compared with other similar algorithms, the accuracy of our algorithm has a great advantage. The following is the specific experimental analysis of each database.

4.1. Experiments on CelebA

CelebA includes faces without occlusion in front and occluded faces. It has a total of 202,599 face images of 10,177 individuals who are well marked with 40 attribute images, such as wearing sunglasses and smiling expressions. Due to the diversity of occlusion, positive samples are defined as faces without occlusion in front, including faces wearing ordinary glasses, and are manually labeled as l in the sample annotation; negative samples are other face images, including wearing glasses, side faces, overlooking, and other face images, and are manually labeled as -l in the sample annotation, as shown in Figure 5. A total of 100 people were selected in the CelebA dataset as the test object, 50 positive samples and 50 negative samples for each person were selected as training samples, and 1000 (500 positive samples, 500 negative samples) were selected as testing samples. Color images of 218 × 178 in the CelebA dataset are preprocessed to 200 × 150 gray images.

Table 3. Average Recognition Rate (%) on CelebA dataset.

Method	Positive Samples	Negative Samples
ESRC [9]	89.30	86.40
NNAODL [10]	96.09	95.58
PCANet [14]	95.20	94.93
2DPCANet [30]	98.02	96.33
L1-2D2PCANet [19]	97.58	97.50
MMPCANet	99.07	98.49

shows the comparison of experimental results on 1000 testing samples. As it is shown, our algorithm obtains a higher average recognition rate than the original algorithm, with an average increase of 3.87% in positive samples and 3.56% in negative samples. It indicates that MMPCANet is more suitable for processing complex images, which proves the advantages of multi-layer fusion features compared with traditional single-layer features. Compared with other similar algorithms, our algorithm also has significant advantages, which are 12.09%, 2.91%, 2.16%, and 0.99% higher than those of ESRC, NNAODL, 2DPCANet, and L1-2D²PCANet, respectively.

In order to verify the performance of the proposed algorithm in all aspects, we conducted time–cost experiments.

Table 4. Performance on CelebA dataset.

	Positive Samples		Negative Samples
	Training Time (s)	Testing Time (s)	Training Time (s)	Testing Time (s)
PCANet [14]	454.83	0.31	499.48	0.30
2DPCANet [30]	374.07	0.23	404.80	0.23
L1-2D2PCANet [19]	554.81	0.17	577.59	0.18
MMPCANet	493.03	0.16	512.69	0.17

shows the average training time and test time of different methods on the CelebA dataset. Compared with PCANet, 2DPCANet extracts image features based on a two-dimensional image matrix, which saves computational time. Therefore, the training time of 2DPCANet is shorter than that of PCANet. MMPCANet takes a long training time, but in turn extracts more useful image features. The short test time of MMPCANet benefits from the random forest classifier.

4.2. Experiments on the AR Database

The AR database contains 3276 images of 126 people, which has an average of 26 per person, collected in two different time periods, including different expressions, illumination, and occlusion changes. The experiment was deployed in the following way and selected 100 individuals (50 males and 50 females) from the AR database as subjects. Eight face images per person in 100 objects were selected as training samples, one of which was not occluded and had no illumination change. Seven of them contained occlusion or illumination factors, as shown in Figure 6a. Testing set I is composed of 300 face images with normal light, strong light on the left, and strong light on the right, as shown in Figure 6b. Testing set II is composed of 300 face images with sunglasses occlusion mixed with normal light, strong light on the left and strong light on the right, as shown in Figure 6c. Testing set III is composed of 300 face images with scarf enclosures that contain normal light, light on the left, and strong light on the right, as shown in Figure 6d.

All images were cut and aligned before the experiment. In the selection of experimental data, testing sets I, II, and III contain common sunglasses, scarves, and posed face occlusions, respectively, in a real uncontrolled environment, and strong illumination makes them more in line with the real scene. In order to verify that the model can effectively deal with occluded faces, we compared the face recognition methods commonly used in ESRC, NNAODL, PCANet, 2DPCANet, and L1-2D2PCANet, as shown in

Table 5. Average Recognition Rate (%) on AR database.

Method	Testing Set I	Testing Set II	Testing Set III	Avg.
ESRC [9]	91.05	89.14	84.64	88.27
NNAODL [10]	95.74	92.44	91.69	93.29
PCANet [14]	93.27	90.48	87.66	90.47
2DPCANet [30]	95.04	90.89	92.90	92.94
L1-2D2PCANet [19]	96.05	95.88	94.43	95.45
MMPCANet	98.95	96.56	97.23	97.58

.

The experimental results show that the recognition rates of the original algorithm and other methods are lower than our method. In testing set I, the recognition rate of MMPCANet is 7.9%, 3.21%, 5.68%, 3.91%, and 2.9% higher than ESRC, NNAODL, PCANet, 2DPCANet, and L1-2D²PCANet, respectively. In testing set II, the recognition rate of MMPCANet is 7.42%, 4.12%, 6.08%, 5.67%, and 0.68% higher than that of ESRC, NNAODL, PCANet, 2DPCANet, and L1-2D²PCANet, respectively. In testing set III, the recognition rate of MMPCANet is 12.59%, 5.54%, 9.57%, 4.33%, and 2.8% higher than that of ESRC, NNAODL, PCANet, 2DPCANet, and L1-2D²PCANet, respectively. It is easy to see that the recognition rate of MMPCANet in self-occlusion (testing set I) is higher than that in scarf occlusion (testing set III), and the recognition rate of scarf occlusion (testing set III) is higher than that of mirror occlusion (testing set II). In these three cases, we suspected that eye features are more recognizable than mouth features. After further verification, we added testing set IV, under which we manually added occlusion to face images. As shown in Figure 7, we divided images into eight occlusion levels, namely, 0% occlusion, 10% occlusion, 20% occlusion, 30% occlusion, 40% occlusion, 50% occlusion, 60% occlusion, and 70% occlusion. Considering 80%, 90%, and 100% occlusion, the image does not contain facial features and therefore was not considered. Then, the accuracy of face recognition was compared based on this dataset. The experimental results are shown in

Table 6. Average Recognition Rate (%) on AR database.

Occlusion Percentage	0%	10%	20%	30%	40%	50%	60%	70%
ESRC [9]	92.79	89.94	85.46	79.70	70.76	53.73	30.52	10.36
NNAODL [10]	95.71	93.48	89.62	83.49	79.99	71.96	44.05	12.58
PCANet [14]	96.46	94.14	91.29	81.72	77.21	70.93	41.98	11.27
2DPCANet [30]	97.21	95.51	91.09	82.19	76.85	67.79	39.39	11.34
L1-2D2PCANet [19]	98.04	97.32	92.78	85.86	78.34	69.93	44.48	12.16
MMPCANet	98.93	97.51	93.53	89.37	85.60	79.32	66.55	13.99

. Figure 8 shows the accuracy of face recognition under different occlusion rates for different face recognition algorithms.

From the test samples and experimental results, it can be seen that, when the face image is unshielded or when the occlusion rate is 10%, each deep learning network can achieve a very high face recognition rate. From 20% occlusion to 60% occlusion, the face recognition rates of other deep learning networks decrease rapidly. The more occlusions that there are, the fewer useful face recognition features that there are that can be extracted. However, in the case of 20%~60% occlusion, eye features can still be extracted, so the proposed method can also effectively capture eye features for face recognition. However, when the occlusion is increased to 70%, the recognition rates of all networks are approximately low (only about 12%), and it can be seen that almost no facial features can be learned by the network.

4.3. Experiments on FERET Database

In this section, the proposed algorithm is tested on the FERET database. The FERET dataset is a face database established in 1993 by the United States Department of Defense Advanced Research Project and FERET project team, which was established by the United States Army Research Experiment. As a result of the image having complex intra-class variability, it is often used to evaluate the performance of face recognition algorithms. It is divided into five parts: Fa (1199 subjects per image), Fb (with expression variation), Fc (image composition under different light conditions), DupI (with age variation), and DupII (a subset of DupI). Figure 9 shows a performance comparison between the various methods and the proposed algorithm. We used face detection tools to cut and standardize the images. The recognition rate comparison results of different algorithms on the FERET database are shown in

Table 7. Average Recognition Rate (%) on FERET database.

Method	Fb	Fc	DupI	DupII
ESRC [9]	91.98	84.05	79.85	72.94
NNAODL [10]	96.21	92.78	88.26	86.22
PCANet [14]	94.44	91.84	85.33	86.40
2DPCANet [30]	95.65	98.98	93.52	90.37
L1-2D2PCANet [19]	97.93	97.82	93.15	92.48
MMPCANet	99.74	99.87	93.70	95.30

.

Through Figure 9 and Table 5, it can be seen that, in most cases, the proposed method outperforms other methods. It is particularly evident in DupI and DupII, indicating that MMPCANet has strong robustness for samples with age variation. The average correct recognition rate in DupII is 8.9% higher than that in PCANet, and the average correct recognition rate in DupI is 8.37% higher than that in PCANet. The smallest improvement is that the proposed algorithm in Fc only increases by 2.05% compared with L1-2D²PCANet. The above experiments show that the proposed algorithm can effectively improve the accuracy of face recognition.

According to the above experiments, most of the traditional algorithms focus on the high-level features of the image when using features to express the image, which is an effective expression method for clean images. However, for occluded images, since occlusion is also a part of the feature, the occlusion of the input model is easily confused with facial features, and it even seriously affects the extraction of key features. It is also easy to classify occluded features as key features, which is the main problem of the low recognition rate of occluded face images. The input of each layer of MMPCANet comes from the output of all previous layers and realizes feature reuse through the parallel connection of features. This is the reason why MMPCANet can achieve good results with a low training time. The current bottleneck that keeps MMPCANet from becoming deeper is that the dimension of the resulting feature increases exponentially with the number of stages. Fortunately, the random forest classifier can solve this problem. Experiments on three datasets verify the effectiveness of the proposed method, and, compared with other methods, it can obtain better classification results.

5. Conclusions

This paper proposes a multi-scale and multi-layer feature fusion based PCANet (MMPCANet) for occluded face recognition. The channel concatenates the original image features and the output features of the first layer, and then it uses the concatenated result as the input of the second layer to regain better spatial information. In addition, in order to avoid the loss of spatial information of the image, a spatial pyramid is used as the feature pooling layer of the network. Finally, random forests that are good at dealing with high-dimensional data are used for classification. Compared with the original PCANet, our algorithm can extract more low-level features of the image and does not require a large number of training samples, but it is more time-consuming. Therefore, it has certain advantages for occluded face recognition. The non-occluded area features of the face were almost fully extracted. The average accuracies are 98.78% on CelebA, 97.58% on AR, and 97.15% on FERET. Our next goal is to add occlusion detection in MMPCANet in order to target occluded faces for feature extraction, thereby reducing the computational cost. The experiments compared ESRC, NNAODL, PCANet, and 2DPCANet, and it achieved the expected results in recognition rate and training efficiency.