A New Approach to Recognize Faces Amidst Challenges: Fusion Between the Opposite Frequencies of the Multi-Resolution Features

Abstract

This paper proposes a new approach to pixel-level fusion using the opposite frequency from the discrete wavelet transform with Gaussian or Difference of Gaussian. The low-frequency from discrete wavelet transform sub-band was fused with the Difference of Gaussian, while the high-frequency sub-bands were fused with Gaussian. The final fusion was reconstructed using an inverse discrete wavelet transform into one enhanced reconstructed image. These enhanced images were utilized to improve recognition performance in the face recognition system. The proposed method was tested against benchmark face datasets such as The Database of Faces (AT&T), the Extended Yale B Face Dataset, the BeautyREC Face Dataset, and the FEI Face Dataset. The results showed that our proposed method was robust and accurate against challenges such as lighting conditions, facial expressions, head pose, 180-degree rotation of the face profile, dark images, acquisition with time gap, and conditions where the person uses attributes such as glasses. The proposed method is comparable to state-of-the-art methods and generates high recognition performance (more than 99% accuracy).

1. Introduction

Face recognition works by matching facial features from the image under investigation with a collection of features known as a database [1]. With the rapid development of technology, the existing recognition performance based on machine learning has reached a certain level of maturity [2]. Due to the technology’s maturity, face recognition is being employed in various applications. One such use is the possibility of facial recognition as a payment option. Payment transactions can be completed using user facial authentication. The benefit of utilizing face recognition as a payment method is the ability to make payments without relying on a smartphone or any device when the user forgets to carry one or the phone is not working [3]. Facial recognition technology is safer than conventional payment methods, including inputting passwords to allow credit card or mobile payments [4].

A few obstacles must be overcome before face recognition can realize its full potential as a quick, safe breakthrough that facilitates payments. These challenges include head orientation, lighting conditions, facial expressions, aging, and hairstyles [5,6,7]. In this research, to overcome challenges and build a face recognition system that is robust and accurate, we propose to tackle these problems using combinations of methods based on image fusion. Image fusion is a technique that creates a single composite image by fusing several images from various sources while maintaining crucial details and improving overall quality. It has attracted much discussion in several areas, including computer vision, remote sensing, and medical imaging, where better analysis depends on integrating complementary data. Image fusion combines significant information from different sensors using various mathematical techniques to create a single composite image. This technique integrates complementary multi-temporal, multi-view, and multi-sensor data into one image with enhanced quality while preserving key features [8]. The fused image may provide more information than the source images. Due to these advantages, image fusion encompasses various tasks such as image enhancement, noise reduction, feature extraction, image matching, object recognition, image restoration, and tracking [9].

In face recognition, image fusion contributes to improving robustness or performance recognition. The approach of image fusion can be through a combination of visible and infrared images [10,11,12]; a fusion of multi-modal 2D and 3D face images [13,14,15]; a fusion of multi-modal biometric face and other biometric traits, e.g., face and ear [16]; face, iris, and fingerprint [17]; face, iris, and thumbprint [18]; or face and voice [19].

This research proposes an image fusion method using visible 2D face images. Several studies that previously investigated this approach are as follows. A study [20] built a facial recognition system using feature fusion based on facial features such as eyes, nose, and mouth and extracted global features using principal component analysis and local features using local binary patterns. The features were fused using a Laplacian pyramid and classified using an artificial neural network. The proposed model was tested against challenges such as variations in pose, illumination, expression, image resolution, and occlusion. The same features were further investigated and fused using multi-resolution singular value decomposition [21] and multi-resolution discrete cosine transform [22]. In [23], multi-resolution feature fusion that combined low- and high-resolution features was amplified using Gabor-feature hallucination. This proposed combination increased performance, especially in low-resolution face recognition. A study [24] proposed a hierarchy feature fusion on end-to-end layers in the convolutional neural network to tackle challenges such as illumination and occlusion. In [25], to improve recognition effectiveness, multi-feature fusion using 2D principal component analysis for extracting global features, and local binary pattern for extracting local features, was proposed. Feature fusion on a local binary pattern, elliptical local binary pattern, median binary pattern, and local phase quantization was proposed in [26] as improved local descriptors.

In [27], an age-invariant face recognition framework was built using feature fusion and decomposition. The authors presented methods of feature fusion based on recurrent neural network and linear feature fusion. The study [28] assembled a face recognition model based on a multi-scale feature fusion network using a multi-scale bilinear pooled convolutional neural network. In [29], the authors proposed estimated features of frontal face images from fused features based on a generative adversarial network to improve recognition of angled face images. An expression-based convolutional neural network was fused in the layer of multi-level feature fusion and transfer learning convolutional neural network [30] to tackle challenges such as facial expressions in face recognition. In [31], multi-modal biometrics of RGB data and texture descriptors from face and periocular were combined using a fusion strategy in the convolutional neural network. In [16], multi-modal biometrics of ear and face profiles represented by local phase quantization and local directional patterns, reduced by principal component analysis, were fused in a feature-level fusion to improve the overall recognition rate.

Looking deeper into image fusion, one strategy to find the best fuse features is, first, to decompose the source images using wavelet decomposition analysis. This process enables us to investigate source images with different scales and frequencies. The study [32] utilized weighted wavelet visual perception fusion. The study aimed to improve underwater images by correcting color and improving global and local contrast of the color-corrected images. The improved-contrast color-corrected images were then decomposed using the wavelet decomposition method. A study [33] investigated three wavelet fusion strategies, max-min, min-max, and mean-mean, to merge charge-coupled device red, green, and blue infrared images and depth-grayscale images. The experiment concluded that the min-max fusion strategy performed best with five levels of decomposition. In [34], feature fusion from the local binary pattern, Gabor, and local directionality pattern of eyebrows, eyes, mouth, and chin features was combined and classified using a support vector machine or artificial neural network. The study [35] proposed employing feature- and pixel-level fusion of wavelet sub-bands for face recognition. The feature-level fusion combined the results of the dimension-reduced approximation sub-band and the results from the pixel-level fusion of detail sub-bands. The best pixel-level fusion coefficients of detailed sub-bands were found using principal component and linear discriminant analyses. A study [36] used a weighted energy ratio to fuse wavelet packet decomposition images. The fused images were extracted using principal component analysis and the Fisher linear discriminant. A study [37] employed quaternion wavelet transform to decompose images into low-frequency and high-frequency coefficients. The low-frequency coefficients were fused using a weighted-average fusion rule based on phase, magnitude, and spatial variance. Meanwhile, the high-frequency coefficients were fused using the maximum fusion rule based on contrast and energy. The inverse quaternion wavelet transform was performed to obtain the final fused image.

Although accuracy and security have significantly increased due to advancements in facial recognition technology, integrating image fusion techniques poses a particular set of challenges. While image fusion results promise better information than the source images, they also introduce the complexity of determining what features or data are being fused. As mentioned, the image fusion approach can be made through a combination of different sensors, multi-modal biometric traits, and dimensional structures. The selected features to be fused need to be as complementary as possible. The overlap of unnecessary information should be minimized [38]. Moreover, it also raises the question of whether the image fusion can retain the visual information of the original image and how to tackle the effect of reduced contrast in the fused image [39].

We proposed using multi-resolution analysis based on discrete wavelet transform (DWT) to decompose images into low-frequency and high-frequency sub-bands. At the same time, the images were filtered using a Gaussian filter to produce blurred images or lower-frequency representation of the images. Then, the Difference of Gaussian was constructed by calculating the differences between the Gaussian blurred images with different sigma. This procedure resulted in high-frequency representation of the images. The low- and high-frequency sub-bands from the decomposition of discrete wavelet transform were then fused with opposite-frequency components from the Gaussian and Difference of Gaussian, i.e., the low-frequency sub-band from DWT was fused with Difference of Gaussian results, and the high-frequency sub-bands from DWT were fused with Gaussian results. These opposite frequencies from the same image reduced the challenges due to the image fusion technique requiring many inputs from several sources and solved the demand that features should be as complementary as possible.

The fusion methods employed Laplacian pyramid image fusion (LP-IF) or discrete wavelet transform image fusion (DWT-IF). The fusion rules were as follows: for LP-IF, averaging the base low-pass and highest absolute value for the fused high-pass coefficient; for DWT-IF, we found that the mean-mean rule fusion for both low- and high-frequency coefficients performed the best. To our knowledge, no research has ever proposed the combination of image fusion using discrete wavelet transform decomposition sub-bands with Gaussian filtered images and Difference of Gaussian coefficients. Using the multi-resolution image fusion also helped us to retain and improve the contrast of the original image. The low-frequency components represent the most energy in an image where they contain the slow change’s structure and the image’s brightness.

The fusion result was reconstructed as an enhanced image using inverse discrete wavelet transform. We used this approach to maintain and improve the information from the original image. A histogram of the oriented gradient was employed, and the extracted features were classified using a support vector machine. Our proposed method was tested against challenges in face recognition such as pose and lighting conditions, facial expressions, dark images, image acquisition variations like time gap and angle of the face profile, and attributes on faces like glasses and makeup. We showed that our proposed method produced high recognition accuracy and was robust against challenges such as lighting conditions, facial expressions, head pose, 180-degree rotation of the face profile, dark images, acquisition with time gap, and conditions where the person uses attributes such as glasses.

To be able to explain our research more clearly, we list several contributions from this paper:

• We proposed a new combination of image fusion using opposite-frequency components from discrete wavelet transform sub-bands and Gaussian-filtered images or Difference of Gaussian. The proposed method needed only the same images to produce the opposite frequency and solve the problems of inputs from multiple sources, and the requirements for features needed to be as complementary as possible.
• We examined the effects of Laplacian pyramid image fusion and discrete wavelet transform image fusion to improve recognition performance.
• We investigated the variation in parameters inside the methods, such as the wavelet family employed in multi-resolution analysis of discrete wavelet transform and the wavelet family and level of decomposition employed in discrete wavelet transform image fusion.
• Due to the high potential of other combinations inside our proposed method, including pixel-level or feature-level fusion of features, we showed that our chosen combination outperformed other combinations’ recognition performance.
• The classification methods were compared using a support vector machine with linear, quadratic, and cubic kernels; nearest neighbor; and neural network.
• Our proposed method was tested against challenges from six face datasets (four datasets and two sub-datasets) and compared with other image fusion and non-fusion methods.

The rest of this paper is organized as follows: Section 2 displays the materials and methods in our research; Section 3 discusses the findings and compares our results against other methods; Section 4 concludes our research.

2. Materials and Methods

We explain the materials in this experiment in Section 2.1 and the proposed methods in Section 2.2.

2.1. Face Datasets

We employed several face datasets for this research:

• The Database of Faces (AT&T) [40]
  The Database of Faces (AT&T Laboratories, Cambridge), formerly known as The ORL Database of Faces, consists of 400 face images from 40 people. The images are in grayscale, with each image having 92 × 112 pixels. The database has variations in lighting conditions and facial expressions, with some variations in glasses and no glasses. Several people were photographed at different times. The background of each image is dark, and the image was taken from the front with an upright position. Figure 1a displays examples of images from this database.
• The BeautyREC Dataset (BeautyREC) [41]
  The BeautyREC Dataset (BeautyREC) is a face makeup transfer dataset containing 3000 color images from various people. It is worth noting that this dataset is imbalanced. The size of each image is 512 × 512 pixels, then resized to 100 × 100 pixels and the color changed into grayscale images. The makeup transferring method is detailed in [41]. The makeup transfer dataset encompasses various makeup styles, face poses, and races. Figure 1b shows an example of images from this dataset.
• The Extended Yale B Face dataset (EYB) [42,43]
  The Extended Yale B (EYB) Face dataset has 2414 frontal face images with variations in pose and lighting conditions. All 2414 images are from a total of 38 people. In this research, we employed only 58 images from each person due to the extremely dark images. Each image has a size of 168 × 192 pixels in grayscale. Figure 1c displays an example of images from this dataset.
• The Extended Yale B Face dataset with dark images (EYB-Dark) [42,43]
  To challenge the method against dark images, we separated four images from each person in the EYB Face dataset and created a subset of EYB-Dark. The four images were taken randomly, but they needed to strictly abide by this rule: one under a slight illumination variation and three dark images. The total of this sub-dataset is 38 people × 4 images = 152 images. The size of each image is the same as the EYB Face dataset, and they are all in grayscale. Figure 1d shows an example of images from this dataset.
• The FEI Face Database (FEI) [44]
  The FEI Face Database contains 2800 images from 200 people; each person has 14 images. Each image is a color image captured in an upright frontal position with a 180-degree rotation of the profile on a uniform white background. The scale may differ by approximately 10%; each image was originally 640 × 480 pixels. This database is a collection of face images from students and employees, aged 19 to 40, from the Artificial Intelligence Laboratory of FEI in São Bernardo do Campo, São Paulo, Brazil. All people have distinct appearances, hairstyles, and jewelry. The ratio between men and women is approximately the same. In this research, we resized the images into 320 × 240 pixels and changed the color into grayscale images. Figure 1e shows the example of images from FEI.
• The FEI Face Database (FEI) Subset Frontal and Expression (FEI-FE) [44]
  The FEI Face Database has a subset that contains only two images per person and 400 images in total, consisting of one frontal face image with a neutral (non-smiling) expression and one frontal face image with a smiling expression. In this research, we resized the images into 180 × 130 pixels and changed the color into grayscale images. Figure 1f shows the example of images from FEI-FE.

2.2. Methods

2.2.1. Proposed Design

Our proposed method can be seen in the flowchart design in Figure 2. The method consisted of discrete wavelet transform (DWT), Gaussian filtering and Difference of Gaussian, image fusion from the output of DWT and both Gaussian filtering and Difference of Gaussian, inverse discrete wavelet transform (IDWT) from the output of image fusion creating an enhanced reconstructed image, and histogram of oriented gradient (HoG); then, the output was finally classified using machine learning algorithms. Each method, which was just mentioned, is explained below.

2.2.2. Multi-Resolution Analysis with Discrete Wavelet Transform (MRA-DWT)

The first process from our proposed method was to transform the input image with discrete wavelet transform (DWT). Multi-resolution analysis with DWT (MRA-DWT) serves a purpose, given that it can break down a signal into several components at different degrees of detail. Wavelets, small waves with varying scales and positions, break down a signal into its component elements. Multiple resolutions were used in this decomposition process to capture both high-frequency and low-frequency parts.

Two-dimensional DWT transformed an image (I) by using the procedure of decomposition and filtering using low-pass filters and high-pass filters twice, once along the column and once along the row, resulting in four different sub-bands: approximation (A), horizontal (H), vertical (V) and diagonal (D) [45]. The four sub-band results can be seen in Figure 3, while the MRA-DWT process is represented in the blue block in Figure 2. It is important to note that the ↓2 symbol in Figure 2 means down-sample columns (keeping the even-indexed columns) after filtering along rows and down-sample rows (keeping the even-indexed rows) after filtering along columns.

The scaling function determines the low-pass filter. It has a non-zero average, thus enabling it to represent the smooth or average components of the signal. The wavelet function determines the high-pass filter. It has a zero average for detecting changes in the image. The scaling function $\varphi \left(t\right)$ and wavelet function $\psi \left(t\right)$ are expressed in Equations (1) and (2), respectively, where h(k) and g(k) are the scaling and wavelet coefficients [46].

$$\varphi (t)={\sum }_{k=-\infty }^{\infty }h\left(k\right)\varphi (2t-k)$$

(1)

$$\psi (t)={\sum }_{k=-\infty }^{\infty }g\left(k\right)\varphi (2t-k)$$

(2)

The scaling and wavelet coefficients depend on the wavelet family. In this research, we examined the MRA-DWT wavelet family using Haar (haar), Daubechies (db2), Symlet (sym2), and biorthogonal wavelet (bior 1.3, bior 2.2, bior 2.6, bior 3.3, and bior 3.7) with one level of decomposition. For example, Figure 4 illustrates the scaling and wavelet functions from the wavelet Haar, and Equations (3) and (4) express the Haar scaling and wavelet functions, respectively.

$$\varphi \left(t\right)=\left\{\begin{array}{c}10\le t<1,\\ 0otherwise.\end{array}\right.$$

(3)

$$\psi \left(t\right)=\left\{\begin{array}{c}10\le t<{\displaystyle \frac{1}{2}},\\ -1{\displaystyle \frac{1}{2}}\le t<1,\\ 0otherwise.\end{array}\right.$$

(4)

The approximation sub-band captures the most energy or information of an image, while the horizontal, vertical, and diagonal sub-bands capture finer details and sharp changes in an image. The approximation sub-band results from two low-pass filters, hence consisting of low-frequency components of the original image. The low-frequency component displays slow changes in the area of the image, and it is where the most energy of the data is located. The horizontal, vertical, and diagonal sub-bands are the results of at least one high-frequency filter, which is why these sub-bands are high-frequency components of the original image. The high-frequency components express small and fast changes in the image and extract details. Figure 3(left) shows the low-frequency component or the approximation sub-band, where we can see the representation of the blurred version of the original image due to two low-pass filters. The other parts of Figure 3 show the high-frequency sub-bands where only the changes and details in the horizontal, vertical, and diagonal directions are shown. Figure 3 shows that the horizontal and vertical sub-bands already represent the details of the original image even though they were filtered by a high-pass filter only once.

2.2.3. Gaussian Filtering and the Difference of Gaussian

The second step was to filter the same input image used in MRA-DWT with a Gaussian filter and create a Gaussian-filtered image and a Difference of Gaussian. Filtering with Gaussian produces an output (G0) of a blurred version of the original input image (I).

$$G0=I\ast {\displaystyle \frac{1}{2\pi {\sigma }^2}}{e}^{{\displaystyle \frac{-x^2+y^2}{2{\sigma }^2}}}$$

(5)

The value of $\sigma$ was 2.7713, as in [47] (p. 245) and [48]. The x and y are the distances from the center point in the x and y directions. The * symbol represents the process of convolution. The following process was to decimate the G0 along both rows and columns by a factor of two. The purpose of decimation by the factor of two was to create the same size for the Gaussian-filtered features (G1_1 and G1_2), Difference of Gaussian features (L), and multi-resolution sub-bands from MRA-DWT (A, H, V, D). After the decimation, the filtering with Gaussian was performed once again for the input of G0 with two sigmas, ${\sigma }_1$ and ${\sigma }_2$, producing G1_1 and G1_2, respectively. The value of ${\sigma }_1$ was 1.2263 and ${\sigma }_2$ was 1.9725, as in [47] (p. 245) and [48]. The kernel size of each Gaussian filter depended on the sigma. The kernel size was $2\times ⌈2\times sigma⌉+1$, where $⌈\dots ⌉$ indicated round up or round toward positive infinity. The kernel size for (5) was 13, that for (6) was 7, and that for (7) was 9.

$$G1\_1=G0\ast {\displaystyle \frac{1}{2\pi {{\sigma }_1}^2}}{e}^{{\displaystyle \frac{-x^2+y^2}{2{{\sigma }_1}^2}}}$$

(6)

$$G1\_2=G0\ast {\displaystyle \frac{1}{2\pi {{\sigma }_2}^2}}{e}^{{\displaystyle \frac{-x^2+y^2}{2{{\sigma }_2}^2}}}$$

(7)

It is worth noting that the Laplacian of Gaussian can be approximated by using the difference of two Gaussians, following the rule in [47]. One of the popular methods that employed the Difference of Gaussian to approximate the Laplacian of Gaussian is SIFT (scale-invariant feature transform), by [49]. A scale-normalized Laplacian of Gaussian (LoG) was calculated in (8) and (9) [47] (pp. 235–236).

$$LoG=\widehat{\lambda }(G1\_1-G1\_2)$$

(8)

$$\widehat{\lambda }={\displaystyle \frac{2{\kappa }^2}{({\kappa }^2-1)}}$$

(9)

The Difference of Gaussians (L) was used to approximate the Laplacian of Gaussians (10).

$$LoG=\widehat{\lambda }(G1\_1-G1\_2)\approx L$$

(10)

The factor $\widehat{\lambda }$ could be omitted from (10) because $\kappa$ was a fixed-scale ratio between two Gaussians at successive scale space levels (i.e., G1_1 and G1_2) [47] (p. 240 and p. 242). In the end, the Difference of Gaussian (L) was calculated by finding the difference between two Gaussians with adjacent scale levels.

$$L=G1\_1-G1\_2$$

(11)

The Gaussian filtering and Difference of Gaussian are represented in the green block in Figure 2. The results of Gaussian filtered features and Difference of Gaussian features can be seen in Figure 5. We can see that Gaussian filtering is a process of blurring the input image, and we consider the Gaussian filtered image to be a low-frequency component of the original image. The Difference of Gaussian creates the distinctness of edges, which brings up other details of the original image. The edges and details are considered the high-frequency component of the original image.

2.2.4. Image Fusion

Image fusion is a technique in computer vision and image processing that merges multiple images into a single composite image that is more informative and suitable for further processing tasks. In this research, the main objective of image fusion was to integrate opposite-frequency components from MRA-DWT and Gaussian filtering or Difference of Gaussian to create new features that had better robustness against facial recognition system challenges than any original image.

In general, according to the data integration phase, there are three types of image fusion: pixel-level fusion, feature-level fusion, and decision-level fusion [50]. In pixel-level fusion, the merging is directly performed on the input images. The essential features are extracted first from the image in feature-level fusion. Later, the process combines these features. Different from pixel-level and feature-level fusion, in decision-level fusion, the image analyses are performed separately until the final process to produce decisions, and these decisions are combined to create the final verdict.

In this research, we compared two different methods of image fusion, i.e., Laplacian pyramid image fusion (LP-IF) and discrete wavelet transform image fusion (DWT-IF). Both methods come from the transform domain image fusion [51]. The LP-IF and DWT/IDWT-IF processes are shown in the red block in Figure 2.

• Laplacian pyramid image fusion (LP-IF)

The Laplacian pyramid was first proposed by Burt and Adelson [52]. It was then used to merge images [53]. The LP-IF creates a pyramid-like structure for the two source images using Gaussian filtering with decimation by a factor of two. This procedure is implemented until it reaches the desired level l. The Laplacian is calculated by subtracting the filtered Gaussian from level l from the filtered Gaussian from level l + 1 that first undergoes a process of un-decimation by a factor of two. Using the popular fusion rule, we chose an averaging for the base low-pass and the highest absolute value for the fused high-pass coefficient. The final fused image is acquired by going through the inverse pyramid transformation. We used level l = 4. All LP-IF codes in this research were credited to the image fusion toolbox by [54].

• Discrete wavelet transform image fusion (DWT/IDWT-IF)

Image fusion by DWT (DWT/IDWT-IF) was first introduced by [55] with a multi-resolution decomposition that shows the frequency and location where the frequency occurs. Like LP-IF, DWT-IF creates a multi-scale representation of the source images, producing four different sub-bands for each image: A, H, V, and D (as explained in Section 2.2.2). The four sub-bands from each source image were fused using max-min, min-max, and mean-mean rules. The max-min rule indicates that the fusion takes the maximum absolute approximation value and minimum absolute value for details. In contrast, the min-max rule is the opposite of the max-min (minimum absolute value for approximation and maximum absolute value for details). The mean-mean rule means we take the mean value for approximation and details [33]. The final fused image is obtained through inverse DWT using the combined coefficients with different rules.

After we acquired four sub-bands from MRA-DWT (A, H, V, and D), the Gaussian filtered image (G1_1) from Equation (2), and the Difference of Gaussian (L) from Equation (4), we proposed to fuse these coefficients to produce an enhanced image. The fusion combined two opposite coefficients’ frequencies, i.e., the low-pass coefficient (which lacks a high-frequency component) was combined with the high-pass coefficient (which lacks a low-frequency component), and vice versa. The idea was to combine the opposite frequencies to complement each other. The approximation sub-band from MRA-DWT (A) consisted of only slow change components, so we combined it with the Difference of Gaussian (L), which had details of distinctness of edges. The detail sub-bands from MRA-DWT (H, V, and D) represented small and fast changes. We combined them with a Gaussian-filtered image (G1_1), where the result also showed slow changes from the original image. In addition, to support our idea, we tried to perform an experiment combining the components of the same frequencies, but the results were unsatisfactory. Figure 6 displays the results of the proposed image fusion.

The fusion rules in our proposed method were as follows (Figure 2):

• Fused A (a low-frequency component) with L (a high-frequency component), resulting in AL;
• Fused H (a high-frequency component) with G1_1 (a low-frequency component), resulting in HG;
• Fused V (a high-frequency component) with G1_1 (a low-frequency component), resulting in VG;
• Fused D (a high-frequency component) with G1_1 (a low-frequency component), resulting in DG.

The image fusion rules were as follows:

• LP-IF: averaging for the base low-pass and highest absolute value for the fused high-pass coefficient;
• DWT/IDWT-IF: max-min, min-max, and mean-mean rules with variations in wavelet family and level of decomposition.

2.2.5. Inverse Discrete Wavelet Transform for Multi-Resolution Analysis (MRA-IDWT)

To reconstruct an enhanced image from the fusion results, we took four fused coefficients, AL, HG, VG, and DG, to be reconstructed using a single-level inverse discrete wavelet transform (MRA-IDWT). The same wavelet was used when we decomposed it into MRA-DWT and reconstructed it with MRA-IDWT. The reconstruction process started by up-sampling the fused coefficients (AL, HG, VG, and DG) by introducing zeros between the samples at odd-indexed columns/rows to increase the image’s resolution in both dimensions. The equivalent synthesis filters, previously employed in the DWT, were subsequently applied to these up-sampled coefficients. The inverse wavelet transform was first applied along the columns of the up-sampled coefficients to rebuild the intermediate row data, merging the AL with HG and VG with DG. The procedure was then applied along the rows of the intermediate data to recreate the enhanced image completely.

2.2.6. Histogram of Oriented Gradient (HoG)

The histogram of oriented gradient (HoG) [56] is a feature descriptor used in image processing and computer vision for object detection and recognition. The technique has become popular due to its robustness and effectiveness in obtaining local shape information while being comparatively unaffected by global variations in position and illumination [57].

The gradient of an enhanced reconstructed image from the previous steps was computed first. The gradients show the edges and contours of the face in the image. Then, the image was divided into small, connected regions called cells. For each cell, a histogram of gradient orientations was created, where the gradient orientation was binned, and the magnitude of the gradient was used to vote for the corresponding orientation bin. The HoG descriptor included a step where groups of cells, known as blocks, were normalized. This normalization process involved adjusting the histogram values to account for changes in lighting conditions, enhancing the descriptor’s invariance to factors such as illumination and contrast. The block normalization was performed by computing the L2 norm of the histogram vector within the block. The final HoG descriptor was a concatenated vector of the normalized histograms from all blocks within the detection window.

2.2.7. Classification and Experiment Setup

The vector of HoG descriptor from each enhanced reconstructed image in the face dataset was reshaped into one row (one image for one row). Then, we combined them into one big matrix, where each row represented features from each image. In the end, we had a big matrix with the size of m × n, where m indicates the number of images inside the dataset and n indicates the size of the feature vector. Afterwards, we trained them using 1-nearest neighbor (1-NN); support vector machine (SVM) with linear, quadratic, and cubic kernels; and neural network with ReLU (NN-ReLU) as classification methods in our machine learning simulations. It is worth noting that in this research, individual people in all datasets represented a class. For example, the Database of Faces (AT&T) consisted of images from 40 people, so we had 40 classes for that dataset. We employed 5-fold cross-validation that divided testing and training data into 20:80 ratios to prevent overfitting. Finally, the average recognition accuracy from each simulation was calculated and compared.

In this research, due to the high potential of other combinations inside our proposed method, including pixel-level or feature-level fusion of features, we showed several possibilities (derivative from the flowchart design in Figure 2) to produce the best results of recognition performance. These several possibilities are shown in Figures S1 and S2. The first possibility (Figure S1) was to let one of the four features of AL, HG, VG, and DG go directly into the next step of HoG and classification. The second possibility (Figure S2) was to fuse the four features of AL, HG, VG, and DG into one concatenated feature vector.

Table 1. The overall experiment designs.

Exp.	Flowchart	Fusion Level	Method of Fusion *	Coefficients
1	Figure S1	Pixel-level	LP-IF	AL, HG, VG, DG (choose one)
2	Figure S2	Pixel- and feature-level	LP-IF	Concatenated vector (c-AL+HG+VG+DG)
3	Figure S1	Pixel-level	DWT-IF **	AL, HG, VG, DG (choose one)
4	Figure S2	Pixel- and feature-level	DWT-IF **	Concatenated vector (c-AL+HG+VG+DG)
5	Figure 2	Pixel-level	LP-IF	Enhanced reconstructed image using MRA-IDWT
6	Figure 2	Pixel-level	DWT-IF **	Enhanced reconstructed image using MRA-IDWT

* Default MRA-DWT/IDWT using Haar wavelet with one level of decomposition. ** Default DWT/IDWT-IF using db2 with five levels of decomposition with the mean-mean fusion rule.

shows the overall experiment designs from our proposed method in Figure 2 and several possibilities from Figures S1 and S2.

It is worth noting that in this research, for some experiments, we might have had a maximum of two pairs of DWT/IDWT with different purposes:

• Multi-resolution analysis with discrete wavelet transform, to divide the input image into four different sub-bands, and inverse discrete wavelet transform, to produce an enhanced reconstructed image after fusion (MRA-DWT/IDWT);
• The discrete wavelet transform and inverse discrete wavelet transform as a method of image fusion (DWT/IDWT-IF).

We compared using Haar (haar), Daubechies (db2), Symlet (sym2), and biorthogonal wavelet (bior 1.3, bior 2.2, bior 2.6, bior 3.3, and bior 3.7) families with one level of decomposition for all MRA-DWT/IDWT. We also compared wavelet families (haar, db2, sym2, and bior 2.6/bior 3.3) and decomposition levels (1, 3, 5, and 7) for DWT/IDWT-IF.

In this research, four different face datasets with two subsets (mentioned in Section 2.1) and their variations were the challenges to be tested with our proposed method.

Table 2. Face datasets and their challenges.

Dataset	Challenges
AT&T	Lighting conditions, facial expressions, glasses and no glasses, image acquisition with a time gap
EYB	Pose, variation in lighting conditions
EYB-Dark	Very dark face images (1 normal and 3 dark images)
BeautyREC	Face makeup (transfer), imbalanced dataset
FEI	180-degree rotation of the face profile
FEI-FE	Neutral and smiling expression

displays the list of face datasets with their challenges.

All simulations in this research were performed using Matlab 2024a (Update 5) with 16 GB installed RAM and Intel(R) Core(TM) i7-7500U CPU @ 2.70 GHz 2.90 GHz.

3. Results and Discussion

We display the results of our research regarding the effect of our proposed method against variations and challenges in the datasets.

3.1. Results for AT&T Face Dataset

3.1.1. Results for All Experiment Designs

First, we explored the results of the six experiment designs from Table 1 against the AT&T Face Dataset. The AT&T Face Dataset has variations such as lighting conditions, facial expressions, face attributes using glasses and no glasses, and image acquisition with a time gap.

Table 3. The average accuracy results (%) for AT&T Face Dataset from all experiments.

Exp.	Coefficients	1-NN	SVM			NN-ReLU
			Linear	Quadratic	Cubic
1	AL	97.0	97.5	97.8	97.8	96.2
	HG	96.8	95.8	97.8	98.0	96.5
	VG	96.5	96.0	97.5	97.2	96.5
	DG	96.2	96.8	98.0	98.2	97.0
2	(c-AL+HG+VG+DG)	97.2	97.5	98.2	98.2	97.8
3	AL	97	97.8	98.2	98.2	97
	HG	96.8	96.2	97.8	97.8	96.8
	VG	97	96.2	97.8	97.5	96.2
	DG	96	96.8	98.2	97.5	97
4	(c-AL+HG+VG+DG)	96.2	97.5	98.2	98.2	97.5
5	I’	96.8	98.2	99.2	99.2	97.2
6	I’	96	97.2	98	97.8	95.5

shows the recognition performance of accuracy results using 1-nearest neighbor (1-NN); support vector machine (SVM) with linear, quadratic, and cubic kernels; and neural network with ReLU (NN-ReLU) classification.

We can see from Table 3 that our proposed method (Exp. 5) produced the highest accuracy (99.2%) among the variations. The options of using only AL, HG, VG, and DG fusion features (Exp. 1 and 3) or concatenating the four features (Exp. 2 and 4) did not produce better results compared with the system using the one enhanced reconstructed image (Exp. 5 and 6). Although they did not perform the best, we learned that Exp. 2 and 4 generally produced results as high as the best accuracies from four features from Exp. 1 and 3 (AL/HG/VG/DG).

In terms of image fusion methods, LP-IF (Exp. 1 and 2) and DWT-IF (Exp. 3 and 4), Table 3 shows that both image fusion methods achieved accuracies higher than 97% (SVM) with LP-IF producing the best. In terms of classification method, overall, the SVM yielded better results compared with 1-NN and NN-ReLU.

In a more detailed analysis,

Table 4. The confusion matrix performance results (%) for the AT&T Face Dataset.

Exp.	Number of Images	SVM Kernel	Accuracy	Precision	Recall	F1-Score
5	10	Quadratic	98.5	98.5	98.64	98.57
		Cubic	98.5	98.5	98.64	98.57
	5	Quadratic	93	93	93.87	93.43
		Cubic	93.5	93.5	94.82	94.16
6	10	Quadratic	97.75	97.75	97.93	97.84
		Cubic	97.75	97.75	97.95	97.85
	5	Quadratic	94	94	95.16	94.57
		Cubic	94	94	95.04	94.52

displays the confusion matrix performance for Exp. 5 and Exp. 6, while the confusion matrix chart can be seen in Figures S3 and S4. Table 4 and Figures S3 and S4 show a balanced performance between accuracy, precision, recall, and F1-Score. Our proposed methods (Exp. 5 and 6) were likely reliable and accurate at discriminating across classes. They generalized well throughout the dataset and did not overfit any one class or subset of the data.

Table 4 also shows the results if there were only five images per individual/class, and the results were compared with using full (10) images per individual/class. There was a decrease in accuracy of approximately 3.75% to 5.5% if our proposed method employed only five images per class.

We calculated the processing time for our proposed methods, Exp. 5 and 6, against the AT&T Face Dataset. These calculations were applied to Exp. 5 and Exp. 6 with variations in level of decomposition in the DWT/IDWT-IF. Figure 7 displays the results of the processing time in seconds. T1 shows the time taken from the input to the creation of the one big matrix (before classification), T2 shows the time taken from the input to the output of accuracy using SVM quadratic, and T3 shows the time taken from the input to the output of accuracy using SVM cubic. We also observed a significant time difference, approximately 87 s, between T1 for Exp. 5 and Exp. 6. DWT/IDWT-IF required much more time than LP-IF. We were not surprised to find an increase in the inference time from Exp. 6a to Exp. 6d. This increase happened due to the rise of the total decomposition levels; the higher the decomposition level in DWT/IDWT-IF, the more time it took. We also calculated the inference time, the time required for one input image processed by our simulation until the system produced the output (prediction). The average inference time was 0.0093 s.

3.1.2. Results for Different Wavelet Families

The experiments in Table 3 used the default settings of employing haar with one level of decomposition for MRA-DWT/IDW and db2 with five levels of decomposition for DWT/IDWT-IF and the mean-mean rule of image fusion. The following investigation shows the results of varying the wavelet family for MRA-DWT/IDWT. It is worth noting that different wavelet filters in the wavelet family for MRA-DWT/IDWT have effects when the process starts to decompose an image into sub-bands and when the process starts to reconstruct features from image fusion into one enhanced reconstructed image.

Different wavelet families for MRA-DWT/IDWT show different characteristics of low- and high-pass filters. The explanation is as follows [58]: wavelet Haar, with only one vanishing moment (haar), is not smooth and has anti-symmetry; Daubechies wavelet, with only two vanishing moments (db2) for the wavelet function, is not considered smooth and does not have symmetry characteristics; Symlet, with two vanishing moments (sym2), is also not considered smooth and has near symmetry characteristics; biorthogonal (bior nr.nd) has exact symmetry and is considered smooth only if the wavelet order is large. In the abbreviation, nr is the number of vanishing moments inside the reconstruction filter, and nd is the number of vanishing moments inside the decomposition filter. In this research, we employed bior 1.3, bior 2.2, bior 2.6, bior 3.3, and bior 3.7.

Figure 8a and Figure 8b display the results of using haar, db2, sym2, bior 1.3, bior 2.2, bior 2.6, bior 3.3, and bior 3.7 with one level of decomposition against the classification methods for Exp. 5 and 6, respectively. Figure 8a shows that the haar wavelet yielded the best result, followed by bior 1.3 and bior 2.2. Figure 8b shows that bior 2.6 and bior 3.3 performed better. They improved the accuracy results by 0.8% (from 98% using haar to 98.8% using bior 2.6/bior 3.3).

In the subsequent investigation, we examined the effect of variations inside DWT/IDWT-IF from Exp. 6. Different from MRA-DWT/IDWT, various wavelet families inside DWT/IDWT-IF have the effects of decomposing input features and reconstructing them into fused features for image fusion purposes.

To investigate DWT/IDWT-IF, we employed bior 3.3 for MRA-DWT/IDWT since it yielded the best results from the previous investigation. First, we checked using the db2 wavelet for one, three, five, and seven levels of decomposition. The peak of the best accuracy results came from five levels of decomposition. Figure 9 shows the results of this investigation. Knowing that five levels of decomposition performed best, we then varied the wavelet family for DWT/IDWT-IF, using haar, db2, sym2, and bior 3.3. The results in Figure 10 show that db2 performed the best among others for DWT/IDWT-IF if paired with five levels of decomposition.

From these wavelet filter variations, we summarized for the AT&T Face Dataset, using haar with one level of decomposition for MRA-DWT/IDWT, yielded 99.2% accuracy for Experiment 5; using bior 3.3 with one level of decomposition for MRA-DWT/IDWT, and db2 with five levels of decomposition for DWT/IDWT-IF, yielded 98.8% accuracy for Experiment 6. As mentioned, the AT&T Face Dataset has variations such as lighting conditions, facial expressions, facial attributes using glasses and no glasses, and image acquisition with a time gap. Our proposed method produced high recognition accuracy against these challenges inside the AT&T Face Dataset.

3.2. Results for Other Face Datasets

3.2.1. Results for EYB and EYB-Dark Face Datasets

The following investigation evaluated the EYB Face Dataset using our proposed method. Having variations in pose and lighting conditions, the EYB Face Dataset was examined. From a total of six experiments, we investigated only Exp. 2, 4, 5, and 6 since the last experiment showed that using concatenated vectors (Exp. 2 and 4) generally performed as well as the highest accuracy from each fused coefficient (Exp. 1 and 3).

We examined Exp. 2, 4, 5, and 6 using only SVM for classification due to the best results produced by the SVM in the last experiment. Figure 11 shows the accuracy results of these settings. We saw that the results from Exp. 2, 4, and 5 produced almost the same highest value. The best recognition performance of accuracy was 99.8%.

To challenge our proposed method further against lighting conditions, we evaluated it against the EYB-Dark Face Dataset. Figure 1d shows that this subset of the EYB Face Dataset has only four images for each individual, where one image is considered to use better lighting, and three other images are dark.

The same experiment was operated for the EYB-Dark Face Dataset, i.e., Exp. 2, 4, 5, and 6. Figure 12 shows the results of recognition performance. Overall, the accuracy was not as high as the EYB Face Dataset and was reduced by approximately 3.7–43.9%. This decrement is understandable, since the EYB-Dark has only four images per person, three of which are dark (Figure 1d). While Exp. 2 and 4 failed to produce high accuracy, our proposed method, Exp. 5 and 6, generated better results with the highest performance, achieving 93.4%.

To further investigate and generate better results, we examined the fusion rule of mean-mean, min-max, and max-min on the EYB-Dark Face Dataset for Experiment 6. Figure 13 displays the results and shows that the mean-mean rule produced the best result between the three different fusion rules. Our results showed different outcomes than those of the study in [33], which found that using the min-max fusion rules provided the best recognition results for hand gesture intention.

Figure 14 shows the results of DMW/IDWT-IF using each fusion rule. Figure 14 shows why the mean-mean rule performed best, followed by the min-max rule. The mean-mean rule produced results that made details previously obscured by the darkness became visible. While the min-max rule also brought up obscured details, it was not as informative as the mean-mean rule.

Moreover, we varied the wavelet family for MRA-DWT/IDWT. The results are shown in Figure 15. This experiment found a better wavelet filter for this challenge inside the EYB-Dark Face Dataset. Using the sym2 wavelet improved the recognition by 2.7%, achieving an accuracy of 96.1%.

Correspondingly, we investigated the effect of the level of decomposition and various wavelet families in DWT/IDWT-IF for Experiment 6 with the same settings as in Figure 9 and Figure 10 for the AT&T Face Dataset. The same conclusion was achieved, which was that using five levels of decomposition produced the best results. These results can be seen in Figure S5.

While using the five levels of decomposition and sym2 wavelet for MRA-DWT/IDWT, we varied the wavelet family for DWT/IDWT-IF for Experiment 6, i.e., haar, sym2, db2, and bior 3.3. The best results (see Figure S6) were also produced using the db2 wavelet. The same outcome was previously obtained from the AT&T Face Dataset.

Against the challenges of dark images, our proposed method was proven to produce a high result of 96.1% accuracy. This result can be compared against the study in [59] using the same subset of the dataset, producing an accuracy of 95.4% but with the help of the contrast-limited adaptive histogram equalization (CLAHE) method inside the recognition system. Our proposed method performed better compared with the system using a contrast adjustment method.

3.2.2. Results for BeautyREC Face Dataset

Moving to the next dataset, the BeautyREC Face Dataset was examined to investigate our proposed method against the challenges of face images using makeup. It is highly desirable to consider that the BeautyREC Face Dataset is a result of transferring the makeup style of a reference image to the corresponding components (skin, lips, eyes) of a source image created by [41]. The dataset contains diverse variations in face pose and race.

First, we tried our proposed Exp. 5 and 6 method against the BeautyREC face dataset to observe the effect of the total number of images. Figure 16 shows the results of recognition accuracy for our proposed method. We noticed that using all 3000 images created a slight decrease in accuracy. Overall, the method has not yet produced satisfactory results for this dataset. The best result was 46% accuracy using only 1820 images and 44.4% using all images.

Next, we observed thirteen variations in our experiments (Figure 17) to find the most suitable setting and parameters to increase recognition performance. We varied the wavelet family for MRA-DWT/IDWT for both fusion with LP-IF (Exp. 5a–Exp. 5e) and DWT/IDWT-IF (Exp. 6a–Exp. 6e). The other wavelet families for MRA-DWT/IDWT (sym2, db2, bior2.6, and bior 3.3) did not improve the accuracy. Using haar for MRA-DWT/IDWT and db2 for DWT/IDWT-IF, we examined the effect of decomposition levels (Exp. 6f–Exp. 6h). Unfortunately, the variations in the level of decomposition also did not improve the results. We then investigated the wavelet family for DWT/IDWT-IF with haar for MRA-DWT/IDWT and five levels of decomposition (Exp. 6i–Exp. 6k). We found that db2 still produced better results. Last, we studied the fusion rules using min-max and max-min (Exp. 6l and Exp. 6m), yet still found that using the mean-mean rule was a better choice.

Although the purpose of [41] in creating BeautyREC was to produce a robust makeup transfer dataset and not to test the face recognition system, we still wanted to challenge our proposed method against this case. Regrettably, our proposed method produced only as high as 44.4% accuracy against this dataset. The most probable reasons were high head pose variations and makeup styles (see Figure 18), a non-uniform background, and the BeautyREC Face Dataset being imbalanced, resulting in our proposed method not producing high accuracy results.

To examine the effect using an imbalanced and a balanced dataset, we observed the performance of the confusion matrix for the BeautyREC Face Dataset. To create a balanced dataset, we gathered 5 images from 30 people in the dataset, resulting in a total of 150 images to be analyzed. It is also worth noting that we observed how imbalanced the BeautyREC Face Dataset is. There was originally a total of 41 people, and everyone provided a very diverse number of images, ranging from only 1 image to 182 images. The average number of images was 73 images, while the standard deviation was as high as 46.22. These values show that this dataset has a wide range and significant variability.

Table 5. The confusion matrix performance results (%) for the BeautyREC Face Dataset.

Exp.	Dataset	SVM Kernel	Accuracy	Precision	Recall	F1-Score
5	Imbalanced	Quadratic	28.82	28.82	38.84	33.09
		Cubic	30.7	30.7	43.76	36.09
	Balanced	Quadratic	53.33	53.33	53.33	53.33
		Cubic	50.67	50.67	48.79	49.71
6	Imbalanced	Quadratic	21.17	21.17	25.7	23.22
		Cubic	22.77	22.77	30.09	25.92
	Balanced	Quadratic	53.33	53.33	53.33	53.33
		Cubic	52	52	50.03	50.99

(Figures S7 and S8) displays the results of the confusion matrix for this dataset. Table 5 shows that overall, our proposed method performed better if the data were balanced and used only five images per class/individual. The accuracy difference between the imbalanced and balanced datasets was around 24.51% to 32.16%. We also observed that the imbalanced dataset had a higher recall. However, the balanced dataset had the same recall or lower recall. High recall in a face recognition system means correctly identifying most faces it is supposed to recognize (fewer false negatives). On the other hand, low recall means the system misses more faces than it should recognize (higher false negatives). In the case of financial transactions using face recognition, generally, the system may prefer lower recall but balanced with higher precision, so the financial transactions can be stricter in their matching.

The accuracy, precision, and recall results varied vastly (Figure S7). The accuracy range was from 3.3% to 100%, the precision range was from 0% to 100%, and the recall range was from 0.2% to 75%. Meanwhile, a balanced dataset showed a narrower range for accuracy, precision, recall, and lowered misclassification results (Figure S8). Overall, after studying the results and understanding the distribution of the original imbalanced BeautyREC Face Dataset more deeply, a balanced dataset was much preferred for better performance in the recognition system.

3.2.3. Results for FEI and FEI-FE Face Database

In our last test, we examined our proposed method, Exp. 5 and 6, against the FEI and FEI-FE Face databases. The FEI Face Database has wider variations compared with the FEI-FE Face Database. The FEI Face Database not only has neutral and smiling face images, but also a 180-degree rotation of face profiles. On the other hand, the FEI-FE Face Database has only a frontal position with a neutral and smiling expression. Because the FEI-FE Face Database consists of only two images per person, we evaluated this data using 2-fold cross-validation.

As we can see from the previous experiments (against the AT&T and EYB Face datasets), the options of using the haar wavelet with one level of decomposition for MRA-DWT/IDWT and db2 with five levels of decomposition and the mean-mean fusion rule for DWT/IWDT-IF produced the best results, so in this investigation we did not vary the wavelet family. Figure 19 and Figure 20 display the recognition results for the FEI and FEI-FE Face Database, respectively. Unsurprisingly, our proposed method performed better on the FEI-FE Face Database than the FEI Face Database. The best accuracy for the FEI Face Database was 97.5%, while the best for the FEI-FE Face Database was 99.5%. The accuracy difference between the two databases was approximately 2–2.3%. As previously mentioned, the FEI-FE Face Database consists of only frontal face images with neutral and smiling expressions. At the same time, the FEI Face Database has variations in the 180-degree rotation of the face profile. This led to higher results for the FEI-FE Face Database. Our proposed method was tested against face expressions, such as neutral and smiling, and face profile rotation (180 degrees). It was proven to yield high accuracy results.

3.3. Comparisons with Other Methods

This section compares our proposed method results against state-of-the-art image and non-image fusion methods inside face recognition.

Table 6. Comparison with other methods (fusion and non-fusion).

Dataset	Research	Fusion/Non-Fusion	Methods	Accuracy/Recognition Rate (%)
AT&T (ORL)	[60]	Fusion	Extended local binary patterns and reduction with local non-negative matrix factorization (on unimodal face)	97.20
	[22]	Fusion	Multi-resolution discrete cosine transform fusion	97.70
	[21]	Fusion	Multi-resolution singular value decomposition fusion	97.78
	[61]	Non-fusion	Convolutional neural network	98
	[20]	Fusion	Laplacian pyramid fusion	98.2
	[62]	Non-fusion	Haar-cascade and convolutional neural network	98.57
	Ours	Fusion	Opposite-frequency features of MRA-DWT/IDWT and image fusion with LP-IF or DWT/IDWT-IF	99.2
EYB	[63]	Fusion (graph)	Auto-weighted multi-view semi-supervised learning method	88.57
	[64]	Non-fusion	Multi-resolution dictionary learning method based on sample expansion	89.59
	[65]	Non-fusion	Local neighborhood difference pattern	97.64
	[66]	Fusion	Pre-processing and local phase quantization and multi-scale local binary pattern with score-level fusion and decision-level fusion	98.30
	Ours	Fusion	Opposite-frequency features of MRA-DWT/IDWT and image fusion with LP-IF or DWT/IDWT-IF	99.8
EYB-Dark	[59]	non-fusion	Histogram of oriented gradient and contrast-limited adaptive histogram equalization	95.4
	Ours	Fusion	Opposite-frequency features of MRA-DWT/IDWT and image fusion with LP-IF or DWT/IDWT-IF	96.1
FEI	[67]	Non-fusion	Permutation coding neural classifier based on random local descriptor	93.57
	[68]	Non-fusion	Integration of the binary-level occurrencematrix and the fuzzy local binary pattern and neural network classifier	95.27
	[69]	Non-fusion	Principal component analysis	96
	Ours	Fusion	Opposite-frequency features of MRA-DWT/IDWT and image fusion with LP-IF or DWT/IDWT-IF	97.5
FEI-FE	[70]	Non-fusion	Using eye region with Gabor transform and nearest neighbor	97
	Ours	Fusion	Opposite-frequency features of MRA-DWT/IDWT and image fusion with LP-IF or DWT/IDWT-IF	99.5

displays the comparisons, and we organized them based on the dataset employed. Our proposed method is comparable to the state-of-the-art methods’ results and produces high recognition performance.

4. Conclusions

In this research, we proposed a pixel-level image fusion of opposite frequencies from MRA-DWT/IDWT sub-bands with Gaussian-filtered results (high-frequency from the wavelet sub-bands and low-frequency from the Gaussian filtered) and MRA-DWT/IDWT sub-bands with Difference of Gaussian (low-frequency from the wavelet sub-bands and high-frequency from the Difference of Gaussian). The pixel-level fusion of creating one enhanced reconstructed image (Exp. 5 and 6) outperformed other possibilities inside the proposed method (Exp. 1–Exp. 4).

The proposed method was challenged against AT&T, EYB, EYB-Dark, BeautyREC, FEI, and FEI-FE Face Datasets. The results showed that our proposed method was robust and accurate against challenges such as lighting conditions, facial expressions, head pose, 180-degree rotation of the face profile, dark images, acquisition with a time gap, and conditions where the person uses attributes such as glasses. The proposed method is comparable to state-of-the-art methods and generates high recognition performance. The proposed method successfully achieved an accuracy of 99.2% for AT&T, 99.8% for EYB, 96.1% for EYB-Dark, 97.5% for FEI, and 99.5% for FEI-FE Face Dataset. The detailed results of the best parameters were as follows: overall, LP-IF and DWT/IDWT-IF both produced the best performance, but notably, DWT/IDWT-IF with sym2 performed better when tested against challenges such as dark images; the mean-mean fusion rule for five levels of decomposition DWT/IDWT-IF always yielded the best accuracy, while the support vector machine was undeniably the best classifier.

Our proposed method did not produce satisfactory results against the makeup face challenge (BeautyREC) due to the high variations in head pose, makeup style, and non-uniform background, and due to this dataset being an imbalanced dataset with a vast range in the number of images. For future development, we suggest the research use pre-processing methods, such as background removal and facial landmark detection. We also know that our proposed algorithm has a relatively long process and sequence of steps, so we aim to simplify the algorithm in future research while maintaining the recognition performance.