A Dataset for Comparing Mirrored and Non-Mirrored Male Bust Images for Facial Recognition

Abstract

Facial recognition, as well as other types of human recognition, have found uses in identification, security, and learning about behavior, among other uses. Because of the high cost of data collection for training purposes, logistical challenges and other impediments, mirroring images has frequently been used to increase the size of data sets. However, while these larger data sets have shown to be beneficial, their comparative level of benefit to the data collection of similar data has not been assessed. This paper presented a data set collected and prepared for this and related research purposes. The data set included both non-occluded and occluded data for mirroring assessment.

Dataset: Due to size, the dataset was split into six files: 10.17632/7w73p32pcn.1; 10.17632/2hxckw6y32.1; 10.17632/fxc4b539hx.1; 10.17632/zhnv3ns8hh.1; 10.17632/c2ggy8fvz8.1; 10.17632/63j23pw3rd.1.

1. Summary

Human and facial recognition [1] have a wide variety of prospective uses. The applications of facial recognition include retail store customer identification [2,3], applicant screening [4], and access control [5]. Facial and human recognition are also used by law enforcement entities [6].

In addition to simply recognizing an individual, facial imagery can be further analyzed to learn or infer about the subject. Age [7] and gender [7,8] can, in many cases, be easily determined from imagery. It can suggest individuals’ interest levels [9] and emotional state [9].

A variety of techniques have been proposed for identification and classification. Neural networks [10] are commonly used for this purpose. Facial and human data also suffers from numerous problems relating to subject state and collection. Thus, a variety of techniques for dealing with a wide number of issues, including lighting conditions, distortion [11,12], and occlusion [11,12,13,14], as well as having only limited training data [15] and subjects’ facial expressions [16], have been developed.

One particular use of facial recognition is to identify an individual for security [17,18,19,20] purposes. Recognition can be used to grant access [21,22] or validate identity [18,23]. When used for security applications, the system must be robust to attacks that may try to confuse or deny recognition capabilities.

Mirroring has been used extensively as part of recognition and, in particular, facial and human recognition studies. Several studies [24,25,26] have used mirroring to expand a training dataset, including removal of the need to photograph subjects from certain perspectives that correspond to the mirror of other ones [27]. Mirroring has also been used to aid recognition by replacing data that is either missing or occluded [28] and to expand a shape model [29] used as part of the recognition process. Studies have also used mirroring in conjunction with other image manipulation techniques, including cropping [30], pixel movement [31,32], rotation [32,33], scaling [32,33,34], blurring [35], and noise addition [35].

Despite the recurrent use of mirroring for facial and other recognition applications, only limited studies of its effectiveness have been performed. Hu et al. [36] evaluate the use of both flip-augmented and non-augmented data sets, showing a limited benefit to augmentation. Miclut et al. [37] compare non-augmented data sets’ effectiveness with mirrored and combined mirrored and rotated augmented data sets. No study was located where the use of augmented data was compared to the use of collected data from the corresponding positions.

This data set provided data that can be used to validate the efficacy of the use of mirroring for training or presenting images for recognition to facial recognition systems. It was designed to facilitate this assessment under a variety of conditions that these systems may face.

To this end, it includes imagery of subjects from multiple perspectives, with multiple lighting conditions and multiple lighting brightness and temperature levels. Most importantly, it includes data for five positions that have been manually collected and two positions that have been generated based on mirroring data from two other manually collected positions. A total of 245 images were included per subject, including 175 collected images and 70 which were generated through mirroring. In addition to these non-occluded images, the dataset includes images where the subject’s face is partially occluded by a hat or a pair of glasses. Thus, the data set includes 735 images for each subject, of which 525 are manually collected and 210 are generated through mirroring.

This dataset:

• Facilitates the testing of the impact of the use of mirrored imagery for testing facial recognition algorithms and for comparing this impact between algorithms.
• Includes mirrored and non-mirrored data for different subject positions, lighting angles, lighting brightness, and lighting temperature conditions.
• Includes mirrored and non-mirrored non-occluded data and data with hat and glasses occlusions.
• The data was collected in a controlled environment with a consistent background.

2. Data Description

The data set was comprised of 735 images in JPEG format, including 525 manually collected images and 210 images that were mirrored images of the images from the third and fourth subject orientation positions, as shown in Figure 1.

The images were of males between 18 and 26 years old. Many had similar skin tones and facial and upper chest characteristics, to facilitate testing of the systems’ ability to correctly identify similar subjects. The JPEG images could be opened and processed by numerous application programs and they contained the original metadata from the camera that was used to collect them. The resolution of the images was 5184 × 3456 pixels.

2.1. Data Organization

The organization convention for the data set is as follows. The top-level directories are named with a number that represents the subject, for example, “3”. In each of these directories are three subfolders called, “vanilla”, “glasses”, and “hat”. “Vanilla” contains all of the subject’s images without them wearing any occlusion, “glasses” contains all of their images with them wearing glasses, and “hat” contains all of their images with them wearing a hat. In each of these directories, are a set of five “position” subdirectories, each of which includes the subject’s orientation positions. These are labeled as pos_0 to pos_4. In addition, the two mirrored positions are labeled as m_pos_3 and m_pos_4. In each position directory, there are lighting angle subdirectories which correspond to each of the light’s positions. These are labeled as angle_1 to angle_7. Finally, in each lighting subdirectory, there are five images of the subject labeled in the format of IMG_####.JPG in the following order: warm, cold, low, medium, and high. In the mirrored directories, the naming format is ###_IMG_####.JPG; however, the ordering still corresponds to warm, cold, low, medium, and high. The labels of the individual images (warm, cold, low, medium, and high) correspond to the lighting settings, which are discussed in detail in the following section.

2.2. Comparison to Other Data Sets

A variety of other facial recognition datasets exist. Learned-Miller et al. [39] created a “Labeled Faces in the Wild” data set that was harvested from websites. This data set included data for 5749 people; however, it only included approximately two images per individual (13,000 images in total). Guo et al. [40] also harvested images from the web for their dataset, “MS-Celeb-1M.” This data set is comprised of 10,000,000 images of 100,000 people. Both of these data sets were collected automatically and could potentially have duplicates that were not detected by the creators’ algorithms, incorrect association of images to a single individual, or other characteristics associated with the curating algorithms. Another data set, entitled “IARPA Janus Benchmark A” [41] has images of 500 people, but only an average of 11.4 images per individual. The “Pgu-Face” dataset [42], similarly, had 896 images covering 224 subjects. The “Extended Cohn-Kanade Dataset (CK+)” dataset [43] had slightly more images per individual, covering 210 individuals with only 23 images per subject. The “FRGC 1.0.4” dataset [44] includes 152 subjects, but only an average of five images per individual. The “Face Recognition Technology (FERET)” data set [45] includes 14,126 images covering 1199 subjects (for about 12 images per subject). Finally, the “ORL Database of Faces” [46] contains only 400 images covering 40 individuals with only 10 images per subject.

The dataset presented herein covers only 11 subjects (which is less than other datasets); however, it includes 735 images per subject, which is far more than any of the other data sets. In terms of the total number of images, the dataset described herein has 8085 images, making it the fourth largest of the datasets. Two of these datasets were automatically generated (“Labeled Faces in the Wild” and “MS-Celeb-1M”), and it is thus the second largest manually collected dataset, from those surveyed (the NIST FERET data set is larger, being comprised of 14,126 images).

3. Methods

The image data set was collected using a Cannon EOS Rebel T2i camera with a 60 mm Ultrasonic EFS lens and set in aperture priority mode with an ISO of 800 and an FSTOP of eight. AI focus, peripheral illumination correction, red eye reduction, auto white balance, and spot metering were enabled, and auto exposure bracketing was set at 01. The flash was disabled. The camera was oriented in landscape mode.

Two Neewer LED500LRC lights were used to illuminate the background and one Yongnuo YN600L light was used to illuminate the subject. A projector screen was used as the backdrop. The camera, subject, and lighting locations are shown in Figure 2. The non-numbered ‘X’ locations are the positions of the background lights. The camera and subject positions are identified by an image of a camera and human, respectively. The numbered ‘X’ locations correspond to the seven subject illumination lighting positions (which create the seven lighting angles). The exact coordinates of the subject, camera, and lights are provided in

Table 1. Camera, lighting, and subject positions (in inches) [38].

Location	X Coordinate	Y Coordinate
Light 1	47.5	65
Light 2	44	87.5
Light 3	61	115
Light 4	84.5	127.5
Light 5	124	116.5
Light 6	198	91
Light 7	149	66
Background 1	43.5	50.5
Background 2	129	47
Camera	97	150
Subject	96.5	63.5

.

The subjects were photographed in five different positions (as shown in Figure 3). Additionally, the data was processed to mirror the position 0 and 1 images (the two leftmost images in Figure 3), to replace the data collected at positions 3 and 4. Figure 4 depicts the mirrored images shown in positions 3 and 4 (the two rightmost images), along with the collected data in positions 0, 1, and 2.

For each of the five positions (including both the mirrored and non-mirrored ones), photos were taken with seven different lighting angles. The seven angles of light are depicted in Figure 2, and images demonstrating the effect of these lighting angles are presented in Figure 5. For each position and lighting angle, images were collected at five different lighting settings. These lighting settings are listed in

Table 2. Lighting configurations and their titles [38].

Configuration Title	Settings
Warm	60% brightness on warm (3200 k)
Cold	60% brightness on cold (5500 k)
Low	10% brightness on warm (3200 k) and 10% brightness on cold (5500 k)
Medium	40% brightness on warm (3200 k) and 40% on brightness on cold (5500 k)
High	70% brightness on warm (3200 k) and 70% brightness on cold (5500 k)

and are visually depicted in Figure 6.

The images that are used in mirrored positions 3 and 4 were created through a pixel-by-pixel replacement from the left of the image to the right and vice versa. Figure 7 depicts this mirroring for the forward-facing position. Note that both images look similar, but are distinguishable by comparing small features, such as looking at the patterns of hair on the left and right of the head.

However, the mirrored images could not just be directly swapped. The lighting angles had to be reversed to most closely replicate or approximate the collected images, both in terms of subject position and in terms of the incident angle of the lighting. For example, to generate the mirrored image for subject position 4, lighting angle 7, the image from subject position 0, lighting angle 1 was used and flipped horizontally.

Even with this, the mirrored images for the subject positions 3 and 4, in this dataset, do not have exactly the same lighting angles of incidence, as the lighting configuration of the experimental setup was not perfectly symmetrical.

Table 3. Lighting angle (in degrees) for subject positions and lighting angles, where 3′ and 4′ refer to the mirrored images.

		Subject Position
		0	1	2	3	3′	4	4′
Light Angle	1	2	17	92	137	138	182	183
	2	25	20	65	110	120	155	165
	3	55	10	35	80	72	125	117
	4	79	34	11	56	56	101	101
	5	117	72	27	18	10	63	55
	6	165	120	75	30	20	15	25
	7	183	138	93	48	17	3	2

lists the angle of incidence for all the lighting and position combinations (including both mirrored and non-mirrored configurations). Figure 8 and Figure 9 depict this visually for positions 3 and 4, respectively.

Mirroring images collected from subject positions 0 and 1, to substitute in place of images collected from the remaining subject positions could be used to offset the image asymmetry created by the lighting placement asymmetry. However, the combination of the mirrored and non-mirrored data produces a greater set of lighting incidence angles than either full collection or half-collection and mirroring alone. Note also that some lighting angles are the same or very similar, while other lighting angles are further removed. Because of this, the data set can be used to assess the impact of mirroring and lighting angle changes, both together and independently, by looking at the impact relative to positions with more and less similar angles of incidence between the collected and mirrored data.

Subjects were asked to keep a neutral face while being photographed, and to face forward relative to their body orientation. The individual rotated to each position and the Yongnuo YN600L light was moved between positions and had its settings altered, while the subject remained stationary. Images were reviewed after each set to ensure that the subject did not move, blink, or show expression and to verify that all required images were present and of suitable quality. Subjects’ attire (shirt) was not controlled and there were minor fluctuations in the subjects’ position and facial state. All other aspects of data collection were controlled.