Signed Fingermark Liveness Detection Method Based on Deep Residual Networks and Multimodal Decision Fusion

Abstract

Fingermarks play an important role in document identification. At the same time, fingermarks on paper documents are often accompanied by signatures and background text, which introduce noise to the original fingermark textures and increase the difficulty of detection. A signed fingermark detection method based on deep residual networks and a decision-level fusion strategy was proposed to defend against spoofing attacks from fake fingermarks. Firstly, the multi-scale structure was introduced in the residual module, which improved the network’s depth and breadth without increasing the parameters. Then, the multi-probability label strategy was refined and employed to enhance the local encoding ability of the feature extraction. A score fusion strategy was designed, with weights allocated based on the difference in signed interference levels of local image blocks. Finally, a model fusion strategy based on evidence theory was suggested, which improved detection accuracy by leveraging complementarity between models. A large-scale fingermark database was established, which included real fingermarks made from real fingers and fake fingermarks made from various materials, and this was divided into two sub databases: signed and unsigned. The experimental results show that the proposed method achieves 96.16% accuracy based on the fingerprint dataset of the global liveness detection competition called LivDet2017 and achieves 99.30% accuracy based on the signed fingermark database, while it has good resistance to spoofing attacks from unknown materials.

1. Introduction

Biometric technology is extensively used in modern society. With the advancements in science and technology, biometric identification has gradually replaced traditional authentication methods such as keys and passwords for personal identity authentication. Fingerprint authentication is highly reliable, easily accessible, unique, and constant, making it the most widely used and trusted method on the market [1]. Fingerprint identification technology has matured, and it is a reliable method of identification. However, it is important to acknowledge that it also presents security risks. The easy accessibility of fingerprints, compared to irises and other biometric features, is both an advantage and a vulnerability. It is crucial to address this susceptibility to spoofing attacks in order to enhance the security of identification systems. Spoofing attacks, particularly those involving artificial fingerprint replicas created using mold-making materials, pose the biggest threat to fingerprint identification systems. It is important to be aware of this common and convenient means of spoofing attack. This is due to the ease with which counterfeiters can imitate the fingerprint image by creating the corresponding fingerprint negative mold. Fingerprint liveness detection technology is a proven solution to the growing problem of spoofing attacks. Extensive research has been conducted on this technology and it is widely implemented.

Since antiquity, fingermarks, generally formed by dipping a finger in pigment or sealing clay and pressing it on paper documents, have often appeared on documents with legal effect. In particular, fingermarks have been employed in private lending and borrowing, such as on the occasion of financing between two natural persons, based on concerns regarding the easy imitation of and the uniqueness of fingermarks, often in accordance with the traditional custom of the mixed use of signatures and fingermarks. This type of fingermark is often accompanied by signatures, text and other background noise interference, also known as signed fingermarks, and the detection of signed fingermarks is thus more difficult. Fake fingermarks can lead to false convictions, not only causing personal losses, but also affecting the credibility of justice. Traditional fingerprint authentication mainly uses manual identification methods based on expert experience, which is highly subjective and time-consuming. The existing literature on computer-based fingermark liveness detection methods is sparse, and the detection accuracy is relatively low. Based on this status quo, it is necessary to further investigate high-performance signed fingermark liveness detection methods.

The main aims of this paper are as follows:

(1)

To propose a network model, SFNet (Signed Fingermark Net), based on deep residual networks and a multi-probability label classification strategy, which achieves the purpose of signed fingermark liveness detection and achieves high accuracy on a fingermark dataset.

(2)

To design and combine a multi-score fusion strategy based on the quality weights of local blocks of fingermark and a model fusion strategy based on evidence theory to further improve the detection accuracy.

2. Related Works

Liveness detection methods for fingerprints can be classified as hardware-based or software-based. Hardware-based methods use additional hardware devices to obtain the biological features unique to real fingers, such as sensors that detect skin temperature [2] or bioelectrical signals [3]. However, fingermarks lack these vital features. Software-based methods use machine learning or deep learning algorithms to extract dynamic or static features from fingerprint images [4]. These methods determine the liveness of the images and are suitable for various applications, including real-time fingerprint detection and verification. They are highly compatible, cost-effective, and allow for algorithm updates at any time, without requiring additional hardware equipment. Fingermark liveness detection is better suited to static features rather than dynamic features. This is because dynamic features are easily influenced by factors such as finger pressure and skin condition, which are continuous and uncontrollable physiological changes. It is impractical to extract dynamic information from fingermarks on paper documents. Therefore, it is recommended to focus solely on static features for fingermark liveness detection.

The ridge texture is the most commonly used static feature for detecting the authenticity of fingermarks. Authentic and fake fingermarks can be distinguished based on differences in continuity, clarity, and ductility. These characteristics provide a strong foundation regarding physical properties for effective liveness detection [5,6,7,8,9,10]. It is crucial to recognize that traditional texture-based liveness detection methods heavily depend on expert knowledge of manually labeled features. However, as science and technology advance, new means of forgery emerge, making it increasingly challenging to define the characteristics of liveness in fingermark standards. If the manual features used to define the standard are targeted and avoided, these types of liveness detection methods will inevitably become ineffective.

Convolutional neural networks [11] have been widely used in computer vision. Researchers have applied deep learning to the field of fingerprint liveness detection, providing better robustness and higher performance than detection methods using manual features. Biometric methods applying deep neural networks can be combined with various score fusion strategies [12] to improve recognition accuracy. MobileNet, proposed in [13], significantly reduces the model parameters compared to VGG-19 [14]. CNN-based cross-layer fusion and multi-model voting strategies were proposed in [15] to achieve a better average performance of the network. The DRN designed in [16] achieved good accuracy based on the benchmark dataset LivDet2017 [17,18].

Through testing, it is found that the accuracy of the method originally used for fingerprint liveness detection is not ideal after migrating to signed fingermark liveness detection. Compared with the fingerprint images obtained by a fingerprint capture device, the fingermarks obtained from paper documents through a document scanner have to pass through multiple layers of media, with relatively fuzzy texture details and noise interference, such as text, signatures, and other elements, and the influence of signatures on liveness detection is still not clear. Exploring this issue is the goal of this paper.

In order to cope with these challenges brought about by signed fingermark spoofing attacks, this paper proposes a network model, SFNet, based on a deep residual network structure and combining a multi-probability label classification strategy, designing and combining a multi-score fusion strategy based on the quality weights of the local blocks of fingermarks and a model fusion strategy based on the theory of evidence, in order to achieve the liveness detection of signed fingermarks and to achieve a higher precision based on the fingermark dataset.

3. Proposed Method

3.1. Input Data Preprocessing

The network input uses local blocks from the original fingermark image after preprocessing. This approach avoids introducing incorrect information into the network, which could result in the loss of discriminative information required for effective network training. Signed fingermarks’ local blocks are preferred over those with rich textures to ensure network training effectiveness. Signed fingermark preprocessing involves foreground extraction, local block extraction, and image enhancement. The foreground region is extracted from the original fingermark image file using the color space method. An average color threshold per unit area is used to identify the effective area of the fingermark, filtering out interference items that may be similar in color to the fingermark (e.g., stamps).

The center of gravity of the foreground region, which represents the original image of the fingermark, is calculated and positioned as the central point. A sliding window with a fixed size of p × p (p = 300 pixels) is then used to intercept the local blocks by sliding in steps of 50 pixels, as illustrated in Figure 1.

To ensure the continuity and consistency of strokes, a larger window size should be selected for signed fingermarks, as they have complex stroke lines. This will avoid limiting the characteristics of stroke lines and reduce the risk of accuracy reduction in liveness detection. It is important to ensure that the training data contains sufficient fingermark information and does not have a large number of blank areas. The accuracy of signed fingermark liveness detection is significantly reduced when the size of the localized blocks is too small or too large. To increase diversity and enrich the data, the intercepted local blocks are augmented with random-direction mirror flipping and random-angle rotation (rotation angles of 0°, 90°, 180°, and 270°) before being sent to the network for training. To enhance network efficiency, we convert local blocks to single-channel grayscale maps without color information. We then perform contrast enhancement based on the grayscale histogram, using the formula given in Equation (1).

$$\mathrm{F}(\mathrm{s})=\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}(\frac{\mathrm{c}\mathrm{d}\mathrm{f}(\mathrm{s})-\mathrm{c}\mathrm{d}{\mathrm{f}}_{\min }}{(\mathrm{w}\times \mathrm{h})-\mathrm{c}\mathrm{d}{\mathrm{f}}_{\min }})\times 255$$

(1)

where w and h refer to the width and height of the image, respectively; cdf is the cumulative distribution function; s is the original gray value; and round represents the rounding function.

Enhancement of the image contrast is performed using grayscale stretching to increase the grayscale difference between the texture of the fingermark and the blank background and to reduce the effect of some of the fingermarks not being visible in the curve of the line due to too light a press.

3.2. SFNet

Figure 2 illustrates the main structure of the proposed signed fingermark liveness detection network, which consists of convolutional layers, average pooling layers, and residual blocks. The fingermark image’s local features are obtained through convolutional layers. The average pooling layer downsamples the image, removes redundant information, and compresses the number of parameters to prevent overfitting. The enhanced residual structure improves the network’s ability to extract high-dimensional features and filter effective features. Batch normalization layers and activation units are added after all of the convolutional layers in the network. This reduces the difference in the distribution of the training data, accelerates data convergence, improves network generalization ability, reduces parameter interdependence, and mitigates possible overfitting. The prediction results are confidently achieved through the classification layer, utilizing the feature maps of the localized blocks to accurately detect the liveness of signed fingermarks.

The rectified linear unit (ReLU) activation function is employed in the network’s activation unit to enhance its expressive power. The loss function of the network is represented by Equation (2) and is expressed as cross entropy.

$$\mathrm{L}=\frac{1}{\mathrm{N}}{\displaystyle \sum _{\mathrm{i}}-[{\mathrm{y}}_{\mathrm{i}}\cdot }\log ({\mathrm{p}}_{\mathrm{i}})+(1-{\mathrm{y}}_{\mathrm{i}})\cdot \log (1-{\mathrm{p}}_{\mathrm{i}})]$$

(2)

where $y_i$ denotes the label of sample i, ${\mathrm{p}}_{\mathrm{i}}$ denotes the probability that sample i is predicted to be true, and $1-{\mathrm{p}}_{\mathrm{i}}$ is the probability that sample i is predicted to be false.

The residual block in the fingermark liveness detection network combines the residual structure proposed in [19] with the multiscale backbone structure proposed in [20]. This fusion results in a structure that is better suited for the signed fingermark liveness detection network. The improved residual block structure is shown in Figure 3. The features are decomposed into multiple groups of equal-length sub-features equally by the decomposition layer, and the class residual structure is introduced between the multi-scale sub-features to deepen the chunking depth. The multiscale residual structure [20] accurately represents multiscale features at a finer granularity level, ensuring consistency in the depth of sub-features across scales. This improvement has been extended to multiple scales through the class residual structure, resulting in a significantly larger sensory field. The connection layer is successfully reorganized into a single feature and connected with the input feature residuals to form a residual block embedding network. This approach effectively improves the depth and breadth of the network without increasing the number of parameters.

3.3. Multimodal Decision-Level Fusion Approach

This paper presents an improved method for detecting the liveness of signed fingermarks. The method uses a multimodal decision-level fusion approach, which includes a score fusion strategy that takes into account the quality weights of the localized blocks of the fingermarks and a model fusion strategy based on the theory of evidence. This approach is highly effective in accurately detecting the liveness of signed fingermarks.

3.3.1. Multi-Probability Labeling Strategy

The traditional classification task performs the final classification through the fully connected layer. As shown in Figure 4a, the classification label confidence prediction of the input fingermark image is performed by the fully connected layer based on the features obtained from the network.

SFNet replaces the first fully connected layer with a convolutional layer for dimensionality reduction prior to final classification, as illustrated in Figure 4b. This technique significantly reduces the number of parameters and computations required without sacrificing network performance.

On this basis, the multi-probability labeling strategy is proposed in combination with the patch-label strategy in [21], as shown in Figure 4c. The patch-label strategy, described in [21], involves preprocessing the image and inputting it into the Inception network. The network performs dimensionality reduction using a fully connected layer and outputs the classification probability through the convolutional layer with confidence. SFNet employs an improved labeling strategy over the patch-label approach described in [21]. The fully connected layer is replaced with a convolutional layer that achieves the same effect. The SFNet subject network takes an image of size 300 × 300 pixels as the input to generate 10 × 10 × 512 feature maps, and the convolutional layer outputs probability distribution maps of size 10 × 10 × 2. The final prediction output of the network is obtained by taking the weighted average of the weights corresponding to the 10 × 10 probability table generated after the SoftMax operation, which is trained using the ground truth label. The probability table represents a local block at the corresponding position of the input image and is utilized to train the local coding capability of the feature extraction network.

3.3.2. Score Fusion Strategy

To ensure prediction accuracy, the network typically receives only a single localized block when classifying and predicting fingermark images. This input strategy, however, can introduce randomness and reduce the robustness of the results. For this reason, we propose the multi-input score fusion strategy. We use a sliding window in the image preprocessing step to intercept multiple local blocks and feed them into the network. This results in multiple independent prediction probabilities, which we combine using the score fusion method to calculate the final classification prediction probability, as shown in Figure 5. The final classification prediction probability of the input fingermark image is determined by utilizing the voting results of multiple prediction inputs. Equation (3) presents the calculation formula for the classification prediction probability.

$${\mathrm{P}}_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}}=\frac{1}{\mathrm{N}}(\mathrm{P}({\mathrm{y}}_1|{\mathrm{x}}_1)+\mathrm{P}({\mathrm{y}}_2|{\mathrm{x}}_2)+\dots +\mathrm{P}({\mathrm{y}}_{\mathrm{N}}|{\mathrm{x}}_{\mathrm{N}}))=\frac{1}{\mathrm{N}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{p}}^{{\mathrm{y}}_{\mathrm{i}}}}{(1-\mathrm{p})}^{1-{\mathrm{y}}_{\mathrm{i}}}$$

(3)

where $\mathrm{P}\left({\mathrm{y}}_{\mathrm{i}}\right|{\mathrm{x}}_{\mathrm{i}})$ denotes the prediction score of the current local block, ${\mathrm{x}}_{\mathrm{i}}$ denotes the current local block input to the network for prediction, ${\mathrm{y}}_{\mathrm{i}}$ is the prediction label of the corresponding local block output by the network, and P is the confidence level corresponding to ${\mathrm{y}}_{\mathrm{i}}$.

However, considering that the local blocks obtained from the sliding window intercepts have an uneven texture distribution and are affected by signatures, the score fusion strategy with equal weights cannot reflect individual differences. The effective area contained in the localized block has large individual differences, and too large a background area will lead to performance degradation of the detection algorithm. In addition, the region impacted by the signed lines will affect the performance of the detection algorithm, i.e., the proportion of the signed lines in the local block is negatively correlated with the credibility of the detection results of the local block. Therefore, in this paper, we design the score fusion method based on the quality weight of fingermark localized blocks and propose the quality score calculation method based on the effective region and the proportion of signed lines, and the calculation formula is shown in Equation (4):

$$\mathrm{Q}=\mathsf{\sigma }{\mathrm{Q}}_{\mathrm{R}\mathrm{O}\mathrm{I}}+\mathsf{\tau }{\mathrm{Q}}_{\mathrm{S}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}}=\mathsf{\sigma }\frac{1}{\mathrm{w}\times \mathrm{h}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{w}}{\displaystyle \sum _{\mathrm{j}=1}^{\mathrm{h}}\mathsf{\epsilon }\left[\mathrm{p}\right(\mathrm{i},\mathrm{j})-{\mathrm{P}}_{\mathrm{R}\mathrm{O}\mathrm{I}}]}}+\mathsf{\tau }\frac{1}{\mathrm{w}\times \mathrm{h}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{w}}{\displaystyle \sum _{\mathrm{j}=1}^{\mathrm{h}}\mathsf{\epsilon }\left[\mathrm{p}\right(\mathrm{i},\mathrm{j})-{\mathrm{P}}_{\mathrm{S}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}}]}}$$

(4)

where $\mathsf{\epsilon }(\mathrm{t}-{\mathrm{t}}_0)$ is the step function whose value is 1 at $\mathrm{t}-{\mathrm{t}}_0\ge 0$ and 0 vice versa; w and h are the width and height of the localized block, respectively; $p(i,j)$ is the pixel value of the current coordinate; ${\mathrm{P}}_{\mathrm{R}\mathrm{O}\mathrm{I}}$ is the threshold for distinguishing between the valid area of the fingermark and the blank background, ${\mathrm{P}}_{\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}}$ is the threshold for distinguishing between the signed line and the valid area; and $\mathsf{\sigma }$ and $\mathsf{\tau }$ are correction coefficients for effective region weights and signed line weights. From this, the quality weight of the current localized block $\omega$ can be obtained, and the calculation formula is shown in Equation (5):

$${\mathsf{\omega }}_{\mathrm{i}}={\mathrm{Q}}_{\mathrm{i}}/{\displaystyle \sum _{\mathrm{j}=1}^{\mathrm{n}}{\mathrm{Q}}_{\mathrm{j}}}$$

(5)

where ${\mathrm{Q}}_{\mathrm{i}}$ is the mass fraction $\mathrm{Q}$ calculated from the current localized block i. The final prediction probability formula is obtained by improving Equation (3) by using the localized block mass fraction of the fingermark as the weight of the fraction fusion, as shown in Equation (6):

$${\mathrm{P}}_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}}={\displaystyle \sum _{\mathrm{i}=\mathrm{i}}^{\mathrm{N}}{\{\mathsf{\varpi }}_{\mathrm{i}}\cdot \left[\mathrm{P}\right({\mathrm{y}}_{\mathrm{i}}| {\mathrm{x}}_{\mathrm{i}})-\mathrm{t}\mathrm{h}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{l}\mathrm{d}\left]\right\}}+\mathrm{t}\mathrm{h}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{l}\mathrm{d}$$

(6)

where $threshold$ denotes the liveness probability judgment threshold. The score fusion strategy takes into account the variation in quality distribution of the fingermark image when making liveness detection decisions using local block quality weights. This approach leverages the individual differences in local blocks to a greater extent, resulting in improved network classification performance.

3.3.3. Model Fusion Strategies Based on Evidence Theory

Fingermark liveness detection accuracy is influenced by various factors, such as the raw material type of the fingermarks and the liveness detection methods employed. Additionally, the accuracy can be affected by the use of models with different classification strategies. This paper argues that network models with different classification strategies have their own strengths and that there is a degree of complementarity between them. The accuracy of forgery detection can be further improved through the method of model fusion. Evidence theory provides a powerful tool for characterizing and fusing uncertain information for decision making. Model fusion strategies based on evidence theory are designed and used to coordinate the complementarities between different models to achieve optimal results. The experiments in this paper only address the fusion of two models.

For a problem, all possible outcomes constitute a discriminative framework denoted by $\mathsf{\theta },$ where $\mathsf{\theta }$ is a non-empty set. The problem in this paper is fingermark liveness discrimination, so $\mathsf{\theta }=\{\mathrm{A},\mathrm{B}\}$, where A is true and B is false. Its power set is $2^{\theta }=\{\mathrm{\varnothing },\left\{\mathrm{A}\right\},\left\{\mathrm{B}\right\},\{\mathrm{A},\mathrm{B}\}\}$.

Another important concept in the theory of evidence is the basic probability assignment (BPA), whose mass function m satisfies the following condition, as shown in Equation (7):

$$\{\begin{array}{c}\mathrm{m}(\mathrm{A})\ge 0,\mathrm{f}\mathrm{o}\mathrm{r} \mathrm{a}\mathrm{l}\mathrm{l} \mathrm{A}\subseteq 2^{\mathsf{\theta }}\\ \mathrm{m}(\varnothing )=\mathrm{m}(\mathrm{A},\mathrm{B})=0\\ {\displaystyle \sum _{\mathrm{A}\subseteq 2^{\mathsf{\theta }}}\mathrm{m}(\mathrm{A})=1}\end{array}$$

(7)

where $\mathrm{m}\left(\mathrm{A}\right)$ denotes the extent to which the evidence supports Proposition A. There are two mass functions, $\mathrm{m}1$ and $\mathrm{m}2$, in this paper. If $\mathrm{m}\left(\mathrm{A}\right)!=0$, then A is said to be a focal element. Assuming that the focal elements are $\{{\mathrm{A}}_1,{\mathrm{A}}_2,\dots ,{\mathrm{A}}_{\mathrm{n}}\}$ and $\{{\mathrm{B}}_1,{\mathrm{B}}_2,\dots ,{\mathrm{B}}_{\mathrm{n}}\}$, in each decision, the two models are two sources of evidence, and the confidence output of their dichotomization is the evidence. In this paper, uncertainty weights are introduced to synthesize m₁ and m₂ into a new probability assignment function m₁₂ according to the Dempster–Shafer combination rule, whose synthesis rule is shown in Equation (8):

$$\begin{array}{l}{\mathrm{m}}_1\oplus {\mathrm{m}}_2={\mathrm{m}}_{12}(\mathrm{A})=\\ \frac{1}{1-\mathrm{K}}{\displaystyle \sum _{{\mathrm{A}}_{\mathrm{i}}\cap {\mathrm{B}}_{\mathrm{j}}=\mathrm{A}}\mathsf{\alpha }\cdot {\mathrm{m}}_1({\mathrm{A}}_{\mathrm{i}})\cdot \mathsf{\beta }\cdot {\mathrm{m}}_2({\mathrm{B}}_{\mathrm{j}})};\\ \mathrm{A}\ne \varnothing ,\mathrm{K}={\mathrm{m}}_1(\mathrm{A}){\mathrm{m}}_2(\mathrm{A})+{\mathrm{m}}_1(\mathrm{B}){\mathrm{m}}_2(\mathrm{B})\end{array}$$

(8)

where $\mathsf{\alpha }$ and $\mathsf{\beta }$ represent the uncertainty weights of the two models, while K is the conflict coefficient, which represents the degree of conflict between the evidence, and the size of K is positively correlated with the degree of conflict. After the BPA calculation, the joint support ${\mathrm{m}}_{12}\left(\mathrm{A}\right)$ and ${\mathrm{m}}_{12}\left(\mathrm{B}\right)$ of the two models for the classification results can be obtained. If ${\mathrm{m}}_{12}\left(\mathrm{A}\right)>{\mathrm{m}}_{12}\left(\mathrm{B}\right)$, the input image decision can then be categorized as a real fingermark, or otherwise a fake one.

4. Experiments

4.1. Database

Publicly available open source fingerprint databases are widely used in liveness detection, such as LivDet2017, FVC2002 and ATVS [22]. To verify the effectiveness of the proposed network for unsigned fingermarks, this paper first tests it on the publicly available fingerprint activity detection dataset LivDet2017, the details of which are shown in

Table 1. Details of LivDet2017 database.

LivDet2017	Train				Test
	Live	Ecoflex	Body Double	WoodGlue	Live	Gelatine	Latex	Liquid Ecoflex
Green Bit	1000	400	400	400	1700	680	680	680
Orcanthus	1000	400	400	400	1700	680	658	680
Digtial Persona	999	400	399	400	1700	679	670	679

.

However, there is no publicly accessible database of signed fingermarks. For this reason, the research team in this paper created a fingermark database containing two subsets: the unsigned fingermark dataset and the signed fingermark dataset. The original fingermarks were made from real fingers or fake fingermarks dipped in sealing clay and pressed onto paper documents. In order to simulate the real scenario, the signed fingermark data were created using paper documents with different types of signed pens, and then fingermarks were pressed, in line with the habit of signing first and pressing later. The original document was scanned by a document scanner at a resolution of 600 dpi, the original fingermark image was obtained and then cropped to a fingermark image of 500 × 300 pixels in size, and the corresponding real value label was created and categorized according to attributes (attributes include: liveness, raw material, date of collection, gender and age of the provider, etc.).

The database of fingermarks includes both live and fake fingermarks. The materials used to make fake fingermarks included all kinds of common chemical raw materials, and the specific types and quantities are shown in

Table 2. Details of fingermark database.

Type		Train		Test
		Signed	Unsigned	Signed	Unsigned
Live Fingermarks		32,667	25,511	8649	6459
Fake Fingermarks	Conductive silicone	6637	7500	1618	2124
	Skin color silicone	6322	3930	1440	1220
	Wood glue	5393	600	1365	160
	Clear silicone	7453	8790	1885	2343
	Fingerprint seals	5141	2280	1256	550
	Total	63,613	48,611	16,213	12,856

.

The fake fingermark materials include conductive silicone, skin-color silicone, wood glue, clear silicone and fingerprint seals, all of which are commonly used in previous spoofing attacks. Because it is usually difficult to distinguish between fingermarks made using different materials with the naked eye, as shown in Figure 6, it is necessary to ensure the diversity of materials in the dataset.

4.2. Experimental Environment and Evaluation Indicators

In this paper, stochastic gradient descent with a batch size of 64 samples was used to train the model. The initial learning rate was set to 0.01, which dynamically reduced as the number of iterations increased. Based on the PyTorch framework with version number 1.1.0.

The experiments in this paper were evaluated in terms of test sample detection accuracy, i.e., the rate of correctly classifying the test set samples (live and fake fingermarks). In this paper, the probability scoring threshold for liveness of fingermarks was set to 0.5, and fingermark images greater than or equal to the threshold were considered as live fingermarks, otherwise they were considered as fake fingermarks.

The training samples of the network with live fingermarks and fake fingermarks were randomly distributed and fed into the network to ensure that the training data were randomized. Each fingermark had a corresponding correct label: the label of the live fingermark was set to 1, and the label of the fake fingermark was set to 0. The test samples were fed into the trained network model to obtain the prediction probabilities corresponding to the two types of data, and the prediction labels were judged according to the probability score threshold. If the predicted label of the test sample was the same as the label of the real value, the prediction was correct; otherwise, the prediction was wrong.

4.3. Analysis of Experimental Results

4.3.1. Fingerprint Verification

Although the fingermark liveness detection method is proposed for signed fingermarks, it still needs to have the basic ability to detect the liveness of unsigned fingerprints. In this paper, the SFNet model was tested on the public fingerprint dataset LivDet2017 to verify the effectiveness of the proposed network structure, and the experimental results are shown in

Table 3. Comparison results of detection accuracy of different models based on the LivDet2017 dataset. Unit: %.

Model	Green Bit	Digital Persona	Orcanthus	Overall
JLW_A	95.08	94.09	93.52	94.23
JLW_B	96.44	95.59	93.71	95.25
ganfp	95.67	93.66	94.16	94.50
ZYL_2	96.26	94.73	93.17	94.72
SFNet	96.82	96.64	95.27	96.16

, and the other four sets of models were from the results of the LivDet2017 competition [23]. It can be seen that compared with the four best performing algorithms in the LivDet2017 competition, SFNet achieves the optimal results in terms of detection accuracy on all sub-datasets, which indicates that SFNet has a better fingerprint liveness detection capability.

4.3.2. Signed Fingermark Verification

Experiments were conducted in the database established in this paper to verify the effectiveness of the proposed SFNet and multi-probability labeling strategy for signed fingermarks. The network was confidently trained with unsigned fingermark data to ensure its ability to detect the liveness of unsigned fingermarks, without any interference from signed noise. SFNet models and an SFNet model incorporating a multi-probability labeling strategy were trained on a comprehensive fingermark dataset containing both signed and unsigned fingermarks (SFNet_1). The test results in

Table 4. Detection accuracy results for SFNet and multi-probability label strategy. Unit: %.

Model	Train	Live	Fake	Overall
SFNet	Signed	98.41	98.68	98.54
	Unsigned	98.96	98.84	98.90
SFNet_1	Signed	98.49	99.00	98.73
	Unsigned	98.97	99.26	99.13
SFNet_2	Signed	70.62	67.96	69.35
	Unsigned	97.98	97.79	97.89

demonstrate that the signed fingermark liveness detection network has a detection accuracy of over 98%, indicating that SFNet exhibits superior liveness detection performance on both signed and unsigned fingermark datasets. The method that incorporated the multi-probability labeling strategy had higher accuracy on both signed and unsigned fingermark datasets, with 99.00% and 99.26%, respectively. This confirms that the multi-probability labeling strategy significantly improves the accuracy of liveness detection and enhances network performance.

In order to verify the existence of signatures interfering with the liveness detection of fingermarks, a set of SFNet models incorporating a multi-probability labeling strategy is trained on the unsigned fingermark training set (SFNet_2). The model’s recognition accuracy for the signed fingermark test set is significantly lower than its performance on the unsigned test set, demonstrating a failure to meet the requirements of liveness detection. This confirms that the presence of a signatures has a negative impact on the accuracy of liveness detection, underscoring the need for a signed fingermark database to be established.

The original purpose of signed fingermark liveness detection algorithms is to defend against spoofing attacks. Therefore, the ability to resist attacks from fake fingermarks made from different materials is one of the important criteria for a liveness detection method. In this paper, we test the accuracy of a signed fingermark dataset made from different materials, as shown in

Table 5. The performance of the network on the signed fingermark database made using different materials. Unit: %.

	Type	SFNet	SFNet_1
Fake	Conductive silicone	98.76	99.07
	Skin color silicone	98.68	98.12
	Wood glue	99.12	99.19
	Clear silicone	98.62	98.99
	Fingerprint seals	99.52	99.28
	Live	98.41	98.49

. It can be seen that SFNet has good detection results for all five materials in the dataset, which indicates that SFNet has high recognition accuracy for known materials and can withstand spoofing attacks from most known materials. In addition, latex was tested as an unknown material, and compared to the known materials, the number of fake signed fingermarks obtained from latex production is lower, but also has a high recognition accuracy, which indicates that SFNet has a certain degree of robustness to unknown materials.

To verify the effectiveness of the proposed local block quality-based score fusion strategy, the effect of SFNet combined with the score fusion strategy was tested on the signed fingermark test set, and the experimental results are shown in

Table 6. Comparison of detection performance of SFNet combined with the score fusion strategy. Unit: %.

Model	Live	Fake	Overall
SFNet	98.49	99.00	98.74
SFNet_3	98.71	99.15	98.92
SFNet_4	99.08	99.27	99.17

. SFNet_3 represents the SFNet combining the multi-probability labeling strategy with the equal-weight score fusion strategy, and SFNet_4 represents the SFNet combining the multi-probability labeling strategy with the image quality-weight score fusion strategy. It can be seen that SFNet_4, which is based on the quality weight score fusion strategy, significantly improves the detection performance of the proposed method compared to SFNet without the score fusion strategy and SFNet_3, which is based on the equal weight score fusion strategy.

To confirm the hypothesis that different network models enhance each other and boost the accuracy of the liveness detection method, we implemented a model fusion strategy based on the theory of evidence. We combined the original SFNet with the SFNet that integrates the multi-probability labeling strategy and the image quality weight score fusion strategy. The resulting ROC and P–R curves, depicted in Figure 7, demonstrate the success of our approach. This section compares three versions of SFNet: the original SFNet (curve a), SFNet_3 (curve b), and SFNet_4 (curve c). SFNet_3 combines the multi-probability labeling strategy and the equal weight score fusion strategy, while SFNet_4 combines the multi-probability labeling strategy and the image quality weight score fusion strategy. The figure clearly demonstrates that the AUC value and the equilibrium point value are significantly higher after fusion than the two curves before fusion, providing strong evidence for the superior performance of the fused model. These results unequivocally establish the complementarity between network models with different strategies and the effectiveness of the model fusion strategy.

4.3.3. Ablation and Migration

In order to demonstrate the effectiveness of the improved residual blocks, corresponding comparison experiments were conducted. The residual block in SFNet combining the multi-probability labeling strategy and the image quality weight score fusion strategy was replaced with a different residual module as the control group, and the replaced residual modules were the original residual block proposed in [19] and the Res2Net module proposed in [20], respectively. The results, as shown in

Table 7. Comparison of detection performance of different residual blocks. Unit: %.

Model	Live	Fake	Overall
ResNet [19]	97.23	97.14	97.19
Res2Net [20]	98.23	98.58	98.39
Our work	99.08	99.27	99.17

, show that the test accuracies of the control group on the comprehensive test set are only 97.19% and 98.39%, while the test accuracy of SFNet combining the multi-probability labeling strategy and the image quality weight score fusion strategy using the improved residual block is 99.17%, which is a significant improvement. This result indicates that the improved residual block has higher performance in the signed fingermark liveness detection task and proves the effectiveness of the improvement.

This conclusion is based on the examination of the impact of local block sizes of training and test data on the network and showcases the importance of carefully selecting the appropriate block size for signed fingermarks. A local block size that is too small increases the effect of signatures in the image block on the fingermark texture. This paper argues that a larger localized block size is needed to obtain sufficient fingermark information and mitigate the negative effects of signed occlusion, which destroys the texture of the fingermarks. This paper presents two comparison experiments using localized block sizes of 112 × 112 pixels and 224 × 224 pixels, which are commonly used in fingermark-related studies. A control group using the original image size of 500 × 300 pixels is also included.

The results of the experiments are displayed in

Table 8. The performance of the network under different local block sizes. Unit: %.

Block Size	Signed	Unsigned	Overall
112 × 112	91.02	96.13	93.26
224 × 224	95.09	97.68	96.23
300 × 300	99.08	99.27	99.17
500 × 300	93.97	95.12	94.47

. It is evident that the effectiveness of the two comparison experiments with smaller sizes drops significantly, particularly for the signed fingermark test set. Notably, the accuracy falls below 90.0% when the local block size is 112 × 112 pixels. Using different local block sizes increases the interference of signatures on the texture of fingermarks in the image block, which affects the effectiveness of the algorithm for signed fingermark liveness detection. It is worth noting that the performance of the original image with a size of 500 × 300 pixels is lower than that of the image block of 224 × 224 pixels. This is due to the significant individual differences exhibited in the original images, with some samples containing numerous blank areas. Furthermore, the average effective information per unit area of the uncropped original images is much lower than that of other groups. The experiment results indicate that a larger size of localized fingermark block should be selected, as long as it contains sufficient valid information per unit area.

Finally, other algorithms are migrated for comparison experiments with this paper’s algorithm. Due to less research on fingermark liveness detection methods, this paper migrates two well-established fingermark liveness detection methods and one fingermark liveness detection method: the Slim-ResCNN [24], the method in [25] based on the CNN architecture, and Inception [21], which combines the patch-label strategy, which is trained on a comprehensive fingermark dataset and tested on a signed fingermark dataset.

The results are shown in

Table 9. The performance of migration methods on signed fingermark database. Unit: %.

Model	Live	Fake	Overall
Slim-ResCNN [24]	92.36	91.03	91.73
An CNN architecture [25]	82.13	84.47	83.24
Inception with patch-label [21]	98.57	98.14	98.37
SFNet after model fusion	99.15	99.46	99.30

, and under the same conditions, the detection accuracy of these methods for signed fingermarks is not satisfactory, and the lowest accuracy is lower than 90.0%, while the SFNet proposed in this paper, which combines the multi-probability labeling strategy and the image quality weight score fusion strategy, achieves the optimal test accuracy.

Counterfeiters may use novel materials to carry out spoofing attacks on the liveness detection algorithm. The ability of a liveness detection algorithm to resist attacks with unknown materials is also one of the criteria for judging the robustness of the algorithm. Unknown materials refer to materials that do not exist in the algorithm’s training dataset. In this paper, a small dataset was created based on five unknown materials, including glass glue, vulcanized silicone, glue, emulsion, and a bionic finger. Among them, the bionic finger was made of mixed materials to simulate a real finger, which has a flexibility and ductility very close to those of real skin, and has a similar appearance and touch to a real finger.

The experimental results are shown in

Table 10. The performance of migration methods on unknown materials. Unit: %.

Type		SFNet after Model Fusion	Inception with Patch-Label [21]	Slim-ResCNN [24]
Fake	Glass glue	96.38	91.30	81.88
	Vulcanized Silicone	97.73	93.18	75.14
	Glue	98.96	91.67	83.32
	Emulsion	92.96	89.63	72.59
	Bionic finger	89.17	83.33	69.17

. It can be seen that SFNet can be effectively protected against attacks from unknown materials, which is better than other methods. It can be seen that SFNet shows a slight degradation in accuracy against bionic fingers, so further research will need to consider investigating how to defend against unknown materials with high ductility and flexibility.

5. Conclusions

This paper proposes an effective method for detecting signed fingermark liveness to counter fingermark spoofing attacks and meet the needs of applications like judicial document authentication. The method is based on deep residual networks and multi-probability labels. The method combines the residual network structure and the multi-scale backbone structure to improve the accuracy of liveness detection. This is achieved through the use of a multi-probability label classification strategy, a multi-input score fusion strategy, and a model fusion strategy. The signed fingermark liveness detection method achieved a remarkable 99.17% success rate on the comprehensive dataset using different strategies, which was further improved to 99.30% after model fusion. These results demonstrate the method’s high effectiveness. It is worth noting that the method can also handle signed fingermarks on special paper backgrounds, such as squares and stripes, in practical scenarios. In the future, we will study liveness detection algorithms for signed fingermarks with complex backgrounds to significantly improve the practicality of fingermark liveness detection.