Multilingual Handwritten Signature Recognition Based on High-Dimensional Feature Fusion

Abstract

Handwritten signatures have traditionally been used as a common form of recognition and authentication in tasks such as financial transactions and document authentication. However, there are few studies on minority languages such as Uyghur and Kazakh used in Xinjiang, China, and no available public dataset for these scripts, which are widely used in banking and other fields. Therefore, this paper addresses this problem by constructing a dataset containing Uyghur, Kazakh, and Han languages and presents an automatic handwritten signature recognition approach based on Uyghur, Kazakh, Han, and public datasets. In the paper, a handwritten signature recognition method that combines local maximum occurrence features (LOMO) and histogram of orientated gradients (HOG) features was proposed. LOMO features use a sliding window to represent the local features of the signature image. The high-dimensional features formed by the combination of these methods are dimensionally reduced by principal component analysis (PCA). The classification is performed using k-nearest neighbors (k-NN), and it is compared with the random forest method. The proposed method achieved a recognition rate of 98.4% using a diverse signature database compared with existing methods. It shows that the method was effective and can be applied to large datasets of mixed, multilingual signatures.

1. Introduction

A handwritten signature is one of the most commonly used authentication techniques for banks and the judiciary [1]. The handwritten signature belongs to behavioral biometrics, which refers to the unique behavioral characteristics that can be used for human authentication. User authentication based on behavioral biometrics has characteristics such as stability and simplicity [2]. Biometric recognition is defined as the automatic identification and verification of persons. Verification is the mode in which the final result is only yes or no, and identification means authenticating a person’s identity. The related research on handwriting recognition can be divided into handwritten text recognition, handwritten signature recognition, etc. Handwritten text recognition refers to the process of automatically recognizing the text written inside an image of a text line, a paragraph, or even whole pages [3]. Handwritten signature recognition is the identification of the signer by the signature. The research in this paper focuses on user identification.

Offline handwritten signature recognition has been a challenging task for computer vision. Offline signature recognition can be addressed with writer-dependent and writer-independent approaches. Writer-dependent (WD) means that the classifier is trained separately for each person’s sample. In other words, when a new user comes in, it needs to be trained independently for that user. Writer-independent (WI) means that the classifier’s training is separate from the testing, which means that the user used for testing can be untrained. WI [4] is used when there are fewer signature maps available for training, although this method also has the potential to miss many writer-specific features. Therefore, the WD method was used in this paper.

Recently, some attempts have been made to solve the problem of failure of feature selection methods performed on a few features. Sharif M et al. [5] presented a new feature optimization selection technique employing a genetic algorithm to select the optimal features directly based on the fitness function of the features. A new extraction method was also introduced, and good results were obtained.

Hadjadji, B. et al. [6] proposed a system for open handwritten signature recognition and combined a classifier of curvilinear wave transform and OC-PCA. In a natural environment, there will not be many signature images used for training, and the writers have studied in WI mode. Therefore, they proposed a new multi-individual OHSIS combination method for density estimation to achieve an efficient system. A design protocol was also presented to select suitable parameters for the new writer.

Mshir, S. et al. [7] put forward a novel signature verification and recognition technique using two datasets to train the pattern through a Siamese network. Offline signature verification uses a convolutional Siamese network, and using the Kaggle dataset, the final recognition rate was 84%. Tests performed on the cross-domain dataset show that the network could handle forgeries in several languages and handwriting styles.

There were also some studies that have been validated using different classifiers. Elssaedi M. M. et al. [8] used five well-known classifiers, such as gradient augmented trees, extracted dynamic and static features (width, height), and utilized RapidMiner for feature selection. Experiments show that on the dataset used, the best classifier for recognition is a neural network, with an accuracy rate of 92.88%. We also validated the dataset in this study using random forest and KNN classifiers, respectively, to verify the proposed method’s effectiveness.

Jagtap, A. B. et al. [9] preprocessed the image first and then extracted the upper and lower envelope features from the preprocessed image. In this, the upper envelope features were extracted by scanning each column of the image from top to bottom. Finally, the extracted features were jointly fused and experimented with on the SVC2004 dataset, and an accuracy of 98.5% was obtained.

Several studies have been conducted to address the issues of accuracy improvement and reduction in the number of required samples. Matsuda K. et al. [10] addressed this issue by proposing a random forest (RF)-based technique—a joint segmentation verification method modification using multiple scripts signature authentications. The tactic of this technique was to perform different fusion methods for multiple signature identifiers. Gradient features were extracted, and shape features were used to represent the user’s pencil stress and speed. Finally, experiments were conducted on the SigComp2011 (Chinese, Dutch) and the SigComp2013 (Japanese) signature datasets. By comparing the three methods of signature image generation, the martingale distance of grayscale, RGB pipeline images, and the histogram of grayscale pictures are employed. Finally, the effectiveness of the presented approach was verified.

Recently, some studies have been conducted on offline signature recognition for minority scripts. For example, Zhang S. J. et al. [11] proposed a BoVW-based feature option algorithm MRMR for offline signature verification. Visual word features were extracted, and the maximum relevance minimum redundancy algorithm (MRMR) was employed for feature selection. K means clustering method was used to cluster the signature images, and support vector machine(SVM) was used for classification. Experiments were conducted on the Uyghur signature dataset in the self-built database and the CEDAR signature dataset [12], obtaining 93.81% and 95.38% recognition rates, respectively. The paper has been studied only for Uyghur, while our study was conducted on Kazakh, Han, and Uyghur datasets. In addition to traditional learning methods, many people have recently explored the use of deep learning methods to solve the offline handwritten signature problem. Tuncer T. et al. [13] studied feature extraction using convolutional neural networks (CNN) and proposed an iterative minimum redundancy maximum relevance IMRMR [14] method for the automatic selection of optimal features, and these features are utilized as an input of the SVM. Some have proposed new deep learning frameworks or models to solve the offline handwritten signature problem [15,16,17,18,19]. To solve the problem of limited training data, the network SigNet CNN inspired by (Krizhevsky et al. [20]) and modified (Hafemann, L. G. et al. [21]) was proposed, in which migration learning was introduced, and the pretrained model was fine-tuned using a limited number of available feature images (target data) in between. Finally, a BF-SVM classifier was used for classification, effectively solving the offline signature identification problem.

In this paper, we adopted the WD method to address the lack of studies on minority scripts and publicly available datasets. We built a dataset containing Uyghur, Kazakh, and Han languages, and our study was conducted on these and public datasets.

Table 1. Comparison of data volumes for different data sets.

Dataset	Signers	Genuine Signature/Signers	Forged Signatures/Signers	Numbers of Genuine Signatures
BHSig-B [22]	100	24	30	2400
BHSig-H [22]	160	24	30	3840
CEDAR [12]	55	24	24	1320
GPDS [23]	4000	24	30	96,000
Uyghur (Ours)	160	24	–	3840
Kazakh (Ours)	151	24	–	3624
Han (Ours)	160	24	–	3840

shows the quantity of data in our self-built dataset compared with the publicly available dataset. We proposed an approach based on the fusion of local maximum occurrence features and a histogram of oriented gradient features for recognition. We performed feature selection and used principal component analysis for the extracted feature vectors to improve recognition efficiency and speed. Finally, random forest and KNN classifiers were used for evaluation, and the CEDAR public dataset was used for testing to demonstrate the efficiency of the presented method.

Our signature dataset contains three subsets: the Uyghur dataset, the Kazakh dataset, and the Han (Chinese) dataset. Uyghur and Han’s datasets contain 160 individual signature samples, the same number of signers as the BHSig260-Hindi dataset. We have followed the same protocol as in GPDS to generate these signatures. Hence, 24 genuine signatures are available for each of the signers. Since we are studying signature recognition, which requires authentic signatures, only genuine signatures are captured, not forged ones.

In this paper, Section 2 is an introduction to the self-built dataset. Section 3 is an explanation of the proposed methods we used. Section 4 shows the results of the experiments. Finally, the conclusion is given in Section 5. Figure 1 shows the main flowchart of offline signature recognition in multiple languages.

2. Offline Handwritten Signature Recognition

2.1. Signature Datasets

Collecting multilingual offline handwritten signatures is a critical and challenging task. After collecting the data, a lot of postprocessing work is needed, such as cutting and labeling. To more comprehensively reflect the characteristics of an individual’s handwritten signature, we chose three time periods: morning, noon, and afternoon. Each period allowed the signer to sign eight signatures, for a total of 24. The signature was signed on A4 paper using a black signature pen, and the signature was finally scanned and cut for all datasets and saved in BMP format. We built a total of three datasets in Uyghur, Han, and Kazakh. Among them, 160 individuals with 24 signatures were collected in Uyghur, 151 individuals with 24 signatures in Kazakh, and 160 individuals with 24 signatures each were collected in Han. A comparison test was also conducted using the data from CEDAR [12], a publicly available dataset comprising 55 individuals with 24 genuine signatures each. Figure 2 shows the sample signature image we used, from left to right, for CEDAR, Han, Uyghur, and Kazakh.

2.2. Preprocessing

Preprocessing is the work that precedes feature extraction. After data acquisition, scanning of still images or other processes may incorporate problems such as noise [5]. Therefore, we needed to preprocess the collected dataset to improve accuracy. Because of the varying size of signature images in the CEDAR dataset, we first normalized the signature images in this paper to make the training process faster. We considered the structure and writing style of Uyghur and Kazakh signature data, and the signatures signed by these texts are longer. So, we chose the size 384 × 96. Second, we gray scaled signature images and used the weighted average method. The weighted average method refers to weighting the three channels of the image R, G, and B by different coefficients and using the weighted values as the grayscale results, with the following expression:

$$\mathrm{Gray}=0.3\times \mathrm{R}+0.59\times \mathrm{G}+0.11\times \mathrm{B}$$

(1)

Next, denoising and smoothing were conducted using a bilateral filter, which can keep the edges and reduce the noise smoothly, making it possible to eliminate the irrelevant information in the image and retain the practical information. Finally, we performed the background removal. The offline signature image has a large amount of blank background, which is useless information. To make the extraction of features more conducive to extracting effective information, we segmented the signature image using the Otsu algorithm to obtain a binary image. Figure 3 shows the preprocessed signature image we used.

3. Proposed Method

In this paper, after the signature images preprocessing, the local maximum occurrence (LOMO) [24,25] features were extracted first, followed by the histogram of oriented gradient (HOG) features. Then, considering the information on the texture, color, and geometric structure of the signature image, we fused LOMO features and HOG [26] features. For a combination of features, we used the concat method, which directly joins two features. For example, the dimensions of the two input features x and y if p and q, and the dimension of the output feature z is p + q. Then, feature selection is performed using principal component analysis. We downscaled the feature vector dimensionality to 100 and 128 dimensions, respectively. The final recognition rate was obtained by feeding the dimensionality-reduced feature vector into the random forest classifier and the k-nearest neighbor algorithm classifier. The offline handwritten signature recognition process included data acquisition, preprocessing, feature extraction, feature fusion if the features extracted here are multifeatures, feature dimensionality reduction, and finally, classification. Figure 4 presents the flowchart of the proposed signature recognition approach.

3.1. Feature Extraction

Feature extraction is the essential stage of offline signature recognition. Good feature selection represents a high recognition rate, and the quality of feature extraction methods directly affects the recognition results. In the paper, we firstly presented a handwritten signature identification method based on improved LOMO [24,25] features. LOMO features describe the signature image’s detailed information. We considered that the signature image has a stable geometric structure, so the HOG gradient direction histogram feature was used. Then, we proposed another handwritten signature recognition approach based on the fusion feature. The extraction process of these two features is explained in detail below.

3.1.1. Local Maximal Occurrence Representation

The LOMO [24,25] feature unites HSV color histogram and SILTP texture feature descriptors and uses the maximum pooling method to obtain a more stable feature representation. It was an efficient feature representation method. In the original LOMO feature, the Retinex method was used to preprocess the image to produce a color image consistent with the scene observed by humans. However, we did not use the Retinex method for preprocessing and instead used the four steps described in Section 2. The HSV color histogram was used for feature extraction of the preprocessed image. Because the signature image does not have much color, the features extracted here were grayscale features. Immediately after that, the scale-invariant ternary mode SILTP [27,28] method was applied to extract the texture feature of the signature image.

We used a sliding window to describe the local details of the signature images. First, the input signature image was horizontally segmented into five horizontal bars, the sliding window size used was 10 × 10, and the step size was smaller than the sliding window size so that each subwindow scan overlaps somewhat. Therefore, the set step size was five pixels, and in every child window, we extracted two scales of SILTP histogram (${\mathrm{SILTP}}_{4,3}^{0.3}$, ${\mathrm{SILTP}}_{4,5}^{0.3}$) and an $8\times 8\times 8-bin$ joint HSV histogram. Immediately after, two $2\times 2$ local mean pooling operations were performed and cascaded all features. The final descriptor has 26,960 dimensions.

The extraction method of SILTP features is described in detail below.

SILTP is a scale-invariant local three-valued pattern derived from LBP texture features and LTP features [28]. LBP feature extraction method and its improved algorithm have achieved remarkable results in texture analysis and pattern recognition applications. SILTP adds a point to LBP to describe the image with only one comparison but is robust to illumination and retains the background texture information of soft shadows, making it more robust. SILTP introduces the LTP tolerance range, thus making it robust. Figure 5 demonstrates the encoding process of LBP.

The SILTP [27,28] algorithm introduces a threshold factor that adaptively generates a comparison range, which reduces the probability of false positives and makes it better to cope with noise and illumination variations. The algorithm compares two values: a threshold value on the central pixel point value and all neighboring pixel points with radius r of the central pixel point. The results are divided into the following three, whose SILTP is encoded as presented in Equation (2).

$${\mathrm{SILTP}}_{Q,r}^{\mathrm{t}}(x_c,y_c)=\underset{d=0}{\overset{Q-1}{\oplus }}{S}_t(P_c,P_d)$$

(2)

$$S_t(P_c,P_d)=\left\{\begin{array}{l}01\to P_d \rangle (1+t)P_c\\ 10\to P_d \langle (1-t)P_c\\ 00\to \mathrm{o}thers\end{array}\right\}$$

(3)

In the above equation, four $P_c$is the central pixel gray value, $P_d$ is the corresponding pixel gray value of the Q domain with r as the radius, ⊕ which means the binary values of all domains are concatenated into a string, and t denotes the threshold range. Figure 6 shows the encoding procedure of the SILTP operator, where the obtained binary result is encoded in the counterclockwise direction according to the formula, and the result is 0001011010000000.

When there is noise in the image, the encoding value of SILTP does not change, and the operator has good robustness. The local three-value model (SILTP) with scale invariance increment the conclusion complexity of the histogram. For example, in a SILTP operator, a certain domain has $3^8$ features and eight domain LBP features are $2^8$ eigenvalues. Therefore, PCA is needed for dimensionality reduction to obtain the best features.

3.1.2. Histogram of Oriented Gradients

HOG [26] features, also known as a histogram of oriented gradient features, extract the gradient information of image edges. Implement the edge gradient information in the form of a histogram of gradients, and finally extract the edge features of the detection target by comprehensive merging. Figure 7 shows the extraction process of HOG features.

To reduce the influence of lighting factors, the whole image needs to be normalized first. The local surface exposure contributes a large proportion of the image texture intensity, so this correction process effectively reduces local shadows and lighting variations in the image. Because color information plays a minor role, it is usually first converted into a grayscale map.

The Gamma correction equation is below:

$$I\left(x,y\right)=I{\left(x,y\right)}^{\mathrm{gamma}}$$

(4)

For example, one can take Gamma = 1/2. For extraction, the image is first divided into small blocks of 10 × 10, then each small pixel’s horizontal gradient and vertical gradient are found. Each pixel point will have two values of gradient direction and intensity. Then, the gradient of every block in the histogram is counted. The HOG feature vector can be obtained by iteratively computing each block immediately after the loop. A person’s different signature images will be slightly different, not identical, but HOG is suitable for such images, and the slight variations will not affect the results.

3.2. Classification

User identification is to identify who this tester is, so classification for offline handwritten signature recognition is a vital stage. Regardless of the classifier chosen, we have to use the training data for training, and after training the optimal model, we can use it for our task. This paper chose a random forest classifier and k-nearest neighbor algorithm for classification.

3.2.1. Principal Component Analysis

After extracting the 26,960-dimensional LOMO features and 3780-dimensional HOG features, the two methods were fused in parallel. Since there would be much redundant information, the principal component analysis method was used for the feature option. The PCA algorithm was used to reduce the feature vectors to 100 and 128 dimensions, respectively, and the best feature vector dimension was derived by comparing experiments.

3.2.2. Random Forest

Random forests are composed of many decision trees, so decision trees are introduced first. Decision trees are supervised learning algorithms with if-then-else rules that are more in line with intuitive human thinking. The logic diagram of the decision tree is demonstrated in Figure 8.

A random forest [29] is composed of many decision trees, such as the one shown above, and each decision tree is not associated with the other. The steps of random forest construction are as follows:

(1) First, generate a random sample of the training subset;

(2) Select attributes in the original signature dataset for node splitting;

(3) Repeat step (2) until no more splitting is possible, and find the optimal solution among the selected feature values as the classification criteria;

(4) Build n decision trees to derive the classification results. The highest result is obtained as the final classification result. Figure 9 demonstrates the flowchart of random forest construction.

3.2.3. K Nearest Neighbors

The k-nearest neighbor algorithm is usually used for classification and is a nonparametric statistical method [30]. Its main idea is that the tested sample is similar to the neighboring Q data, and the class of the tested sample is the same as the class of Q samples. Therefore, the value of k is also extremely important, and the KNN method reclassifies decisions with only a minimal number of neighboring samples. This paper also used different k values to test the KNN model that is most suitable for our datasets.

4. Experimental Results

A total of four language datasets were used in this experiment. Among them, there were 160 signers with 24 signature images each for Uyghur and Han languages. In other words,320 signers provided 7680 offline handwritten signature images. Kazakh had 151 signers with 24 signature images each. In order to verify the effectiveness of the experiment, this paper also uses the open Latin offline handwritten signature dataset CEDAR [12]. Since the main purpose of this study was to determine which signers the signature to be tested belongs to, only all genuine signature samples in CEDAR (55 signers with 24 signature images each) were used in this paper, for a total of 1320 signature images. In this paper, S (=6, 9, 12, 15, and 18) denotes the S signature images of each signer as training data; the remaining were used as test data, and the signature images have no order.

4.1. Evaluation Criteria

The performance of this presented approach was assessed using the average accuracy (ACC), as demonstrated in Equation (3).

$$\mathrm{ACC}=\frac{1}{m}{\displaystyle {\sum }_{i=1}^m\frac{R}{D}}$$

(5)

where D is the quantity of all figures involved in the test, R denotes the number of correct predictions of all data in the test, and m = 10. In this paper, each experiment was conducted ten times, and the ACC was obtained by taking the average value.

4.2. Analysis of Experimental Results

4.2.1. Experimental Results Based on the LOMO Method

The datasets used for the experiments in this paper were four languages (Kazakh, Han, Uyghur, CEDAR) with a total of 526 signers. The experiments on the offline signature recognition method based on the local maximum occurrence (LOMO) feature were conducted first. KNN and RF classifiers were used, respectively. Among them, the recognition rates of Han, Kazakh, Uyghur, public dataset CEDAR, and multilingual mixed languages based on Lomo features are shown in Figure 10 below.

In Figure 10, we found that the most significant identification results were obtained when the extraction method based on local maximum occurrence features was used and the training set S was set to 16. When using the RF classifier, the recognition rate of a single language was 92.83%. when using the KNN classifier, the recognition rate can reach 94.15%. The recognition results of RF and KNN classifiers were compared, and we can conclude that the identification rate of the LOMO feature-based recognition approach using the KNN classifier was slightly higher than the recognition rate used in RF classifier.

The previous set of experiments concluded that the LOMO feature-based extraction method has the best recognition when using the KNN classifier. Therefore, we have conducted a set of experiments based on the KNN classifier recognition method.

Table 2. Recognition results of Lomo features of different dimensions.

Dataset	Dimension		Train Numbers
	100	128	S = 14	S = 15	S = 16	S = 18
CEDAR	√		97.2	97.6	97.8	98.3
		√	97	97.3	97.2	97.7
Han (Chinese)	√		88.64	88.83	88.85	89.96
		√	88.01	88.1	88.22	88.91
Uyghur	√		90.49	91.1	91.7	92.24
		√	90.4	91.1	91.95	92.45
Kazakh	√		92.38	92.57	92.3	94.15
		√	92.85	92.94	92.72	94.59
Uyghur + Kazakh	√		86.33	86.68	87.37	87.83
		√	86.94	87.66	88.55	89.04
Han (Chinese) + Uyghur	√		82.49	83.23	83.13	83.91
		√	83.07	83.8	83.39	83.89

demonstrates the results of the LOMO feature-based recognition method using a KNN classifier with 100 and 128-dimensional feature vectors, respectively. This set of experiments was performed to validate the most suitable feature vector dimension of Lomo features based on our datasets. It was finally concluded that when the extracted feature vectors are downscaled to 100 dimensions, it can make our dataset achieves a better recognition rate.

4.2.2. Recognition Results Based on HOG Feature Extraction

Figure 11 shows the results of the identification method based on gradient histogram HOG features. In order to have a more visual representation, the results we have not only shown in tabular form but also used histograms. When using this recognition method, to obtain a better recognition model, we tried to input two feature vectors of different dimensions into the classifier separately and compared the better combination. It was clear from the above figure that when the KNN classifier was used, the recognition rate of the 100-dimensional HOG feature vector was slightly higher than that of the 128-dimensional HOG features. It can be seen from the comparison that when the dimension was reduced to 100 dimensions, the recognition rate on the CEDAR dataset reached 94.62%. Combined with the histogram, one can see that the recognition method has better recognition results on our self-built dataset, reaching 97.48% on the Kazakh signature dataset.

Table 3. Single language recognition results are based on HOG features.

Dataset	Classifier
	S = 14		S = 15		S = 16
	KNN	RF	KNN	RF	KNN	RF
Han(Chinese)	93.47	91.5	93.53	91.75	94.49	96.46
CEDAR	98.32	95.19	94.65	95.99	94.21	93.48
Uyghur	95.78	94.17	96.44	92.38	95.97	95.35
Kazakh	97.09	95.18	97	92.38	96.74	94.88

and

Table 4. Multilingual Mixed Signature Recognition Results Based on HOG Features.

Dataset	Dimension		Train Numbers
	100	128	S = 14	S = 15	S = 16	S = 18
Han + Uyghur	√		93.86	93.83	94.36	94.46
		√	93.42	93.52	93.92	93.83
Uyghur + Kazakh	√		95.02	95.5	95.72	96.49
		√	94.13	94.68	94.8	95.83

show the results of extracting HOG features from signature images and recognizing offline signatures of a single language and offline signatures of two languages mixed. Table 3 shows the results of experiments in which 100-dimensional HOG features are fed into two classifiers separately. It can be seen that when the KNN classifier is used, the recognition rate on the single-language data set can reach 97.09%, while the recognition rate is 96.77% when the RF classifier is used. Finally, it can be concluded that the identification approach based on the KNN classifier was better. Therefore, on the mixed, multilingual dataset, we have performed a set of recognition experiments that send 100-dimensional and 128-dimensional feature vectors only to the KNN classifier. Table 4 shows the results of the experiments we made. It can be seen that when S = 18, the recognition rate of mixed multilingual (Uyghur + Kazakh) reaches 96.49% when using 100-dimensional feature vectors and 95.83% when using 128-dimensional feature vectors. Through comparison, we can conclude that the recognition rate of 100-dimensional feature vectors is higher than that of 128-dimensional feature vectors. Finally, it is concluded that the proposed method is also applicable to mixed, multilingual datasets.

4.2.3. Results of a Handwriting Signature Recognition Approach Based on Fused Features

Table 5. Used the RF classifier to identify the result after the fusion of the two features.

Dataset	Train Numbers
	S = 14	S = 15	S = 16	S = 18
CEDAR	94.35	95.54	95.44	95.74
Han(Chinese)	91.95	92.38	92.86	93.1
Uyghur	93	94.55	94.47	96.65
Kazakh	95.02	95.93	94.88	95.54
Han + Uyghur	90.73	91.23	91.58	92.87
Han + Uyghur + Kazakh	88.33	89.57	90.21	91.21
Uyghur + Kazakh	90.37	91.23	91.58	92.87
Han + Kazakh	91.13	91.2	91.63	92.48

and

Table 6. Used the KNN classifier to identify the result after the fusion of the two features.

	S = 14	S = 15	S = 16	S = 18
CEDAR	94.22	94.08	94.33	93.99
Han (Chinese)	93.74	93.95	95.11	94.98
Uyghur	95.81	96.23	96.65	96.05
Kazakh	96.9	96.28	96.8	96.76
Han + Uyghur	94.74	95.34	95.46	96.38
Han + Uyghur + Kazakh	93.62	94.4	94.89	95.11
Uyghur + Kazakh	94.74	95.34	95.46	96.38
Han + Kazakh	94.53	94.37	94.66	95.92

show the result of the recognition approach based on fused local maximum occurrence features and gradient histogram features. Table 5 shows the recognition results based on the RF classifier, while Table 6 demonstrates the recognition results using the KNN classifier. It can be seen from Table 5 that when S = 18, the recognition rate of a single language can reach 96.65%, while the recognition rate of a mixed multilanguage is about 92%. Similarly, it can be seen from Table 6 that the recognition rate of a single language is 96.76%, while the result of a mixed multilanguage can reach 96.38%. By comparing the two tables, it can be seen that the recognition rate using the KNN classifier is higher by 1–3% than that using the RF classifier.

Figure 12 and

Table 7. Comparison of recognition results of CEDAR datasets based on different features.

Method	Dimension		Train Numbers
	100	128	S = 15		S = 16		S = 18
			RF	KNN	RF	KNN	RF	KNN
HOG	√		95.99	94.65	96.46	94.21	96.77	94.62
		√	96.26	92.78	96.14	92.28	97.85	91.4
LomoHOG	√		95.54	94.33	95.44	94.08	95.74	94.22
		√	95.55	91.3	94.76	91.34	96.66	90.58
LOMO	√		97.3	97.6	96.5	97.8	98.4	98.3
		√	97	97.3	97.1	97.2	97.8	97.7

compare the experimental results of the recognition method based on the local maximum LOMO feature, the recognition method based on the HOG feature, and the recognition method based on the fusion feature of the public dataset CEDAR. From the comparison, we can see the advantages of the Lomo algorithm, whether using a 100-dimensional LOMO feature vector or a 128-dimensional LOMO feature vector for handwritten signature recognition. Whether founded on the KNN classifier or RF classifier, its recognition rate for the public dataset CEDAR was much higher than the results of the gradient histogram-based recognition method. When the training set S is set to 18, the recognition rate based on local maximum occurrence features reached 98.4%, which was higher than the recognition rate of HOG features alone by 1%. Figure 12 shows the effectiveness of the LOMO feature in handwritten signature recognition.

Table 8. Mixed multilanguage signature recognition results were based on different features.

Dataset	Method	Train Numbers
Han (Chinese) + Uyghur		S = 14	S = 15	S = 16
	LomoHOG	90.73	91.23	91.58
	Lomo	76.35	76.7	77.81
	HOG	88.8	89.5	90.72

shows the ablation experiments we performed to compare the results for the three recognition methods applied to the mixed multilanguage dataset (Han (Chinese)+ Uyghur) separately. The table shows that the recognition method based on fused local maximum occurrence and histogram of oriented gradient features performs better than the other two recognition methods. Although the authentication approach based on LOMO features has achieved good results on the public dataset CEDAR, the recognition results on the dataset we collected were not very satisfactory. In order to solve this problem, we introduced the gradient histogram feature and improved the recognition rate by fusing the two features. When S = 16, the recognition rate of the recognition method based on the fusion feature is higher than that of the Lomo feature alone by 13.77% and higher than that of the HOG feature alone by 2.07%. This shows that the proposed method was effective not only for single-language datasets but also for multilingual datasets.

Table 9. Results of different language recognition based on LomoHOG features.

Dataset	Train Numbers
	S = 15		S = 16		S = 18
	RF	KNN	RF	KNN	RF	KNN
CEDAR	95.55	91.3	94.76	91.34	96.66	90.58
Han (Chinese)	90.92	93.01	91.98	93.86	92.47	94.77
Uyghur	95.71	96.65	94.84	96.23	96.23	95.18
Kazakh	95.54	97.29	96.16	96.98	95.54	97.67
Han + Uyghur	91.01	95.39	91.12	95.46	91.22	94.46
Han + Uyghur + Kazakh	89.18	94.75	90.43	95.06	91.06	95.32
Uyghur + Kazakh	91.01	95.39	91.12	95.46	93.42	96.38
Han + Kazakh	90.67	94.31	91.44	94.47	92.8	95.27

demonstrates the experimental results of the recognition approach based on fusion features of single-language and mixed, multilanguage datasets. We found that when fused with LOMO features and HOG features, the recognition rate can reach 97.67% for a single language dataset, 96.38% for two languages mixed, and 95.32% for three languages mixed when using the KNN classifier in WD mode. When using the RF classifier for recognition, the recognition rate was 96.66% for a single language dataset, 93.42% for a mixture of two languages, and 91.06% for a combination of three languages.

In this study, we presented a recognition approach based on local maximum occurrence features, an authentication approach founded on the histogram of gradient direction features, and a recognition method fusing LOMO features and HOG features for multilingual offline handwritten signature recognition, respectively. Through ablation experiments, it can be concluded that the recognition method combining LOMO features and HOG features proposed by us is better than the first two methods, both for single-language and mixed multilanguage recognition. From the above recognition results, it may be noted that whether the RF classifier or KNN classifier was chosen for classification, regarded 14, 15, and 16 signatures per person were optioned as the training set and the rest as the test set, where the higher the number of training sets, the higher the correct recognition results. The recognition results of the KNN classifier outperformed the recognition results of the RF classifier when different features were extracted from the same training set, which proved the advantage of the KNN classifier.

4.3. Comparison of Experimental Results

With the aim of showing the effectiveness of the presented approach, we compared our experimental results with the existing experimental results.

Table 10. Results of Uyghur datasets and comparative experiments.

Reference	Database	Dataset Size/Training Data	Feature	Classifier	ACC
Aini, Z. et al. [31]	Uyghur	600 samplestrain 300 samples	Orientation feature	Euclidean distance	92.58%
Shu-Jing, Z., et al. [11]	Uyghur	train 480 samples	MRMR	SVM	93.81%
Ubul, K., et al. [32]	Uyghur	2500 samples, train 2000 samples	4MCLF-48 and LCDC	weighted Manhattan distance	94.60%
Xamxidin, N. [33]	Uyghur	4000 sample, train 2900 samples	-	IDN	94.32%
Ours	Uyghur	3355 samples train 2240 samples	LomoHOG	KNN	96.41%
Ours	Uyghur + Kazakh	6397 samples train 4269 samples	LomoHOG	KNN	94.74%

and

Table 11. Comparison of other research results on CEDAR datasets.

Reference	Database	Dataset Size/Training Data	Feature	Classifier	ACC
Batool, F. E., et al. [34]	CEDAR	1320 samples Train 840 samples	GLCM	SVM	96.45%
Souza, V. L. F. et al. [35]	CEDAR	–	–	DCNN	96.73%
Kumari, K. et al. [36]	CEDAR	-	-	SVM	94.9%
Culqui-Culqui, G. et al. [37]	CEDAR	-	-	CNN-HDR	93.92%
Ours	CEDAR	1320 samples train 825 samples	LomoHOG	RF	96.66%
Ours	CEDAR	1320 samples train 825 samples	Lomo	RF	98.4%

show the comparison results. It has been observed that both our presented offline signature recognition method based on local maximum occurrence features and the recognition method based on fusion features are better than some existing results on the public dataset CEDAR. Our approach put forward was optimal for the recognition of offline multilingual signature datasets. The above comparative analysis shows that the approach put forward in this study has some general utility in offline handwritten signature recognition, which provided a reference for the research of multilingual handwritten signature recognition.

5. Conclusions

This paper handled the problem of the lack of studies on offline signature recognition of minority languages and no publicly available dataset. First, a dataset including three languages (Han, Uyghur, and Kazakh) was built. Next, a handwritten signature recognition method based on local maximum occurrence features (LOMO) was proposed. This method achieved a recognition rate of 98.4% on the public dataset CEDAR. Then, considering the great differences in writing style and font structure between the public data set and the self-built data set, an offline signature recognition method based on fusing the local maximum occurrence feature and histogram of oriented gradient features was proposed. This method achieved a recognition rate of 96.8% for single-language signatures on the self-built dataset and 96.41% for the mixed, multilingual signature dataset. Meanwhile, by setting the number of training data, the dimensionality of the feature vector, and the classifier as independent variables, the best offline recognition model was found. In order to reflect the robustness of the method in this paper, recognition was performed not only on the KNN classifier but also on the RF classifier. By comparing and analyzing the final recognition rate of the two classifiers on the dataset, it was concluded that the recognition result of the KNN classifier was slightly higher than that of the RF classifier. It can be seen that the research of this paper fills the blank of signature recognition of minority languages in China. In the future, we want to extend the dataset and collect other languages, such as Kirghiz. Moreover, different algorithms are considered to improve the accuracy further and combine with our method.