Non-Contact Heart Rate Detection Based on Hand Vein Transillumination Imaging

Abstract

Vein recognition technology identifies human vein characteristics under near-infrared light and compares it with stored vein information for personal identification. Although this has high anti-counterfeiting performance, it is possible to fabricate artificial hands that simulate vein characteristics to deceive the identity authentication system. In view of this potential deficiency, we introduced heart rate information to vein authentication, a means of living body detection, which can further improve the anti-counterfeiting effect of vein authentication. A hand vein transillumination imaging experiment was designed to prove its effectiveness. In the proposed method, a near-infrared light source is used to transilluminate the hand, and the transillumination images are collected by a common camera. Then, the region of interest is selected for gray-scale image processing, the feature value of each frame is extracted by superimposing and averaging the images, and then the one-dimensional pulse wave is drawn. Furthermore, the baseline drift phenomenon is filtered by morphological methods, and the maximum percentage frequency is determined by Fast Fourier Transform, that is, the pulse wave frequency. The heart rate value is then calculated, and finally, the stability of the heart rate detection result is evaluated. The experiment shows that the method produces accurate and stable results, demonstrating that it can provide living information (heart rate value) for vein authentication, which has great application prospects and development opportunities in security systems.

1. Introduction

The heart rate is an indicator of the physical health of the human body and has important applications in medicine, emotional analysis and other fields. In the medical field, monitoring the heart rate can play a role in preventive care [1,2], such as the screening of patients with early cardiovascular disease. The premise of affective computing is that changes in physiological states such as heart rate are closely linked to people’s emotions [3,4], so heart rate parameters are essential for building a comprehensive emotion recognition system. In addition, in recent years, heart rate detection has attracted increasing attention in the field of biometric authentication [5,6], and it can be used as an important parameter to improve the anti-spoofing systems.

Common heart rate monitoring methods include the electrocardiographic signal method, the arterial blood pressure method, photoplethysmography (PPG), and imaging photoplethysmography (iPPG), among others. The electrocardiographic signal method [7] is based on the sinus node, which rhythmically controls the systolic and diastolic contractions to pump blood to the trunk. The electrical signal gradually spreads to the body surface and can be measured by electrodes on the skin surface. The arterial blood pressure method [8] uses a pressure sensor to convert the pressure signal corresponding to the regular fluctuation of arterial blood pressure into a heart rate signal. In 1938, Hertzman proposed PPG [9], which detects the pulse by using light to transmit or reflect blood flowing in blood vessels. In 2000, AT Wu et al. proposed iPPG [10] on the basis of PPG, which collects reflected light images with a black-and-white camera and analyzes the picture sequence to obtain pulse wave signals. Since traditional heart rate monitoring methods, such as electrocardiographic signal and arterial blood pressure methods, require contact with the skin, users may experience discomfort, and it is not suitable for obtaining heart rate parameters when applied in the field of biometrics. On the other hand, with the development of technology, cameras have become ubiquitous, so their use in heart rate detection has many potential applications.

Biometric technology combines computers with high-tech approaches, such as optics, acoustics, biosensors, and biostatistics principles, and uses the inherent physiological characteristics of the human body (fingerprints, iris, gait, veins, etc.) to confirm a person’s identity [11,12,13,14]. Traditional identification methods include identification items (keys, certificates, ATM cards, etc.), and security information (such as usernames and passwords). However, because these methods mainly rely on external objects or memory, once the identification item or security information that proves identity is stolen or forgotten, the individual can be easily impersonated or replaced by others. Biometrics is more secure, confidential and convenient than traditional identification methods. Biometric identification technology has several advantages: it does not require memorization, has good anti-counterfeiting performance, is difficult to forge or steal, and is “carried” with the individual, and is usable anytime, anywhere. Among these techniques, vein recognition technology [15] is highly effective against counterfeiting. Veins are hidden in the body, and are inherent features that are difficult to forge and steal. Vein authentication is a second-generation biometric recognition technology [16], with great application prospects and development opportunities in security fields.

Although vein authentication has a high anti-counterfeiting ability, if an individual has permission to access the internal database and print out the vein characteristic data, it is possible to deceive the identity authentication system by creating a fake hand that simulates the vein characteristics [17,18,19,20,21]. Therefore, vein authentication can be improved by combing heart rate information obtained based on the iPPG method to perform living body recognition. This approach can not only prevent internal information leakage and fraud, but also assess the emotional state of the human person. If the heartbeat value is not within the normal range [3,4], it may indicate intimidation or extreme irritation, in which case, the user is unable to perform normal authentication, thus ensuring the security of the system.

Based on the above considerations, this paper proposes a non-contact heart rate detection method based on hand vein transillumination imaging. Near-infrared light is used as the light source, and a fisheye lens camera obtains high-information images, which are then further processed. The eigenvalue is used to obtain the one-dimensional pulse wave signal, which is filtered through morphological processing [22]. Then, the pulse wave frequency is obtained through the Fourier transform, which realizes the detection of the heart rate. This method provides heart rate information during vein authentication, thereby ensuring that the subject is a living body, which has high a practical application value.

2. Materials and Methods

2.1. Acquisition of Vein Transmission Images

The transmission image acquisition device used to obtain the pulse wave of a hand is shown in Figure 1. It consists of an infrared light-emitting tube, an 850 nm narrowband filter (manufacturer: Hua Shang Laser, model: HS-850LGP-F4.5), a 150° fisheye lens camera (model: 3200_720P), a computer (model: HP Pavilion Gaming Desktop 690-05xx), and MATLAB software (MATLAB version: R2018a) which is employed to further process images. The infrared light-emitting tube light source used in the experiment has an emission angle of 45°, a voltage of 1.3–1.6 V, a current of 20–30 mA, and a peak wavelength of 850 nm. The center wavelength of the narrow-band filter is 850 ± 10 nm, the half-band is 30 nm, the peak transmittance is > 86%, and the ring depth is <1%. The camera frame rate is 20 fps, the output resolution is set to 1280 * 720, and the output format is set to JPG. In the process of data collection, the light source directly illuminates the hand, and the light passing through the hand is received by the camera and then transferred to the PC via USB for storage and post-processing. The entirety of the above-mentioned experiment was carried out in a closed environment to avoid interference from ambient light.

This experiment was performed on 10 subjects, and 10 sets of data were collected for each participant. Each set of data includes a heart rate measured by a blood pressure meter, which is treated as the true heart rate. Before data collection, the subjects were instructed to relax for a period of time while maintaining steady breathing and a stable emotional state. During data collection, the subjects held the near-infrared light source and tried to keep their hands steady. The data were collected by camera, and the measurement time of each subject in the experiment was 20 s. Image acquisition was performed on the basis of the experimental device to obtain the original vein transmission image (a set of data is used as an example for illustration below).

Infrared light serves as the experimental light source. The light source was held in the palm and the camera was located on the other side. The light source was turned on and video images of the light transmitted through the hand were recorded. A 20 s video was obtained, from which 400 image frames were extracted. Since the data obtained in the initial stage of acquisition may be unstable, 300 frames of the intermediate acquisition stage are selected for experimental data processing and analysis, one of which is shown in Figure 2a. The region of interest (ROI) was determined, and the corresponding interception for each frame was performed to obtain the image corresponding to the ROI, which is expressed as $I_{\mathrm{i}}$ (i = 1, 2…m, m = 300), as shown in Figure 2b. A gray-scale operation was performed to obtain a single-channel image corresponding to the ROI, denoted as $I_{gra{y}_i}$, one of which is shown in Figure 2c.

2.2. Acquisition and Pre-Processing of One-Dimensional Pulse Wave Signal

2.2.1. Acquisition of One-Dimensional Pulse Wave Signal

All pixels in each frame of ROI image ($s_i=sum(sum(I_i))$) were accumulated to obtain the feature value of each frame. All feature value points constitute a one-dimensional pulse wave signal (i = 1, 2…m, m = 1000). The schematic diagram is shown in Figure 3.

2.2.2. Elimination of Baseline Drift Based on Morphological Operation

In the process of collecting video images for a period of time with an ordinary camera, noise is inevitably be introduced due to the influence of external factors (camera instability, light intensity, etc.), which contaminates the collected signal and affects subsequent processing. In addition, the heart rate signal, the PPG signal also contain a series of low-frequency noises, from sources such as breathing, body tremor, and skeletal muscle contraction. Therefore, the signal-to-noise ratio should be improved to obtain a more accurate blood volume. The pulse waveform requires the source signal to be denoised. In this research, a morphological operation was used to denoise the signal.

Mathematical morphology theory states that open and close operations move structural elements below and above the signal, respectively. The open operation can cut off the signal “wave crest”, which is contractible, eliminates isolated signal points, suppresses the positive impulse noise, and makes the signal smooth; the close operation can fill the signal “valley”, with expansion, and can suppress negative impulse noise. The open and close operations both have low-pass characteristics. The purpose of the morphological filter is to eliminate the peaks and troughs of a specific width in the signal by combining open and close operations. The components that are actually filtered out of the signal are related to the structural elements used in the calculation. For example, the width of the structural element is M, and the width of the peak or trough in the signal is N, as shown in Equation (1).

$$\{\begin{array}{c}N>M, signal is retained\\ N<M, signal is filtered out\end{array}$$

(1)

In order to simultaneously remove the positive and negative pulse noises in the signal, a cascade combination of open and close operations was adopted to define the open-close and close-open filters, as shown in Equation (2).

$$\{\begin{array}{c}Morphological opening-closing: OC(f)=(f\circ k)•k\\ Morphological closing-opening: CO(f)=(f•k)\circ k\end{array}$$

(2)

where, $f$ represents the original signal, and $k$ represents the structural element.

Due to the shrinkage of the open operation, the output of the morphological open-close filter is too small, and due to the expandability of the close operation, the output of the morphological close-open filter is too large, which greatly affects the filter’s ability to efficiently remove noise. To reduce the occurrence of offsets and improve filtering effects, they are generally not used alone, instead the signal is filtered through the cascade of morphological open-close and close-open operations. The mathematical expression of the signal $f$ after morphological filtering is shown in Equation (3).

$$f^{’}=\frac{1}{2}[OC(f)+CO(f)]$$

(3)

The mathematical morphology filter adopts the linear structure element. The width of the linear structure element is set to 20, which is greater than the width of the characteristic pulse waveform. Morphological open-close and close-open filters are applied to the pulse wave signal to simultaneously remove both positive and negative impulse noises in the signal, respectively. The signal is filtered through the cascade of morphological open-close and the close-open operations, thereby improving the filter’s high-efficiency noise removal performance and achieving better filtering effects. The pulse wave signals before and after filtering are shown in Figure 4a,b.

2.3. FFT-Based Heart Rate Calculation

FFT [23] is the general term for an efficient and fast calculation method that uses a computer to calculate the discrete Fourier transform (DFT). The signal is transformed from the time domain to the frequency domain through FFT, and then the frequency spectrum structure and the change law of the signal are analyzed.

The full-period pulse wave signal is determined by peak-to-peak value detection, and the full-period FFT is further performed on the obtained pulse wave signal. The frequency domain diagram of the pulse wave is shown in Figure 5. Then, the heart rate value is determined by the maximum frequency ratio in the frequency chart, the maximum frequency ($f$) = 1.467 Hz, that is, the corresponding heart rate (HR) = f × 60 = 1.467 × 60 ≈ 88 bpm.

2.4. Performance Evaluation of Heart Rate Detection Algorithm

Accuracy and robustness are important indicators for judging the performance of a heart rate detection algorithm. Accuracy indicates the closeness of the measurement result of the heart rate detection algorithm to the heart rate reference value. Robustness refers to the performance of the heart rate detection algorithm under different interference conditions. To determine the degree of stability, also known as anti-interference, this article will evaluate the performance of the heart rate detection algorithm using four quantitative indicators and one qualitative indicator.

2.4.1. Qualitative Index

In modern science, the feasibility of a device or technology is assessed using a complete evaluation system, called a consistency evaluation, to determine whether two methods are interchangeable. Generally, the consistency evaluation method is used to determine the agreement between two methods. This article uses the Bland-Altman evaluation method to verify whether the non-contact heart rate test results are consistent with the reference values, thereby determining the effectiveness and accuracy of the algorithm.

The Bland–Altman analysis is named after the two scholars who proposed it in 1986 [24]. The main steps are to calculate the consistency limit of the two methods, display the calculation results graphically, and finally determine whether the two are consistent.

2.4.2. Quantitative Index

The error between the measured heart rate ($H{R}_g$) and the actual heart rate ($H{R}_{erro}$) is shown in Equation (4).

$$H{R}_{erro}=H{R}_m-H{R}_g$$

(4)

(1) Mean Error (M)

M refers to the average heart rate measurement error, which reflects the deviation of the measurement result from the reference heart rate. It is negatively related to the accuracy of the heart rate detection method.

(2) Standard Deviation (SD)

In order to improve the reliability of the average error index, as much sample data as possible should be used for the calculation in the analysis. SD represents the standard deviation of the measurement error value of the heart rate detection, which indicates the degree of dispersion of the heart rate error values. It is inversely related to the stability of the heart rate detection algorithm.

(3) Root Mean Squared Error (RMSE)

RMSE is the square root of the squared sum of the measurement error and the ratio of the number of video samples used for heart rate detection. The calculation formula is shown in Equation (5). RMSE is an indicator that measures the degree of deviation between the detected heart rate detection value and the true value; thus, the RMSE value reflects the accuracy of the heart rate detection algorithm, and is inversely related to the heart rate detection effect.

$$RMSE=\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N{(H{R}_i^{erro})}^2}}$$

(5)

where, $N$ is the number of video samples used for heart rate detection, $H{R}_i^{erro}$ represents the error value of the i-th video heart rate measurement, and $M$ is the average error.

(4) Pearson Correlation Coefficient (r)

The Pearson correlation coefficient (r) is the ratio of the covariance of the measured heart rate and the real heart rate and the product of the standard deviation of the two. The closer its value is to 1, the stronger the positive correlation. The calculation formula is shown in Equation (6). The larger the absolute value of r, the stronger the correlation between the measured value and the actual value, and the higher the accuracy of heart rate detection.

$$r=\frac{Cov(H{R}_m,H{R}_g)}{SD(H{R}_m)*SD(H{R}_g)}=\frac{{\displaystyle \sum _{i=1}^N((H{R}_m^i-\overline{H{R}_m^i})}{\displaystyle \sum _{i=1}^N(H{R}_g^i-\overline{H{R}_g^i}))}}{\sqrt{{(H{R}_m^i-\overline{H{R}_m^i})}^2*{(H{R}_g^i-\overline{H{R}_g^i})}^2}}$$

(6)

where, $N$ is the number of video samples used for heart rate detection, $Cov$ refers to the covariance, $SD$ represents the standard deviation, $H{R}_m^i$ and $H{R}_g^i$ represent the measured and true values in the i-th video, respectively, $\overline{H{R}_m^i}$ and $\overline{H{R}_g^i}$ are the average value of the detected heart rate of all sample videos and the average of the actual values of the tested sample videos, respectively.

3. Results and Discussion

In this study, we obtained heart rate information through hand vein transmission images, introduced living body information to vein authentication, and further improved the anti-counterfeiting effect of the vein authentication system. If the purpose is only to determine whether the body is alive, then it is only necessary to confirm whether the heart rate can be detected; however, we plan to further use heart rate information to assess the emotional state of the subject. If the heart rate value is not within the normal range, the individual may be under duress or experiencing extreme anger, in which case, the user is be unable to perform normal authentication, which can further ensure the security of the system. Based on the above considerations, we also have certain requirements for the accuracy of the heart rate obtained through the vein transmission image described in this article. Therefore, in this section, we further analyze the anti-interference ability and accuracy of the proposed heart rate detection method. Accuracy is the closeness of the measurement result of the heart rate detection algorithm to the heart rate reference value. Robustness refers to the stability of the heart rate detection performance under different interference conditions, that is, the anti-interference ability.

First, we analyzed the results theoretically. In the image acquisition process, we used a near-infrared light source for active illumination and collected hand vein images through transmission, so a high signal-to-noise ratio and information-rich transmission image can be obtained. Before obtaining the one-dimensional pulse wave signal, we used spatial superposition averaging. When obtaining the single-channel image, the three-channel image superimposition average was used; when the one-dimensional pulse wave signal was obtained, all of the pixels in the image frame were used. Spatial superposition and averaging were performed, thereby largely suppressing the interference of random noise. In pulse wave preprocessing, the morphological filtering method was used to reduce baseline drift. In summary, the aim is to obtain pulse wave signals with a high signal-to-noise ratio and high stability from multiple perspectives of image acquisition and processing.

Secondly, we verified the anti-interference ability and accuracy of the heart rate monitoring method through further data analysis. The results are analyzed and explained from two perspectives: qualitative and quantitative indicators.

3.1. Qualitative Index

This section uses the Bland-Altman method to verify whether the non-contact heart rate test results are consistent with the reference values, so as to confirm the effectiveness and accuracy of the method. The differences between the estimated values from the contact and non-contact methods are plotted with the average value of HR, and 100 pairs of measurement data of 10 participants are analyzed with the Bland-Altman method, as shown in Figure 6a,b.

First, a scatter plot is drawn based on the true heart rate and the measured value obtained by the camera, and the image is used to analyze the correlation between the two. As shown in Figure 6a, the relationship between the true heart rate and the measured value obtained by the camera is approximately linear.

Second, based on the number of data points outside the 95% consistency limit and the maximum difference within the agreement limit, and the degree of acceptability, consistency evaluation can be performed. It can be seen from Figure 6b that the 95% consistency is limited to -5.3~5.8 bpm, and 7.00% (7/100) of points are outside the 95% consistency limit. Within the consistency limit, the true heart rate is compared with the value measured by the camera. The maximum absolute value of the difference is 7 bpm, and the average error is 2.28 bpm. According to the pharmaceutical industry standard of the People’s Republic of China (error <= 5 bpm), the error is acceptable. Thus, the value measured by the camera is considered to be in good agreement with the true heart rate. In addition, the average HR difference is 0.3 bpm. The closeness of the true heart rate to the measured value obtained by the camera is indicated by the blue solid line, which represents the average HR difference. Close to the line (orange dashed line) represents an average difference of 0, which also reflects the high consistency between the two datasets.

3.2. Quantitative Index

In order to quantitatively evaluate the accuracy of the pulse wave detection method based on hand transmission imaging from various aspects, this section presents a quantitative analysis from four perspectives: mean error (M), standard deviation (SD), root mean squared error (RMSE), and the Pearson correlation coefficient (r).

Table 1. Statistical results of indicators.

Index	M	SD	RMSE	r
value	2.2814	1.1373	2.8254	0.7522

reports the evaluation results using different indicators. The values of M, SD, and RMSE are 2.2814, 1.1373, and 2.8254, respectively. Based on the work in [25], an index value of about 4 or less demonstrates the feasibility of the algorithm for heart rate detection. When the value of the index is close to or greater than 10, the algorithm is completely invalid for heart rate detection. The three indices are negatively correlated with the accuracy of heart rate detection, and thus, the results indicate the high accuracy of the heart rate values obtained by the method in this paper. The r value is 0.7522, which is positively correlated with the accuracy of the heart rate detection and also verifies the high accuracy of the heart rate results obtained in this study.

4. Conclusions

This paper proposes a non-contact heart rate detection method based on hand vein transillumination that can be used in vein authentication. The experimental results prove the effectiveness and stability of the method for detecting heart rate. The hand vein transillumination image is collected by near-infrared illumination, the ROI is extracted and grayed out, and the one-dimensional pulse wave signal is drawn by superimposition and averaging. The baseband drift noise is further filtered by the morphological method, and finally the pulse wave frequency is obtained through frequency domain analysis, resulting in the heart rate value. This article takes vein authentication as the starting point and aims to introduce living body identification information so as to further improve its anti-counterfeiting effect, which has potential value and significance for the application and development of vein authentication in the security field.