iBVP Dataset: RGB-Thermal rPPG Dataset with High Resolution Signal Quality Labels

Abstract

Remote photo-plethysmography (rPPG) has emerged as a non-intrusive and promising physiological sensing capability in human–computer interface (HCI) research, gradually extending its applications in health-monitoring and clinical care contexts. With advanced machine learning models, recent datasets collected in real-world conditions have gradually enhanced the performance of rPPG methods in recovering heart-rate and heart-rate-variability metrics. However, the signal quality of reference ground-truth PPG data in existing datasets is by and large neglected, while poor-quality references negatively influence models. Here, this work introduces a new imaging blood volume pulse (iBVP) dataset of synchronized RGB and thermal infrared videos with ground-truth PPG signals from ear with their high-resolution-signal-quality labels, for the first time. Participants perform rhythmic breathing, head-movement, and stress-inducing tasks, which help reflect real-world variations in psycho-physiological states. This work conducts dense (per sample) signal-quality assessment to discard noisy segments of ground-truth and corresponding video frames. We further present a novel end-to-end machine learning framework, iBVPNet, that features an efficient and effective spatio-temporal feature aggregation for the reliable estimation of BVP signals. Finally, this work examines the feasibility of extracting BVP signals from thermal video frames, which is under-explored. The iBVP dataset and source codes are publicly available for research use.

1. Introduction

The foundation of optical sensing for the blood volume pulse signal laid by [1] proved to be significantly useful in clinical care settings, and the past decade also witnessed the proliferation of health-tracking devices and smart watches that monitor heart-rate and heart-rate-variability metrics. Since Verkruysse [2]’s pioneering investigation on the feasibility of extracting photo-plethysmography signals from RGB cameras in a contactless manner, increasing attention has been given to a wide range of imaging-based physiological sensing methods and their promising applications and contexts where non-invasive and contactless measurement techniques are preferred, such as stress and mental workload recognition [3,4] and biometric authentication [5].

Several rPPG datasets have been made available for academic use, and this recent review, [6], overviews these datasets. Some of the widely used datasets include MANHOB-HCI [7], PURE [8], MMSE-HR [9], VIPL-HR [10], UBFC-rPPG [11], UBFC-Phys [12], V4V [13], and SCAMPS [14]. These datasets consist of the RGB videos with resolution ranging from 320 × 240 to 1920 × 1080, and frame rates from 20 frames per second (FPS) to 120 FPS. The ground-truth data often consist of the photo-plethysmography (PPG) signal and/or electro-cardiography (ECG) signal, along with their computed pulse rate (PR) or heart rate (HR) metrics. The majority of these datasets are acquired in laboratory settings with controlled lighting conditions [7,9,11,12], and varying head movement, poses, and emotional changes. PURE [8], ECG-Fitness [15], and VIPL-HR [10] datasets introduce illumination changes for recording videos. VIPL-HR [10] and MANHOB-HCI [7] deploy multiple cameras to capture videos with different resolution as well as face poses. Unlike other datasets, the SCAMPS [14] dataset consists of synthetically generated videos with randomly sampled appearance attributes such as skin texture, hair, clothing, lighting, and environment, making it more suited to train supervised methods, rather than to evaluate rPPG methods. Most of the datasets focus on the RGB imaging modality.

As the ground-truth signals for rPPG datasets are collected using contact-based PPG or ECG sensors, it is essential to screen noise artifacts [16,17] present in such signals. Data collection scenarios including altered physiological states, varying ambient conditions, and head movement offer rich real-world representations, enabling robust training of supervised methods, as well as the realistic validation of both supervised and unsupervised rPPG extraction methods. However, these scenarios actively involving participants’ movement result in noises in the reference PPG signals as well due to varying light issues on the contact points. Representative noise artifacts present in ground-truth PPG signals of the existing datasets can be observed in Figure 1.

While researchers have proposed de-noising algorithms to remove artifacts from ECG as well as PPG signals [18,19], such methods are often limited to the signals that are not severely corrupted and are therefore reparable, while insufficient to denoise signals with substantial artifacts. Despite the availability of several rPPG datasets, the signal quality of the ground-truth signals is often neglected. Besides misleading the training of supervised methods, poor-quality ground-truth PPG signals can lead to the inappropriate evaluation of rPPG methods.

In this work, our collected RGB-thermal rPPG dataset (which we call iBVP dataset) is first assessed for signal quality of the ground-truth PPG signals. Signal-quality assessment is conducted both manually through the visual inspection of the PPG signals as well as using an automated approach. For the latter, we deploy SQA-PhysMD, adapted from a recent work [20], in which the CNN-based decoder module is replaced with a more efficient matrix-decomposition-based module [21] and the inference is made per sample, making it a dense 1D segmentation task. SQA-PhysMD shows significantly improved performance against the existing SOTA methods for an automated PPG-signal-quality assessment [22,23] on the iBVP dataset (preliminary results presented in

Table A4. Performance evaluation of PPG-signal-quality assessment methods on bench-marking and iBVP dataset.

SQ Method	PPG DaLiA Testing Set		iBVP Dataset
	DICE Score (%)	Accuracy (%)	DICE Score (%)	Accuracy (%)
Segade [23]	86.47	85.14	48.61	96.93
TinyPPG [22]	87.03	85.99	52.56	97.59
SQA-PhysMD (ours)	87.61	86.62	67.47	98.72

in Appendix C). The noisy segments can be flexibly removed from the ground-truth PPG signals as well as the corresponding video frames for training as well as evaluating rPPG methods. We make the original dataset available with high-resolution-signal-quality labels generated using SQA-PhysMD as well as the ones marked manually.

There have been several rPPG methods proposed over the decade, which can be categorized as unsupervised or model-based, feature-based supervised, and end-to-end supervised methods. Some of the notable foundational work with unsupervised methods includes Green [2], ICA [24], CHROM [25], PBV [26], POS [27], and LGI [28]. The research focus then progressed to feature-based supervised methods, which include HR-CNN [15], DeepPhys [29], MTTS-CAN [30], RhythmNet [31], NAS-HR [32], PulseGAN [33], and Dual-GAN [34]. More recent rPPG development focuses on end-to-end supervised methods, which can be sub-categorized into the methods using convolution networks such as PhysNet [35], 3DCNN [36], SAM-rPPGNet [37], and RTrPPG [38], and transformer-network-based methods such as PhysFormer [39], PhysFormer++ [40], and EfficientPhys [41]. In addition to RGB frames, researchers have also shown the benefits of combining RGB with additional imaging modalities such as near-infrared (NIR) and long-wave infrared (thermal IR). One recent work in this direction proposed an Information-enhanced Network (IENet) [42], highlighting the superior performance of the model that combines RGB and NIR frames. To address the challenges in training end-to-end approaches as well as to optimize the usage of computing resources, a recent work proposed a multi-stage network, MSDN [43], which implements three stages that can be trained independently.

In this work, we first present an evaluation of the introduced iBVP dataset, and then show preliminary results of a novel 3D-CNN-based end-to-end learning approach, iBVPNet, for estimating PPG signals. iBVPNet effectively captures blood volume pulse (BVP)-related spatial and temporal features from video frames. To evaluate iBVPNet and existing state-of-the-art (SOTA) models, we leverage maximum amplitude of cross-correlation (MACC) [44] as an additional metric while reporting commonly used metrics. In spite of MACC being highly relevant in evaluating rPPG estimation, it has not been discussed sufficiently in the literature. In summary, we make the following contributions:

• Introducing the iBVP dataset comprising RGB and thermal facial video data with signal-quality-assessed ground-truth PPG signals.
• Presenting and validating a new rPPG framework, iBVPNet, for estimating the BVP signal from RGB as well as thermal video frames.
• Discovering MACC [44] as an effective evaluation metric to assess rPPG methods.

2. iBVP Dataset

2.1. Data Collection Protocol

The data acquisition was conducted with an objective of inducing variations in physiological states, as well as head movement. Each participant experienced four conditions, including (a) rhythmic slow breathing and rest, (b) an easy math task, (c) a difficult math task, and (d) a guided head movement task, as depicted in Figure 2A. While a higher agreement between the rPPG and the ground-truth PPG can be achieved in the absence of these variances, the inclusion of the same in the data acquisition protocol enables simulating real-world physiological variations.

Cognitively challenging math tasks with varying degrees of difficulty levels were chosen, as these have been reported to alter the physiological responses [45,46,47]. The achieved distribution of heart rate computed from ground-truth PPG signals can be observed in Figure 2B. Furthermore, as wearable sensors are less reliable under significant motion conditions [48], we added an experimental condition that involved guided head movement. Each condition lasted for 3 min, with 1 min of rest after each condition. To randomize the sequence of conditions, we interchanged “A” with “D” and “B” with “C”. The study protocol was approved by the University College London Interaction Centre ethics committee (ID Number: UCLIC/1920/006/Staff/Cho).

2.2. Participants

PPG signals were collected from 33 participants (23 females) recruited through an online recruitment platform for research. Participants represented multiple ethnic groups and skin types with age ranging from 18 to 45 (27 ± 5.8). All participants reported having no known health conditions, provided informed consent ahead of the study, and were compensated for their time following the study. Detailed demographic information of study participants is provided in the

Table A5. Participant Demographics.

PID	Gender	Age	Ethnicity †
p01	M	33	A
p02	F	19	A
p03	M	21	A
p04	F	20	A
p05	F	32	D
p06	F	19	C
p07	M	18	C
p08	F	30	C
p09	F	25	C
p10	F	23	A
p11	M	32	A
p12	F	30	A
p13	F	25	C
p14	F	23	A
p15	F	24	A
p16	F	20	A
p17	M	28	E
p18	F	24	C
p19	F	21	A
p20	M	27	C
p21	F	27	C
p22	M	28	A
p23	F	34	A
p24	F	25	C
p25	F	27	C
p26	F	22	B
p27	M	45	A
p28	F	31	C
p29	F	28	A
p30	F	35	B
p31	M	33	A
p32	F	24	A
p33	M	29	D

† Ethnic groups are coded as follows: A: Asian/Asian British (Indian, Pakistani, Bangladeshi, Chinese, any other Asian background); B: Black/Black British (African, Caribbean, any other Black/African/Caribbean background); C: Caucasian; D: Mixed / Multiple ethnic groups; E: Other.

in Appendix D. After being welcomed and briefed, participants were asked to remove any bulky clothing (e.g., winter coats, jackets) and seated comfortably in front of a 65-by-37-inch screen, where they were fitted with PhysioKit [20] sensors. The PPG sensor was attached to participants’ left or right ear with a metal clip. Of the 33 participants, one was excluded from the dataset due to insufficient consent for sharing the identifiable data with the research community. While the dataset comprises 32 participants, this work utilized data from the first 30 participants owing to logistic limitations.

2.3. Data Acquisition

As depicted in Figure 2C, RGB and thermal cameras were positioned in front of the participant at around 1 m distance. Logitech BRIO 4K UHD webcam (Logitech International, Lausanne, Switzerland) was used to capture RGB video frames with 640 × 480 resolution, while thermal infrared frames were captured using thermal camera (A65SC, FLIR systems Inc., Wilsonville, OR, USA) having 640 × 512 resolution. Frame rate for both RGB and thermal video acquisition was set to 30 FPS. Customized software was developed using C++ programming language, which utilized the FLIR-provided Spinnaker library (https://www.flir.co.uk/products/spinnaker-sdk/, accessed on 7 November 2022) to acquire thermal infrared frames, while using OpenCV library functions to acquire RGB frames. The PhysioKit toolkit [20] was adapted to acquire the ground-truth PPG signals in synchronization with RGB and thermal frames. Implementation was adapted such that RGB frames, thermal frames, and PPG signal were acquired in separate and dedicated threads, while sharing a common onset trigger and a timer to stop the acquisition in a synchronized manner. PPG signals were acquired with a higher sampling rate of 250.

2.4. Morphology and Time Delay of PPG Signals

The majority of existing rPPG datasets have ground-truth PPG signals acquired using a finger probe or wrist watch, making it challenging to match the morphology as well as phase [49] of ground-truth and extracted rPPG signals from facial video frames. The morphology of PPG waveform is site-dependent [50] and therefore it is crucial to acquire ground-truth signals for rPPG from a site that is closest to the face. In addition, a recent study highlights the significant delay, equivalent to a half pulse duration between the PPG signal acquired from the finger and the rPPG signals [49]. With these considerations for morphological resemblance between PPG and rPPG signals and minimum time delay or phase difference, we carefully chose the ear as the sensor site for acquiring ground-truth PPG signals and attached the sensor clip to the upper or lower lobe of the ear based on the best fit and comfort of the participants. This makes the iBVP dataset highly suitable for training as well as evaluating the deep learning-based models that estimate BVP signals. It can be argued that the models that can reliably estimate the BVP signals offer significant advantages over the models trained to directly estimate heart rate or other BVP-derived metrics.

2.5. Pre-Processing and Signal Quality Assessment

A band-pass filter (0.5–2.5 Hz) of the second order was applied to PPG signals, which were then re-sampled to a sampling rate of 30, to match it with the FPS of RGB and thermal video frames. The band-pass filter was further applied after re-sampling the signal to reduce the sampling artifacts. The cut-off for higher frequency was chosen as 2.5 Hz to preserve only the pulsating waveform with systolic peaks, while discarding the features related to the dicrotic notch and diastolic peak as rPPG signals may not contain these characteristic features.

While PPG signals acquired from the ear tend to have good signal quality [20,51], it is still prone to noise artifacts due to head movement. Figure 2D shows a comparison of head movement across different experimental conditions, computed as the inter-frame rotation of facial frames. It is not only important to assess the quality of ground-truth PPG signals, but also to measure skin perfusion for the validity of the PPG signals. Skin-perfusion values were computed as a ratio of pulsatile amplitude (AC) to non-pulsatile (DC) amplitude from raw PPG signals [52]. For signal quality, the ground-truth PPG signals of our iBVP dataset were first assessed manually through visual inspection by one author and the same were meticulously reviewed by another author. Manually marked signal-quality labels were added to the dataset. To prevent subjective bias in marking the signal quality of the ground-truth PPG signals, we investigated automated signal-quality assessment methods.

The conventional signal-quality assessment methods for PPG signals rely on extracting (i) frequency components to compute the signal-to-noise ratio [53], (ii) different measures of signal-quality indices (SQIs) [54] including relative power SQI [47], or (iii) analyzing morphological features and comparing with the template signal [55]. In several real-world settings, frequency-based SQI measures of signal quality can be misleading due to overlapping frequency components of noise artifacts [56]. Morphological-feature-based signal-quality assessment is challenging owing to several factors [55,57] that include the following: (i) it is required to accurately segment the pulses to match the template, (ii) as morphological features vary significantly between different individuals, a generalized pulse template cannot be used as a reference to match, and (iii) some noise artifacts resemble the pulse morphology, making it more challenging to discriminate between a good-quality PPG signal and noise artifacts.

Machine learning- and deep learning-based methods for PPG-signal-quality assessment have recently attracted wider attention among researchers [22,23,57,58,59]. These developments have been captured in a recent survey [60] that reviews signal-quality assessment methods for contact-based as well as imaging-based PPG. While the majority of works focus on developing a classifier model [58,61,62,63] with binary inference for quality of the length of a PPG signal, a few recent works have proposed models that offer high-temporal-resolution-signal-quality assessment [20,22,23,64]. In this work, we first extend SQA-Phys, the signal-quality assessment method deployed in [20], such that the inference for signal quality is made per sample, resulting in a 1D dense segmentation task, and secondly we replace the CNN-based decoder of the encoder–decoder architecture with a decoder based on matrix decomposition [21].

A recent work on 2D semantic segmentation [21] shows the efficacy of a matrix-decomposition-based decoder in capturing the global context. Inspired from the approach of low-rank discovery through matrix decomposition [21], this work adapts the hamburger module by implementing the Non-negative Matrix Factorization (NMF) for 1D features. Combining the 1D-CNN encoder module of SQA-Phys [20] and a matrix-decomposition-based decoder [21], we refer to this new architecture as SQA-PhysMD, as depicted in Figure 3. Datasets used for the training of the SQA-PhysMD model include the PPG DaLiA [65] training set, WESAD dataset [66], and TROIKA [17] dataset. The signal-quality labels for this training dataset are provided by the authors of [23] and are available as a part of their repository [67]. Training parameters and the model validation were conducted in line with the recent state-of-the-art works [22,23].

SQA-PhysMD outperformed the SOTA models on the bench-marking PPG DaLiA testing set as well as showed very promising generalization on the PPG signals of the iBVP dataset, which can be observed in Appendix B. Signal-quality labels generated using the SQA-PhysMD model were therefore additionally added to the dataset to enable researchers to discard the noisy PPG signal segments along with their corresponding video frames. SQA-PhysMD can be further used with any existing rPPG dataset to clean the ground-truth signal and thereby eliminate the corresponding video frames.

As most of the rPPG methods deploy face detection as an initial step of the processing pipeline, we pre-process the dataset by first cropped facial regions. We use Python Facial Expression Analysis Toolbox (Py-Feat) [68] along with RetinaFace [69] to detect the facial frame in the RGB images. We then pick a cropping pixel dimension as 256 × 256 as it could contain the largest detected facial frame dimension with margins. This cropping reduces the image dimensions without incurring loss of information in temporal or spatial dimensions. Inspired by a recent work that explored different resolution of images for rPPG estimation using 3D CNN networks (RTrPPG) [38], we used 64 × 64 resolution for training and evaluation purposes. As thermal video frames were well acquired in alignment with the RGB video frames, the same cropping was used to pre-process the thermal video frames.

2.6. Comparison with Existing Datasets

Table 1. Comparison of different rPPG datasets.

Dataset	Modality	Subjects	Tasks	Duration (min)	Varying Illumination	SQ Labels	Resolution	Compression	FPS	Free Access
PURE [8]	RGB	10	S, M, T	60	Y	N	640 × 480	None	30	Yes
OBF * [70]	RGB, NIR	106	M	1000	N	N	640 × 480	None	30	No
MANHOB-HCI [7]	RGB	27	E	350	N	N	1040 × 1392	None	24	Yes
MMSE-HR [9]	RGB, 3D Thermal	40	E	935	N	N	RGB: 1040 × 1392; Thermal: 640 × 480	None	25	No
VIPL-HR [10]	RGB, NIR	107	S, M, T	1235	Y	N	Face-cropped	MJPG	25	Yes
UBFC-rPPG [11]	RGB	43	S, C	86	Y	N	640 × 480	None	30	Yes
UBFC-Phys [12]	RGB	56	S, C, T	504	N	N	1024 × 1024	JPEG	35	Yes
iBVP (Ours)	RGB, Thermal	32	B, C, M	381	N	Y	RGB: 640 × 480; Thermal: 640 × 512	None	30	Yes

B: Rhythmic breathing; E: Facial expression; M: Head movement; S: Stable; T: Talking; C: Controlled; SQ: Signal quality. * OBF dataset is temporarily unavailable at the time of submission.

presents a comparison of different rPPG datasets (non-exhaustive), highlighting various aspects of each dataset. The advancements in rPPG research have been made owing to the availability of these datasets. A higher number of participants and varying scenarios including illumination conditions and tasks performed by the participants offer several advantages including the reliable validation of rPPG methods, as well as robust training of supervised rPPG algorithms. The key highlight of the iBVP dataset is its labels that are assessed for the signal quality, making it a highly reliable bench-marking dataset as well as a good candidate to train supervised models. Additionally, most of the existing datasets have captured ground-truth PPG signals from the finger or wrist, introducing not just a phase difference but also morphological differences [50] with the rPPG signal that is extracted from facial regions. While the phase difference can be easily adjusted, when combined with morphological differences, it can not be optimally synchronized with the facial rPPG signals. The iBVP dataset therefore is more suitable for evaluating as well as training the models that estimate PPG signals in contrast to the models trained to estimate heart rate or related metrics with an end-to-end approach. For an exhaustive discourse and description of different rPPG datasets, it it recommended to refer to a recent review article [6].

3. Validation of iBVP Dataset

To evaluate the iBVP dataset, we chose the models that can be trained to infer BVP signals in an end-to-end manner. Among such models, 3D-CNN-architecture-based models including PhysNet3D [35] and RTrPPG [38] were found to be the most suitable to learn spatio-temporal features from facial video frames. The evaluation of the rPPG models trained with the iBVP dataset is performed with an objective to validate the iBVP dataset, supporting the use of the dataset as a bench-marking as well as training dataset.

We further introduce a novel 3D-CNN framework, iBVPNet, as illustrated in Figure 4. iBVPNet is a fully convolutional architecture designed with an objective of effectively learning spatio-temporal features for BVP estimation. It consists of three blocks, with each block distinctly learning spatial and temporal features. The first block aggregates spatial features, while encoding temporal features. The second block deploys large temporal kernels to encode the long-range temporal information. The final block further aggregates the spatial dimension while decoding the temporal features. Below, we describe the experiments and present the preliminary results, highlighting the efficacy of the proposed iBVPNet model.

3.1. Experiments

PhysNet3D [35], RTrPPG [38], and iBVPNet are trained using the iBVP dataset with a subject-wise 10-fold cross-validation approach. Models are separately trained and evaluated for RGB and thermal video frames. In each fold, data of 3 out of 30 participants are left out for validation, and the models are trained with the remaining data of 27 participants. A 20 s video segment and the corresponding ground-truth PPG signal are used for the training. A total of 600 face-cropped video frames are stacked and provided as input to the models, while the ground-truth PPG signals are resampled to 30 samples per second to match the count of video frames.

A batch size of 8 is used across all the experiments, and the learning rate is initialized to 1 × 10⁻⁴, with a step-size of 2 iterations and gamma of 0.95. Models are trained for 100 iterations in each fold, with the cosine similarity (CS) loss function. Empirically, CS loss was found to achieve stable convergence in comparison with the negative Pearson correlation, which has been used in earlier works [35,38]. For augmenting the RGB video frames, video AugMix [71,72] was used to apply transforms that include changes related to contrast, equalization, rotation, shear, translation, and brightness. Thermal video frames are augmented using only rotation, shear, and translation transforms.

3.2. Evaluation Metrics

rPPG methods are commonly evaluated for HR measurement [6], whereas the methods aimed at estimating the BVP signals use the metrics that measure similarity between two time series signals. To evaluate the accuracy of HR measurement, widely used metrics include the Mean Absolute Error (MAE), Root Mean Square Error, and Pearson correlation coefficient [6]. Among the rPPG methods focused on BVP estimation, predominantly used metrics include the Template Match Correlation (TMC) [73] and signal-to-noise ratio (SNR) [35,38]. The performance of TMC can be affected by the accuracy of segmenting an individual pulse waveform from the PPG signals [54,73]. In this work, we propose using the metrics that align the two time-series signals without requiring to segment the waveform based on morphological features. Specifically, we compute cross-correlation between the ground-truth PPG signal and the estimated BVP signal at multiple time-lags, with an assumption that maximum amplitude of cross-correlation (MACC) [44] is achieved at the optimal alignment between the two signals. MACC computed for an individual pair of estimated and ground-truth PPG signals is further averaged across the testing dataset. We present the evaluation results using MACC and SNR metrics to assess the quality of estimated BVP signals. In addition, we also compute metrics based on HR measurement including root mean squared error (RMSE) and Pearson correlation between the HR values computed from ground-truth and estimated BVP signals.

3.3. Results

Evaluation metrics are first averaged for each fold out of 10-fold cross-validation and then further averaged across all the folds.

Table 2. Performance evaluation for rPPG estimation with RGB frames of iBVP dataset.

3D CNN Models	MACC (Avg)	SNR (Avg)	RMSE (HR)	Corr (HR)
PhysNet3D [35]	0.781	0.511	4.283	0.848
RTrPPG [38]	0.702	0.283	6.901	0.704
iBVPNet (ours)	0.784	0.677	2.717	0.813

compares the averaged metrics for different end-to-end rPPG models trained with RGB video frames. For MACC, SNR, and RMSE (HR), the proposed iBVPNet shows superior performance, while the PhysNet3D shows the highest correlation for HR. Detailed fold-wise results for models trained with RGB video frames are presented in

Table A1. Detailed performance evaluation for rPPG estimation with RGB video frames of iBVP dataset.

Folds		MACC (Avg)				SNR (Avg)				RMSE (HR)				Corr (HR)
		PhysNet3D	RTrPPG	iBVPNet (Ours)		PhysNet3D	RTrPPG	iBVPNet (Ours)		PhysNet3D	RTrPPG	iBVPNet (Ours)		PhysNet3D	RTrPPG	iBVPNet (Ours)
0		0.767	0.669	0.790		0.532	0.250	0.762		2.829	6.058	1.476		0.846	0.568	0.860
1		0.734	0.654	0.710		0.373	0.190	0.423		8.412	12.480	5.325		0.538	0.258	0.376
2		0.830	0.773	0.860		0.709	0.475	0.972		2.937	6.213	1.412		0.888	0.587	0.934
3		0.718	0.637	0.660		0.305	0.113	0.291		5.848	7.591	4.542		0.800	0.674	0.679
4		0.851	0.763	0.836		0.637	0.402	0.740		2.330	3.993	1.681		0.955	0.879	0.945
5		0.867	0.801	0.853		0.601	0.373	0.808		2.092	3.508	1.113		0.966	0.905	0.973
6		0.780	0.689	0.824		0.573	0.297	0.825		5.114	7.682	2.342		0.898	0.826	0.945
7		0.821	0.751	0.821		0.603	0.342	0.806		2.943	5.051	2.652		0.903	0.781	0.830
8		0.702	0.603	0.744		0.329	0.113	0.604		4.103	11.395	2.692		0.772	0.655	0.724
9		0.743	0.680	0.746		0.445	0.271	0.535		6.222	5.044	3.932		0.909	0.911	0.870

.

The 33% increase in the SNR for the BVP signals estimated with trained iBVPNet models compared with the existing SOTA method is noteworthy. In Figure 5, we present estimated BVP waveforms for rhythmic breathing and head movement conditions to qualitatively compare the outcomes from our proposed models as well as the SOTA methods. It can be observed that head movement affects the quality of estimated rPPG signals both during rhythmic breathing as well as head movement conditions. Figure 5 further highlights that while the performance of RTrPPG [38] is not reliable, PhysNet3D [35] and the proposed iBVPNet show comparable estimated signals.

In

Table 3. Performance evaluation of SOTA models on iBVP dataset.

Unsupervised Models	MAE (HR)	RMSE (HR)	Corr (HR)	SNR (BVP)	MACC (BVP)		Supervised Models (Trained on PURE [8])	MAE (HR)	RMSE (HR)	Corr (HR)	SNR (BVP)	MACC (BVP)
ICA [24]	10.25	14.11	0.10	−9.10	0.25		DeepPhys [29]	7.93	11.96	0.40	−8.04	0.34
GREEN [2]	11.84	15.85	0.15	−10.18	0.22		TS-CAN [30]	6.68	10.27	0.36	−7.70	0.36
CHROM [25]	4.18	8.33	0.56	−4.94	0.46		PhysNet3D [35]	4.85	8.69	0.56	−4.59	0.44
POS [27]	3.51	6.98	0.71	−4.57	0.47		PhysFormer [39]	8.05	13.07	−0.02	−8.32	0.34
LGI [28]	6.08	11.23	0.41	−6.38	0.39		EfficientPhys [41]	5.06	9.23	0.65	−6.15	0.44
PBV [26]	9.94	14.47	0.23	−8.74	0.27		IBVPNet (ours)	3.60	6.94	0.71	−3.35	0.50

, we further compare the performance of conventional and some of the recent SOTA models on the iBVP dataset. Pre-trained weights of the supervised models trained on the PURE dataset [8], as provided by rPPG-Toolbox [74], were used to run the inferences. The iBVPNet model was trained on the PURE dataset to compare its performance with the SOTA models on the iBVP dataset. It is noteworthy to observe that the unsupervised method POS [27] achieves state-of-the-art performance on the iBVP dataset, specifically when evaluated with the MAE (HR) metric. The iBVPNet model proposed in this work demonstrates the highest performance on most of the metrics including RMSE (HR), Corr (HR), SNR, and MACC. The MACC metric [44] suggested in this work for evaluating rPPG methods shows good agreement with the existing metrics.

To compare the intra-dataset performance of five SOTA models on the existing datasets, we compiled a

Table A3. Performance comparison of SOTA rPPG methods on existing bench-marking datasets and iBVP Dataset.

Datasets	rPPG Method	RMSE (HR)	Corr (HR)
PURE [8]	PhysNet3D [35]	2.60	0.99
	rPPGNet [76]	1.21	1.00
	SAM-rPPGNet [37]	1.21	1.00
MANHOB-HCI [7]	PhysNet3D [35]	8.76	0.69
	rPPGNet [76]	5.93	0.88
VIPL-HR [10]	PhysNet3D [35]	14.80	0.20
	AutoHR [78]	8.68	0.72
iBVP Dataset (ours)	PhysNet3D [35]	4.28	0.85
	RTrPPG [38]	6.90	0.70
	iBVPNet (ours)	2.72	0.81

Note: Only 3D CNN methods that estimate BVP signals are chosen. For iBVP dataset, only RGB imaging modality is considered for this comparison, as it is the only common modality among the datasets).

in Appendix B, based on the data from a recent review article [6]. Table A3 also presents a comparison of two SOTA methods together with our proposed method on the introduced iBVP dataset, validating the usefulness of iBVP dataset in extracting BVP signals and HR.

Lastly, we performed the evaluation on infrared thermal image frames, which the iBVP dataset offers with higher temporal resolution (30 FPS) than the existing datasets. It can be noted from Table 1 that MMSE-HR is the only other dataset that offers thermal infrared image frames for an rPPG estimation task; however, the frame rate for the same is 25 FPS.

Table 4. Performance evaluation for rPPG estimation using thermal frames of iBVP Dataset.

3D CNN Models	MACC (Avg)	SNR (Avg)	RMSE (HR)	Corr (HR)
PhysNet3D [35]	0.360	−0.135	6.339	−0.019
iBVPNet (ours)	0.413	0.159	5.400	0.095

compares the metrics averaged across multiple folds, for different end-to-end rPPG models trained with thermal video frames. Although the iBVPNet showed superior performance across all evaluation metrics as compared with the SOTAs, the overall quality of BVP estimation was not found satisfactory. Detailed fold-wise results for models trained with thermal video frames are presented in

Table A2. Detailed performance evaluation for rPPG estimation with thermal video frames of iBVP dataset.

Folds		MACC (Avg)			SNR (Avg)			RMSE (HR)			Corr (HR)
		PhysNet3D	iBVPNet (Ours)		PhysNet3D	iBVPNet (Ours)		PhysNet3D	iBVPNet (Ours)		PhysNet3D	iBVPNet (Ours)
0		0.377	0.469		−0.099	0.363		6.496	3.144		0.092	0.136
1		0.352	0.403		−0.110	0.109		6.932	5.557		0.286	0.065
2		0.389	0.437		−0.071	0.266		5.599	4.731		−0.139	−0.218
3		0.378	0.409		−0.151	0.171		5.856	5.037		0.093	0.589
4		0.367	0.401		−0.120	0.138		5.475	5.401		0.065	−0.060
5		0.368	0.442		−0.149	0.232		5.628	4.856		−0.141	−0.046
6		0.350	0.430		−0.114	0.213		6.815	5.865		0.014	0.365
7		0.338	0.386		−0.150	0.113		5.453	6.015		−0.238	−0.247
8		0.358	0.431		−0.144	0.264		6.409	4.245		−0.063	0.238
9		0.326	0.322		−0.238	−0.279		8.732	9.152		−0.162	0.129

. Figure 6 qualitatively compares the outcomes of different rPPG methods in estimating BVP waveform from thermal video frames for rhythmic breathing and head movement conditions. This highlights that the BVP information extracted from the thermal frames was not strong. Similar results have been reported in [10] from NIR-based BVP estimation. However, a very recent work, [42], has shown highly promising results with the combination of RGB and NIR, indicating the further need for investigating the combination of RGB-thermal modalities for improving the performance on rPPG estimation.

4. Discussion and Conclusions

The experiments with the SOTA methods and the proposed iBVPNet model highlight the usefulness of the introduced iBVP dataset in training and validating the rPPG methods. While most of the rPPG methods estimate HR, it is advantageous to estimate the PPG signal, which can then reliably be used to extract various HR- and HRV-related metrics. The presence of noise artifacts in the ground-truth PPG signals can obscure the training stage for end-to-end supervised models. Here, the SQA-PhysMD implemented in this work for inferring a dense-signal-quality measure for PPG signals (an extended version of SQA-Phys [20]) has played a key role in eliminating noisy segments not only from the ground-truth PPG signals but also from the corresponding video frames.

As the SQA-PhysMD can assess signal quality for any type of PPG signals, it can be independently applied to the existing datasets for producing high-resolution-signal-quality labels as in the iBVP dataset. This can help with automatically removing noisy segments in the existing rPPG datasets, reducing tedious manual work and efforts, which are otherwise required to be made by researchers [38,75]. Furthermore, the ground-truth PPG signals acquired from the ear lobe closely match the phase and the morphology of the rPPG signals extracted from the facial video frames. Therefore, the iBVP dataset can significantly contribute toward improving robustness of rPPG methods. It is, however, to be noted that the dataset presents limited diversity with respect to skin color and varying lighting conditions. Further, the dataset does not present well-balanced gender distribution, which may slightly bias the training of the supervised rPPG methods. Future work to extend this dataset shall take these limitations into consideration.

Some of the existing RGB imaging-based rPPG datasets are available after applying the video compression techniques (e.g., motion JPEG, JPEG). It is noteworthy that the performance of SOTA models can be severely affected owing to the loss of BVP information from the compressed videos [35,37]. To circumvent this, one recent work implemented a generative method to reconstruct the original video frames from the compressed video frames as an initial step, followed by an architecture to estimate BVP signals [76]. However, this approach adds significant overhead in processing the video frames, and therefore alternative ways are required to address the BVP extraction from the compressed videos. Thus, the iBVP dataset offers raw RGB-thermal image frames, without the compression methods.

In agreement with a previous finding on rPPG with infrared (IR) video frames [10], this work reports poor performance of the existing SOTA rPPG methods as well as our proposed iBVPNet model on thermal video frames. It is worth noting that thermal video frames require tailored pre-processing since various factors including ambient temperature and quantization methods [44] can significantly impact the rPPG extraction. Alternatively, raw thermal frames augmented with a thermal augmentation module such as that proposed in a recent work on the segmentation of thermal facial frames [77] can be incorporated while training the models for rPPG estimation. In addition, approaches combining the RGB and thermal modalities can be investigated to compare the model performance with that trained individually on RGB and thermal modalities. Further investigations on assessing the potential of thermal infrared imaging in extracting BVP signals are therefore required, to which our dataset can contribute in future work.

Lastly, we highlight the use of MACC as an effective metric, which can be used along with the widely used metrics to report rPPG methods. The unique benefit MACC offers is that it is a more direct metric to compare the waveforms, and does not become impacted by the method used to compute the heart rate (e.g., based on FFT or counting of peaks in the waveform).