Deep Transfer Learning Approach in Smartwatch-Based Fall Detection Systems ^†

Abstract

This study introduces a fall detection system utilizing an affordable consumer smartwatch and smartphone with edge computing capabilities for implementing AI algorithms. Due to the widespread use of these devices, the system as a whole is extremely accepted, easy to use, requires no tuning of any kind, and guarantees extended functioning for a long period. From a technical standpoint, falls are identified using AI techniques to analyze 3D raw data acquired by the smartwatch’s built-in accelerometer. However, existing AI models for fall detection are often trained on simulated falls involving young people, which may not accurately represent the falls of elderly in unhealthy conditions, such as arthritis or Parkinson’s disease, leading to limitations in detecting falls in this population. Additionally, variations in hardware features among different smartwatches can result in inconsistencies in accelerometer data measurements across X, Y, and Z orientations, further complicating accurate fall detection. To address the challenge of limited and device-specific datasets and to enhance model generalization across various devices, a Deep Transfer Learning approach is proposed. This method proves effective when data are poor. Specifically, the Continuous Wavelet Transform (CWT) is applied to raw accelerometer signals to convert them into 2D images, enabling the use of deep architectures for Transfer Learning. By employing CWT on 5 s time windowed raw accelerometer signals, heat maps (scalograms) are generated. Real-time accelerations sampled at 50 Hz are collected using a smartwatch application, transmitted via Bluetooth to a smartphone app, and converted into scalograms. These serve as input for pre-trained Deep Learning models to estimate fall probabilities. Preliminary tests on the Wrist Early Daily Activity and Fall Dataset (WEDA-FALL) show promising results with an accuracy of approximately 98%, underscoring the efficacy of utilizing wrist-worn wearable devices for processing raw accelerometer data.

1. Introduction

According to the latest report of the World Health Organization (WHO) [1], an average of 37.5 million falls require medical assistance each year. In particular, about 30% of the population over-65s falls each year, reaching 40% in the over-70s. Falling is the second most frequent reason for accidental death worldwide, and adults over 60 years of age are the most affected [1,2]. So, for this kind of user, a fall is a critical and unsafe situation requiring rapid assistance to reduce the consequences from health, social, welfare, and economic points of view. It has been shown [3,4] that elderly people having previous falls develop a deep-seated anxiety of falling, resulting in a decrease in their level of mobility and quality of life itself.

In the last 15 years, fall detection systems received significant focus from the scientific community, and several systems [5,6,7,8] have been implemented (often integrating simple threshold-based or rules-based approaches [9,10] with different detection rates and levels of acceptability/usability). Presently, two groups of fall detection sensors have been identified in the scientific community. A first group of fall detection sensors is based on ambient devices, such as floor pressure sensors [11,12], acoustic sensors [13], video cameras [14,15,16], etc. Such devices are expensive; sensitive to surrounding conditions such as, for example, uniform lighting and the presence of furniture in the environment, which can lead to false alarms and decrease their accuracy [17]; and are dependent on number and positioning [7]. Furthermore, they may be subject to privacy issues inhibiting their use in the home (especially in bedrooms and bathrooms, where many falls typically occur). The second group involves wearable sensor-based methods used to evaluate speed and unexpected position changes. Usually, accelerometers, gyroscopes, or magnetometers are employed for fall detection [18,19,20], also using commercially available devices [21,22,23]. In contrast to ambient sensors, wearable sensors have clear benefits such as user-friendliness, size, and low cost, due to the presence of these sensors in devices already available to end-users (e.g., smartphones and wristbands). Several commercial smartwatches have technology to support elderly with health problems, detecting the main activities of daily living (ADLs) such as walking, sitting and lying down.

2. Related Works

Fall detection via smartwatches is becoming increasingly popular in recent years, as evidenced by the number of recent scientific publications using different algorithimic approaches. In [24], a fall detection solution is proposed, consisting of a smartwatch paired with a smartphone, a cloud-based storage system, and data analysis packages using a Machine Learning (ML) approach training a Support Vector Machine (SVM) algorithm using the following features: the resulting minimum and maximum acceleration in a sliding window of 750 ms; the Euclidean norm of the difference between the maximum and minimum acceleration in the same sliding window; and the acceleration vector length at the time of sampling. An average accuracy of 93.8% was obtained with this model. The main limitation of this study is that only data from young and healthy individuals were used for training. In [25], a Deep Learning (DL) strategy to effectively detect four different types of fall and nine common ADLs was investigated. The performance of four different DL models and six ML models in classifying different human activities other than falls is compared, and Gated Recurrent Units (GRU) model was selected as the best model, with an average accuracy of around 90.5% on a dataset proposed in the same paper which, however, contains data on falls performed by seven young subjects aged between 21 and 55, which presents generalization limitations. A DL approach for fall detection is also introduced in [26]. First, a data augmentation methodology is used due to data imbalance. Subsequently, a BiLSTM network is trained using, as input, (1) only the acceleration values, (2) only the gyroscope values, and (3) their combination. The paper reports both the obtained accuracy in classifying specific ADLs (sitting, squatting, walking, running) and the obtained accuracy to differentiate the fall from the activities. The achieved results show that the fall can only be classified using the measured acceleration from the smartwatch. However, also in this work, data were acquired with end-users aged between 19 and 25 years. A very interesting approach is presented in [27] using a combination of smartwatch and smartphone. In particular, an Android application called “SmartFall” is implemented that acquires accelerometer data from a smartwatch, and processes it using a DL architecture. Two standard ML algorithms, SVM and Naive Bayes (NB), are compared with a GRU model. Training was performed using data from three different benchmark datasets containing more general training data compared to previously work in the literature. The highest average accuracy obtained was 79%. Additionally, the related studies were summarized according to the used methodology, fall classification accuracy, and limitations, as shown in

Table 1. Summary of related studies.

Paper	Method	Accuracy (%)	Limitations
[24]	SVM	93.8	Only data from young and healthy individuals
[25]	GRU	90.5	Data from seven young subjects
[26]	BiLSTM	97.35	End-users aged between 19 and 25 years
[27]	GRU	79	Not high accuracy, but more general training data

.

As highlighted, most of the proposed work has limitations in data type used to train learning patterns, since young subjects are normally used to replicate ADLs and falls. However, it is known that the movements of the elderly can be affected due to motor pathologies that increase violent movements in the smartwatch, which can lead to the generation of patterns in the accelerometer data uncorrelated with falling. To overcome the problem of the lack of datasets related to falls by elderly subjects, the proposed work introduces the application of a Transfer Learning (TL) strategy applied to one-dimensional signals (e.g., raw accelerometers on three axes) and based on the use of the Continuous Wavelet Transform (CWT). Using the CWT, the classification problem of 1D signals is reduced to the image classification (the typical input of a deep architecture used for TL), representing the same input signal in two dimensions. For this purpose, in order to more accurately classify the fall event, three different pre-trained architectures among the most widely used in the literature are compared: DenseNet201, VGG16, and ResNet50.

The remainder of this paper is organized as follows. Section 3 reports some details on the implemented algorithmic pipeline. The preliminary results are included in Section 4, while the conclusions are reported in Section 5.

3. Materials and Methods

In this paper, an algorithmic pipeline for real-time fall detection using the raw accelerometer data returned by any consumer smartwatch is introduced. The architecture of the proposed system is shown in Figure 1.

First, an Android application was developed and installable on any commercial smartphone to collect estimated acceleration values from the smartwatch in real time, along all three axes. The app acts as a server, receiving data from an app installed on the smartwatch reading the raw accelerometer data sampled at 50 Hz and encapsulating it in a JSON message in 5 s packets. The collected data from the smartwatch are sent to the cloud, for use by any processing unit on which TL’s fall detection strategy is implemented. More details can be found in the following subsections. In IoT applications, data privacy is a major issue in healthcare monitoring systems. In the developed application, the collected data are all de-identified and indexed with a user id, whose association to the end-user is only known to the associated caregiver.

As described above, the goal is to use this approach on any commercial smartwatch that can be paired with Android-based smartphones. However, at this prototyping stage, the Fitbit Sense [28] was chosen as the wristwatch device due to the variety of supported sensors and its low cost. In fact, this smartwatch is capable of detecting heart rate, electrodermal activity, skin temperature, sleep monitoring, respiratory rate, SpO2, GPS position and, of course, it has an accelerometer and a gyroscope, from which it is possible to acquire a different sampling rate. The Samsung Galaxy A32 5G smartphone [29] was selected to execute our fall detection app, and to receive sensor data from the smartwatch via a Bluetooth communication protocol. It has a 2GHz octa-core processor and 4 gigabytes of RAM that allows for real-time data reception and processing over a long period.

3.1. Accelerometer Signals Processing Technique

As described in Section 1, DL strategies have been increasingly used for fall detection algorithms in recent years, although classical approaches, such as SVM and Decision Trees (DT), are still more popular due to the limited size of fall detection datasets. However, artificial neural networks avoid the feature extraction and selection phase, which can be a laborious task. Furthermore, the problem of small data size can be solved using TL techniques [30], involving the use of pre-trained Convolutional Neural Networks (CNN).

Since such networks require images as input, it is necessary to transform 1-D accelerometer signals into 2-D images. To achieve this, various approaches exist. In this work, CWT was applied, which allows us to obtain a time-frequency representation (a scalogram) of the signal, and to simultaneously capture its peculiarities in different frequency bandwidths, which is useful to process non-stationary signals.

For a time series $x\left(t\right)$, CWT is expressed as follows:

$$X_{\omega }(a,b)={\displaystyle \frac{1}{\sqrt{a}}}{\int }_{-\infty }^{+\infty }x\left(t\right){\phi }^{*}\left({\displaystyle \frac{t-b}{a}}\right)dt$$

(1)

where a and b are the scaling and translational value, respectively ($a>0$, a and $b\in \mathbb{R}$), while ${\phi }^{*}\left(t\right)$ is the complex conjugate of the analyzed mother wavelet and, finally, $1/\sqrt{a}$ is a normalization factor [31]. Applying CWT to time windows of accelerometer signals, a heat map known in the scientific literature as a scalogram can be obtained. A scalogram represents the absolute value of the CWT coefficients of a signal, with the b and a values along the x- and y-axis, respectively, and the intensity of each point measured by Equation (1). In the proposed approach, the scalograms were extracted from the time series of the three raw accelerometer signals split into 5 s time windows. Normally, a smaller value of the scaling quantity enables better evaluation of abrupt changes, while a higher value provides more information that can lead to better classification accuracy. The most commonly used scaling values in this context are 32, 64, 128, and 256. In the present work, the scaling value was set to 256, while Morlet was used as a wavelet. The resulting scalograms are then resized to 224 × 224 to fit with the input layer of the selected deep architectures shown in the next section.

Figure 2 shows an example of a scalogram obtained for walking (a) and fall (b) with the corresponding raw accelerometer signals, respectively. For the sake of brevity, only x-axis of the accelerometer signal is reported.

3.2. Model Selection for Transfer Learning

CNN architectures suffer from a major drawback, namely the need for a large amount of data for training purposes. This problem is more evident when decisions must be performed on rare data, as for fall detection. To overcome this problem, a TL strategy is applied in our proposed fall detection system. The main idea of TL is to extract knowledge from a dataset (i.e., “source domain”) and then transfer it to a new dataset (i.e., “target domain”) to improve learning [32]. In the fall detection context via accelerometer data, the principal reason to use TL is the lack of much data in the application domain, especially falls of elderly subjects. Furthermore, the annotation of these data are expensive and sometimes subjective, especially for areas requiring experts for data labeling and with a high inter-/intra-class variability.

To select a CNN model, three popular architectures were tested, namely DenseNet201, VGG16, and ResNet50 which, in the literature, have achieved important results in the classification of scalogram images. All of these models were trained on all generated scalograms. Each input sample consisted of 3 scalograms (one scalogram for each signal channel).

Each network was modified and tuned for the fall detection problem. Each of the original networks was trained to classify images for 1000 classes on the ImageNet dataset, but in this work, there is a binary classification problem (Fall/No Fall). Therefore, the last fully connected layer was replaced with a new classification layer structured through six new blocks (flatten:1 and dense:5) producing two outputs (fall and no fall). In

Table 2. Hyperparameters for the proposed TL models.

Hyperparameter	Model Architecture
	DenseNet201	VGG16	ResNet50
Learning rate	0.002	0.001	0.002
Batch size	128	128	128
Optimizer	Adam	Adam	Adam
Output activation layer	softmax	softmax	softmax
Number of epochs	50	70	50

, optimized hyperparameters for the three trained models are shown.

For the sake of brevity, Figure 3 shows the graphical representation of the proposed TL scheme for the DenseNet201 network only.

3.3. Dataset

To validate the proposed architecture, a series of tests were performed using the Wrist Early Daily Activity and Fall Dataset (WEDA) [33] as benchmark dataset. It uses the commercial Fitbit Sense^® smartwatch to provide accelerometric and gyroscopic data on 25 users (14 young and 11 elderly over 80). Data are sampled at different frequencies from 5 to 50 Hz and labeled with 8 types of fall and 11 ADLs, shown in

Table 3. Types of falls and ADLs in the considered dataset.

Types of Falls	Types of ADL
Fall forward while walking caused by a slip	Walking
Lateral fall while walking caused by a slip	Jogging
Fall backward while walking caused by a slip	Walking up and downstairs
Fall forward while walking caused by a trip	Sitting on a chair, wait a moment, and get up
Fall backward when trying to sit down	Sitting a moment, attempt to get up and collapse into a chair
Fall forward while sitting, caused by fainting or falling asleep	Crouching (bending at the knees), tie shoes, and get up
Fall backward while sitting, caused by fainting or falling asleep	Stumble while walking
Lateral fall while sitting, caused by fainting or falling asleep	Gently jump without falling (trying to reach high object)
	Hit table with hand
	Clapping Hands
	Opening and closing door

.

In our tests, accelerometer data sampled at 50 Hz were used, and training experiments were performed on a Dell^™ Precision 7920 Rack workstation, with 256GB RAM, dual Intel Xeon Gold 5218R [email protected] processors, three NVIDIA^™ RTX A2000 12 GB GPUs, using Python version 3.8 and the following libraries: Tensorflow (2.10), pandas (2.0.3), scikit-learn (1.2.1), spkit (0.0.9.6.7).

4. Results and Discussions

Firstly, to achieve the objectives of this study, and to evaluate the goodness of the proposed approach in fall detection, three different models (DenseNet201, VGG16, and ResNet50) were trained. The performance of these models were evaluated using a 10-cross validation [34]. Consequently, each model was trained with 80% of the data, and testing was performed with the remaining 20% and, to avoid over-fitting, 10% of the testing set was used as validation. The entire process was replicated 10 times, using different training and testing sets to prevent the occurrence of the same samples in the training and testing set at the same time. Four well-known metrics were calculated for the numerical evaluation of performance, such as accuracy, precision, recall, and F1 score.

Table 4. Comparison of the performance for each model.

Model	Accuracy	Precision	Recall	F1-Score
DenseNet201	0.9756	0.9743	0.9674	0.9741
VGG16	0.9524	0.9614	0.9520	0.9686
ResNet50	0.9619	0.9713	0.9620	0.9750

shows the obtained results for each considered model, demonstrating the goodness of the proposed approach, with an average accuracy in the range of 95.24% for VGG16 to 97.56% for DenseNet201.

For brevity, Figure 4 shows, respectively, the loss and accuracy during training and validation of our best performing model. In particular, from Figure 4, it is evident how our model improves at varying epochs, fixed at 50 in this work, while Figure 5 reports the confusion matrices of the obtained average accuracies for each considered model.

Furthermore,

Table 5. Comparison of the performance for each model.

Model	Accuracy
LightGBM [36]	0.9530
kNN [35]	0.9805
Our DenseNet201	0.9756

compares the classification performance (in terms of accuracy) obtained with the proposed approach with two other research papers based on the same dataset. As can be seen, the obtained accuracy is similar to [35], with the advantage of a high degree of generalization approach due to the TL usage and, consequently, with a greater adaptability in real-world contexts.

Furthermore, in [35] kNN is used, a supervised learning method requiring a large amount of labeled training data with the difficulty of classifying against new data, resulting in overfitting. Finally, another obvious difference between [35] and our approach is the duration of the sliding window (9 s versus 5 s), a very important parameter given the need to detect a fall in the shortest possible time.

5. Conclusions

In this work, a system for automatic fall detection was designed and implemented, using a commercial smartwatch for data collection and a smartphone as the processing unit. TL was used for the analysis of accelerometric data extracted from the wearable device, due to the lack of large amounts of data in the considered application context. Three different pre-trained models were employed and it was shown that the DenseNet201 model performed best in terms of accuracy in classifying the different types of analyzed falls.

However, the proposed system has limitations compared to commercial solutions already on the market. The most important limitation derives from the use of the cloud, increasing its cost and making it impossible to detect a fall if there is no network connection. A second limitation relates to the impossibility of sending a fall detected in the event of no network connection to the smartphone. A further limitation is the necessary proximity of the smartphone to the smartwatch due to the Bluetooth connection required for the correct transmission of the raw data.

Possible future developments will focus on the use of additional benchmark datasets (as MobiFall Dataset [37], SisFall [38], and tFall [39]) to validate the algorithmic methodology’s performance, try to reduce the cost of the entire solution, and for methodology implementation directly on the smartphone using the network model previously trained on high-performance computing hardware to avoid the use of the cloud, minimizing possible data transmission and privacy issues. In addition, further deep architectures will be compared in addition to the three considered to evaluate whether DenseNet201 continues to be the best choice in terms of overall accuracy. In the end, to improve the quality of the results and to estimate the performance on new subjects, a leave-one-out subject cross-validation will be implemented.