Next Article in Journal
Feature Extraction of the Brain’s Dynamic Complex Network Based on EEG and a Framework for Discrimination of Pediatric Epilepsy
Next Article in Special Issue
Ground Contact Time Estimating Wearable Sensor to Measure Spatio-Temporal Aspects of Gait
Previous Article in Journal
Machine Learning-Based Software Defect Prediction for Mobile Applications: A Systematic Literature Review
Previous Article in Special Issue
Wearable Feet Pressure Sensor for Human Gait and Falling Diagnosis
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The Validity of the Energy Expenditure Criteria Based on Open Source Code through two Inertial Sensors

by
Jaime Martín-Martín
1,2,†,
Li Wang
3,†,
Irene De-Torres
1,4,
Adrian Escriche-Escuder
1,5,
Manuel González-Sánchez
1,5,
Antonio Muro-Culebras
1,5,
Cristina Roldán-Jiménez
1,5,
María Ruiz-Muñoz
1,6,
Fermín Mayoral-Cleries
1,7,
Attila Biró
8,
Wen Tang
9,
Borjanka Nikolova
10,
Alfredo Salvatore
11 and
Antonio I. Cuesta-Vargas
1,5,12,*
1
Biomedical Research Institute of Málaga (IBIMA), 29010 Málaga, Spain
2
Legal and Forensic Medicine Area, Department of Human Anatomy, Legal Medicine and History of Science, Faculty of Medicine, University of Málaga, 29071 Málaga, Spain
3
Faculty of Media and Communication, Bournemouth University, Bournemouth BH12 5BB, UK
4
Physical Medicine and Rehabilitation Unit, Regional Universitary Hospital of Málaga, 29010 Málaga, Spain
5
Department of Physiotherapy, University of Málaga, 29071 Málaga, Spain
6
Department of Nursing and Podiatry, University of Málaga, 29071 Málaga, Spain
7
Mental Health Unit, Regional Universitary Hospital of Málaga, 29010 Málaga, Spain
8
ITWare, 1117 Budapest, Hungary
9
Faculty of Science and Technology, Bournemouth University, Bournemouth BH12 5BB, UK
10
Arthaus, Production Trade and Service Company Arthaus Doo Import-Export Skopje, 1000 Skopje, North Macedonia
11
Sensor ID Snc., 86021 Boiano, Italy
12
School of Clinical Science, Faculty of Health Science, Queensland University Technology, Brisbane 400, Australia
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Sensors 2022, 22(7), 2552; https://doi.org/10.3390/s22072552
Submission received: 17 January 2022 / Revised: 8 March 2022 / Accepted: 24 March 2022 / Published: 26 March 2022
(This article belongs to the Special Issue Wearable Sensors for Gait and Falls Monitoring)

Abstract

:
Through this study, we developed and validated a system for energy expenditure calculation, which only requires low-cost inertial sensors and open source R software. Five healthy subjects ran at ten different speeds while their kinematic variables were recorded on the thigh and wrist. Two ActiGraph wireless inertial sensors and a low-cost Bluetooth-based inertial sensor (Lis2DH12), assembled by SensorID, were used. Ten energy expenditure equations were automatically calculated in a developed open source R software (our own creation). A correlation analysis was used to compare the results of the energy expenditure equations. A high interclass correlation coefficient of estimated energy expenditure on the thigh and wrist was observed with an Actigraph and Sensor ID accelerometer; the corrected Freedson equation showed the highest values, and the Santos-Lozano vector magnitude equation and Sasaki equation demonstrated the lowest one. Energy expenditure was compared between the wrist and thigh and showed low correlation values. Despite the positive results obtained, it was necessary to design specific equations for the estimation of energy expenditure measured with inertial sensors on the thigh. The use of the same formula equation in two different placements did not report a positive interclass correlation coefficient.

1. Introduction

The estimation of energy expenditure (EE) is fundamental to determine the level of physical activity of a person. The current society we live in has eminently sedentary behaviour, which can produce serious problems, and along with other unhealthy lifestyle habits, means a change in lifestyle is necessary [1,2]. Likewise, the use of behaviour modification techniques alone can produce an increase in physical activity [3]. Use of these, in conjunction with physical activity monitors and suggests an increase in physical activity. Despite the possible technical difficulties that may exist during the process, a motivation to be physically active is increased through self-monitoring and goal setting, among others [4]. It is, therefore, necessary to design valid systems for the estimation of EE that are accessible and reliable.
Energy expenditure is the most popular way to assess physical activity carried out by the subject throughout the day, as well as to record the prolonged periods of inactivity (sleeping or seated) [5]. In general, this consumption is recorded in the metabolic equivalents (METs) unit. METs is a direct adaptation of “mass-specific energy costs, computed by taking the energy costs (VO2 mL∙kg−1∙min−1) and dividing them by 3.5 mL∙kg−1∙min−1”, according to the Compendium of Physical Activities [6]. In other words, an indirect calorimetry can be used as an estimation instrument to determine the energy expenditure of different activities. However, the formula mentioned above cannot be applied directly to all subjects because it does not consider individual variables, such as age, weight, height, or gender. Thus, correction factors should be used [7].
Physical activities are categorised based on their intensity and consumption of EE. In this research, the categories are established as followings: ˂1.5 METs light intensity, ˂3 METs moderate intensity, ˂6 METs vigorous intensity, ˃6 METs very vigorous intensity [8]. Nowadays, equipped with low-cost inertial sensors, wearable devices (e.g., activity monitors) have become powerful [9], and they are capable of recording physical activities and estimating them by METs in a real-time manner. However, detailed procedure information is still missing in published studies, for using these low-cost devices. For example, the following details are still unclear to others: accelerometer brand, placement, duration of measurement, data processing, and interpretation [10], which makes it difficult to reproduce similar results. In addition, most of the manufacturers choose to design their own systems, especially for calculating calories consumed or energy expenditure, and there are no fair comparisons between these low-cost devices [11]. In other words, the reliability of wearable devices for measuring fitness-related indicators has been analysed, but the estimation accuracy of energy consumption is still inadequate [12]. Besides, machine learning systems have been used to calculate energy expenditure. However, it is not reproducible in daily life as the high predictive precision of the models is calibrated under strict laboratory conditions [13]. Therefore, it is necessary to apply nonlinear models to obtain a reliable and easier EE estimation for daily practise [14,15].
Previous studies showed the measurement of energy consumption in different devices (Apple Watch 2, Samsung Gear S3, Jawbone Up3, Fitbit Surge, Huawei Talk Band B3, and Xiaomi Mi Band 2) was inadequate, and their mean absolute percentage errors reached 0.44. This shows the low reliability of devices, such as the Fitbit or the Jawbone, to estimate energy expenditure [16]. In addition, measurement accuracies varied between subjects individually, from 0.10 to 0.50, depending on the activity performed [8]. Previous studies have also demonstrated the existence of significant differences in energy expenditure or physical activity when climbing or descending stairs and even walking at low speeds when comparing specific devices designed for this purpose [17]. In this sense, the estimation of energy expenditure with smartwatches or similar devices in comparison with a gold standard (gas analysis data from indirect calorimetry or oxygen volume consumption) have also shown significant differences. These differences are smaller when the outcome variable heart rate is taken as a reference [18]. Likewise, specific device designs based on inertial sensors for energy expenditure estimation compared to gas analysis can obtain better results than those obtained by a conventional smartwatch or an activity-specific smartwatch [19]. Despite the improvement in the results based on specific systems, there are still differences in the estimation of energy expenditure that could be reduced.
Two of the most used inertial sensors for estimating energy expenditure are ActiGraph [20] and Genea [21], which are regarded as the gold standard when compared to new devices appearing in the market [22,23,24]. However, these devices have enclosed software and signatures with multiple modes for energy expenditure calculation, which remains unclear how these devices calculate the calory consumption using inertial sensors [25]. Meanwhile, their high purchasing cost is not affordable for many clinical applications, such as physical activity promotion, breaking sedentary behaviour, and technical use for weight loss [26]. Therefore, this study aims to develop an open source system for the calculation of energy expenditure, which is based on low-cost inertial sensors Lis2DH12 (STMicroelectronics, Gienbra, Suiza) and ActiGraph GT9X Link (ActiGraph LLC, Pensacola, FL, USA).

2. Materials and Methods

2.1. Design

An analytical cross-sectional study using three inertial measurement units (IMU) or sensors used to record physical activities was conducted to design an open source R-code for estimating energy expenditure. Five healthy adults were recruited to perform twice a set of ten physical activities. The informed consent was approved accordingly.
Three IMUs per participant were used for the present study—two Actigraph GT9X Links [27] and one SensorID [28]—following two configurations. First configuration: each participant was equipped with an Actigraph with a band and a SensorID in the wrist with a double tape band and another Actigraph sensor placed on the thigh with a band. Second configuration: each participant was equipped with an Actigraph with a band and a SensorID on the thigh with a double tape band and another Actigraph sensor placed in the wrist with a band.
A set of ten physical activities on a treadmill were conducted in each configuration. The set of activities were walk or running at 10 different speeds: 1.4 km/h, 2.9 km/h, 4.3 km/h, 5 km/h, 6 km/h, 6.5 km/h, 7 km/h, 8 km/h, 9 km/h and 10 km/h, respectively. Participants performed each physical activity on the treadmill for 3 min, and no data was recorded in the first one minute (to ignore some unstable activity pattern); the next two minutes were recorded.

2.2. Devices

The ActiGraph GT9X Link [27] sensor is equipped with its own software and it was designed to estimate EE. It can be programmed to measure each activity, which makes it possible to be automatically activated for recording accelerometer data in a specific period. The sampling frequency of the device is 30–100 Hz with a dynamic range of measure ±8 g, and its data storage is 4 GB [27]. In the present study, it was configured from the manufacturer’s software to extract the RAWdata format from the accelerometer (x-, y-, z-axes). This sensor is widely used for the estimation of physical activities [10].
The IMU sensor of the SensorID uses the Lis2DH12 [29] chipset model and Bluetooth 5.0 for wireless connection, and the accelerometers of activities can only be recorded in a real-time manner. The device has a measuring range of ±16 g and a range of sampling frequency 1–1620 Hz. Additionally, the x-, y-, and z-axes data from the accelerometer were obtained.
The sampling rate of both ActiGraph sensors and the IMU sensor from SensorID was 30Hz in all experiments. Each participant completed the sets of physical activities.

2.3. R-Code Development

An R-Code was developed in order to estimate EE based on x-, y-, z-axes data obtained from the sensors. The average activity count value (counts per minute) of each activity for EE computation is calculated through the sum of activity count values (counts per second) divided by two (each activity lasts two minutes). For Crouter et al.’s equation [30], the average activity count value (counts per 10 s) is calculated by the sum of activity count values (counts per second) divided by 12 (2 min is equal to 12 multiply 10 s). The VM is calculated by the square root of the square of the three axes of data. For Santos-Lozano et al.’s equations [20], we choose the equations designed for all age groups. All the programming codes designed for reading the accelerometer data, EE computation and EE correction are written in R language by two of our researchers (J.M.-M. and L.W.) (available code: https://doi.org/10.6084/m9.figshare.12080730; accessed on 30 April 2021), and a public library called “activityCounts” is used to calculate the activity counts (counts per second) [31].

2.4. Data Analysis

In our experiments, five participants wearing three sensors completed two configurations of a set of ten physical activities, and each activity lasted two minutes. Therefore, we have 300 records of activity data (5 participants × 3 sensors × 2 configuration × 10 activities) and 600 min of data in total. To obtain a more precise estimation of EE, descriptive and anthropometric variables of the sample were obtained (age, body mass, height, and gender). The estimation of EE by equations usually does not concern the individual variations, such as age and body mass, which leads to an inaccurate predicted energy cost of activities within a specific individual [6]. We further corrected the EE values by considering the variation of each participant (Table 1).
After the data collection, a data cleaning procedure proceeded. The incorrect activity data due to initialization errors and device malfunction were removed during this procedure. We removed four invalid monitor data (removed t2221, t221, t2130, and t3127), which finally led to 296 valid recorded activities. To select the appropriate predictions of EE, we reviewed up-to-date EE estimation methods using accelerometer data for adults in order to best match the physical activities conducted in this work. After the literature review, five different equations were chosen for calculating EE (in METs) which were Freedson et al. [32], Crouter et al. [30], Santos-Lozano et al. [20], the equations of the vertical axis (VT), and vector magnitude (VM) and Sasaki et al. [33]. The equations used in the present studies were validated in the right hip [20,30,32,33].

2.5. Statistical Analysis

To assess the reliability of the EE estimation between the two devices, intraclass correlation analysis elements (ICC) were performed: alpha mixed pattern of two factors. The statistical analysis employs the covariances among the items. A high intraclass correlation coefficient (ICC) close to 1 indicates high similarity between values from the same group. ICC is a reliability index that reflects both degrees of correlation and agreement between measurements [34].

3. Results

An interclass correlation analysis was performed to compare the calculated EE, which was based on the accelerometer data obtained by the two inertial sensors: SensorID (Lis2DH12) and ActiGraph. Ten different equations were applied to calculate the EE for different speeds of activities like walking and running. In six out of ten equations in our experiments, no significant differences were observed in the comparisons of EE estimated with the inertial sensors in the same placements (see Table 2 and Table 3).
The average correlation index between the standardised elements of energy consumption on the wrist was 0.946 (Table 2). The speeds obtained for the best interclass correlation index (ICC) on the wrist were 2.9 km/h and 4.3 km/h. The speed of 6 km/h was the one that obtained the lowest standardised interelement correlation index on the wrist. The calculation of the corrected EE of Freedson et al. 1998 [32] and Santos-Lozano et al. VT 2013 [20] obtained the best correlation rates between inertial sensors. The EE equations of Sasaki et al. 2011 [33] and corrected Santos-Lozano et al. VT 2013 [35] obtained the lowest correlation rates in our system, ICC:0.111, and ICC:0.306, respectively (see Table 2).
The ICC for the EE calculated on the thigh is shown in Table 3. The average ICC for the standardised elements of different speeds was 0.935. The speeds that offered the best rates were 2.9 km/h and 4.3 km/h. The corrected EE equations of Crouter et al. 2010 [30] and Freedson et al. 1998 [32] (0.982 and 0.979, respectively) were the best correlation results when placing the inertial sensor (SensorID) on the thigh. The lowest correlation rates among elements were those obtained by the corrected equations of Santos-Lozano et al. VM 2013 [20] (ICC:0.022) and Sasaki et al. 2011 [24] (ICC:0.044). The speed with the lowest correlation index among standardised elements on the thigh position was 7 km/h (see Table 3). Significant differences observed in the EE comparison were the wrist and thigh with the ActiGraph sensor (see Table 4 for details).

4. Discussion

The ICC of six EE equations (ten in total) shows similar results when applied to the accelerometer data from two different inertial sensors. Therefore, both sensors are valid for the EE estimation. The processing and analysis of the data in R software proved to be an automated and efficient calculation method. Based on the results obtained, the reliability of low-cost sensors (Lis2DH12) is well recognised when compared to a high-performance sensor (like ActiGraph).
The results of the interclass correlation interval on the wrist (Table 2) and thigh (Table 3) were consistent at most of the activity speeds, but it was slightly better on the wrist. The estimation of EE at a low-speed (2.3 km/h and 4.3 km/h) had a higher interelement correlation index than higher speeds (9 km/h and 10 km/h). The results suggested that it is not possible to use the same energy expenditure equation for the wrist and thigh (Table 4). To avoid the possible underestimation and reflect reliable energy expenditure calculation, we corrected the EE equations by considering the individual variables, such as gender, body mass index and age, for all participants in our experiments (Corrected METs—Compendium of Physical Activities).
Before interpreting the results of this study, the following aspects should be considered. The results in relation to ICC and the estimation of EE should be taken with caution because of two factors. The estimated results of energy expenditure applying the different equations were not compared with a VO2 indirect calorimetry system due to the lack of necessary equipment; VO2 systems for data collection are expensive and therefore difficult to use in daily life for the calculation of energy expenditure, making it necessary to identify alternative systems [36]. The sample size of this study was five participants, which is small. However, each participant walked on the treadmill twice at all selected speeds. This involves a large amount of data. Hence, the established sample size defines a confidence level of 80% and a margin error of 20%.
In the present study, we used EE equations and their corresponding corrected versions. For example, we named the corrected version of “Freedson et al. 1998” as the “corrected Freedson et al. 1998” [32]. Freedson et al.’s equation obtained the best correlation on the wrist with and without correction among five different EE equations. However, on the thigh, the best equation with and without correction was Crouter et al. 2010 [30]. It is important to consider the usage of differential equations and their reliability in any system designed for estimating energy expenditure [9,10,35]. Moreover, standard EE systems should still be used when the energy cost of physical activities is computed for population sets [6,7].
Previous studies have analysed EE based on equations considering VM, body weight and the heart rate reserve of individuals in two different allocations. No significant differences were observed in the variables weight, dead height or body mass index among the participants in the present study. As shown in Table 1, gender and body mass index are considered in the Crouter et al. [30] and Santos-Lozano et al. [20] equations. In all the corrected equations, age is considered, in addition to the aforementioned variables. An earlier study [25] shows that a significantly improved accuracy of estimated EE is observed when the Freedson et al. equation considers individual variables, such as vector magnitude and body weight. However, we did not notice such a significant difference in ICC in our experiments between the traditional Freedon et al. equation and the corrected equation (adapted to vector magnitude and body weight). In contrast, the results of the Freedson et al. equation from the wrist and thigh showed a strong ICC (wrist: 0.945; thigh: 0.972). Results of greater consistency were obtained when the corrected version of the EE equation was applied (ICC wrist: 0.960; thigh: 0.979). Moreover, the energy expenditure data estimated by an inertial sensor and an EE equation should still be taken with caution, as the predicted EE could be significantly underestimated compared to the actual EE in different population groups. Any underestimation could increase along with the increasing intensity of the activities; multiple-stage analysis regression models revealed that age and weight were related to actual EE in both the older and younger groups [37]. Meanwhile, similar studies have shown that the corrected formula translated to a better energy expenditure prediction by adding a heart rate reserve [38,39].
The heart rate parameters with a wrist accelerometer improved energy expenditure estimation significantly [18,40]. Related studies have compared two inertial sensors on EE estimations using the same equations as in our study [41]. They demonstrated that outpatient activities (walking on 3.2 km/h, 4.2 km/h, 6.0 km/h, and jogging at 8.4 km/h) have the best agreement between the Active Style Pro sensor and ActiGraph, including any four EE equations used in the ActiGraph software. In our study, these four same equations and speeds of similar gait were also used, and we observed positive correlations when measuring with two different inertial sensors (see Table 2 and Table 3). The inertial sensors placed on the hip are more likely to calculate accurate EE when participants are doing physical activities that involve mobility of the lower limbs (e.g., household, stairs, walking, and running) and the upper limbs (e.g., laundry, window washing, dusting, dishes and sweeping) [38,42]. However, a better overall prediction of EE is obtained when the inertial sensors are placed on the wrist in our study [42]. However, Slade et al. have used systems based on two inertial sensors (shank and thigh) for the estimation of energy expenditure, which produces a big improvement. Even using a single sensor on the thigh can significantly reduce error in the estimation of energy expenditure obtained by the smartwatch [19].
Besides, we also observed that the results obtained from the wrist and thigh cannot be compared fairly due to the interclass relation coefficient’s weakness (see Table 4). Similar to this, recent studies have demonstrated the need to design specific energy expenditure equations when measured on the thigh [43]. In this sense, according to the results obtained in the present study, the Lis2DH12 sensor compared with the ActiGraph showed a high ICC in the calculation of energy expenditure used in both the thigh and the wrist in six of the equations used (Table 3 and Table 4). In this regard, it is important to consider the evolution of the inertial sensor itself, which is improving in technology and increasing in recording accuracy. For this reason, it is difficult to compare the use of two inertial sensors for the estimation of energy expenditure if they are not the same and if there is no third reliable measurement for the estimation of energy expenditure.
The proposed EE calculation system is developed in an open source R software, which is under a general public license [44], and this licence guarantees the freedom for all users to share and change it. In addition, the advantages of our system are that we only need a CSV file as input, which records the raw accelerometer data (including the time, accelerometer x-axis, accelerometer y-axis, accelerometer z-axis, and the vector of magnitude), and the calculation is a numerical computation which needs low computing resources. Thus, the system is suitable for low-cost or high-cost inertial sensors, and it can be easily further implemented in smart devices, and is independent of specific software. The users can modify the provided EE equations or add new EE equations freely.
The results obtained in this study demonstrate the feasibility of using different equations to estimate energy expenditure in an open source system. This system could be used by any manufacturer able to implement it in their devices and allow the development of third-party applications based on these systems. Devices, such as the Jawbone Up3 (Jowbone, San Francisco, CA, USA) and Fitbit Surge (Fitbit, San Francisco, CA, USA), showed positive correlations greater than 0.8 when using step counting or accelerometer steps; however, energy expenditure was underestimated compared to indirect and direct calorimetry [16]. Furthermore, there is no doubt that a system based on biomechanical analysis is more reliable for the estimation of energy expenditure, with an error under 13%, which is very far from the 44% obtained by the smartwatch [19]. However, the use of such a system on a daily basis could be a limitation for the end-user to carry multiple recording devices instead of just one. In this way, a machine learning system could also be used to estimate the EE; using nonlinear models based on dynamic recognition allows for a better functioning of these systems [13]. These systems can increase the accuracy of EE recognition by up to 7% [45]. However, it is necessary to first validate the sensor data [46].
Our future research will apply specific energy expenditure equations to the thigh, for example, recent EE equations designed for daily living, sports activities and machine learning systems. This would produce a more precise system of energy expenditure calculation for a greater number of activities. This would be essential to compare the energy expenditure obtained by equations with the VO2 via indirect calorimetry [19,47]. The applications of the proposed method can be related to any EE estimation for physical activities, such as physical activity promotion for walking/running, clinical applications, like sedentary behaviour change for health care, and technical use for losing weight. We noticed that the proposed EE calculation system is capable of running automat, and only requires low-cost inertial sensors and limited computational resources, thus we could develop a completely autonomous software that is easily implanted into smart devices with inertial sensors on the market.
The present study has some limitations. For example, this study cannot determine which one of the two inertial sensors (Lis2DH12 or ActiGraph) or which one of the equations is better for the EE estimation because an indirect calorimeter or doubly labelled water method was not used as a validation site. The present study has a limited sample of participants, which could produce a type 2 error: hypotheses null not rejected. However, ten different speeds have been tested, and two positions and ten different energy expenditure calculation formulas generated an extensive database. The present study did not use the heart rate reserve; however, we used two allocations for the sensors. These placements are more comfortable and easier to record physical activities. In addition, this study shows that low-cost accelerometers can be used to design a new computational EE calculation system, which is written in open source R software for energy expenditure research. Furthermore, this system could replace standard measurements, such as the ActiGraph software.

5. Conclusions

In our study, two inertial sensors (SensorID and ActiGraph) were used to calculate the EE for different speeds of physical activities, such as walking and running, and the way we estimated EE was by using an automated system written in open source R software. The conclusion of this study is that not all formulas for EE estimations provide the same reliability when using different inertial sensors. However, we noticed that similar energy expenditure results were produced by the EE equations of Freedson et al., Crouter et al., Santos-Lozano et al., and their corrected versions, which indicates that they can be used by two different inertial sensors. Based on the results obtained, equations like Freedson et al. and Crouter et al. should not be used on the wrist and thigh at the same time as they show negative correlations in our experiments. The proposed EE calculation system can be easily further improved to be a more precise system for a wide range of applications, such as physical activity promotion and daily behaviour change. Moreover, its nature of open source software, automation and low requirements of computation resources makes it suitable to be implanted into smart devices with inertial sensors.

Author Contributions

Conceptualisation, C.R.-J. and A.I.C.-V.; data curation, L.W., A.M.-C. and M.R.-M.; formal analysis, J.M.-M., L.W., M.G.-S. and A.M.-C.; funding acquisition, F.M.-C. and A.B.; investigation, J.M.-M. and A.E.-E.; methodology, J.M.-M., C.R.-J. and A.I.C.-V.; project administration, F.M.-C., A.B., W.T., B.N., A.S. and A.I.C.-V.; resources, F.M.-C., W.T., A.S. and A.I.C.-V.; writing—original draft, J.M.-M., I.D.-T., M.G.-S., M.R.-M. and A.I.C.-V.; writing—review and editing, I.D.-T., M.G.-S., F.M.-C., A.B., W.T., B.N., A.S. and A.I.C.-V. All authors have read and agreed to the published version of the manuscript.

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 823871 (iGame).

Institutional Review Board Statement

This study was approved by the Malaga provincial Research Ethics Committee of the Andalusian Health Service, with the protocol code: RCT-iGAME.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The datasets for this study can be found in: https://doi.org/10.6084/m9.figshare.12080730 (accessed on 30 April 2021).

Conflicts of Interest

Alfredo Salvatore is the executive director of SensorID and provided the inertial measurement unit equipped with a Lis2DH12 sensor provided by ST Microelectronics and derived by Sensor ID original firmware. The remaining authors have no conflict of interest to declare.

References

  1. Kinoshita, K.; Ozato, N.; Yamaguchi, T.; Sudo, M.; Yamashiro, Y.; Mori, K.; Ishida, M.; Katsuragi, Y.; Sasai, H.; Yasukawa, T.; et al. Association of sedentary behaviour and physical activity with cardiometabolic health in Japanese adults. Sci. Rep. 2022, 12, 2262. [Google Scholar] [CrossRef]
  2. Curran, F.; Blake, C.; Cunningham, C.; Perrotta, C.; van der Ploeg, H.; Matthews, J.; O’Donoghue, G. Efficacy, characteristics, behavioural models and behaviour change strategies, of non-workplace interventions specifically targeting sedentary behaviour; a systematic review and meta-analysis of randomised control trials in healthy ambulatory adults. PLoS ONE 2021, 16, e0256828. [Google Scholar] [CrossRef]
  3. Martín-Martín, J.; Roldán-Jiménez, C.; De-Torres, I.; Muro-Culebras, A.; Escriche-Escuder, A.; Gonzalez-Sanchez, M.; Ruiz-Muñoz, M.; Mayoral-Cleries, F.; Biró, A.; Tang, W.; et al. Behavior Change Techniques and the Effects Associated with Digital Behavior Change Interventions in Sedentary Behavior in the Clinical Population: A Systematic Review. Front. Digit. Health 2021, 3, 620383. [Google Scholar] [CrossRef] [PubMed]
  4. Creaser, A.V.; Clemes, S.A.; Costa, S.; Hall, J.; Ridgers, N.D.; Barber, S.E.; Bingham, D.D. The Acceptability, Feasibility, and Effectiveness of Wearable Activity Trackers for Increasing Physical Activity in Children and Adolescents: A Systematic Review. Int. J. Environ. Res. Public Health 2021, 18, 6211. [Google Scholar] [CrossRef] [PubMed]
  5. Saeidifard, F.; Medina-Inojosa, J.R.; Supervia, M.; Olson, T.P.; Somers, V.K.; Erwin, P.J.; Lopez-Jimenez, F. Differences of energy expenditure while sitting versus standing: A systematic review and meta-analysis. Eur. J. Prev. Cardiol. 2018, 25, 522–538. [Google Scholar] [CrossRef] [PubMed]
  6. Ainsworth, B.E.; Haskell, W.L.; Herrmann, S.D.; Meckes, N.; Bassett, D.R., Jr.; Tudor-Locke, C.; Greer, J.L.; Vezina, J.; Whitt-Glover, M.C.; Leon, A.S. 2011 Compendium of Physical Activities: A Second Update of Codes and MET Values. Med. Sci. Sports Exerc. 2011, 43, 1575–1581. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Byrne, N.M.; Hills, A.P.; Hunter, G.R.; Weinsier, R.L.; Schutz, Y. Metabolic equivalent: One size does not fit all. J. Appl. Physiol. 2005, 99, 1112–1119. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  8. Weir, J.B.D.B. New methods for calculating metabolic rate with special reference to protein metabolism. J. Physiol. 1949, 109, 1–9. [Google Scholar] [CrossRef] [PubMed]
  9. Plasqui, G.; Bonomi, A.G.; Westerterp, K.R. Daily physical activity assessment with accelerometers: New insights and validation studies. Obes. Rev. 2013, 14, 451–462. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  10. Montoye, A.H.K.; Moore, R.W.; Bowles, H.R.; Korycinski, R.; Pfeiffer, K.A. Reporting accelerometer methods in physical activity intervention studies: A systematic review and recommendations for authors. Br. J. Sports Med. 2018, 52, 1507–1516. [Google Scholar] [CrossRef]
  11. Lyden, K.; Kozey, S.L.; Staudenmeyer, J.W.; Freedson, P.S. A comprehensive evaluation of commonly used accelerometer energy expenditure and MET prediction equations. Eur. J. Appl. Physiol. 2011, 111, 187–201. [Google Scholar] [CrossRef] [PubMed]
  12. Xie, J.; Wen, D.; Liang, L.; Jia, Y.; Gao, L.; Lei, J. Evaluating the Validity of Current Mainstream Wearable Devices in Fitness Tracking Under Various Physical Activities: Comparative Study. JMIR mHealth uHealth 2018, 6, e94. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Farrahi, V.; Niemelä, M.; Kangas, M.; Korpelainen, R.; Jämsä, T. Calibration and validation of accelerometer-based activity monitors: A systematic review of machine-learning approaches. Gait Posture 2019, 68, 285–299. [Google Scholar] [CrossRef] [PubMed]
  14. O’Driscoll, R.; Turicchi, J.; Hopkins, M.; Horgan, G.W.; Finlayson, G.; Stubbs, J.R. Improving energy expenditure estimates from wearable devices: A machine learning approach. J. Sports Sci. 2020, 38, 1496–1505. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Ponce, D.; de Goes, C.R.; de Andrade, L.G.M. Proposal of a new equation for estimating resting energy expenditure of acute kidney injury patients on dialysis: A machine learning approach. Nutr. Metab. 2020, 17, 96. [Google Scholar] [CrossRef] [PubMed]
  16. Evenson, K.R.; Goto, M.M.; Furberg, R.D. Systematic review of the validity and reliability of consumer-wearable activity trackers. Int. J. Behav. Nutr. Phys. Act. 2015, 12, 159. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  17. Henrikisen, A.; Johansson, J.; Hartvigsen, G.; Grimsgaard, S.; Hopstock, L. Measuring Physical Activity Using Triaxial Wrist Worn Polar Activity Trackers: A Systematic Review. Int. J. Exerc. Sci. 2020, 13, 438–454. [Google Scholar]
  18. Shcherbina, A.; Mattsson, C.M.; Waggott, D.; Salisbury, H.; Christle, J.W.; Hastie, T.; Wheeler, M.T.; Ashley, E.A. Accuracy in Wrist-Worn, Sensor-Based Measurements of Heart Rate and Energy Expenditure in a Diverse Cohort. J. Pers. Med. 2017, 7, 3. [Google Scholar] [CrossRef]
  19. Slade, P.; Kochenderfer, M.J.; Delp, S.L.; Collins, S.H. Sensing leg movement enhances wearable monitoring of energy expenditure. Nat. Commun. 2021, 12, 4312. [Google Scholar] [CrossRef] [PubMed]
  20. Santos-Lozano, A.; Santín-Medeiros, F.; Cardon, G.; Torres-Luque, G.; Bailón, R.; Bergmeir, C.; Ruiz, J.R.; Lucia, A.; Garatachea, N. Actigraph GT3X: Validation and Determination of Physical Activity Intensity Cut Points. Int. J. Sports Med. 2013, 34, 975–982. [Google Scholar] [CrossRef]
  21. Duncan, M.J.; Wilson, S.; Tallis, J.; Eyre, E. Validation of the Phillips et al. GENEActiv accelerometer wrist cut-points in children aged 5–8 years old. Eur. J. Pediatr. 2016, 175, 2019–2021. [Google Scholar] [CrossRef]
  22. McClain, J.J.; Hart, T.L.; Getz, R.S.; Tudor-Locke, C. Convergent Validity of 3 Low Cost Motion Sensors With the ActiGraph Accelerometer. J. Phys. Act. Health 2010, 7, 662–670. [Google Scholar] [CrossRef] [PubMed]
  23. Esliger, D.W.; Rowlands, A.; Hurst, T.L.; Catt, M.; Murray, P.; Eston, R. Validation of the GENEA Accelerometer. Med. Sci. Sports Exerc. 2011, 43, 1085–1093. [Google Scholar] [CrossRef] [Green Version]
  24. Romanzini, M.; Petroski, E.L.; Ohara, D.; Dourado, A.C.; Reichert, F.F. Calibration of ActiGraph GT3X, Actical and RT3 accelerometers in adolescents. Eur. J. Sport Sci. 2014, 14, 91–99. [Google Scholar] [CrossRef] [PubMed]
  25. John, D.; Sasaki, J.; Hickey, A.; Mavilia, M.; Freedson, P.S. ActiGraph™ Activity Monitors: “The Firmware Effect”. Med. Sci. Sports Exerc. 2014, 46, 834–839. [Google Scholar] [CrossRef] [PubMed]
  26. Asimina, S.; Chapizanis, D.; Karakitsios, S.; Kontoroupis, P.; Asimakopoulos, D.N.; Maggos, T.; Sarigiannis, D. Assessing and enhancing the utility of low-cost activity and location sensors for exposure studies. Environ. Monit. Assess. 2018, 190, 155. [Google Scholar] [CrossRef] [PubMed]
  27. ActiGraph Link. ActiGraph. Available online: https://www.actigraphcorp.com/actigraph-link/ (accessed on 25 September 2019).
  28. Sensor ID—Proximity Wireless Technologies Made in Italy. Available online: http://www.sensorid.it/ (accessed on 25 September 2019).
  29. LIS2DH12. Available online: https://www.st.com/content/st_com/en/products/mems-and-sensors/accelerometers/lis2dh12.html (accessed on 25 September 2019).
  30. Crouter, S.E.; Kuffel, E.; Haas, J.D.; Frongillo, E.A.; Bassett, D.R. Refined Two-Regression Model for the ActiGraph Accelerometer. Med. Sci. Sports Exerc. 2010, 42, 1029–1037. [Google Scholar] [CrossRef] [Green Version]
  31. Brønd, J.C.; Andersen, L.B.; Arvidsson, D. Generating ActiGraph Counts from Raw Acceleration Recorded by an Alternative Monitor. Med. Sci. Sports Exerc. 2017, 49, 2351–2360. [Google Scholar] [CrossRef] [PubMed]
  32. Freedson, P.S.; Melanson, E.; Sirard, J. Calibration of the Computer Science and Applications, Inc. accelerometer. Med. Sci. Sports Exerc. 1998, 30, 777–781. [Google Scholar] [CrossRef] [PubMed]
  33. Sasaki, J.E.; John, D.; Freedson, P.S. Validation and comparison of ActiGraph activity monitors. J. Sci. Med. Sport 2011, 14, 411–416. [Google Scholar] [CrossRef] [PubMed]
  34. Koo, T.K.; Li, M.Y. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016, 15, 155–163. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  35. Aguilar-Farias, N.; Peeters, G.M.E.E.G.; Brychta, R.J.; Chen, K.Y.; Brown, W.J. Comparing ActiGraph equations for estimating energy expenditure in older adults. J. Sports Sci. 2019, 37, 188–195. [Google Scholar] [CrossRef]
  36. Ingraham, K.A.; Ferris, D.P.; Remy, C.D. Evaluating physiological signal salience for estimating metabolic energy cost from wearable sensors. J. Appl. Physiol. 2019, 126, 717–729. [Google Scholar] [CrossRef] [PubMed]
  37. Nagayoshi, S.; Oshima, Y.; Ando, T.; Aoyama, T.; Nakae, S.; Usui, C.; Kumagai, S.; Tanaka, S. Validity of estimating physical activity intensity using a triaxial accelerometer in healthy adults and older adults. BMJ Open Sport Exerc. Med. 2019, 5, e000592. [Google Scholar] [CrossRef] [PubMed]
  38. O’Driscoll, R.; Turicchi, J.; Beaulieu, K.; Scott, S.; Matu, J.; Deighton, K.; Finlayson, G.; Stubbs, R.J. How well do activity monitors estimate energy expenditure? A systematic review and meta-analysis of the validity of current technologies. Br. J. Sports Med. 2018, 54, 332–340. [Google Scholar] [CrossRef] [PubMed]
  39. Chang, C.-H.; Lin, K.-C.; Ho, C.-S.; Huang, C.-C. Accuracy of the energy expenditure during uphill exercise measured by the Waist-worn ActiGraph. J. Exerc. Sci. Fit. 2019, 17, 62–66. [Google Scholar] [CrossRef] [PubMed]
  40. Ho, C.-S.; Chang, C.-H.; Lin, K.-C.; Huang, C.-C.; Hsu, Y.-J. Correction of estimation bias of predictive equations of energy expenditure based on wrist/waist-mounted accelerometers. PeerJ 2019, 7, e7973. [Google Scholar] [CrossRef]
  41. Yano, S.; Koohsari, M.J.; Shibata, A.; Ishii, K.; Frehlich, L.; McCormack, G.R.; Oka, K. Physical Activity and Sedentary Behavior Assessment: A Laboratory-Based Evaluation of Agreement between Commonly Used ActiGraph and Omron Accelerometers. Int. J. Environ. Res. Public Health 2019, 16, 3126. [Google Scholar] [CrossRef] [Green Version]
  42. Ellis, K.; Kerr, J.; Godbole, S.; Lanckriet, G.; Wing, D.; Marshall, S. A random forest classifier for the prediction of energy expenditure and type of physical activity from wrist and hip accelerometers. Physiol. Meas. 2014, 35, 2191–2203. [Google Scholar] [CrossRef]
  43. White, T.; Westgate, K.; Hollidge, S.; Venables, M.; Olivier, P.; Wareham, N.; Brage, S. Estimating energy expenditure from wrist and thigh accelerometry in free-living adults: A doubly labelled water study. Int. J. Obes. 2019, 43, 2333–2342. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  44. R Project. The R Project for Statistical Computing. Available online: http://www.r-project.org/ (accessed on 3 April 2020).
  45. Omar, A.; Shaghayegh, Z.; Aerin, P.; Chul-Gyu, T.; Edward, J.P. Improving Energy Expenditure Estimation through Activity Classification and Walking Speed Estimation Using a Smartwatch. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Montreal, QC, Canada, 20–24 July 2020. [Google Scholar]
  46. Sama, A.J.; Hillstrom, H.; Daluiski, A.; Wolff, A. Reliability and agreement between two wearable inertial sensor devices for measurement of arm activity during walking and running gait. J. Hand Ther. 2022, 35, 151–154. [Google Scholar] [CrossRef] [PubMed]
  47. Dibben, G.O.; Gandhi, M.M.; Taylor, R.S.; Dalal, H.M.; Metcalf, B.; Doherty, P.; Tang, L.H.; Kelson, M.; Hillsdon, M. Physical activity assessment by accelerometry in people with heart failure. BMC Sports Sci. Med. Rehabil. 2020, 12, 47. [Google Scholar] [CrossRef] [PubMed]
Table 1. Equations applied to calculate energy expenditure.
Table 1. Equations applied to calculate energy expenditure.
Freedson et al. 1998 [32] E E = 1.439008 + 7.95 e 4 × A C V T
Crouter et al. 2010 [30]Walking/Running E E = 2.294275 × exp ( 8.4679 e 5 ) × A C V T
Santos-Lozano et al. VT 2013 [20]Adults E E = 3.4002 + 5.3 e 4 × A C V T 5.564 e 2 × B M + 1.2789 × G e n d e r
Santos-Lozano et al. VM 2013 [20]Adults E E = 2.8323 + 5.4 e 4 × A C V M 5.912 e 2 × B M + 1.4410 × G e n d e r
Sasaki et al. 2011 [33] E E = 8.63 e 4 × A C V M + 0.668876
Correction_METs Female H B e q u a t i o n = 655.0955 + 1.8496 × H e i g h t × 100 + 9.5634 × B M 4.6756 × A g e
Male H B e q u a t i o n = 66.4730 + 5.0033 × H e i g h t × 100 + 13.7516 × B M 6.7550 × A g e
Corrected C o r r e c t e d _ E E = o t h e r   m e t h o d s   ( M E T s ) × 3.5 H B e q u a t i o n × ( 1000 1440 × 5 × B M )
V T : vertical axis; V M : vector of magnitude;
A C V T : activity counts on vertical axis, counts per min;
A C V T : activity counts on vertical axis, counts per 10 s; This is a special A C V T for Crouter et al. 2010;
A C V M : activity counts on vector of magnitude (three axes), counts per min;
B M : body mass (kg);
H e i g h t : cm;
A g e : year;
G e n d e r : 1 : female, 2 : male.
Table 2. ICC results for energy expenditure computed from two inertial sensors (ActiGraph and Sensor ID) on the wrist.
Table 2. ICC results for energy expenditure computed from two inertial sensors (ActiGraph and Sensor ID) on the wrist.
SpeedCrombach’s
Standardised Items
Energy Expenditure Calculation System
Freedson et al. 1998 [32]Crouter et al. 2010 [30]Santos-Lozano et al. VT 2013 [20]Santos-Lozano et al. VM 2013 [20]Sasaki et al. 2011 [33]Corrected Freedson et al. 1998 [32]Corrected Crouter et al. 2010 [30]Corrected Santos-Lozano et al. VT 2013 [20]Corrected Santos-Lozano et al. VM 2013 [20]Corrected Sasaki et al. 2011 [33]
1.4 km/h0.9180.9970.7230.9970.928−0.3310.9970.6510.9970.9040.457
2.9 km/h0.9920.9920.9410.9940.9280.9790.9930.8000.9940.9580.926
4.3km/h0.9820.9350.8350.9510.9570.9290.9320.9320.9500.9570.929
5 km/h0.9570.9560.9560.9830.9150.7440.9850.9910.9830.8990.875
6 km/h0.8500.9510.9580.9460.518−0.0050.9570.9680.9560.110−0.310
6.5 km/h0.9430.9570.9440.9610.1460.0100.9570.9480.963−0.119−0.019
7 km/h0.9550.9410.9310.936−0.359−0.1570.9620.9490.960−0.357−0.015
8 km/h0.9610.8650.8790.846−0.1510.0150.8710.8610.841−0.1380.177
9 km/h0.9400.9890.9890.989−0.613−0.7090.9910.9910.989−0.3010.032
10 km/h0.9610.8650.8190.858−0.366−0.3610.9590.8970.9330.1480.225
MEAN0.9460.9450.8980.9460.3910.1110.9600.8990.9570.3060.328
Table 3. ICC results for energy expenditure computed from two inertial sensors (ActiGraph and Sensor ID) on the thigh.
Table 3. ICC results for energy expenditure computed from two inertial sensors (ActiGraph and Sensor ID) on the thigh.
SpeedCrombach’s
Standardised Items
Energy Expenditure Calculation System
Freedson et al. 1998 [32]Crouter et al. 2010 [30]Santos-Lozano et al. VT 2013 [20] Santos-Lozano et al. VM 2013 [20] Sasaki et al. 2011 [33]Corrected Freedson et al. 1998 [32] Corrected Crouter et al. 2010 [30]Corrected Santos-Lozano et al. VT 2013 [20]Corrected Santos-Lozano et al. VM 2013 [20] Corrected Sasaki et al. 2011 [33]
1.4 km/h0.8870.9900.9870.9970.804−0.6890.9820.9990.995−0.007−0.260
2.9 km/h0.9930.9890.9930.9940.9150.8790.9890.9940.9940.9130.881
4.3 km/h0.9930.9940.9950.9930.7210.5410.9950.9960.9940.7260.608
5 km/h0.9740.9860.9900.9810.195−0.1760.9890.9910.987−0.035−0.279
6 km/h0.9530.9500.9550.949−0.198−0.3610.9680.9680.964−0.607−0.544
6.5 km/h0.9540.9960.9960.989−0.327−0.4410.9890.9870.984−0.702−0.555
7 km/h0.7160.9160.9300.8970.139−0.0710.9520.9530.947−0.763−0.660
8km/h0.9390.9730.9790.9680.2880.2970.9820.9840.981−0.1380.139
9 km/h0.9660.9550.9620.9470.7560.9210.9650.9650.9600.6420.781
10 km/h0.9790.9670.9770.9610.6300.6540.9780.9830.9760.1940.330
MEAN0.9350.9720.9760.9680.3920.1550.9790.9820.9780.0220.044
Table 4. ICC results for energy expenditure computed from the ActiGraph sensor on two placements (thigh and wrist).
Table 4. ICC results for energy expenditure computed from the ActiGraph sensor on two placements (thigh and wrist).
SpeedCrombach’s
Standardized Items
Energy Expenditure Calculation System
Freedson et al. 1998 [32]Crouter et al. 2010 [30]Santos-Lozano et al. VT 2013 [20] Santos-Lozano et al. VM 2013 [20] Sasaki et al. 2011 [33]Corrected Freedson et al. 1998 [32] Corrected Crouter et al. 2010 [30]Corrected Santos-Lozano et al. VT 2013 [20]Corrected Santos-Lozano et al. VM 2013 [20] Corrected Sasaki et al. 2011 [33]
1.4 km/h0.853−0.352−0.7600.3310.8020.437−0.468−0.6750.0510.6730.455
2.9 km/h0.961−0.143−0.0570.2590.8190.7490.0420.0980.2450.8220.771
4.3 km/h0.944−0.237−0.0640.0760.7850.649−0.1070.2300.0660.6900.654
5 km/h0.948−0.018−0.0220.1850.5290.4730.1730.2720.2840.5490.632
6 km/h0.9460.0500.0680.1870.4040.4840.1940.2320.2750.6120.710
6.5 km/h0.755−0.368−0.420−0.0370.0530.344−0.212−0.217−00.300.2570.521
7 km/h0.923−0.293−0.296−0.2320.8190.843−0.330−0.292−0.3320.7090.796
8 km/h0.9570.2430.2270.1440.5080.5420.5010.3680.4470.2790.419
9 km/h0.9290.0340.056−0.119−0.073−0.0080.3390.2300.280−0.257−0.099
10 km/h0.9220.2430.2510.198−0.585−0.5330.3840.3210.315−0.583−0.496
MEAN0.921−0.084−0.1020.0990.4060.3980.0520.0570.1330.3750.436
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Martín-Martín, J.; Wang, L.; De-Torres, I.; Escriche-Escuder, A.; González-Sánchez, M.; Muro-Culebras, A.; Roldán-Jiménez, C.; Ruiz-Muñoz, M.; Mayoral-Cleries, F.; Biró, A.; et al. The Validity of the Energy Expenditure Criteria Based on Open Source Code through two Inertial Sensors. Sensors 2022, 22, 2552. https://doi.org/10.3390/s22072552

AMA Style

Martín-Martín J, Wang L, De-Torres I, Escriche-Escuder A, González-Sánchez M, Muro-Culebras A, Roldán-Jiménez C, Ruiz-Muñoz M, Mayoral-Cleries F, Biró A, et al. The Validity of the Energy Expenditure Criteria Based on Open Source Code through two Inertial Sensors. Sensors. 2022; 22(7):2552. https://doi.org/10.3390/s22072552

Chicago/Turabian Style

Martín-Martín, Jaime, Li Wang, Irene De-Torres, Adrian Escriche-Escuder, Manuel González-Sánchez, Antonio Muro-Culebras, Cristina Roldán-Jiménez, María Ruiz-Muñoz, Fermín Mayoral-Cleries, Attila Biró, and et al. 2022. "The Validity of the Energy Expenditure Criteria Based on Open Source Code through two Inertial Sensors" Sensors 22, no. 7: 2552. https://doi.org/10.3390/s22072552

APA Style

Martín-Martín, J., Wang, L., De-Torres, I., Escriche-Escuder, A., González-Sánchez, M., Muro-Culebras, A., Roldán-Jiménez, C., Ruiz-Muñoz, M., Mayoral-Cleries, F., Biró, A., Tang, W., Nikolova, B., Salvatore, A., & Cuesta-Vargas, A. I. (2022). The Validity of the Energy Expenditure Criteria Based on Open Source Code through two Inertial Sensors. Sensors, 22(7), 2552. https://doi.org/10.3390/s22072552

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop