Breathing Analysis Using Thermal and Depth Imaging Camera Video Records
Abstract
:1. Introduction
2. Methods
2.1. Data Acquisition
2.2. Data Processing
- videorecording of the face area during a selected time range,
- extraction of thermographic frames with the selected sampling frequency (of 10 Hz) and a given resolution,
- automatic determination of temperature ranges in each thermographic frame and the adaptive calibration of each thermal image,
- detection of the mouth area using the selected number of initial frames with the largest temperature changes and the adaptive update of this ROI for each subsequent thermal image,
- evaluation of the mean temperature in the specified window of a changing position and size in each frame.
3. Results
4. Conclusions
Supplementary Materials
Acknowledgments
Author Contributions
Conflicts of Interest
References
- Procházka, A.; Schatz, M.; Vyšata, O. Microsoft kinect visual and depth sensors for breathing and heart rate analysis. Sensors 2016, 16, 1–11. [Google Scholar] [CrossRef] [PubMed]
- Lee, J.; Hong, M.; Ryu, S. Sleep monitoring system using kinect sensor. Int. J. Distrib. Sens. Netw. 2015, 2015. [Google Scholar] [CrossRef]
- Procházka, A.; Vyšata, O.; Vališ, M.; Ťupa, O.; Schatz, M.; Mařík, V. Bayesian classification and analysis of gait disorders using image and depth sensors of Microsoft Kinect. Digit. Signal Prog. 2015, 47, 169–177. [Google Scholar] [CrossRef]
- Procházka, A.; Vyšata, O.; Vališ, M.; Ťupa, O.; Schatz, M.; Mařík, V. Use of Image and depth sensors of the Microsoft Kinect for the detection of gait disorders. Neural Comput. Appl. 2015, 26, 1621–1629. [Google Scholar] [CrossRef]
- Erden, F.; Velipasalar, S.; Alkar, A.; Cetin, A. Sensors in assisted living. IEEE Signal Process. Mag. 2016, 33, 36–44. [Google Scholar] [CrossRef]
- Procházka, A.; Schätz, M.; Centonze, F.; Kuchyňka, J.; Vyšata, O.; Vališ, M. Extraction of breathing features using MS Kinect for sleep stage detection. Signal Image Video Process. 2016, 10, 1278–1286. [Google Scholar] [CrossRef]
- Appel, V.; Belini, V.; Jong, D.; Magalhães, D.; Caurin, G. Classifying emotions in rehabilitation robotics based on facial skin temperature. In Proceedings of the IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics, Sao Paulo, Brazil, 12–14 August 2014; pp. 276–281. [Google Scholar]
- Boccanfuso, L.; Wang, Q.; Leite, I.; Li, B.; Torres, C.; Chen, L.; Salomons, N.; Foster, C.; Barney, E.; Ahn, Y.; et al. A thermal emotion classifier for improved human–robot interaction. In Proceedings of the 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, USA, 26–31 August 2016; pp. 718–723. [Google Scholar]
- Kwaśniewska, A.; Rumiński, J. Face detection in image sequences using a portable thermal camera. In Proceedings of the 13th Quantitative Infrared Thermography Conference, Gdansk, Poland, 4–8 July 2016. [Google Scholar]
- Latif, M.; Md. Yusof, H.; Sidek, S.; Rusli, N.; Fatai, S. Emotion detection from thermal facial imprint based on GLCM features. ARPN-JEAS 2016, 11, 345–350. [Google Scholar]
- Nguyen, H.; Kotani, K.; Chen, F.; Le, B. Estimation of human emotions using thermal facial information. In Proceedings of the SPIE—The International Society for Optical Engineering, ICGIP 2013, Hong Kong, China, 26–27 October 2013. [Google Scholar]
- Rahulamathavan, Y.; Phan, R.C.V.; Chambers, J.A.; Parish, D.J. Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans. Affect. Comput. 2013, 4, 83–92. [Google Scholar] [CrossRef]
- Cheong, Y.; Yap, V.; Nisar, H. A novel face detection algorithm using thermal imaging. In Proceedings of the 2014 IEEE Symposium on Computer Applications and Industrial Electronics, ISCAIE, Penang, Malaysia, 7–8 April 2014; pp. 208–213. [Google Scholar]
- Liu, P.; Yin, L. Spontaneous facial expression analysis based on temperature changes and head motions. In Proceedings of the11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2015, Ljubljana, Slovenia, 4–8 May 2015. [Google Scholar]
- Cardone, D.; Pinti, P.; Merla, A. Thermal infrared imaging-based computational psychophysiology for psychometrics. Comput. Math. Method Med. 2015, 2015, 1–8. [Google Scholar] [CrossRef] [PubMed]
- Ioannou, S.; Gallese, V.; Merla, A. Thermal infrared imaging in psychophysiology: Potentialities and limits. Psychophysiology 2014, 51, 951–963. [Google Scholar] [CrossRef] [PubMed]
- Nhan, B.; Chau, T. Classifying affective states using thermal infrared imaging of the human face. IEEE Trans. Biomed. Eng. 2010, 57, 979–987. [Google Scholar] [CrossRef] [PubMed]
- Hong, K.; Hong, S. Real-time stress assessment using thermal imaging. Vis. Comput. 2016, 32, 1369–1377. [Google Scholar] [CrossRef]
- Engert, V.; Merla, A.; Grant, J.; Cardone, D.; Tusche, A.; Singer, T. Exploring the use of thermal infrared imaging in human stress research. PLoS ONE 2014, 9, e90782. [Google Scholar] [CrossRef] [PubMed]
- Kim, H.; Kim, J.-Y.; Im, C.-H. Fast and robust real-time estimation of respiratory rate from photoplethysmography. Sensors 2016, 16, 1494. [Google Scholar] [CrossRef] [PubMed]
- Zhang, X.; Ding, Q. Respiratory rate estimation from the photoplethysmogram via joint sparse signal reconstruction and spectra Psion. Biomed. Signal Process. Control 2017, 35, 1–7. [Google Scholar] [CrossRef]
- Hu, M.H.; Zhai, G.T.; Li, D.; Fan, Y.Z.; Chen, X.H.; Yang, X.K. Synergetic use of thermal and visible imaging techniques for contactless and unobtrusive breathing measurement. J. Biomed. Opt. 2017, 22, 1–11. [Google Scholar] [CrossRef] [PubMed]
- Lin, Y.-D.; Chien, Y.-H.; Chen, Y.-H. Wavelet-based embedded algorithm for respiratory rate estimation from PPG signal. Biomed. Signal Process. Control 2017, 36, 138–145. [Google Scholar] [CrossRef]
- Carpagnano, G.E.; Foschino-Barbaro, M.P.; Crocetta, C.; Lacedonia, D.; Saliani, V.; Zoppo, L.D.; Barnes, P.J. Validation of the exhaled breath temperature measure: Reference values in healthy subjects. Chest 2017, 151, 855–860. [Google Scholar] [CrossRef] [PubMed]
- Khalidi, F.Q.; Saatchi, R.; Burke, D.; Elphick, H.; Tan, S. Respiration rate monitoring methods: A review. Pediatr. Pulmonol. 2011, 46, 523–529. [Google Scholar] [CrossRef] [PubMed]
- Adib, F.; Mao, H.; Kabelac, Z.; Katabi, D.; Miller, R.C. Smart homes that monitor breathing and heart rate. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 2015, Seoul, Korea, 18–23 April 2015; pp. 837–846. [Google Scholar]
- Heck, D.H.; McAfee, S.S.; Liu, Y.; Babajani-Feremi, A.; Rezaie, R.; Freeman, W.J.; Wheless, J.W.; Papanicolaou, A.C.; Ruszinko, M.; Sokolov, Y.; Kozma, R. Breathing as a fundamental rhytm of brain function. Front. Neural Circuits 2017, 10, 115. [Google Scholar] [CrossRef] [PubMed]
- Murthy, R.; Pavlidis, I.; Tsiamyrtzis, P. Touchless Monitoring of breathing function. In Proceedings of the 26th Annual International Conference of the IEEE EMBS, San Francisco, CA, USA, 1–5 September 2004. [Google Scholar]
- Al-Obaisi, F.; Alqatawna, J.; Faris, H.; Rodan, A.; Al-Kadi, O. Pattern recognition of thermal images for monitoring of breathing function. Int. J. Control Autom. 2015, 8, 381–392. [Google Scholar] [CrossRef]
- Folke, M.; Cernerud, L.; Ekström, M.; Hök, B. Critical review of non-invasive respiratory monitoring in medical care. Med. Biol. Eng. Comput. 2003, 41, 377–383. [Google Scholar] [CrossRef] [PubMed]
- Usamentiaga, R.; Venegas, P.; Guerediaga, J.; Vega, L.; Molleda, J.; Bulnes, F.G. Infrared Thermography for temperature measurement and non-destructive testing. Sensors 2014, 14, 12305–12348. [Google Scholar] [CrossRef] [PubMed]
- Xia, J.; Siochi, R.A. A real-time respiratory motion monitoring system using microsoft kinect sensor. Med. Phys. 2012, 39, 2682–2685. [Google Scholar] [CrossRef] [PubMed]
- Griessenberger, H.; Heib, D.P.J.; Kunz, A.B.; Hoedlmoser, K.; Schabus, M. Assessment of a wireless headband for automatic sleep scoring. Sleep Breath. 2013, 17, 747–752. [Google Scholar] [CrossRef] [PubMed]
- Kolb, A.; Barth, E.; Koch, R.; Larsen, R. Time-of-flight sensors in computer graphics. In Eurographics 2009—State of the Art Reports; Pauly, M., Greiner, G., Eds.; The Eurographics Association: Geneva, Switzerland, 2009; pp. 119–134. [Google Scholar]
- Charvátová, H.; Procházka, A.; Vaseghi, S.; Vyšata, O.; Vališ, M. GPS-based analysis of physical activities using positioning and heart rate cycling data. Signal Image Video Process. 2017, 11, 251–258. [Google Scholar] [CrossRef]
Thermo Camera Specifications | MS Kinect Specifications | |||
---|---|---|---|---|
Feature | Description | Feature | Description | |
Thermal sensor resolution | RGB stream resolution | |||
Detection distance | 300 m | Depth stream resolution | ||
Temperature range | −40–330 °C | Infrared stream resolution | ||
Frame rate | <9 Hz | Depth range | 0.4–4 m | |
Microbolometer | Vanadium Oxide | Frame rate | <30 Hz | |
Lens material | Chalcogenide | |||
Pixel pitch | 12 m | |||
Spectral range | 7.5–14 m |
Test | Fixed ROI | Moving ROI | |||||
---|---|---|---|---|---|---|---|
T (°C) | R (°C) | F (bpm) | T (°C) | R (°C) | F (bpm) | ||
1 | 26.49 | 3.49 | 14.79 | 27.19 | 10.07 | 14.79 | |
2 | 26.26 | 3.17 | 15.61 | 27.02 | 9.12 | 15.61 | |
3 | 26.79 | 5.66 | 15.41 | 27.09 | 10.61 | 15.41 | |
4 | 27.33 | 4.31 | 15.82 | 27.47 | 11.38 | 15.82 | |
5 | 26.14 | 4.00 | 16.23 | 26.95 | 9.44 | 16.23 | |
6 | 27.55 | 4.20 | 14.58 | 27.32 | 9.07 | 14.58 | |
7 | 27.54 | 4.13 | 16.64 | 27.40 | 9.98 | 16.64 |
Experiment | Temperature Evolution | Frequency Evolution | |||||||
---|---|---|---|---|---|---|---|---|---|
Reg. Coeff. [°C/min] | S [%] | Aver. Reg. Coeff. | Reg. Coeff. [bpm] | S [%] | Aver. Reg. Coeff. | ||||
Mean | STD | Mean | STD | ||||||
1 | −0.252 | 0.001 | −0.513 | 0.401 | |||||
2 | −0.182 | 0.001 | −1.571 | 0.476 | |||||
3 | −0.135 | 0.001 | −0.162 | 0.059 | −0.250 | 0.055 | −0.720 | 0.619 | |
4 | −0.092 | 0.003 | −0.117 | 0.508 | |||||
5 | −0.148 | 0.001 | −1.150 | 0.358 |
Breathing Feature | Segment | Mean Deleay (s) | STD |
---|---|---|---|
Frequency | Load | 76 | 17 |
Rest | 98 | 47 | |
Temperature | Load | 188 | 59 |
Rest | 130 | 34 |
© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Procházka, A.; Charvátová, H.; Vyšata, O.; Kopal, J.; Chambers, J. Breathing Analysis Using Thermal and Depth Imaging Camera Video Records. Sensors 2017, 17, 1408. https://doi.org/10.3390/s17061408
Procházka A, Charvátová H, Vyšata O, Kopal J, Chambers J. Breathing Analysis Using Thermal and Depth Imaging Camera Video Records. Sensors. 2017; 17(6):1408. https://doi.org/10.3390/s17061408
Chicago/Turabian StyleProcházka, Aleš, Hana Charvátová, Oldřich Vyšata, Jakub Kopal, and Jonathon Chambers. 2017. "Breathing Analysis Using Thermal and Depth Imaging Camera Video Records" Sensors 17, no. 6: 1408. https://doi.org/10.3390/s17061408
APA StyleProcházka, A., Charvátová, H., Vyšata, O., Kopal, J., & Chambers, J. (2017). Breathing Analysis Using Thermal and Depth Imaging Camera Video Records. Sensors, 17(6), 1408. https://doi.org/10.3390/s17061408