Skip to main content
SpringerLink
Log in

Deep Learning-Based Multi-modal COVID-19 Screening by Socially Assistive Robots Using Cough and Breathing Symptoms

  • Conference paper
  • First Online:
Social Robotics (ICSR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13818))

Included in the following conference series:

Abstract

In this paper, we present the development of a novel autonomous social robot deep learning architecture capable of real-time COVID-19 screening during human-robot interactions. The architecture allows for autonomous preliminary multi-modal COVID-19 detection of cough and breathing symptoms using a VGG16 deep learning framework. We train and validate our VGG16 network using existing COVID datasets. We then perform real-time non-contact preliminary COVID-19 screening experiments with the Pepper robot. The results for our deep learning architecture demonstrate: 1) an average computation time of 4.57 s for detection, and 2) an accuracy of 84.4% with respect to self-reported COVID symptoms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://youtu.be/EFb6rHAmvRU.

References

  1. Avgousti, S., Christoforou, E.G., Masouras, P., Panayides, A.S., Tsekos, N.V.: Robotic systems on the frontline against the pandemic. In: Ahram, T., Taiar, R. (eds.) IHIET 2021. LNNS, vol. 319, pp. 1105–1112. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-85540-6_142

    Chapter  Google Scholar 

  2. Li, Y., Yan, J., Hu, B.: Mask detection based on efficient-YOLO. In: 2021 40th Chinese Control Conference (CCC), IEEE, pp. 4056–4061 (2021)

    Google Scholar 

  3. Mucchiani, C., Cacchione, P., Johnson, M., Mead, R., Yim, M.: Deployment of a socially assistive robot for assessment of COVID-19 symptoms and exposure at an elder care setting. In: 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), IEEE, pp. 1189–1195 (2021)

    Google Scholar 

  4. Shen, Y., et al.: Robots under COVID-19 pandemic: a comprehensive survey. IEEE Access 9, 1590–1615 (2020)

    Article  Google Scholar 

  5. Wei, W., Wang, J., Ma, J., Cheng, N., Xiao, J.: A real-time robot-based auxiliary system for risk evaluation of COVID-19 infection. arXiv preprint arXiv:2008.07695 (2020)

  6. Bertsimas, D., et al.: An aggregated dataset of clinical outcomes for COVID-19 patients (2020). http://www.covidanalytics.io/datasetdocumentation (2020)

  7. Coppock, H., Gaskell, A., Tzirakis, P., Baird, A., Jones, L., Schuller, B.: End-to-end convolutional neural network enables COVID-19 detection from breath and cough audio: a pilot study. BMJ Innov. 7(2), 356–362 (2021)

    Google Scholar 

  8. Fakhry, A., Jiang, X., Xiao, J., Chaudhari, G., Han, A., Khanzada, A.: Virufy: a multi-branch deep learning network for automated detection of COVID-19, arXiv preprint arXiv:2103.01806 (2021)

  9. Banerjee, A., Nilhani, A.: A residual network based deep learning model for detection of COVID-19 from Cough Sounds. arXiv preprint arXiv:2106.02348 (2021)

  10. Laguarta, J., Hueto, F., Subirana, B.: COVID-19 artificial intelligence diagnosis using only cough recordings. IEEE Open J. Eng. Med. Biol. 1, 275–281 (2020)

    Article  Google Scholar 

  11. Rao, S., Narayanaswamy, V., Esposito, M., Thiagarajan, J.I., Spanias, A.: COVID-19 detection using cough sound analysis and deep learning algorithms. Intell. Decis.Technol. 15, 1–11 (2021)

    Google Scholar 

  12. Rao, S., Narayanaswamy, V., Esposito, M., Thiagarajan, I., Spanias, A.: Deep learning with hyper-parameter tuning for COVID-19 cough detection. In: 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), IEEE, pp. 1–5 (2021)

    Google Scholar 

  13. Imran, A., et al.: AI4COVID-19: AI enabled preliminary diagnosis for COVID-19 from cough samples via an app. Informat. Med. Unlock. 20, 100378 (2020)

    Article  Google Scholar 

  14. Chetupalli, S.R., et al.: Multi-modal point-of-care diagnostics for COVID-19 based on acoustics and symptoms, arXiv preprint arXiv:2106.00639 (2021)

  15. Effati, M., Sun, Y.-C., Naguib, H.E., Nejat, G.: Multimodal detection of COVID-19 symptoms using deep learning & probability-based weighting of modes. In: 2021 17th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), IEEE, pp. 151–156 (2021)

    Google Scholar 

  16. Gong, Z., et al.: SHUYU robot: an automatic rapid temperature screening system. Chin. J. Mech. Eng. 33(1), 1–4 (2020)

    Article  Google Scholar 

  17. Huang, H.-W., et al.: Agile mobile robotic platform for contactless vital signs monitoring (2020)

    Google Scholar 

  18. Happich, J.: 5G edge patrol robots deployed in China to detect Covid-19 cases. eeNews Europe, March 2020

    Google Scholar 

  19. Elsayed, E.K., Alsayed, A.M., Salama, O.M., Alnour, A.M., Mohammed, H.A.: Deep learning for covid-19 facemask detection using autonomous drone based on IoT, In: 2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE), IEEE, pp. 1–5 (2021)

    Google Scholar 

  20. Mohammed, M., Hazairin, N.A., Al-Zubaidi, S., AK, S., Mustapha, S., Yusuf, E.: Toward a novel design for coronavirus detection and diagnosis system using IoT based drone technology, Int. J. Psychosoc. Rehabil. 24(7), 2287–2295 (2020)

    Google Scholar 

  21. Ackerman, E.E., Shi, F.: Video Friday: robots help keep medical staff safe at COVID-19 Hospital–IEEE Spectrum. Accessed Nov 2020

    Google Scholar 

  22. Robotics, U.: Aimbot: Ubtech’s Anti-Epidemic Solution (2020)

    Google Scholar 

  23. Simonyan, K., Zisserman, A,: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  24. McFee, B., et al.: Librosa: Audio and music signal analysis in python. In: Proceedings of the 14th Python in Science Conference, vol. 8: Citeseer, pp. 18–25 (2015)

    Google Scholar 

  25. Nassif, A.B., Shahin, I., Bader, M., Hassan, A., Werghi, N.: COVID-19 detection systems using deep-learning algorithms based on speech and image data. Mathematics 10(4), 564 (2022)

    Article  Google Scholar 

  26. Brown, C., et al.: Exploring automatic diagnosis of covid-19 from crowdsourced respiratory sound data (2020). arXiv preprint arXiv:2006.05919

  27. Orlandic, L., Teijeiro, T., Atienza, D.: The COUGHVID crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms. Sci. Data 8(1), 1–10 (2021)

    Article  Google Scholar 

  28. Corey, R.M., Jones, U., Singer, A.C.: Acoustic effects of medical, cloth, and transparent face masks on speech signals. J. Acoust. Soc. Am. 148(4), 2371–2375 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Autonomous Systems and Biomechatronics Laboratory (ASBLab), Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, Canada

    Meysam Effati & Goldie Nejat

  2. Toronto Rehabilitation Institute, Toronto, Canada

    Goldie Nejat

  3. Rotman Research Institute, Baycrest Health Sciences, North York, Canada

    Goldie Nejat

Authors
  1. Meysam Effati
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Goldie Nejat
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Meysam Effati .

Editor information

Editors and Affiliations

  1. University of Florence, Florence, Italy

    Filippo Cavallo

  2. Qatar University, Doha, Qatar

    John-John Cabibihan

  3. University of Florence, Florence, Italy

    Laura Fiorini

  4. University of Florence, Florence, Italy

    Alessandra Sorrentino

  5. Wichita State University, Wichita, KS, USA

    Hongsheng He

  6. Qingdao University, Qingdao, China

    Xiaorui Liu

  7. National Institute of Advanced Industrial Science and Technology, Tsukuba, Japan

    Yoshio Matsumoto

  8. National University of Singapore, Singapore, Singapore

    Shuzhi Sam Ge

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Effati, M., Nejat, G. (2022). Deep Learning-Based Multi-modal COVID-19 Screening by Socially Assistive Robots Using Cough and Breathing Symptoms. In: Cavallo, F., et al. Social Robotics. ICSR 2022. Lecture Notes in Computer Science(), vol 13818. Springer, Cham. https://doi.org/10.1007/978-3-031-24670-8_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-24670-8_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-24669-2

  • Online ISBN: 978-3-031-24670-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation