Skip to main content
SpringerLink
Log in

Tell It Your Way: Technology-Mediated Human-Human Multimodal Communication

  • Conference paper
  • First Online:
Wireless Mobile Communication and Healthcare (MobiHealth 2021)

  • 396 Accesses

Abstract

Communication plays a pivotal role in our daily lives. With the advances of technology we are now able to use it to communicate with others at a distance. However, while in direct human-human communication we are able to adjust how we pass a message based on our context and the perceived context of the receiver. When we do it at a distance, using a messaging tool, one of the most popular choices, nowadays, this becomes harder. In fact, most messaging tools, such as Whatsapp or Messenger, provide some degree of flexibility regarding the way a message is sent (e.g., text, audio, image), but the receiver is limited to receiving it in the format of sender choice. In this regard, providing more flexibility in such technology-mediated communication scenarios might foster increased adaptability of these tools to multiple user abilities and contexts, and provide important alternatives for those with some disability (e.g., aphasia, blindness). The work presented here adopts a user-centered approach to design and develop a first proof-of-concept for a multimodal messaging system than enables message modality conversion regardless of the format used by the sender.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.statista.com/statistics/258749/most-popular-global-mobile-messenger-apps/.

  2. 2.

    https://aph-alarm-project.com/.

  3. 3.

    https://flutter.dev/.

  4. 4.

    https://firebase.google.com/docs/auth.

  5. 5.

    https://www.w3.org/TR/eventsource/.

  6. 6.

    https://flask.palletsprojects.com/en/2.0.x/.

  7. 7.

    https://gtts.readthedocs.io/en/latest/.

  8. 8.

    https://pypi.org/project/SpeechRecognition/.

  9. 9.

    https://wicg.github.io/speech-api/.

  10. 10.

    https://github.com/pjreddie/darknet/blob/master/data/coco.names.

References

  1. Almeida, N., Teixeira, A., Silva, S., Ketsmur, M.: The AM4I architecture and framework for multimodal interaction and its application to smart environments. Sens. (Switz.) 19(11), 1–30 (2019). https://doi.org/10.3390/s19112587

    Article  Google Scholar 

  2. Amarasinghe, A., Wijesuriya, V.B.: Stimme: a chat application for communicating with hearing impaired persons. In: 2019 IEEE 14th International Conference on Industrial and Information Systems: Engineering for Innovations for Industry 4.0, ICIIS 2019 - Proceedings, pp. 458–463 (2019)

    Google Scholar 

  3. Bryan-Kinns, N., Hamilton, F.: One for all and all for one? Case studies of using prototypes in commercial projects. In: ACM International Conference Proceeding Series, vol. 31, pp. 91–100 (2002). https://doi.org/10.1145/572020.572032

  4. Cooper, A., Reimann, R., Cronin, D.: About Face 3: The Essentials of Interaction Design, vol. 3 (2007)

    Google Scholar 

  5. Daems, J., Bosch, N., Solberg, S., Dekelver, J., Kultsova, M.: AbleChat: development of a chat app with pictograms for people with intellectual disabilities. In: Engineering for Society - Leuven 2016 - Proceedings, pp. 25–32 (2016)

    Google Scholar 

  6. Liu, L., et al.: Deep learning for generic object detection: a survey. Int. J. Comput. Vis. 128(2), 261–318 (2019)

    Article  Google Scholar 

  7. Martins, A.I., Rosa, A.F., Queirós, A., Silva, A., Rocha, N.P.: European Portuguese validation of the system usability scale (SUS). Proc. Comput. Sci. 67, 293–300 (2015)

    Article  Google Scholar 

  8. Miller, G.A.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)

    MATH  Google Scholar 

  9. Mirzaei, M.R., Ghorshi, S., Mortazavi, M.: Helping deaf and hard-of-hearing people by combining augmented reality and speech technologies. In: Proceedings of 9th International Conference on Disability, Virtual Reality and Associated Technologies, pp. 10–12 (2012)

    Google Scholar 

  10. Nielsen, J.: Heuristic Evaluation. Usability Inspection Methods (1994). Edited by: Nielsen J, Mack RL

    Google Scholar 

  11. Ramesh, A., et al.: Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 (2021)

  12. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2016-December, pp. 779–788, June 2015

    Google Scholar 

  13. Samonte, M.J.C., Gazmin, R.A., Soriano, J.D.S., Valencia, M.N.O.: BridgeApp: an assistive mobile communication application for the deaf and mute. In: ICTC 2019–10th International Conference on ICT Convergence: ICT Convergence Leading the Autonomous Future, pp. 1310–1315, October 2019

    Google Scholar 

  14. Sevens, L., Vandeghinste, V., Schuurman, I., Eynde, F.V.: Less is more: a rule-based syntactic simplification module for improved text-to-pictograph translation. Data Knowl. Eng. 117, 264–289 (2018)

    Article  Google Scholar 

Download references

Acknowledgement

This work was supported by EU and national funds through the Portuguese Foundation for Science and Technology (FCT), in the context of project AAL APH-ALARM (AAL/0006/2019) and funding to the research unit IEETA (UIDB/00127/2020).

Author information

Authors and Affiliations

  1. IEETA – Institute of Electronics and Informatics Engineering of Aveiro, DETI – Department of Electronics, Telecomunications and Informatics, University of Aveiro, Aveiro, Portugal

    Helena Cardoso, Nuno Almeida & Samuel Silva

Authors
  1. Helena Cardoso
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nuno Almeida
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Samuel Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Helena Cardoso .

Editor information

Editors and Affiliations

  1. Chongqing University of Posts and Telecommunications, Chongqing, China

    Xinbo Gao

  2. The University of Sydney, Sydney, NSW, Australia

    Abbas Jamalipour

  3. Chongqing University of Posts and Telecommunications, Chongqing, China

    Lei Guo

Rights and permissions

Reprints and permissions

Copyright information

© 2022 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cardoso, H., Almeida, N., Silva, S. (2022). Tell It Your Way: Technology-Mediated Human-Human Multimodal Communication. In: Gao, X., Jamalipour, A., Guo, L. (eds) Wireless Mobile Communication and Healthcare. MobiHealth 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 440. Springer, Cham. https://doi.org/10.1007/978-3-031-06368-8_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06368-8_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06367-1

  • Online ISBN: 978-3-031-06368-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation