Skip to main content
SpringerLink
Log in

A realtime portable and accessible aiding system for the blind – a cloud based approach

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the rise of AI and Deep Learning technologies, it is now possible to give the visually impaired a sense of sight. This work intends to propose a system, which helps the blind to perceive their surrounding without any extra hand. The system harnesses the power of revolutionary cloud technology, cutting-edge artificial intelligence systems and state of the art language translation technologies for the inevitable cause of assisting the blind. This work mainly focusses on developing a simple gesture-controlled cloud based mobile application, which would allow them to capture their surroundings and help them to navigate through their surroundings in real-time. In this work a real case system architecture is proposed which would analyse the spatial reference of objects in the image and also the custom trained NLP engine generates a description that is narrated in their own native language which stands the unique aspect of the work. The proposed application proves to be a one-stop solution for the visually impaired with real case analysis. To strengthen the analysis of the work, results pertaining to the system architecture emphasising its real-time performance and accessibility are done.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The manuscript has no associated data. Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

References

  1. Ahmed F, Mahmud MS, Al-Fahad R, Alam S, Yeasin M (2018) Image captioning for ambient awareness on a sidewalk. In: 2018 1st International Conference on Data Intelligence and Security (ICDIS). IEEE, pp 85–91

  2. Arora A, Grover A, Chugh R, Reka SS (2019) Real time multi object detection for blind using single shot multibox detector. Wirel Pers Commun 107(1):651–661

    Article  Google Scholar 

  3. Bagwan SMR, Sankpal LJ (2015) VisualPal: a mobile app for object recognition for the visually impaired. In: 2015 International Conference on Computer, Communication and Control (IC4). IEEE, pp 1–6

  4. Bai J, Liu D, Su G, Fu Z (2017) A cloud and vision-based navigation system used for blind people. In: Proceedings of the 2017 international conference on artificial intelligence, automation and control technologies, pp 1–6

  5. Bakshi AM, Simson J, de Castro C, Yu CC, Dias A (2019) Bright: an augmented reality assistive platform for visual impairment. In: 2019 IEEE Games, Entertainment, Media Conference (GEM). IEEE, pp 1–4

  6. Berger A, Maly F (2019) Smart google glass solution used as education support tool. In: 2019 International Symposium on Educational Technology (ISET). IEEE, pp 265–267

  7. Chaudhuri A, Mandaviya K, Badelia P, Ghosh SK (2017) Optical character recognition systems. In: Optical character recognition systems for different languages with soft computing. Springer, Cham, pp 9–41

  8. Cutter M, Manduchi R (2017) Improving the accessibility of mobile OCR apps via interactive modalities. ACM Trans Accessible Comput (TACCESS) 10(4):1–27

    Article  Google Scholar 

  9. Dong J, Li X, Lan W, Huo Y, Snoek CG (2016) Early embedding and late reranking for video captioning. In: Proceedings of the 24th ACM international conference on multimedia, pp 1082–1086

  10. Evans G, Miller J, Pena MI, MacAllister A, Winer E (2017) Evaluating the Microsoft HoloLens through an augmented reality assembly application. In: Degraded environments: sensing, processing, and display 2017, vol 10197. International Society for Optics and Photonics, pp 101970

  11. Fiannaca A, Apostolopoulous I, Folmer E (2014) Headlock: a wearable navigation aid that helps blind cane users traverse large open spaces. In: Proceedings of the 16th international ACM SIGACCESS conference on computers & accessibility, pp 19–26

  12. Fusco G, Tekin E, Ladner RE, Coughlan JM (2014) Using computer vision to access appliance displays. In: Proceedings of the 16th international ACM SIGACCESS conference on computers & accessibility, pp 281–282

  13. Gaudissart V, Ferreira S, Thillou C, Gosselin B (2004) SYPOLE: mobile reading assistant for blind people. In: 9th Conference Speech and Computer

  14. Gupta P, Shukla M, Arya N, Singh U, Mishra K (2022) Let the blind see: an AIIoT-based device for real-time object recognition with the voice conversion. In: Machine learning for critical internet of medical things. Springer, Cham, pp 177–198

  15. Gurari D, Zhao Y, Zhang M, Bhattacharya N (2020) Captioning images taken by people who are blind. arXiv preprint arXiv:2002.08565

  16. Hammami A, Ben Hamida A, Ben Amar C (2021) Blind semi-fragile watermarking scheme for video authentication in video surveillance context. Multimed Tools Appl 80(5):7479–7513

    Article  Google Scholar 

  17. Hasnine MN, Flanagan B, Akcapinar G, Ogata H, Mouri K, Uosaki N (2019) Vocabulary learning support system based on automatic image captioning technology. In: International conference on human-computer interaction. Springer, Cham, pp 346–358

  18. Hu M, Chen Y, Zhai G, Gao Z, Fan L (2019) An overview of assistive devices for blind and visually impaired people. Int J Robot Autom 34(5):580–598

    Google Scholar 

  19. Huang J, Lin S, Wang N, Dai G, Xie Y, Zhou J (2019) Tse-cnn: a two-stage end-to-end cnn for human activity recognition. IEEE J biomedical health Inf 24(1):292–299

    Article  Google Scholar 

  20. Jain A, Dubey A, Gupta R, Jain N, Tripathi P (2013) Fundamental challenges to mobile based ocr. vol, 2(5), 86–101

  21. Jasmine GS, Marry DM, Lakshmi SS, Rishiwanth R, Sreehariprasath K, Surendhar J (2021) Camera based text and product lable reading for blind people. In: 2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS), vol 1. IEEE, pp 1122–1126

  22. Khan MA, Paul P, Rashid M, Hossain M, Ahad MAR (2020) An AI-based visual aid with integrated reading assistant for the completely blind. IEEE Trans Human-Machine Syst 50(6):507–517

    Article  Google Scholar 

  23. Kumar S, Mathew S, Anumula N, Chandra KS (2020) Portable camera-based assistive device for real-time text recognition on various products and speech using android for blind people. In: Innovations in electronics and communication engineering. Springer, Singapore, pp 437–448

  24. Kuriakose B, Shrestha R, Sandnes FE (2020) Smartphone navigation support for blind and visually impaired people-a comprehensive analysis of potentials and opportunities. In: International conference on human-computer interaction. Springer, Cham, pp 568–583

  25. Landa Y (2011) Videation assistant for blind and cognitively-impaired users

  26. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D … Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision. Springer, Cham, pp 740–755

  27. Lu D, Fang Y (2021) Audi-exchange: AI-guided hand-based actions to assist human-human interactions for the blind and the visually impaired. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 1718–1726

  28. Maćkowski M, Brzoza P, Żabka M, Spinczyk D (2018) Multimedia platform for mathematics’ interactive learning accessible to blind people. Multimed Tools Appl 77(5):6191–6208

    Article  Google Scholar 

  29. Maheshan MS, Harish BS, Nagadarshan N (2020) A convolution neural network engine for sclera recognition. Int J Interact Multimed Artif Intell 6(1)

  30. Martinez Gutierrez MF (2019) Automated image captioning: exploring the potential of microsoft computer vision for English and Spanish (Doctoral dissertation, University of Geneva)

  31. Matusiak K, Skulimowski P, Strurniłło P (2013) Object recognition in a mobile phone application for visually impaired users. In: 2013 6th International Conference on Human System Interactions (HSI). IEEE, pp 479–484

  32. Murali M, Sharma S, Nagansure N (2020) Reader and object detector for blind. In: 2020 International Conference on Communication and Signal Processing (ICCSP). IEEE, pp 0795–0798

  33. Neto R, Fonseca N (2014) Camera reading for blind people. Procedia Technol 16:1200–1209

    Article  Google Scholar 

  34. Onyejegbu LN, Ikechukwu OA (2016) Optical character recognition as a cloud service in azure architecture. Int J Comput Appl 146(13):14–20

    Google Scholar 

  35. Pamparău C, Vatavu RD (2021) FlexiSee: flexible configuration, customization, and control of mediated and augmented vision for users of smart eyewear devices. Multimed Tools Appl 80(20):30943–30968

    Article  Google Scholar 

  36. Peters JP, Thillou C, Ferreira S (2004) Embedded reading device for blind people: a user-centered design. 33rd Applied Imagery pattern recognition workshop (AIPR’04). IEEE, pp 217–222

  37. Price LC, Chen J, Park J, Cho YK (2021) Multisensor-driven real-time crane monitoring system for blind lift operations: lessons learned from a case study. Autom Constr 124:103552

    Article  Google Scholar 

  38. Qureshi TA, Rajbhar M, Pisat Y, Bhosale V (2021) AI based app for blind people. Int Res J Eng Technol 8(3):2883–2887

    Google Scholar 

  39. Rahman MW, Tashfia SS, Islam R, Hasan MM, Sultan SI, Mia S, Rahman MM (2021) The architectural design of smart blind assistant using IoT with deep learning paradigm. Internet Things 13:100344

    Article  Google Scholar 

  40. Raj VG, Vigneswaran EE, Deshnaa M, RajPrasanth K (2022) Virtual smart glass for blind using object detection. In: 2022 4th international conference on smart systems and inventive technology (ICSSIT). IEEE, pp 1419–1424

  41. Sandhya BR, Sahana S, Sneha S (2021) Third eye for blind. Perspectives in Communication, Embedded-systems and Signal-processing-PiCES 4(11):280–283

  42. Schauerte B, Martinez M, Constantinescu A, Stiefelhagen R (2012) An assistive vision system for the blind that helps find lost things. In: International conference on computers for handicapped persons. Springer, Berlin, pp 566–572

  43. Singh N, Ji G (2021) Computer vision assisted, real time blind spot detection based collision warning system for two wheelers. In: 2021 5th International Conference on Electronics, Communication and Aerospace Technology (ICECA), pp 1179–1184. IEEE

  44. Smith R (2007) An overview of the Tesseract OCR engine. In: Ninth international conference on document analysis and recognition (ICDAR 2007), vol 2. IEEE, pp 629–633

  45. Tamimi AA, Hammad AAA, Abdalla AM, Al-Allaf ON (2019) A system for the detection and identification of objects and their distances to aid blind people. In: Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), pp 98–104

  46. Vaithiyanathan D, Muniraj M (2019) Cloud based text extraction using google cloud vison for visually impaired applications. In: 2019 11th International Conference on Advanced Computing (ICoAC). IEEE, pp 90–96

  47. Vázquez SR, Fitzpatrick D, O’Brien S (2018) Is web-based Computer-Aided Translation (CAT) software usable for blind translators?. In: International Conference on Computers Helping People with Special Needs. Springer, Cham, pp 31–34

  48. Verma KK, Singh BM, Mandoria HL, Chauhan P (2020) Two-stage human activity recognition using 2D-ConvNet. Int J Interact Multimed Artif Intell 6(2):125–135

    Google Scholar 

  49. Wang Q, Chan AB (2018) Cnn + cnn: convolutional decoders for image captioning. arXiv preprint arXiv:1805.09019

  50. Wang M, Song L, Yang X, Luo C (2016) A parallel-fusion RNN-LSTM architecture for image caption generation. In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE, pp 4448–4452

  51. Yang CS, Yang YH (2017) Improved local binary pattern for real scene optical character recognition. Pattern Recognit Lett 100:14–21

    Article  Google Scholar 

  52. Yin XC, Zuo ZY, Tian S, Liu CL (2016) Text detection, tracking and recognition in video: a comprehensive survey. IEEE Trans Image Process 25(6):2752–2773

    Article  MathSciNet  MATH  Google Scholar 

Download references

Funding

There is no funding involved for the work

Author information

Authors and Affiliations

  1. School of Computer Science Engineering, Vellore Institute of Technology, Chennai, India

    S. Venkat Ragavan & A. H. Tarun

  2. School of Electronics Engineering, Vellore Institute of Technology, Chennai, India

    S. Yogeeshwar & B. S. Vishwath Kumar

  3. Centre for Smart Grid Technologies, School of Electronics Engineering, Vellore Institute of Technology, Chennai, India

    S. Sofana Reka

Authors
  1. S. Venkat Ragavan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. H. Tarun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. S. Yogeeshwar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. B. S. Vishwath Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. S. Sofana Reka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Sofana Reka.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Venkat Ragavan, S., Tarun, A.H., Yogeeshwar, S. et al. A realtime portable and accessible aiding system for the blind – a cloud based approach. Multimed Tools Appl 82, 20641–20654 (2023). https://doi.org/10.1007/s11042-023-14419-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14419-9

Keywords

Search

Navigation