Skip to main content
SpringerLink
Log in

Is Tiny Deep Learning the New Deep Learning?

  • Conference paper
  • First Online:
Computational Intelligence and Data Analytics

Part of the book series: Lecture Notes on Data Engineering and Communications Technologies ((LNDECT,volume 142))

Abstract

The computing everywhere paradigm is paving the way for the pervasive diffusion of tiny devices (such as Internet-of-Things or edge computing devices) endowed with intelligent abilities. Achieving this goal requires machine and deep learning solutions to be completely redesigned to fit the severe technological constraints on computation, memory, and power consumption typically characterizing these tiny devices. The aim of this paper is to explore tiny machine learning (TinyML) and introduce tiny deep learning (TinyDL) for the design, development, and deployment of machine and deep learning solutions for (an ecosystem of) tiny devices, hence supporting intelligent and pervasive applications following the computing everywhere paradigm.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    This issue will be addressed in Sect. 5 by introducing on-device learning mechanisms for ML and DL.

  2. 2.

    What is described here can easily be extended to other families of DL solutions.

  3. 3.

    Here, 242 Kb is the memory required to store all of the weights of all of the processing layers, whereas 37 Kb is the memory required to store the intermediate processing results.

References

  1. Alippi C, Disabato S, Roveri M (2018) April) Moving convolutional neural networks to embedded systems: the AlexNet and VGG-16 Case. In: 17th ACM/IEEE International conference on information processing in sensor networks (IPSN). IEEE, Porto, pp 212–223

    Google Scholar 

  2. Alippi C, Roveri M (2017) The (Not) far-away path to smart cyber-physical systems: an information-centric framework. Computer 50(4):38–47

    Article  Google Scholar 

  3. Cai H, Gan C, Zhu L, Han S (2020) Tiny transfer learning: towards memory-efficient on-device learning

    Google Scholar 

  4. Canonaco G, Bergamasco A, Mongelluzzo A, Roveri M (2021) Adaptive federated learning in presence of concept drift. In: 2021 International joint conference on neural networks (IJCNN). IEEE, New York, pp 1–7

    Google Scholar 

  5. Disabato S, Roveri M (2018) Reducing the computation load of convolutional neural networks through gate classification. In: 2018 International joint conference on neural networks (IJCNN). IEEE, New York, pp 1–8

    Google Scholar 

  6. Disabato S, Roveri M (2020) Incremental on-device tiny machine learning. In: Proceedings of the 2nd International workshop on challenges in artificial intelligence and machine learning for internet of things, pp 7–13

    Google Scholar 

  7. Disabato S, Roveri M, Alippi C (2021) Distributed deep convolutional neural networks for the internet-of-things. IEEE Trans Comput

    Google Scholar 

  8. Ditzler G, Roveri M, Alippi C, Polikar R (2015) Learning in nonstationary environments: a survey. IEEE Comput Intell Maga 10(4):12–25

    Article  Google Scholar 

  9. Falbo V, Apicella T, Aurioso D, Danese L, Bellotti F, Berta R, Gloria AD (2019) Analyzing machine learning on mainstream microcontrollers. In: International conference on applications in electronics pervading industry, environment and society. Springer, Berlin, pp 103–108

    Google Scholar 

  10. Frankle J, Carbin M (2018) The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635

  11. Fukushima K (1980) Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern 36:193–202

    Article  Google Scholar 

  12. Gholami A, Kim S, Dong Z, Yao Z, Mahoney MW, Keutzer K (2021) A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630

  13. Higginbotham S (2019) Machine learning on the edge-[internet of everything]. IEEE Spectrum 57(1):20

    Article  Google Scholar 

  14. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  15. Hu D, Krishnamachari B (2020) Fast and accurate streaming CNN inference via communication compression on the edge. In: 2020 IEEE/ACM fifth international conference on internet-of-things design and implementation (IoTDI). IEEE, New York, pp 157–163

    Google Scholar 

  16. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50\(\times \) fewer parameters and <0.5 mb model size. arXiv preprint arXiv:1602.07360

  17. Ivakhnenko AG (1971) Polynomial theory of complex systems. IEEE Trans Syst Man Cybern 4:364–378

    Article  MathSciNet  Google Scholar 

  18. Kephart JO, Chess DM (2003) The vision of autonomic computing. Computer 36(1):41–50

    Article  MathSciNet  Google Scholar 

  19. Khan LU, Saad W, Han Z, Hossain E, Hong CS (2021) Federated learning for internet of things: recent advances, taxonomy, and open challenges. IEEE Commun Surv Tutorials

    Google Scholar 

  20. Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Review 51(3):455–500

    Article  MathSciNet  Google Scholar 

  21. Konečnỳ J, McMahan B, Ramage D (2015) Federated optimization: distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575

  22. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25

    Google Scholar 

  23. Lai L, Suda N, Chandra V (2018) CMSIS-NN: efficient neural network kernels for ARM Cortex-M CPUs. arXiv preprint arXiv:1801.06601

  24. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551

    Article  Google Scholar 

  25. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  26. Liang T, Glossner J, Wang L, Shi S, Zhang X (2021) Pruning and quantization for deep neural network acceleration: a survey. Neurocomputing 461:370–403

    Article  Google Scholar 

  27. Liu J, Tripathi S, Kurup U, Shah M (2020) Pruning algorithms to accelerate convolutional neural networks for edge applications: a survey. arXiv preprint arXiv:2005.04275

  28. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26

    Article  Google Scholar 

  29. McMahan B, Moore E, Ramage D, Hampson S, Arcas BA (2017) Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. PMLR, pp 1273–1282

    Google Scholar 

  30. Nagel M, Fournarakis M, Amjad RA, Bondarenko Y, van Baalen M, Blankevoort T (2021) A white paper on neural network quantization. arXiv preprint arXiv:2106.08295

  31. Pouyanfar S, Sadiq S, Yan Y, Tian H, Tao Y, Reyes MP, Shyu ML, Chen SC, Iyengar SS (2018) A survey on deep learning: algorithms, techniques, and applications. ACM Comput Surveys (CSUR) 51(5):1–36

    Article  Google Scholar 

  32. Ren H, Anicic D, Runkler TA (2021) Tinyol: Tinyml with online-learning on microcontrollers. In: 2021 International joint conference on neural networks (IJCNN). IEEE, New York, pp 1–8

    Google Scholar 

  33. Reuther A, Michaleas P, Jones M, Gadepally V, Samsi S, Kepner J (2019) Survey and benchmarking of machine learning accelerators. In: 2019 IEEE high performance extreme computing conference (HPEC). IEEE, New York, pp 1–9

    Google Scholar 

  34. Sanchez-Iborra R, Skarmeta AF (2020) TinyMLL-enabled frugal smart objects: challenges and opportunities. IEEE Circuits Syst Maga 20(3):4–18

    Article  Google Scholar 

  35. Scardapane S, Scarpiniti M, Baccarelli E, Uncini A (2020) Why should we add early exits to neural networks? Cogn Comput 12(5):954–966

    Article  Google Scholar 

  36. Shakarami A, Ghobaei-Arani M, Shahidinejad A (2020) A survey on the computation offloading approaches in mobile edge computing: a machine learning-based perspective. Comput Networks 182:107496

    Article  Google Scholar 

  37. STMicroelectronics: X-cube-ai (2021) https://www.st.com/en/embedded-software/x-cube-ai.html

  38. Sze V, Chen YH, Yang TJ, Emer JS (2017) Efficient processing of deep neural networks: a tutorial and survey. Proc IEEE 105(12):2295–2329

    Article  Google Scholar 

  39. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

    Google Scholar 

  40. Teerapittayanon S, McDanel B, Kung HT (2017) Distributed deep neural networks over the cloud, the edge and end devices. In: 2017 IEEE 37th international conference on distributed computing systems (ICDCS). IEEE, New York, pp 328–339

    Google Scholar 

  41. Verbraeken J, Wolting M, Katzy J, Kloppenburg J, Verbelen T, Rellermeyer JS (2020) A survey on distributed machine learning. ACM Comput Surveys (CSUR) 53(2):1–33

    Article  Google Scholar 

  42. Warden P, Situnayake D (2019) TinyML. O’Reilly Media (Incorporated)

    Google Scholar 

  43. Xu R, Wunsch D (2005) Survey of clustering algorithms. IEEE Trans Neural Networks 16(3):645–678

    Article  Google Scholar 

  44. Yang Q, Liu Y, Chen T, Tong Y (2019) Federated machine learning: concept and applications. ACM Trans Intell Syst Technol (TIST) 10(2):1–19

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dipartimento di Elettronica, Informazione e Bioingegneria Politecnico di Milano, Milan, Italy

    Manuel Roveri

Authors
  1. Manuel Roveri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manuel Roveri .

Editor information

Editors and Affiliations

  1. Director of the Cloud Computing and Distributed Systems (CLOUDS) Laboratory, University of Melbourne, Melbourne, VIC, Australia

    Rajkumar Buyya

  2. Computer Science School (FI), Technical University of Madrid, Madrid, Spain

    Susanna Munoz Hernandez

  3. Department of Information Technology, Vasavi College of Engineering, Hyderabad, India

    Ram Mohan Rao Kovvur

  4. Department of Information Technology, Vasavi College of Engineering, Hyderabad, India

    T. Hitendra Sarma

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Roveri, M. (2023). Is Tiny Deep Learning the New Deep Learning?. In: Buyya, R., Hernandez, S.M., Kovvur, R.M.R., Sarma, T.H. (eds) Computational Intelligence and Data Analytics. Lecture Notes on Data Engineering and Communications Technologies, vol 142. Springer, Singapore. https://doi.org/10.1007/978-981-19-3391-2_2

Download citation

Publish with us

Policies and ethics

Search

Navigation