Abstract
Modern embedded and cyber-physical applications consist of critical and non-critical tasks co-located on multiprocessor systems on chip (MPSoCs). Co-location of tasks results in contention for shared resources, resulting in interference on interconnect, processing units, storage, etc. Hence, machine learning-based resource managers must operate even non-critical tasks within certain constraints to ensure proper execution of critical tasks. In this paper we demonstrate and evaluate countermeasures based on backup policies to enhance rule-based reinforcement learning to enforce constraints. Detailed experiments reveal the CPUs’ performance degradation caused by different designs, as well as their effectiveness in preventing constraint violations. Further, we exploit the interpretability of our approach to further improve the resource manager’s operation by adding designers’ experience into the rule set.
Funding source: DFG
Award Identifier / Grant number: HE4584/7-2
About the authors
Florian Maurer received his BSc and MSc degrees from the Technical University of Munich, Munich, Germany, in 2016 and 2018, respectively. He is currently a PhD candidate at the Technical University of Munich, Munich, Germany and working toward the PhD degree in electrical and computer engineering. His research interests include self-aware and self-adaptive multi- and many-core systems.
Moritz Thoma received his BEn degree from the Baden-Wuerttemberg Cooperative State University Stuttgart, Germany and his MSc degree from the Technical University of Munich, Germany. He is currently a PhD candidate at BMW Group Munich, Germany pursuing a PhD degree in electrical and computer engineering. His research interests include efficient AI deployment and efficient MLOps.
Anmol Surhonne received the BE degree from the PES Institute of Technology, and the MSc degree from Nanyang Technological University and Technical University of Munich. He is currently working towards a PhD degree in electrical and computer engineering at the Technical University of Munich. His research interests include self aware multi/many core systems and machine learning.
Prof. Bryan Donyanavard is an Assistant Professor in the Computer Science department at San Diego State University. Prior to that, he was a researcher in the IoT and Cyber-physical Systems group at Ericsson in Stockholm. His research focuses on runtime management of resource-constrained systems in software and architecture. He received his Ph.D. in Computer Science from UC Irvine in 2019. He has previously worked as a software engineer at Sun Microsystems and Google, and spent time as a visiting researcher at TU Munich.
Prof. Dr. sc.techn. Andreas Herkersdorf is a professor and the Head of Department Computer Engineering at Technical University of Munich (TUM). He received a Dr. degree from ETH Zurich, Switzerland, in 1991. Between 1988 and 2003, he has been in technical and management positions with the IBM Research Laboratory in Rüschlikon, Switzerland. Since 2003, Dr. Herkersdorf is the Chair Professor of Integrated Systems at TUM. He is a senior member of the IEEE, member of National Academy of Science and Engineering (acatech) and serves as editor for Springer and De Gruyter journals for design automation and information technology. His research interests include application-specific multi-processor architectures, IP network processing, Network on Chip and self-adaptive fault-tolerant computing.
-
Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
-
Research funding: Funded by DFG (grant number HE4584/7-2).
-
Conflict of interest statement: The authors declare no conflicts of interest regarding this article.
References
[1] J.-P. Lozi, B. Lepers, J. Funston, F. Gaud, V. Quéma, and A. Fedorova, “The linux scheduler: a decade of wasted cores,” in Proceedings of the Eleventh European Conference on Computer Systems, EuroSys ’16, New York, NY, USA, Association for Computing Machinery, 2016.10.1145/2901318.2901326Search in Google Scholar
[2] R. H. Dennard, F. H. Gaensslen, H.-N. Yu, V. L. Rideout, E. Bassous, and A. R. Leblanc, “Design of ion-implanted mosfet’s with very small physical dimensions,” IEEE J. Solid State Circ., vol. 9, no. 5, pp. 256–268, 1974. https://doi.org/10.1109/JSSC.1974.1050511.Search in Google Scholar
[3] L. Costero, A. Iranfar, M. Zapater, F. D. Igual, K. Olcoz, and D. Atienza, “Mamut: multi-agent reinforcement learning for efficient real-time multi-user video transcoding,” in 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2019, pp. 558–563.10.23919/DATE.2019.8715256Search in Google Scholar
[4] N. Mishra, C. Imes, J. D. Lafferty, and H. Hoffmann, “Caloree: learning control for predictable latency and low energy,” in Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’18, New York, NY, USA, Association for Computing Machinery, 2018, pp. 184–198.10.1145/3173162.3173184Search in Google Scholar
[5] B. Donyanavard, T. Mück, A. M. Rahmani, et al.., “Sosa: self-optimizing learning with self-adaptive control for hierarchical system-on-chip management,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’52, New York, NY, USA, Association for Computing Machinery, 2019, pp. 685–698.10.1145/3352460.3358312Search in Google Scholar
[6] J. Zeppenfeld, A. Bouajila, W. Stechele, and A. Herkersdorf, “Learning classifier tables for autonomic systems on chip,” in INFORMATIK 2008. Beherrschbare Systeme - dank Informatik. Band 2, H.-G. Hegering, A. Lehmann, H. J. Ohlbach, and C. Scheideler, Eds., Bonn, Gesellschaft für Informatik e. V, 2008, pp. 771–778.Search in Google Scholar
[7] M. Seo and F. Kurdahi, “Efficient tracing methodology using automata processor,” ACM Trans. Embed. Comput. Syst., vol. 18, no. 5s, pp. 1–18, 2019, https://doi.org/10.1145/3358200.Search in Google Scholar
[8] E. A. Rambo, B. Donyanavard, M. Seo, et al.., “The self-aware information processing factory paradigm for mixed-critical multiprocessing,” IEEE Trans. Emerg. Topics Comput., vol. 10, no. 1, pp. 250–266, 2022, https://doi.org/10.1109/tetc.2020.3011663.Search in Google Scholar
[9] A. Ganapathi, K. Datta, A. Fox, and D. Patterson, “A case for machine learning to optimize multicore performance,” in First USENIX Workshop on Hot Topics in Parallelism (HotPar’09), 2009.Search in Google Scholar
[10] L. Nardi, D. Koeplinger, and K. Olukotun, “Practical design space exploration,” in 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), 2019, pp. 347–358.10.1109/MASCOTS.2019.00045Search in Google Scholar
[11] J. Zuckerman, D. Giri, J. Kwon, P. Mantovani, and L. P. Carloni, “Cohmeleon: learning-based orchestration of accelerator coherence in heterogeneous socs,” in MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’21, New York, NY, USA, Association for Computing Machinery, 2021, pp. 350–365.10.1145/3466752.3480065Search in Google Scholar
[12] P. Hernández, L. Costero, K. Olcoz, and F. D. Igual, “Applying game-learning environments to power capping scenarios via reinforcement learning,” in Cloud Computing, Big Data & Emerging Topics, E. Rucci, M. Naiouf, F. Chichizola, L. De Giusti, and A. De Giusti, Eds., Cham, Springer International Publishing, 2022, pp. 91–106.10.1007/978-3-031-14599-5_7Search in Google Scholar
[13] S. (Justin) Lu, R. Tessier, and W. Burleson, “Reinforcement learning for thermal-aware many-core task allocation,” in Proceedings of the 25th Edition on Great Lakes Symposium on VLSI, GLSVLSI ’15, New York, NY, USA, Association for Computing Machinery, 2015, pp. 379–384.Search in Google Scholar
[14] F. J. Gomez, D. Burger, and R. Miikkulainen, “A neuro-evolution method for dynamic resource allocation on a chip multiprocessor,” in IJCNN’01. International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222), vol. 4, 2001, pp. 2355–2360.Search in Google Scholar
[15] B. Maity, B. Donyanavard, A. Surhonne, A. Rahmani, A. Herkersdorf, and N. Dutt, “Seams: self-optimizing runtime manager for approximate memory hierarchies,” ACM Trans. Embed. Comput. Syst., vol. 20, no. 5, pp. jul–26, 2021, https://doi.org/10.1145/3466875.Search in Google Scholar
[16] R. Jain, P. R. Panda, and S. Subramoney, “Machine learned machines: adaptive co-optimization of caches, cores, and on-chip network,” in 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2016, pp. 253–256.10.3850/9783981537079_0083Search in Google Scholar
[17] L. Costero, A. Iranfar, M. Zapater, F. D. Igual, K. Olcoz, and D. Atienza, “Resource management for power-constrained hevc transcoding using reinforcement learning,” IEEE Trans. Parallel Distr. Syst., vol. 31, no. 12, pp. 2834–2850, 2020, https://doi.org/10.1109/tpds.2020.3004735.Search in Google Scholar
[18] LLVM, LLVM “test-Suite” Repository, 2016, Available at: https://github.com/llvm/llvm-test-suite/tree/main/SingleSource/Benchmarks/St anford.Search in Google Scholar
[19] Frontgrade Gaisler AB, GRLIB IP Library User’s Manual, 2023, Available at: https://gaisler.com/products/grlib/grlib.pdf.Search in Google Scholar
[20] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA, The MIT Press, 2020.Search in Google Scholar
[21] N. Sperling, A. Bendrick, D. Stöhrmann, et al.., “Information processing factory 2.0 - self-awareness for autonomous collaborative systems,” in 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2023, pp. 1–6.10.23919/DATE56975.2023.10137006Search in Google Scholar
[22] J. Eschmann, Reward Function Design in Reinforcement Learning, Cham, Springer International Publishing, 2021, pp. 25–33.10.1007/978-3-030-41188-6_3Search in Google Scholar
[23] T. Back, Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms, New York, NY, Oxford University Press, 1996.10.1093/oso/9780195099713.001.0001Search in Google Scholar
[24] J. H. Holland, Adaptation in Natural and Artificial Systems, Cambridge, MA, The MIT Press, 1992.10.7551/mitpress/1090.001.0001Search in Google Scholar
[25] S. Mahadevan and J. Connell, “Automatic programming of behavior-based robots using reinforcement learning,” in Proceedings of the Ninth National Conference on Artificial Intelligence - Volume 2, AAAI’91, AAAI Press, 1991, pp. 768–773.Search in Google Scholar
[26] R. S. Sutton, “Generalization in reinforcement learning: successful examples using sparse coarse coding,” in Advances in Neural Information Processing Systems, vol. 8, 1995.Search in Google Scholar
[27] F. Maurer, B. Donyanavard, A. M. Rahmani, N. Dutt, and A. Herkersdorf, “Emergent control of mpsoc operation by a hierarchical supervisor/reinforcement learning approach,” in 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2020, pp. 1562–1567.10.23919/DATE48585.2020.9116574Search in Google Scholar
[28] J. Garcia and F. Fernández, “Safe exploration of state and action spaces in reinforcement learning,” J. Artif. Intell. Res., vol. 45, pp. 515–564, 2012, https://doi.org/10.1613/jair.3761.Search in Google Scholar
© 2023 Walter de Gruyter GmbH, Berlin/Boston
