Kento Kimura
ABSTRACT
It is difficult to completely avoid system failures in recent large-scale and complex systems. Therefore, it is important to detect system faults rapidly and accurately and recover from them. Fault recovery is categorized into external one from remote hosts and internal one with processes or the operating system (OS) inside a target system. However, both methods are subject to system faults. If fault recovery fails, a hardware reset is required and can lead to losing system data and states. This paper proposes GPUfas for recovering from system faults by indirectly controlling OS behavior from a GPU, which is not easily affected by system faults. GPUfas attempts fault recovery by rewriting OS data in main memory and leveraging the capabilities of the OS itself. For example, it can mimic signal sending and process scheduling to force termination of the processes that consume excessive resources. It can also mimic unlocking to recover from some kind of deadlock. We have implemented GPUfas using the Linux kernel, CUDA, and LLVM to enable a GPU to rewrite OS data transparently. Then, we confirmed the effectiveness and efficiency of fault recovery by GPUfas.
- Amazon Web Services, Inc. 2020. Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region. https://aws.amazon.com/message/11201/.Google Scholar
- A. Beekhof. [n.d.]. Pacemaker. https://clusterlabs.org/pacemaker/.Google Scholar
- A. Bohra, I. Neamtiu, P. Gallard, F. Sultan, and L. Iftode. 2004. Remote Repair of Operating System State Using Backdoors. In Proceedings of the 1st International Conference on Autonomic Computing. 256--263.Google Scholar
- P. Chen, W. Ng, S. Chandra, C. Aycock, G. Rajamani, and D. Lowell. 1996. The Rio File Cache: Surviving Operating System Crashes. In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems. 74--83.Google Scholar
- F. David, J. Carlyle, and H. Campbell. 2007. Exploring Recovery from Operating System Lockups. In Proceedings of the 2007 USENIX Annual Technical Conference. 351--356.Google Scholar
- A. Depoutovitch and M. Stumm. 2010. Otherworld: Giving Applications a Chance to Survive OS Kernel Crashes. In Proceedings of the 5th European Conference on Computer Systems. 181--194.Google Scholar
- Y. Fu and Z. Lin. 2013. EXTERIOR: Using a Dual-VM Based External Shell for Guest-OS Introspection, Configuration, and Recovery. In Proceedings of the 9th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments. 97--110.Google Scholar
- Intel, Hewlett-Packard, NEC, and Dell. 2004. Intelligent Platform Management Specification Second Generation v2.0.Google Scholar
- J. Leners, H. Wu, W. Hung, M. Aguilera, and M. Walfish. 2011. Detecting Failures in Distributed Systems with the Falcon Spy Network. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles. 279--294.Google Scholar
- NVIDIA Corporation. 2018. CUDA Toolkit Documentation v10.0.130. https://docs.nvidia.com/cuda/archive/10.0/.Google Scholar
- NVIDIA Corporation. 2022. Developing a Linux Kernel Module Using RDMA for GPUDirect. Technical Report TB-06712-001 v11.7. NVIDIA.Google Scholar
- Y. Ozaki, S. Kanamoto, H. Yamamoto, and K. Kourai. 2019. Detecting System Failures with GPUs and LLVM. In Proceedings of the 10th ACM SIGOPS Asia-Pacific Workshop on Systems. 47--53.Google Scholar
- A. Taherin, T. Patel, G. Georgakoudis, I. Laguna, and D. Tiwari. 2021. Examining Failures and Repairs on Supercomputers with Multi-GPU Compute Nodes. In Proceedings of the 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 305--313.Google Scholar
- The LLVM Foundation. [n.d.]. The LLVM Compiler Infrastructure. https://llvm.org/.Google Scholar
- Tokyo Stock Exchange, Inc. 2020. Report on the Cash Equity Trading System Failure on Oct. 1. https://www.jpx.co.jp/english/corporate/news/news-releases/0060/20201019-01.html.Google Scholar
- K. Yamakita, H. Yamada, and K. Kono. 2011. Phase-based Reboot: Reusing Operating System Execution Phases for Cheap Reboot-based Recovery. In Proceedings of the 41st IEEE/IFIP International Conference on Dependable Systems and Networks. 168--180.Google Scholar
- T. Yoshimura, H. Yamada, and K. Kono. 2012. Is Linux Kernel Oops Useful or Not?. In Proceedings of the 8th USENIX Workshop on Hot Topics in System Dependability.Google Scholar
- Y. Zhu, Y. Li, J. Xue, T. Tan, J. Shi, Y. Shen, and C. Ma. 2012. What is System Hang and How to Handle it. In Proceedings of the 23rd IEEE International Symposium on Software Reliability Engineering. 141--150.Google Scholar
Index Terms
- GPU-based first aid for system faults
Recommendations
Reliability-aware performance model for optimal GPU-enabled cluster environment
Given that the reliability of a very large-scaled system is inversely related to the number of computing elements, fault tolerance has become a major concern in high performance computing including the most recent deployments with graphic processing ...
Algorithm-Based Fault Location and Recovery for Matrix Computations on Multiprocessor Systems
Algorithm-based fault-tolerance (ABFT) is an inexpensive method of incorporating fault-tolerance into existing applications. Applications are modified to operate on encoded data and produce encoded results which may then be checked for correctness. An ...
Warped-slicer: efficient intra-SM slicing through dynamic resource partitioning for GPU multiprogramming
ISCA '16: Proceedings of the 43rd International Symposium on Computer ArchitectureAs technology scales, GPUs are forecasted to incorporate an ever-increasing amount of computing resources to support thread-level parallelism. But even with the best effort, exposing massive thread-level parallelism from a single GPU kernel, ...
Comments