UMFS: An efficient user-space file system for non-volatile memory
Introduction
Non-volatile memory (NVM) [1], [2], [3], [4], [5] is expected to be a mainstream storage media in embedded systems for its advanced features, such as low-power consumption, high density, near-DRAM speed, and byte-addressability. Conventional disk-oriented file systems [6] cannot fully exploit the benefits of NVM for their long I/O stacks have large software overhead [7]. To this end, in-memory file systems [8], [9], [10], [11], [12], [13], [14], [15], [16] are proposed to achieve high-performance file accesses taking advantages of the byte addressability and high speed of NVM.
Nevertheless, existing in-memory file systems mostly access the file data in the kernel space. The mode changing and software stacks, such as the library and Virtual File System (VFS) layer, brings large costs for file accesses [17], [18]. Previous works have proposed several approaches to enable user-space file accesses, such as memory-mapped I/O (mmap) [12], [15], [19], Aerie [20], [21], and FUSE [22], [23]. Existing user-space file access approaches, however, are add-ons of kernel-level file systems. They have large overhead in supporting the user-space file accesses. For example, mmap needs to build an additional mapping table aside from the index structure of the file.
This paper proposes a new design of genuine user-space file system to achieve high-performance file accesses. A new file system, User-space in-Memory File System (UMFS), is implemented in Linux based on the proposed design. The file system operations of UMFS are re-designed to avoid large overhead of traversing the kernel stacks in file data accesses. To coexist with existing file systems and programs in Linux, a POSIX-compatible library libumfs is implemented to support the operations in UMFS. The files in UMFS can be exposed to user space in a constant time independent from the file size. File accesses in UMFS utilizes the contiguous virtual address space of user process and the address translation hardware (e.g., MMU) to achieve high throughput. The proposed UMFS is designed for the persistent memories that can be connected to the memory bus and directly accessed by CPU. Thus, UMFS can be built on NVDIMM [24] and emerging NVMs [1], [2], [3].
UMFS provides data consistency guarantees for user-space file data by a new design of user-space journaling. The user-space journaling records the updates in a new structure, “uLog”, in user space. uLog is located in NVM but mapped to the process virtual address space when the file is opened. We avoid execution in the kernel as much as possible by moving the operations that do not require hardware privileges to user space. In the cases of system failure, UMFS can be recovered by scanning through the uLogs during the next system reboot.
We evaluate UMFS with microbenchmarks and macrobenchmarks, including flexible I/O (FIO) [25] and Filebench [26]. UMFS is compared with typical in-memory file systems, including EXT4-DAX [11], NOVA [13], and SIMFS [10]. Experimental results show that UMFS significantly outperforms existing file systems for sequential/random read and write. For example, for random read by eight threads, UMFS is faster than EXT4-DAX, NOVA, and SIMFS by 3.2 times, 3.1 times, and 2.1 times on average, respectively. For random writes, UMFS provides 1.5 times, 2.8 times, and 1.2 times higher throughput than EXT4-DAX, NOVA, and SIMFS on average, respectively. To the authors’ knowledge, UMFS achieves the best known result for file systems in the literature.
The main contributions of this paper are as follows:
- •
We measure and analyze the costs of the software stacks of the kernel and the existing user-space file accesses.
- •
We propose a new approach to enable user-space file accesses while maintaining the semantics of existing file systems and applications.
- •
We present a consistency mechanism for user-space file data.
- •
We implement a real User-space in-Memory File System (UMFS) in Linux system. Extensive experiments show that UMFS outperforms existing in-memory file systems, including EXT4-DAX, NOVA, and SIMFS.
The rest of the paper is organized as follows. The background and motivation are introduced in Section 2. Then, we present the design overview in Section 3. Section 4 describes the implementation details of UMFS. The experimental results are presented in Section 5. Finally, Section 6 concludes the paper.
Section snippets
Background and motivation
We aim to design a high-performance file system that can read/write file data using the virtual address space of user process. This section first reviews the emerging NVM technologies and exiting file systems for NVM. Then, we show the cost of kernel stacks for file accesses. Finally, we discuss existing approaches for user space file accesses and conclude the challenges.
UMFS design
We design and implement a User-space in-Memory File System (UMFS) with the goal of providing fast and consistent file data accesses directly from user mode without modifying the existing applications. Specifically, we try to tackle the following targets in this work:
- •
The overhead in supporting user-space file accesses should be minimized.
- •
The existing semantics of file systems should be preserved from the view of applications.
- •
The consistency of the file data in user space should be guaranteed.
An
Implementation
UMFS is implemented on Linux 3.11.8 for x86-64 processors. In this section, we present the implementation details of the file system layout, virtual space management, user-space data consistency, and file system operations.
Experiments
Our evaluation answer the following three main questions about UMFS:
- •
How does UMFS perform against existing in-memory file systems?
- •
What is the overhead of uLog?
- •
What kind of operations gain most benefit from UMFS?
Conclusion
In recent years, many in-memory file systems have been proposed to manage NVM for its a promising mainstream storage media. We observed that existing in-memory file systems are designed in kernel space. They have large overhead for traversing the software layers of kernel. In this paper, we presented a design of user-space in-memory file system, UMFS. The files in UMFS are exposed to user-space for fast file accesses. We also proposed a data consistency mechanism to ensure consistent updates in
Acknowledgment
This work is partially supported by the National 863 Program of China (grant no. 2015AA015304), the National Nature Science Foundation of China (grant no. 61472052), and China Postdoctoral Science Foundation (grant no. 2017M620412).
Xianzhang Chen received Ph.D. degree from the College of Computer Science, Chongqing University, China in 2017. He obtains B.S. and M.S. degrees from the College of Computer Science and Engineering, Southeast University, China. He is currently a research assistant in Chongqing University. His research interests include non-volatile memory, embedded systems, and operating systems. He received IEEE Transactions on Computers “Editor’s pick of the year 2016”, IEEE ICCD Best Paper Award (2017),
References (47)
- et al.
Towards greener data centers with storage class memory
Future Gener. Comput. Syst.
(2013) - et al.
Fine grained, direct access file system support for storage class memory
J. Syst. Archit.
(2017) - et al.
Efficient memory-mapped i/o on fast storage device
ACM Trans. Storage
(2016) - 2015....
- et al.
Storage-class memory: the next storage system technology
Ibm J. Res. Develop.
(2008) - et al.
Design of a large-scale storage-class rram system
Proceedings of the 27th International ACM Conference on International Conference on Supercomputing (ICS)
(2013) - et al.
Object-based scm: An efficient interface for storage class memories
IEEE 27th Symposium on Mass Storage Systems and Technologies (MSST)
(2011) - et al.
The new ext4 filesystem: current status and future plans
Proceedings of Linux Symposium
(2008) - et al.
Enabling write-reduction strategy for journaling file systems over byte-addressable nvram
DAC
(2017) - et al.
Scmfs: A file system for storage class memory and its extensions
ACM Trans. Storage
(2013)
A new design of in-memory file system based on file virtual address framework
IEEE Trans. Comput.
System software for persistent memory
Proceedings of the Ninth European Conference on Computer Systems (EuroSys)
Nova: A log-structured file system for hybrid volatile/non-volatile main memories
Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST)
Shortcut-jfs: A write efficient journaling file system for phase change memory
IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST)
A high performance file system for non-volatile main memory
Proceedings of the Eleventh European Conference on Computer Systems (EuroSys)
A unified framework for designing high performance in-memory and hybrid memory file systems
J. Syst. Archit.
Towards high-performance application-level storage management
Proceedings of the 6th USENIX Conference on Hot Topics in Storage and File Systems (HotStorage)
Optimizing file systems for fast storage devices
Proceedings of the 8th ACM International Systems and Storage Conference (SYSTOR)
Storage-class memory needs flexible interfaces
Proceedings of the 4th Asia-Pacific Workshop on Systems
Aerie: Flexible file-system interfaces to storage-class memory
Proceedings of the Ninth European Conference on Computer Systems (EuroSys)
Terra incognita: on the practicality of user-space file systems
Usenix Conference on Hot Topics in Storage and File Systems (HotStorage)
Cited by (8)
Delay-based I/O request scheduling in SSDs
2019, Journal of Systems ArchitectureCitation Excerpt :Flash-based SSDs have been gaining ever-increasing prevalence in electronic devices due to its attractive features, such as low standby power, high density, and shock resistance [1–14].
LLLFS: A Low-latency Library File System for Persistent Memory
2022, Proceedings - 2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing, UCC 2022Simurgh: A fully decentralized and secure nvmm user space file system
2021, International Conference for High Performance Computing, Networking, Storage and Analysis, SCA scalable file management protocol for high speed storage system
2020, Proceedings of 2020 IEEE 3rd International Conference of Safe Production and Informatization, IICSPI 2020
Xianzhang Chen received Ph.D. degree from the College of Computer Science, Chongqing University, China in 2017. He obtains B.S. and M.S. degrees from the College of Computer Science and Engineering, Southeast University, China. He is currently a research assistant in Chongqing University. His research interests include non-volatile memory, embedded systems, and operating systems. He received IEEE Transactions on Computers “Editor’s pick of the year 2016”, IEEE ICCD Best Paper Award (2017), NVMSA Best Paper Award (2015), etc.
Edwin H.-M. Sha received Ph.D. degree from the Department of Computer Science, Princeton University, USA in 1992. From August 1992 to August 2000, he was with the Department of Computer Science and Engineering at University of Notre Dame, USA. Since 2000, he has been a tenured full professor at the University of Texas at Dallas. He has published more than 380 research papers in refereed conferences and journals. His work has been cited over 3300 times. He received Teaching Award, Microsoft Trustworthy Computing Curriculum Award, NSF CAREER Award, NSFC Overseas Distinguished Young Scholar Award, etc.
Qingfeng Zhuge received her Ph.D. from the Department of Computer Science at the University of Texas at Dallas in 2003. She obtained her B.S. and M.S. degrees in Electronics Engineering from Fudan University, Shanghai, China. She is currently a professor at East China Normal University, Shanghai, China. She received Best Ph.D. Dissertation Award in 2003. She has published more than 90 research articles in premier journals and conferences. Her research interests include parallel architectures, embedded systems, supply-chain management, real-time systems, optimization algorithms, compilers, and scheduling.
Ting Wu received the B.S. degrees in computer science and technology from the China University of and Technology Mining, Xuzhou, China, in June 2013, and received M.S. degrees in computer system architecture from the Lanzhou University, Lanzhou, China, in June 2016. He is currently a Ph.D. student in the College of Computer Science at Chongqing University, China. His advisor is Dr. Edwin H.-M Sha. His research interests include non-volatile memory, virtual machine, and operating system.
Weiwen Jiang received the B.E. degree from the Department of Computer Science, Nanjing Agricultural University, Nanjing, China, in 2008. He is currently working towards the Ph.D. degree in the Department of Computer Science at Chongqing University and Department of Electrical and Computer Engineering, University of Pittsburgh. His research interests include self-timed system optimization, embedded systems, and non-volatile memory.
Xiaoping Zeng received the B.E., M.S., and Ph.D. degrees in Electrical Engineering from Chongqing University, Chongqing, China in 1982, 1987, and 1996, respectively. He is now a professor and Ph.D. supervisor at the College of Communication Engineering, Chongqing University, China. His research interests include system design, aeronautical information network, multiple-input multiple-output systems and vehicle-to-vehicle communications.
Lin Wu received the B.S. degree from the College of Computer Science and Engineering, Southwest University of Science and Technology, Sichuan, China, in 2013. He is currently pursuing the Ph.D. degree under the supervision of Dr. Q. Zhuge with Chongqing University. His current research interests include distributed systems, file systems, and operating systems.