Daniel Fryer
Ashvin Goel
Abstract
File system bugs that corrupt metadata on disk are insidious. Existing reliability methods, such as checksums, redundancy, or transactional updates, merely ensure that the corruption is reliably preserved. Typical workarounds, based on using backups or repairing the file system, are painfully slow. Worse, the recovery may result in further corruption.
We present Recon, a system that protects file system metadata from buggy file system operations. Our approach leverages file systems that provide crash consistency using transactional updates. We define declarative statements called consistency invariants for a file system. These invariants must be satisfied by each transaction being committed to disk to preserve file system integrity. Recon checks these invariants at commit, thereby minimizing the damage caused by buggy file systems.
The major challenges to this approach are specifying invariants and interpreting file system behavior correctly without relying on the file system code. Recon provides a framework for file-system specific metadata interpretation and invariant checking. We show the feasibility of interpreting metadata and writing consistency invariants for the Linux ext3 file system using this framework. Recon can detect random as well as targeted file-system corruption at runtime as effectively as the offline e2fsck file-system checker, with low overhead.
- Arnold, J. and Kaashoek, M. F. 2009. Ksplice: automatic rebootless kernel updates. In Proceedings of the ACM SIGOPS European Conference on Computer Systems (EuroSys'09). 187--198. Google Scholar
Digital Library
- Bairavasundaram, L. N., Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., Goodson, G. R., and Schroeder, B. 2008. An analysis of data corruption in the storage stack. ACM Trans. Storage 4, 3, 1--28. Google ScholarDigital Library
- Bairavasundaram, L. N., Sundararaman, S., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2009. Tolerating file-system mistakes with EnvyFS. In Proceedings of the USENIX Technical Conference. 87--100. Google ScholarDigital Library
- Bonwick, J. and Moore, B. 2004. ZFS: the last word in file systems. http://opensolaris.org/os/community/zfs/docs/zfs_last.pdf.Google Scholar
- Btrfs. 2012. Wikipedia page. http://btrfs.wiki.kernel.org.Google Scholar
- Chen, F. and Roşu, G. 2007. Mop: an efficient and generic runtime verification framework. In Proceedings of the ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA'07). 569--588. Google ScholarDigital Library
- Custer, H. 1994. Inside the Windows NT File System. Microsoft Press. Google ScholarDigital Library
- Danial, A. 2012. CLOC: count lines of code. http://cloc.sourceforge.net/.Google Scholar
- Demsky, B. and Rinard, M. C. 2006. Goal-directed reasoning for specification-based data structure repair. IEEE Trans. Softw. Engin. 32, 12, 931--951. Google ScholarDigital Library
- Ganger, G. R., McKusick, M. K., Soules, C. A. N., and Patt, Y. N. 2000. Soft updates: a solution to the metadata update problem in file systems. ACM Trans. Comput. Syst. 18, 2, 127--153. Google ScholarDigital Library
- Gunawi, H. S., Prabhakaran, V., Krishnan, S., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2007. Improving file system reliability with I/O shepherding. In Proceedings of the Symposium on Operating Systems Principles (SOSP'07). 293--306. Google ScholarDigital Library
- Gunawi, H. S., Rajimwale, A., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2008. SQCK: a declarative file system checker. In Proceedings of the Operating Systems Design and Implementation (OSDI'08). 131--146. Google ScholarDigital Library
- Hagmann, R. 1987. Reimplementing the Cedar file system using logging and group commit. In Proceedings of the Symposium on Operating Systems Principles (SOSP'87). 155--162. Google ScholarDigital Library
- Henson, V., van de Ven, A., Gud, A., and Brown, Z. 2006. Chunkfs: using divide-and-conquer to improve file system reliability and repair. In Proceedings of the Workshop on Hot Topics in System Dependability (HotDep'06). Google ScholarDigital Library
- Hitz, D., Lau, J., and Malcolm, M. 1994. File system design for an NFS file server appliance. In Proceedings of the USENIX Technical Conference. Google ScholarDigital Library
- Iptables. 2012. Wikipedia page. http://en.wikipedia.org/wiki/Iptables.Google Scholar
- Kaashoek, F. M., Engler, D. R., Ganger, G. R., Briceno, H. M., Hunt, R., Mazikres, D., Pinckney, T., Grimm, R., Jannotti, J., and Mackenzie, K. 1997. Application Performance and Flexibility on Exokernel Systems. In Proceedings of the Symposium on Operating Systems Principles (SOSP'97). 52--65. Google ScholarDigital Library
- Meyer, D. T. and Bolosky, W. J. 2010. A study of practical deduplication. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST'10). 1--14. Google ScholarDigital Library
- Perkins, J. H., Kim, S., Larsen, S., Amarasinghe, S. P., Bachrach, J., Carbin, M., et al. 2009. Automatically patching errors in deployed software. In Proceedings of the Symposium on Operating Systems Principles (SOSP'09). 87--102. Google ScholarDigital Library
- Prabhakaran, V., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2005a. Model-based failure analysis of journaling file systems. In Proceedings of the IEEE Dependable Systems and Networks (DSN'05). 802--811. Google ScholarDigital Library
- Prabhakaran, V., Bairavasundaram, L. N., Agrawal, N., Gunawi, H. S., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2005b. Iron file systems. In Proceedings of the Symposium on Operating Systems Principles (SOSP'05). 206--220. Google ScholarDigital Library
- Rubio-González, Cindy, Gunawi, S., H., Liblit, B., Arpaci-Dusseau, H., R., Arpaci-Dusseau, and C., A. 2009. Error propagation analysis for file systems. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'09). 270--280. Google ScholarDigital Library
- Sivathanu, G., Sundararaman, S., and Zadok, E. 2006. Type-safe disks. In Proceedings of the Operating Systems Design and Implementation (OSDI'06). 15--28. Google ScholarDigital Library
- Sivathanu, M., Prabhakaran, V., Popovici, F. I., Denehy, T. E., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2003. Semantically-smart disk systems. In USENIX Conference on File and Storage Technologies (FAST'03). 73--88. Google ScholarDigital Library
- Sokolsky, O., Sammapun, U., Lee, I., and Kim, J. 2006. Run-time checking of dynamic properties. Electron. Notes Theor. Comput. Sci. 144, 91--108. Google ScholarDigital Library
- Sundararaman, S., Subramanian, S., Rajimwale, A., Arpaci-dusseau, A. C., Arpaci-dusseau, R. H., and Swift, M. M. 2010. Membrane: Operating system support for restartable file systems. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST'10). 281--294. Google ScholarDigital Library
- Sweeney, A., Doucette, D., Hu, W., Anderson, C., Nishimoto, M., and Peck, G. 1996. Scalability in the XFS file system. In Proceedings of the USENIX Technical Conference. 1--14. Google ScholarDigital Library
- Tweedie, S. C. 1998. Journalling the ext2fs filesystem. In Proceedings of the 4th Annual Linux Expo.Google Scholar
- Yang, J., Sar, C., and Engler, D. 2006a. EXPLODE: a lightweight, general system for finding serious storage system errors. In Proceedings of the Operating Systems Design and Implementation (OSDI'06). 131--146. Google ScholarDigital Library
- Yang, J., Sar, C., Twohey, P., Cadar, C., and Engler, D. 2006b. Automatically generating malicious disks using symbolic execution. In Proceedings of the IEEE Symposium on Security and Privacy. 243--257. Google ScholarDigital Library
- Yang, J., Twohey, P., Engler, D., and Musuvathi, M. 2006c. Using model checking to find serious file system errors. ACM Trans. Comput. Systems 24, 4, 393--423. Google ScholarDigital Library
- Zhang, Y., Rajimwale, A., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2010. End-to-end data integrity for file systems: a ZFS case study. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST'10). 29--42. Google ScholarDigital Library
Index Terms
- Recon: Verifying file system consistency at runtime
Recommendations
Checking the Integrity of Transactional Mechanisms
Special Issue on Usenix Fast 2014Data corruption is the most common consequence of file-system bugs. When such corruption occurs, offline check and recovery tools must be used, but they are error prone and cause significant downtime. Previously we showed that a runtime checker for the ...
Recon: verifying file system consistency at runtime
FAST'12: Proceedings of the 10th USENIX conference on File and Storage TechnologiesFile system bugs that corrupt file system metadata on disk are insidious. Existing file-system reliability methods, such as checksums, redundancy, or transactional updates, merely ensure that the corruption is reliably preserved. The typical workarounds,...
Scalable testing of file system checkers
EuroSys '12: Proceedings of the 7th ACM european conference on Computer SystemsFile system checkers (like e2fsck) are critical, complex, and hard to develop, and developers today rely on hand-written tests to exercise this intricate code. Test suites for file system checkers take a lot of effort to develop and require careful ...
Comments