Skip to main content
SpringerLink
Log in

Fatman: Building Reliable Archival Storage Based on Low-Cost Volunteer Resources

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

We present Fatman, an enterprise-scale archival storage based on volunteer contribution resources from under-utilized web servers, usually deployed on thousands of nodes with spare storage capacity. Fatman is specifically designed for enhancing the utilization of existing storage resources and cutting down the hardware purchase cost. Two major concerned issues of the system design are maximizing the resource utilization of volunteer nodes without violating service level objectives (SLOs) and minimizing the cost without reducing the availability of archival system. Fatman has been widely deployed on tens of thousands of server nodes across several datacenters, providing more than 100 PB storage capacity and serving dozens of internal mass-data applications. The system realizes an efficient storage quota consolidation by strong isolation and budget limitation, to maximally support resource contribution without any degradation on host-level SLOs. It novelly improves data reliability by applying disk failure prediction to minish failure recovery cost, named fault-aware data management, dramatically reduces the mean time to repair (MTTR) by 76.3% and decreases file crash ratio by 35% on real-life product workload.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Sathiamoorthy M, Asteris M, Papailiopoulos D S, Dimakis A G, Vadali R, Chen S, Borthakur D. XORing elephants: Novel erasure codes for big data. In Proc. the 39th VLDB, Aug. 2013, pp.325–336.

  2. Huang C, Simitci H, Xu Y, Ogus A, Calder B, Gopalan P, Li J, Yekhanin S. Erasure coding in windows Azure storage. In Proc. USENIX ATC, Jun. 2012.

  3. Vrable M, Savage S, Voelker G M. Cumulus: Filesystem backup to the cloud. In Proc. the 7th USENIX Conf. File and Storage Technologies, Feb. 2009, pp.225–238.

  4. Vrable M, Savage S, Voelker G M. BlueSky: A cloud-backed file system for the enterprise. In Proc. the 10th USENIX Conf. File and Storage Technologies, Feb. 2012, pp.19:1–19:14.

  5. Reed I S, Solomon G. Polynomial codes over certain finite fields. Journal of the Society for Industrial and Applied Mathematics, 1960, 8(2): 300–304.

    Article  MATH  MathSciNet  Google Scholar 

  6. Khan O, Burns A, Plank J, Pierce W, Huang C. Rethinking erasure codes for cloud file systems: Minimizing I/O for recovery and degraded reads. In Proc. the 10th USENIX Conf. File and Storage Technologies, Feb. 2012, pp.20:1–20:14.

  7. Cipar J, Corner M D, Berger E D. TFS: A transparent file system for contributory storage. In Proc. the 5th USENIX Conf. File and Storage Technologies, Feb. 2007, pp.215–229.

  8. McKusick M K, Joy W N, Leffler S J, Fabry R S. A fast file system for UNIX. ACM Trans. Comput. Syst., 1984, 2(3): 181–197.

    Article  Google Scholar 

  9. Hoelzle U, Barroso L A. The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines (1st edition). Morgan and Claypool Publishers, 2009.

  10. Schroeder B, Gibson G A. Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? In Proc. the 5th USENIX Conf. File and Storage Technologies, Feb. 2007, pp.1:1–1:16.

  11. Barham P, Dragovic B, Fraser K, Hand S, Harris T L, Ho A, Neugebauer R, Pratt I, Warfield A. Xen and the art of virtualization. In Proc. the 19th SOSP, Oct. 2003, pp.164–177.

  12. Soltesz S, P¨otzl H, Fiuczynski M E, Bavier A C, Peterson L L. Container-based operating system virtualization: A scalable, high-performance alternative to hypervisors. In Proc. the 2nd EuroSys, Mar. 2007, pp.275–287.

  13. Ghemawat S, Gobioff H, Leung S T. The Google file system. In Proc. the 19th SOSP, Oct. 2003, pp.29–43.

  14. Jiang W, Hu C, Zhou Y, Kanevsky A. Are disks the dominant contributor for storage failures? A comprehensive study of storage subsystem failure characteristics. In Proc. the 6th USENIX Conf. File and Storage Technologies, Feb. 2008, pp.111–125.

  15. Xin Q, Schwarz T J E, Miller E L. Disk infant mortality in large storage systems. In Proc. the 13th MASCOTS, Sept. 2005, pp.125–134.

  16. Rodrigues R, Liskov B. High availability in DHTS: Erasure coding vs. replication. In Proc. the 4th IPTPS, Feb. 2005, pp.226–239.

  17. Tamo I, Papailiopoulos D S, Dimakis A G. Optimal locally repairable codes and connections to matroid theory. In Proc. CoRR, Jan. 2013.

  18. Vishwanath K V, Nagappan N. Characterizing cloud computing hardware reliability. In Proc. the 1st SoCC, Jun. 2010, pp.193–204.

  19. Zhu B, Wang G, Liu X, Hu D, Lin S, Ma J. Proactive drive failure prediction for large scale storage systems. In Proc. the 29th MSST, Jun. 2013.

  20. Paris J, Schwarz T J E, Long D. Evaluating the reliability of storage systems. Technical Report, UH-CS-06-08, Department of Computer Science, University of Houston, 2006.

  21. Larson S M, Snow C D, Shirts M, Pande V S. Folding@home and Genome@home: Using distributed computing to tackle previously intractable problems in computational biology. arXiv: 0901.0866, 2009. http://arxiv.org/abs/0901.0866, Jan. 2015.

  22. Durrani M N, Shamsi J A. Volunteer computing: Requirements, challenges, and solutions. J. Network and Computer Applications, 2014, 39: 369–380.

    Article  Google Scholar 

  23. Hamerly G, Elkan C. Bayesian approaches to failure prediction for disk drives. In Proc. the 18th ICML, Jun. 2001, pp.202–209.

  24. Hughes G F, Murray J F, Kreutz-Delgado K, Elkan C. Improved disk-drive failure warnings. IEEE Transactions on Reliability, 2002, 51(3): 350–357.

    Article  Google Scholar 

  25. Murray J F, Hughes G F, Kreutz-Delgado K. Machine learning methods for predicting failures in hard drives: A multiple-instance application. Journal of Machine Learning Research, 2005, 6: 783–816.

    MATH  MathSciNet  Google Scholar 

  26. Bitton D, Gray J. Disk shadowing. In Proc. the 14th VLDB, Aug. 1988, pp.331–338.

  27. Chen P M, Lee E L, Gibson G A, Katz R H, Patterson D A. RAID: High-performance, reliable secondary storage. ACM Comput. Surv., 1994, 26(2): 145–185.

    Article  Google Scholar 

  28. Plank J S. A tutorial on Reed-Solomon coding for faulttolerance in RAID-like systems. Software — Practice & Experience, 1997, 27(9): 995–1012.

    Article  Google Scholar 

  29. Qin A, Hu D, Liu J, Yang W, Tan D. Fatman: Cost-saving and reliable archival storage based on volunteer resources. In Proc. the 40th VLDB, Sept. 2014, pp.1748–1753.

Download references

Author information

Authors and Affiliations

  1. Baidu Inc., Beijing, 100193, China

    An Qin, Dian-Ming Hu, Jun Liu, Wen-Jun Yang & Dai Tan

Authors
  1. An Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dian-Ming Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wen-Jun Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dai Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to An Qin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, A., Hu, DM., Liu, J. et al. Fatman: Building Reliable Archival Storage Based on Low-Cost Volunteer Resources. J. Comput. Sci. Technol. 30, 273–282 (2015). https://doi.org/10.1007/s11390-015-1521-6

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-015-1521-6

Keywords

Search

Navigation