Skip to main content
SpringerLink
Log in

The Impact of Distributed Data in Big Data Platforms on Organizations

  • Conference paper
  • First Online:
Proceedings of the Future Technologies Conference (FTC) 2018 (FTC 2018)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 881))

Included in the following conference series:

Abstract

Big data is an established platform used worldwide by many organizations for exploring and analyzing business inputs in order to reach better understanding and capabilities. Our research is focused on how organizations’ data-accumulating procedures may influence the processing of data in a big data environment. In this paper, we present a use case which examines the impact of data structure, due to big data architecture characteristics (specifically on HDFS), and how it can reflect on business processes and performance. The main contribution of this research is to point out why an organization that uses big data platforms needs to take into consideration the big data storage architecture.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.google.org/.

  2. 2.

    https://www.mysql.com/.

  3. 3.

    https://www.microsoft.com/en-us/sql-server/default.aspx.

  4. 4.

    https://www.postgresql.org/.

  5. 5.

    https://hadoop.apache.org/.

  6. 6.

    https://hive.apache.org/.

  7. 7.

    https://pig.apache.org/.

References

  1. Apache™ Hadoop®! project. http://hadoop.apache.org/

  2. Bansal, S.K.: Towards a semantic extract-transform-load (ETL) framework for big data integration. In: 2014 IEEE International Congress on Big Data (BigData Congress), pp. 522–529. IEEE, June 2014

    Google Scholar 

  3. Benjelloun, F.Z., Lahcen, A.A., Belfkih, S.: An overview of big data opportunities, applications and tools. In: Intelligent Systems and Computer Vision (ISCV), pp. 1–6. IEEE, March 2015

    Google Scholar 

  4. Bhandarkar, M.: MapReduce programming with apache Hadoop. In: 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), pp. 1. IEEE, April 2010

    Google Scholar 

  5. Chen, H., Li, T., Luo, C., Horng, S.J., Wang, G.: A rough set-based method for updating decision rules on attribute values’ coarsening and refining. IEEE Trans. Knowl. Data Eng. 26(12), 2886–2899 (2014)

    Article  Google Scholar 

  6. Clegg, B.: Big Data: How the Information Revolution Is Transforming Our Lives (Hot Science) (2017)

    Google Scholar 

  7. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)

    Article  Google Scholar 

  8. Engelberg, G., Koren, O., Perel, N.: Big data performance evaluation analysis using apache pig. Int. J. Softw. Eng. Its Appl. 10(11), 429–440 (2016)

    Google Scholar 

  9. Fan, J., Han, F., Liu, H.: Challenges of big data analysis. Natl. Sci. Rev. 1(2), 293–314 (2014)

    Article  Google Scholar 

  10. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: ICML, vol. 96, pp. 148–156, July 1996

    Google Scholar 

  11. Gao, P., Wang, M., Ghiocel, S.G., Chow, J.H., Fardanesh, B., Stefopoulos, G.: Missing data recovery by exploiting low-dimensionality in power system synchrophasor measurements. IEEE Trans. Power Syst. 31(2), 1006–1013 (2016)

    Article  Google Scholar 

  12. Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system, vol. 37, no. 5, pp. 29–43. ACM (2003)

    Google Scholar 

  13. Kendal, D., Koren, O., Perel, N.: Pig vs. hive use case analysis. Int. J. Database Theory Appl. 9(12), 267–276 (2016)

    Article  Google Scholar 

  14. Khan, S.: Ethem Alpaydin. Introduction to Machine Learning (Adaptive Computation and Machine Learning Series). The MIT Press (2004), 415 p. Natural Language Engineering 14(1), 133 (2008)

    Google Scholar 

  15. Leyva, E., González, A., Perez, R.: A set of complexity measures designed for applying meta-learning to instance selection. IEEE Trans. Knowl. Data Eng. 27(2), 354–367 (2015)

    Article  Google Scholar 

  16. Lu, R., Zhu, H., Liu, X., Liu, J.K., Shao, J.: Toward efficient and privacy-preserving computing in big data era. IEEE Netw. 28(4), 46–50 (2014)

    Article  Google Scholar 

  17. McAfee, A., Brynjolfsson, E., Davenport, T.H., Patil, D.J., Barton, D.: Big data: the management revolution. Harv. Bus. Rev. 90(10), 60–68 (2012)

    Google Scholar 

  18. Nghiem, P.P., Figueira, S.M.: Towards efficient resource provisioning in MapReduce. J. Parallel Distrib. Comput. 95, 29–41 (2016)

    Article  Google Scholar 

  19. Oosterhof, N.N., Hölzenspies, P.K., Kuper, J.: Application patterns. In: Trends in Functional Programming, pp. 370–382 (2005)

    Google Scholar 

  20. Papadimitriou, S., Sun, J.: DisCo: distributed co-clustering with map-reduce: a case study towards petabyte-scale end-to-end mining. In: Eighth IEEE International Conference on Data Mining, ICDM 2008, pp. 512–521. IEEE, December 2008

    Google Scholar 

  21. Rumbold, J.M., Pierscionek, B.K.: What are data? A categorization of the data sensitivity spectrum. Big Data Res. 12, 49–59 (2017)

    Article  Google Scholar 

  22. Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  23. Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–10. IEEE, May 2010

    Google Scholar 

  24. Sowmya, R., Suneetha, K.R.: Data mining with big data. In: 2017 11th International Conference on Intelligent Systems and Control (ISCO), pp. 246–250. IEEE, January 2017

    Google Scholar 

  25. Srikant, R., Agrawal, R.: Mining quantitative association rules in large relational tables. In: ACM Sigmod Record, vol. 25, no. 2, pp. 1–12. ACM, June 1996

    Article  Google Scholar 

  26. Tanenbaum, A.S., Van Steen, M.: Distributed Systems: Principles and Paradigms. Prentice-Hall, Upper Saddle River (2007)

    MATH  Google Scholar 

  27. Viktor, M.S., Kenneth, C.: Big Data: A Revolution That Will Transform How We Live, Work, and Think. Houghton Mifflin Harcourt, Boston (2013)

    Google Scholar 

  28. Wamba, S.F., Akter, S., Edwards, A., Chopin, G., Gnanzou, D.: How ‘big data’can make big impact: findings from a systematic review and a longitudinal case study. Int. J. Prod. Econ. 165, 234–246 (2015)

    Article  Google Scholar 

  29. Wang, J., Wang, R., Yin, J., Zhu, H., Yang, Y.: Reliability analysis on shifted and random declustering block layouts in scale-out storage architectures. In: 2014 9th IEEE International Conference on Networking, Architecture, and Storage (NAS), pp. 148–157. IEEE, August 2014

    Google Scholar 

  30. Wang, J., Wu, H., Wang, R.: A new reliability model in replication-based big data storage systems. J. Parallel Distrib. Comput. 108, 14–27 (2017)

    Article  Google Scholar 

  31. White, T.: Hadoop: The definitive guide, 4th edn. O’Reilly Media Inc., Sebastopol (2015)

    Google Scholar 

  32. Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: Data mining: practical machine learning tools and techniques. Morgan Kaufmann, Burlington (2016)

    Google Scholar 

  33. Zhang, H., Chen, G., Ooi, B.C., Tan, K.L., Zhang, M.: In-memory big data management and processing: a survey. IEEE Trans. Knowl. Data Eng. 27(7), 1920–1948 (2015)

    Article  Google Scholar 

Download references

Acknowledgment

The authors would like to thank Mr. Brian Christopher Poll for his valuable comments and suggestions to improve the quality of the paper.

Author information

Authors and Affiliations

  1. School of Industrial Engineering and Management, Shenkar – Engineering. Design. Art., 12 Anne Frank St., Ramat-Gan, Israel

    Oded Koren, Matan Binyaminov & Nir Perel

Authors
  1. Oded Koren
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matan Binyaminov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nir Perel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nir Perel .

Editor information

Editors and Affiliations

  1. Saga University, Saga, Japan

    Kohei Arai

  2. The Science and Information (SAI) Organization, Bradford, West Yorkshire, UK

    Rahul Bhatia

  3. The Science and Information (SAI) Organization, Bradford, West Yorkshire, UK

    Supriya Kapoor

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Koren, O., Binyaminov, M., Perel, N. (2019). The Impact of Distributed Data in Big Data Platforms on Organizations. In: Arai, K., Bhatia, R., Kapoor, S. (eds) Proceedings of the Future Technologies Conference (FTC) 2018. FTC 2018. Advances in Intelligent Systems and Computing, vol 881. Springer, Cham. https://doi.org/10.1007/978-3-030-02683-7_76

Download citation

Publish with us

Policies and ethics

Search

Navigation