Abstract
With a greater ability for Big Data to obtain and compute such data sets, came a loss in the ability to ensure accurate data collection using traditional methods. This study aims to develop a model for data integrity, accurately detecting and compensating for errors and data drift. We will be working with a real-time water monitoring device that measures data in regular intervals. The data is managed in a decentralized manner by a Cloud-based Database Management System. We begin by exploring various drift compensation techniques. Afterwards, several algorithms implemented in similar studies are investigated. Upon choosing the optimal algorithm with any proper modifications, it is then implemented into an environmental sensing web application. In addition, new data visualization techniques are proposed to promote awareness for when data begins to drift.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adelman, J., Lamaute, N., Reicher, D., Van Norden, D., Ganis, M.: Remote Sensing in a Body of Water Using an Adafruit Feather. Seidenberg School of Computer Science & Information Systems, Pace University Pleasantville, NY 10570 (May 2017)
Andari, S., Caruso, M., Ganis, M., Robbins, C.B., Whit, C., Zada, A.: Web Application for Environmental Sensing. Seidenberg School of Computer Science & Information Systems, Pace University Pleasantville, NY 10570 (Dec 2017)
Caruso, M., Hassan, J., Keeley, L., Nikam, S., Zada, A.J.: Web Application for Environmental Sensing: Monitoring and Analyzing Water Temperatures. Seidenberg School of Computer Science & Information Systems, Pace University Pleasantville, NY 10570 (Feb 2018)
Taming data drift – the silent killer of data integrity. StreamSets Inc. https://19ttqs47cfw33zkecq3dz58m-wpengine.netdna-ssl.com/wp-content/uploads/2016/07/Taming-Data-Drift-White-Paper.pdf
Prabhakar, A.: Continuous ingest in the face of data drift (Part 1). Cloudera. http://vision.cloudera.com/continuous-ingest-in-the-face-of-data-drift/ (1 Feb 2016)
Picard, K.: The Jefferson project turns lake george into the world’s smartest lake. Seven Days. https://www.sevendaysvt.com/vermont/the-jefferson-project-turns-lake-george-into-the-worlds-smartest-lake/Content?oid=18412829 (25 Jul 2018)
Johnson, S.K.: Science by Robot: outfitting the world’s “smartest” lake. Ars Technica. https://arstechnica.com/science/2015/04/science-by-robot-outfitting-the-worlds-smartest-lake/ (18 Apr 2015)
In the Lab. The Jefferson project at lake George. Department of Biological Sciences, BT2149 Rensselaer Polytechnic Institute Troy, NY 12180 http://jeffersonproject.rpi.edu/lab. Accessed 1 Oct 2018
River and Estuary Observatory Network. Beacon Institute for Rivers and Estuaries. https://www.bire.org/river-and-estuary-observatory-network/. Accessed 1 Oct 2018
Why REON? Beacon Institute. https://www.thebeaconinstitute.org/approach/whyreon.php. Accessed 1 Oct 2018
Real-Time Hydrologic Sensing. REON. http://rths.us. Accessed 1 Oct 2018
Martin, B.: Tech-Tip – ensuring water quality data integrity. American Water Works Association. https://www.awwa.org/resources-tools/water-and-wastewater-utility-management/partnership-for-safe-water/partnership-resources/partnership-resources-details/articleid/4134/tech-tip-ensuring-water-quality-data-integrity.aspx (12 Apr 2016)
Larose, D.T.: An Introduction to Data Mining. Wiley-Interscience, John Wiley & Sons Inc, Hoboken, NJ (2005)
Pancha, G.: Big data’s hidden scourge: data drift. CMS Wire. https://www.cmswire.com/big-data/big-datas-hidden-scourge-data-drift/ (8 Apr 2016)
Wold, S., Antti, H., Lindgren, F., Öhman, J.: Orthogonal signal correction of near-infrared spectra. Chemom. Intell. Lab. Syst. 44, 1–2 (1998)
Artursson, T., Eklov, T., Lundström, I., Mårtensson, P., Sjöström, M., Holmberg, M.: Drift correction for gas sensors using multivariate methods. J. Chemom., Special Issue: Proceedings of the SSC6. 14(5–6), 711–723 (2000)
Gutierrez-Osuna, R.: Signal processing methods for drift compensation. In: PRISM 2nd NOSE II Workshop, Department of Computer Science, Texas A&M University College Station, TX 77843 (May 2003)
Hourly Data. NOAA National Centers for Environmental Information, Westchester Co Airport. Station WBAN:94745 Hourly Data. https://www.ncdc.noaa.gov/cdo-web/datatools/lcd. Accessed 17 Oct 2018
Piermont Pier Hydrologic Station Data. Hudson River Environmental Conditions Observing System. www.hrecos.org/ (2018). Accessed 17 Oct 2018
Keeley, L.: Sonetteira/ADA-data-viz: ADA Data Visualizations – Release 1 (Version v1.0.1). Zenodo. https://doi.org/10.5281/zenodo.2533272 (7 Jan 2019)
Beal, B.: Examining the relationship between dissolved oxygen and water temperature. Project Watershed. http://projectwatershed.org/sites/projectwatershed.org/files/Relat_dissolved_oxygen_temperature.pdf. Accessed 2 Nov 2018
Keeley, L.: Sonetteira/ADA: ADA Website – Release 1 (Version v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.2533186 (7 Jan 2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Siegel, D.T., Keeley, L.D., Shelke, P.P., Cotoranu, A., Ganis, M. (2019). Data Integrity Model for Environmental Sensing. In: Latifi, S. (eds) 16th International Conference on Information Technology-New Generations (ITNG 2019). Advances in Intelligent Systems and Computing, vol 800. Springer, Cham. https://doi.org/10.1007/978-3-030-14070-0_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-14070-0_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-14069-4
Online ISBN: 978-3-030-14070-0
eBook Packages: EngineeringEngineering (R0)