Abstract
We consider a setting where applications, such as websites or games, need causal access to objects available in geo-replicated cloud data stores. Common ways of implementing causal consistency involve hiding objects while waiting for their dependencies or waiting for server replicas to synchronize. To minimize delays and retrieve objects faster, applications may try to reach different server replicas at once. This entails a cost because providers charge for each reading request, including reading misses where the causal copy of the object is unavailable. Therefore, latency and cost are conflicting goals, which we control by selecting where to read and when. We formulate this challenge as a multi-criteria optimization problem and propose five non-dominated reading strategies, four of which are Pareto optimal, in a setting constrained to two server replicas. We validate these solutions on the following real cloud storage services: AWS S3, DynamoDB and MongoDB. Savings of as much as 50% on reading costs, with no significant or even a positive impact on latency, demonstrate that both clients and cloud providers could benefit from richer services compatible with these retrieval strategies.
Article PDF
Similar content being viewed by others
Data Availability
The datasets generated during the current study are available from the corresponding author upon reasonable request.
References
Abadi, D.: Consistency tradeoffs in modern distributed database system design: CAP is only part of the story. IEEE Comput. 45(2), 37–42 (2012). https://doi.org/10.1109/MC.2012.33
Amazon: Cloud Object Storage - Amazon S3 - Amazon Web Services. https://aws.amazon.com/s3/?nc1=h_ls. Accessed 21 May 2022 (2006)
Amazon Web Services, Inc. or affiliates. Amazon ElastiCache- In-memory data store and cache. https://aws.amazon.com/elasticache/. Accessed 13 July 2022 (2011)
Amazon Web Services, Inc. or affiliates, Amazon DynamoDB. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html. Accessed 13 July 2022 (2012)
Attiya, H., Welch, J.L.: Sequential consistency versus linearizability. ACM Trans. Comput. Syst. (TOCS) 12(2), 91–122 (1994)
Bailis, P., Fekete, A., Ghodsi, A., Hellerstein, J.M., Stoica, I.: The potential dangers of causal consistency and an explicit solution. In: Proceedings of the 3rd ACM Symposium on Cloud Computing, SoCC ’12. ISBN 9781450317610. https://doi.org/10.1145/2391229.2391251. Association for Computing Machinery, New York (2012)
Bailis, P., Ghodsi, A., Hellerstein, J.M., Stoica, I.: Bolt-on causal consistency. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD ’13. ISBN 9781450320375. https://doi.org/10.1145/2463676.2465279, pp 761–772. Association for Computing Machinery, New York (2013)
Banker, K., Garrett, D., Bakkum, P., Verch, S.: MongoDB in action: covers MongoDB version 3.0. Manning, Shelter Island, NY. ISBN 978-1617291609 (2016)
Barbara, L., Ladin, R.: Highly-available distributed services fault-tolerant distributed garbage collection. In: Proceedings of the 5th Symposium on the Principles of Distributed Computing, pp 29–39. ACM, Canada (1986)
Bermbach, D., Kuhlenkamp, J.: Consistency in distributed storage systems. In: Gramoli, V., Guerraoui, R. (eds.) Networked Systems. ISBN 978-3-642-40148-0, pp 175–189. Springer, Berlin (2013)
Bermbach, D., Tai, S.: Eventual consistency: How soon is eventual? An evaluation of amazon S3’s consistency behavior. In: Proceedings of the 6th Workshop on Middleware for Service Oriented Computing, MW4SOC 2011, Lisbon, Portugal, December 12-16, 2011. https://doi.org/10.1145/2093185.2093186, p 1 (2011)
Birman, K.P., Joseph, T.A.: Reliable communication in the presence of failures. ACM Trans. Comput. Syst. 5(1), 47–76 (1987). ISSN 0734-2071. https://doi.org/10.1145/7351.7478https://doi.org/10.1145/7351.7478
Bravo, M., Rodrigues, L.E.T., Van Roy, P.: Saturn: a distributed metadata service for causal consistency. In: Proceedings of the 12th European Conference on Computer Systems, EuroSys 2017, Belgrade, Serbia, April 23-26, 2017. https://doi.org/10.1145/3064176.3064210https://doi.org/10.1145/3064176.3064210, pp 111–126 (2017)
Brzezinski, J., Sobaniec, C., Wawrzyniak, D.: Session guarantees to achieve PRAM consistency of replicated shared objects. In: International Conference on Parallel Processing and Applied Mathematics, pp 1–8. Springer (2003)
Brzezinski, J., Sobaniec, C., Wawrzyniak, D.: From session causality to causal consistency. In: 12th Euromicro Workshop on Parallel, Distributed and Network-Based Processing (PDP 2004), 11–13 February 2004, A Coruna, Spain. https://doi.org/10.1109/EMPDP.2004.1271440https://doi.org/10.1109/EMPDP.2004.1271440, pp 152–158 (2004)
Carlson, J.: Redis in action. Manning, Shelter Island, NY. ISBN 978-1617290855 (2013)
Cowling, J.: Inside the MagicPocket - Dropbox. https://dropbox.tech/infrastructure/inside-the-magic-pocket. Accessed 14 July 2022 (2016)
Cristian, F.: Probabilistic clock synchronization. Distrib. Comput. 3(3), 146–158 (1989). ISSN 0178-2770. https://doi.org/10.1007/BF01784024
Weeks, D.C.: S3mper: Consistency in the cloud. https://netflixtechblog.com/s3mper-consistency-in-the-cloud-b6a1076aa4f8https://netflixtechblog.com/s3mper-consistency-in-the-cloud-b6a1076aa4f8, Accessed 13 Aug 2022 (2014)
DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: Amazon’s highly available key-value store. ACM SIGOPS Oper. Syst. Rev. 41(6), 205–220 (2007)
Défago, X., Schiper, A., Urbán, P.: Total order broadcast and multicast algorithms: Taxonomy and survey. ACM Comput. Surv. 36(4), 372–421 (2004). ISSN 0360-0300. https://doi.org/10.1145/1041680.1041682
Du, J., Iorgulescu, C., Roy, A., Zwaenepoel, W.: Gentlerain: Cheap and scalable causal consistency with physical clocks. In: Proceedings of the ACM Symposium on Cloud Computing, SOCC ’14. ISBN 9781450332521. https://doi.org/10.1145/2670979.2670983, pp 1–13. Association for Computing Machinery, New York (2014)
Duan, Y., Koufaty, D., Torrellas, J.: SCsafe: Logging sequential consistency violations continuously and precisely. In: 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp 249–260. IEEE (2016)
Farrell, A.K.: A deadlock detection scheme for Argus. Bachelor’s Thesis July 1989 MIT Dept. of Electrical Engineering and Computer Science
Python Software Foundation: pymongo ⋅ pypi. https://pypi.org/project/pymongo/. Accessed 14 July 2022 (2009)
Gilbert, S., Lynch, N.: Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News 33(2), 51–59 (2002). ISSN 0163-5700. https://doi.org/10.1145/564585.564601
Golab, W.: Proving PACELC. SIGACT News 49(1), 73–81 (2018). ISSN 0163-5700. https://doi.org/10.1145/3197406.3197420https://doi.org/10.1145/3197406.3197420
Golab, W., Li, X., Shah, M.A.: Analyzing consistency properties for fun and profit. In: Proceedings of the 30th Annual ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing, pp 197–206 (2011)
Google: Cloud Computing Services — Google Cloud. https://cloud.google.com/. Accessed 14 July 2022 (2008)
Google: Datastore – Google Cloud. https://cloud.google.com/datastore/. Accessed 14 July 2022 (2013)
Guerraoui, R., Schiper, A.: Fault-tolerance by replication in distributed systems. In: International Conference on Reliable Software Technologies, pp 38–57. Springer (1996)
Hasan, R., Tucek, J., Stanton, P., Yurcik, W., Brumbaugh, L., Rosendale, J., Boonstra, R.: The techniques and challenges of immutable storage with applications in multimedia. In: Lienhart, R.W., Babaguchi, N., Chang, E.Y. (eds.) Storage and Retrieval Methods and Applications for Multimedia 2005. https://doi.org/10.1117/12.588103, vol. 5682, pp 41–52. International Society for Optics and Photonics, SPIE, Bellingham (2005)
Herlihy, M.P., Wing, J.M.: Linearizability: A correctness condition for concurrent objects. ACM Trans. Program. Lang. Syst. 12(3), 463–492 (1990). ISSN 0164-0925. https://doi.org/10.1145/78969.78972
Hewitt, E.: Cassandra: The Definitive Guide, (Revised) 3rd edn.: Distributed Data at Web Scale. O’Reilly Media, Inc., Sebastopol (2022). ISBN 978-1492097143
Hsu, T.-Y., Kshemkalyani, A.D., Shen, M.: Causal consistency algorithms for partially replicated and fully replicated systems. Future Gener. Comput. Syst. 86, 1118–1133 (2018). ISSN 0167-739X. https://doi.org/10.1016/j.future.2017.04.044. https://www.sciencedirect.com/science/article/pii/S0167739X17308166
Hwang, D.J.-H.: Constructing a highly-available location service for a distributed environment. Master’s thesis, MIT Dept. of Electrical Engineering and Computer Science, Cambridge, Massachusetts, USA (1987)
IBM: IBM Cloud Storage. https://www.ibm.com/cloud/storage. Accessed 14 July 2022 (2013)
Jose, J., Subramoni, H., Luo, M., Zhang, M., Huang, J., Wasi-ur Rahman, M., Islam, N.S., Ouyang, X., Wang, H., Sur, S., Panda, D.K.: Memcached design on high performance RDMA capable interconnects. In: 2011 International Conference on Parallel Processing. https://doi.org/10.1109/ICPP.2011.37, pp 743–752 (2011)
Kermarrec, A.-M., Kuz, I., van Steen, M., Tanenbaum, A.S.: A framework for consistent, replicated Web objects. In: The 18th International Conference on Distributed Computing Systems. https://doi.org/10.1109/ICDCS.1998.679725https://doi.org/10.1109/ICDCS.1998.679725, pp 276–291. IEEE Computer Society, Amsterdam (1998)
Kreps, J., Narkhede, N., Rao, J., et al.: Kafka: A distributed messaging system for log processing. In: Proceedings of the NetDB, vol. 11, pp 1–7 (2011)
Kshemkalyani, A.D., Singhal, M.: Necessary and sufficient conditions on information for causal message ordering and their optimal implementation. Distrib. Comput. 11(2), 91–111 (1998). ISSN 0178-2770. https://doi.org/10.1007/s004460050044https://doi.org/10.1007/s004460050044
Ladin, R.: A method for constructing highly available services and a technique for distributed garbage collection. PhD thesis, MIT Dept. of Electrical Engineering and Computer Science (1989)
Ladin, R., Liskov, B., Shrira, L., Ghemawat, S.: Providing high availability using lazy replication. ACM Trans. Comput. Syst. 10(4), 360–391 (1992). https://doi.org/10.1145/138873.138877https://doi.org/10.1145/138873.138877
Lamport, L.: Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21(7), 558–565 (1978)
Liskov, B., Scheifler, R., Walker, E., Weihl, W.: Orphan detection (extended abstract). In: Proceedings of the 17th International Symposium on Fault-Tolerant Computing, Pittsburgh (1987)
Lloyd, W., Freedman, M.J., Kaminsky, M., Andersen, D.G.: Don’t settle for eventual: scalable causal consistency for wide-area storage with COPS. In: Proceedings of the 23rd ACM Symposium on Operating Systems Principles 2011, SOSP 2011, Cascais, Portugal October 23–26 2011. https://doi.org/10.1145/2043556.2043593https://doi.org/10.1145/2043556.2043593, pp 401–416 (2011)
Prince, M., Lorenzo, A., Dahlin, M.: Consistency, availability, and convergenc. Technical Report UTCS TR-11-22, Department of Computer Science, The University of Texas at Austin (2011)
McCoy, K.: VMS File System Internals (VAX - VMS Series). Digital Press, Maynard, Massachusetts USA. ISBN 1555580564 (1990)
Microsoft: Azure Blob Storage documentation. https://docs.microsoft.com/en-us/azure/storage/blobs/. Accessed 14 july 2022 (2008)
Microsoft: Microsoft Azure. https://azure.microsoft.com/services/storage. Accessed 14 July 2022 (2008)
Microsoft: Consistency levels in Azure Cosmos DB. https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels. Accessed 14 July 2022 (2022)
Evaggelia, P., Bharat, B.: Maintaining consistency of data in mobile distributed environments. In: Proceedings of 15th International Conference on Distributed Computing Systems, pp 404–413. IEEE (1995)
Ren, K., Li, D., Abadi, D.J.: Slog: serializable, low-latency, geo-replicated transactions. Proc. VLDB Endowment 12(11), 1747–1761 (2019)
Roohitavaf, M., Demirbas, M., Kulkarni, S.: Causalspartan: Causal consistency for distributed data stores using hybrid logical clocks. In: 2017 IEEE 36th Symposium on Reliable Distributed Systems (SRDS). https://doi.org/10.1109/SRDS.2017.27, pp 184–193 (2017)
Soni, M.: Practical AWS Networking: Build and manage complex networks using services such as Amazon VPC, Elastic Load Balancing, Direct Connect, and Amazon Route 53. ISBN 978-1788398299 (2018)
Tanenbaum, A.S., van Steen, M.: Distributed systems - principles and paradigms, 2nd edn. Pearson Education, Upper Saddle River (2007). ISBN 978-0-13-239227-3
Terry, D.B., Demers, A.J., Petersen, K., Spreitzer, M., Theimer, M., Welch, B.B.: Session guarantees for weakly consistent replicated data. In: Proceedings of the 3rd International Conference on Parallel and Distributed Information Systems (PDIS 94), Austin, Texas, USA, September 28–30, 1994. https://doi.org/10.1109/PDIS.1994.331722, pp 140–149 (1994)
Wada, H., Fekete, A., Zhao, L., Lee, K., Liu, A.: Data consistency properties and the trade-offs in commercial cloud storages: the consumers’ perspective. In: 5th Biennial Conference on Innovative Data Systems Research (CIDR 2011) (2011)
Wasabi Technologies Inc. Cloud Object Storage by Wasabi — 1/5th the Price — Wasabi. https://wasabi.com. Accessed 14 July 2022 (2017)
Weihl, W.E.: Distributed version management for read-only actions. IEEE Trans. Softw. Eng. E-13(1), 55–64 (1987)
Acknowledgements
The authors would like to thank the Centre for Informatics and Systems of the University of Coimbra (CISUC) for providing the conditions for this work.
Funding
Open access funding provided by FCT—FCCN (b-on). This work is funded by national funds through the FCT – Foundation for Science and Technology, I.P., within the scope of the project CISUC – UID/CEC/00326/2020, by the European Social Fund, through the Regional Operational Program Centro 2020 and by research grants of the programs: Science Without Borders (Ciências sem Fronteiras - CsF), Brazilian Space Agency (Agência Espacial Brasileira - AEB).
Author information
Authors and Affiliations
Contributions
Stanley Lima and Filipe Araujo started the work on the paper and designed the first experiments. Filipe Araujo later reformulated the paper with the help of Miguel Guerreiro. While both have contributed to the design and execution of the experiments, Andre Bento and Jaime Correia participated in discussions regarding the objectives and the evaluation of results. Stanley Lima, Filipe Araujo, and Miguel Guerreiro wrote most of the paper. Raul Barbosa was involved in the review process. All authors approved the final manuscript.
Corresponding author
Ethics declarations
Consent for Publication
All authors gave their consent for this publication.
Competing interests
The authors are not aware of any competing interests regarding this work.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Lima, S., Araujo, F., Guerreiro, M.d.O. et al. Efficient Causal Access in Geo-Replicated Storage Systems. J Grid Computing 21, 8 (2023). https://doi.org/10.1007/s10723-022-09640-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10723-022-09640-z