Abstract
In recent years, there has been an increase in the number of software applications developed using the microservices architectural pattern. This trend is due to the benefits derived from the more traditional N-tier architectural patterns that use monolithic designs for each tier. The value of using the microservices architectural pattern, particularly in the cloud, has been pioneered by companies such as Netflix and Google. These companies have created protocols and tools to support the development of cloud-based applications. However, the testing of microservices applications continues to be challenging due to the added complexity of network communication between the collaborating services. In addition, an increasing number of tools are being used to test microservices-based applications, which makes selecting the most appropriate tool(s) a challenging task. In this article, we compare several open-source tools used to support the testing of microservices based on testing levels, the scaffolding required, languages used for test cases, and the type of interface used to interact with the applications under test. We describe a prototype for a microservices-based application called Rideshare that allows users to reserve rides from available drivers. Using the Rideshare application, we performed a study using a subset of selected open-source tools to determine the overhead added by these tools. We present the results of the study and describe our experiences in configuring the tools to test the Rideshare application using different testing approaches.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ammann, P., & Offutt, J. (2016). Introduction to Software Testing (2nd ed.). New York, NY, USA: Cambridge University Press.
Apache JMeter. (2019). JMeter. http://jmeter.apache.org/, [Online; accessed 25-Jan-2019].
Arquillian Project. (2019a). Arquillian. http://arquillian.org/, [Online; accessed 5-Nov-2019].
Arquillian Project. (2019b). ShrinkWrap. https://arquillian.org/modules/shrinkwrap-shrinkwrap/, [Online; accessed 15-Nov-2019].
Atkinson, C., Groß, H. G. (2002). Built-in contract testing in model-driven, component-based development. In: ICSR Workshop on Component-Based Development Processes.
Bertolino, A., Calabr, A., De Angelis, G., Gallego, M., García, B., Gortzar, F. (2018). When the testing gets tough, the tough get elastest. In: 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion), pp. 17–20.
Beust, C. (2019). TestNG. https://testng.org/, [Online; accessed 14-Feb-2020].
Bozkurt, M., Harman, M., Hassoun, Y. (2010). Testing web services: A survey. Tech. Rep. TR-10-01, Centre for Research on Evolution, Search & Testing, King’s College London.
Bueno, A., Gumbrecht, A., Porter, J. (2018). Testing Java Microservices: Using Arquillian, Hoverfly, AssertJ, JUnit, Selenium, and Mockito. Manning Publications Company, https://books.google.com/books?id=ugYovgAACAAJ
Byars, B. (2020). Mountebank. http://www.mbtest.org/, [Online; accessed 14-Feb-2020].
Clemson, T. (2014) Testing strategies in a microservice architecture. https://martinfowler.com/articles/microservice-testing/, [Online; accessed 10-Oct-2021].
Colantonio, J. (2020). Top api testing tools for 2020. https://testguild.com/12-open-source-api-testing-tools-rest-soap-services/, [Online; accessed 14-Feb-2020].
Community, O. (2019). OAuth 2.0. https://oauth.net/2/, [Online; accessed 10-Jan-2019].
Cucumber Developer Team. (2019). Cucumber. https://cucumber.io/, [Online; accessed 2-Nov-2019].
Cucumber Developer Team. (2021). Gherkin syntax. https://cucumber.io/docs/gherkin/, [Online; accessed 15-Oct-2021].
Datawire Inc. (2019). Telepresence. https://www.telepresence.io, [Online; accessed 10-Jan-2019].
Dehghani, Z. (2018). How to break a monolith into microservices. https://martinfowler.com/articles/break-monolith-into-microservices.html, [Online; accessed 21-Jan-2019].
Dijkstra, B. (2014). The test automation pyramid. https://www.ontestautomation.com/the-test-automation-pyramid/, [Online; accessed 10-Apr-2019].
Docker Development Team. (2018). Docker Community Edition. https://docs.docker.com/, [Online; accessed 12-Feb-2019].
European Commission Project. (2019). Elastest. https://elastest.io/docs/, [Online; accessed 30-Oct-2019].
Fat, N., Vujovic, M., Papp, I., Novak, S. (2016). Comparison of angularjs framework testing tools. In: 2016 Zooming Innovation in Consumer Electronics International Conference (ZINC), pp. 76–79. https://doi.org/10.1109/ZINC.2016.7513659
Forsgren, N., Smith, D., Humble, J., Frazelle, J. (2019). Accelerate state of devops report. DORA & Google Cloud. https://services.google.com/fh/files/misc/state-of-devops-2019.pdf
Fowler, M. (2014). Unittest. https://martinfowler.com/bliki/UnitTest.html, [Online; accessed 6-Oct-2019].
Fowler, M., Lewis, J. (2018). Microservices a definition of this new architectural term. https://martinfowler.com/articles/microservices.html, [Online; accessed 5-Apr-2018].
Garlan, D., Perry, D. E. (1995). Introduction to the special issue on software architecture. IEEE Transactions on Software Engineering 21(4), 269–274. http://dl.acm.org/citation.cfm?id=205313.205314
Gatling Corp. (2019). Gatling. https://gatling.io/, [Online; accessed 25-Jan-2019].
Ghani, I., Wan-Kadir, W. M., Mustafa, A. (2019). Web service testing techniques: A systematic literature review. International Journal of Advanced Computer Science and Applications 10(8). http://dx.doi.org/10.14569/IJACSA.2019.0100858
Google. (2019a). Angular Js. https://angularjs.org/, [Online; accessed 10-Jan-2019].
Google. (2019b). Google Maps Platform. https://cloud.google.com/maps-platform/, [Online; accessed 10-Jan-2019].
Google. (2019c). Minikube. https://github.com/kubernetes/minikube, [Online; accessed 10-Jan-2019].
Google. (2019d). The Go Programming Language. https://golang.org/, [Online; accessed 10-Jan-2019].
Google. (2021). About Google. https://about.google/, [Online; accessed 22-Sept-2021].
Gremlin Inc. (2021). Gremlin. https://www.gremlin.com/, [Online; accessed 15-Oct-2021].
Habl, A., Kipouridis, O., Fottner, J. (2017). Deploying microservices for a cloud-based design of system-of-systems in intralogistics. In: 2017 IEEE 15th International Conference on Industrial Informatics (INDIN), IEEE, pp. 861–866.
Haleby, J. (2020). Rest assured. http://rest-assured.io/, [Online; accessed 14-Feb-2020].
Heckel, R., & Lohmann, M. (2005). Towards contract-based testing of web services. Electronic Notes in Theoretical Computer Science, 116, 145–156.
Heorhiadi, V., Rajagopalan, S., Jamjoom, H., Reiter, M., Sekar, V. (2016). Gremlin: Systematic resilience testing of microservices. In: Proceedings - 2016 IEEE 36th International Conference on Distributed Computing Systems, ICDCS 2016, Institute of Electrical and Electronics Engineers Inc., vol 2016-August, pp. 57–66. https://doi.org/10.1109/ICDCS.2016.11
Hussain, S., Wang, Z. S., & Toure, I. K. (2013). Survey of the tools for web service testing. Applied Mechanics and Materials, Trans Tech Publ, 427, 2951–2955.
IBM Cloud Team. (2021). Soa vs. microservices: What’s the difference? https://www.ibm.com/cloud/blog/soa-vs-microservices, [Online; accessed 9-Oct-2021].
Java Development Journal. (2019). Testing in Spring Boot. https://www.javadevjournal.com/spring-boot/testing-in-spring-boot/, [Online; accessed 17-Sep-2019].
Jenkins Development team. (2020). Jenkins. https://jenkins.io/, [Online; accessed 14-Feb-2020].
JS Foundation. (2019). Appium. http://appium.io/, [Online; accessed 25-Jan-2019].
Katalon LLC. (2019). Katalon Studio. https://www.katalon.com/, [Online; accessed 2-Nov-2019].
Kieffer, T. (2019). Writing Integration Tests with Docker Compose and JUnit. https://blog.codecentric.de/en/2017/03/writing-integration-tests-docker-compose-junit/, [Online; accessed 16-Sep-2019].
King, T. M., Clarke, P. J., Akour, M., Ganti, A. S. (2014). Validating autonomic service-driven applications: Challenges and approaches. In: Ramanathan R, Raja K (eds) Handbook of Research on Architectural Trends in Service-Driven Computing, IGI Global, chap 9, pp. 197–219.
Meszaros, G. (2007). xUnit Test Patterns: Refactoring Test Code. Addison-Wesley Signature Series, Addison-Wesley, https://www.safaribooksonline.com/library/view/xunit-test-patterns/9780131495050/
Mikowski, P. (2019). Testing rest apis with hoverfly. https://dzone.com/articles/testing-rest-apis-with-hoverfly, [Online; accessed 11-Sep-2019].
Nagy, T., Nagyné Víg, A. (2008). Erlang testing and tools survey. In: Proceedings of the 7th ACM SIGPLAN Workshop on ERLANG, ACM, New York, NY, USA, ERLANG ’08, pp. 21–28. http://doi.acm.org/10.1145/1411273.1411277
Netflix Development Team. (2019a). Chaos monkey. https://github.com/Netflix/chaosmonkey, [Online; accessed 1-Nov-2019].
Netflix Development Team. (2019b). Eureka. https://github.com/Netflix/eureka, [Online; accessed 1-Nov-2019].
Netflix Development Team. (2019c). Netflix. https://www.netflix.com/, [Online; accessed 18-Jan-2019].
Newman, S. (2015) Building microservices: designing fine-grained systems. O’Reilly Media, Inc.
Niclausse, N. (2019). Tsung. http://tsung.erlang-projects.org/, [Online; accessed 25-Jan-2019].
NPM Team. (2021). Protractor Cucumber Framework. https://www.npmjs.com/package/protractor-cucumber-framework, [Online; accessed 18-Mar-2021].
Oracle. (2020). Apiary. https://github.com/apiaryio, [Online; accessed 14-Feb-2020].
Pact Foundation. (2021). Pact. https://docs.pact.io/, [Online; accessed 15-Oct-2021].
Palantir Technologies. (2018). Docker Compose Rule Library. https://github.com/palantir/docker-compose-rule, [Online; accessed 15-Oct-2018].
Pivotal Software, Inc. (2019a). Spring Boot. https://spring.io/projects/spring-boot, [Online; accessed 25-Jan-2019].
Pivotal Software, Inc. (2019b). Spring Cloud Contract. http://spring.io/projects/spring-cloud-contract, [Online; accessed 25-Jan-2019].
Postman Inc. (2019). Postman. https://www.getpostman.com/, [Online; accessed 10-Feb-2019].
Raines, G. (2009). Cloud computing and soa. Tech. Rep. MTR090026, The MITRE Corporation, Bedford, Massachusetts.
Richards, M. (2015). Software architecture patterns. Incorporated: O’Reilly Media.
Rimal, B. P., Choi, E., Lumb, I. (2009). A taxonomy and survey of cloud computing systems. In: 2009 Fifth International Joint Conference on INC, IMS and IDC, pp. 44–51. https://doi.org/10.1109/NCM.2009.218
Russo, D., & Ciancarini, P. (2017). Towards antifragile software architectures. Procedia Computer Science, 109, 929–934.
Serenity/JS Team. (2021). Serenity/JS. https://serenity-js.org/, [Online; accessed 15-Oct-2021].
Sikuli Development Team. (2012). Sikuli. http://doc.sikuli.org/, [Online; accessed 2-Nov-2019].
Singh, I., & Tarika, B. (2014). Comparative analysis of open source automated software testing tools: Selenium, sikuli and watir. International Journal of Information & Computation Technology, 4(15), 1507–1518.
Situ, T. (2016). Consumer-driven contract testing with spring cloud contract. https://specto.io/blog/2016/11/16/spring-cloud-contract/, [Online; accessed 5-Nov-2019].
Smartbear. (2019). SoapUI. https://www.soapui.org/open-source.html, [Online; accessed 14-Feb-2020].
Software Freedom Conservancy (SFC). (2021). Selenium IDE. https://www.selenium.dev/, [Online; accessed 15-Oct-2021].
Software Test Help. (2021a). Top 15+ most popular web service testing tools in 2021. https://www.softwaretestinghelp.com/web-services-testing-tools/, [Online; accessed 10-Oct-2021].
Software Testing Help. (2019). 20+ best open source automation testing tools in 2020. https://www.softwaretestinghelp.com/open-source-testing-tools/, [Online; accessed 14-Feb-2020].
Software Testing Help. (2021b). How to choose the best automation testing tool (a complete guide). https://www.softwaretestinghelp.com/automation-testing-tutorial-4/, [Online; accessed 15-Oct-2021].
Soldani, J., Tamburri, D. A., Heuvel, W. J. V. D. (2018). The pains and gains of microservices: A systematic grey literature review. Journal of Systems and Software 146, 215–232. https://doi.org/10.1016/j.jss.2018.09.082,http://www.sciencedirect.com/science/article/pii/S0164121218302139
SonarSource SA. (2019). SonarQube. https://docs.sonarqube.org/, [Online; accessed 18-Jan-2019].
Sotomayor, J. P., Allala, S. C., Alt, P., Phillips, J., King, T. M., Clarke, P. J. (2019). Comparison of runtime testing tools for microservices. In: 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), vol 2, pp. 356–361. https://doi.org/10.1109/COMPSAC.2019.10232
SpectoLabs. (2018). Hoverfly. https://hoverfly.readthedocs.io/en/latest/index.html, [Online; accessed 14-Dec-2018].
SpectoLabs. (2019). Hoverfly. https://hoverfly.io/, [Online; accessed 18-Jan-2019].
Stolberg, S. (2009). Enabling agile testing through continuous integration. In: 2009 Agile Conference, pp. 369–374. https://doi.org/10.1109/AGILE.2009.16
Suliman, D., Paech, B., Borner, L., Atkinson, C., Brenner, D., Merdes, M., Malaka, R. (2006). The morabit approach to runtime component testing. In: 30th Annual International Computer Software and Applications Conference (COMPSAC’06), vol 2, pp. 171–176. https://doi.org/10.1109/COMPSAC.2006.169
Sundar, A. (2018). An insight into microservices testing strategies. https://www.infosys.com/it-services/validation-solutions/white-papers/documents/microservices-testing-strategies.pdf, [Online; accessed 15-Jan-2019].
Taibi, D., Lenarduzzi, V., & Pahl, C. (2017). Processes, motivations, and issues for migrating to microservices architectures: An empirical investigation. IEEE Cloud Computing, 4(5), 22–32. https://doi.org/10.1109/MCC.2017.4250931
Tal, L. (2019). A comprehensive guide to contract testing apis in a service oriented architecture. https://medium.com/@liran.tal/a-comprehensive-guide-to-contract-testing-apis-in-a-service-oriented-architecture-5695ccf9ac5a, [Online; accessed 13-Sep-2019].
The AngularJS Team - Google. (2019). Protractor. https://www.protractortest.org, [Online; accessed 2-Nov-2019].
The AngularJS Team - Google. (2021). Karma. https://karma-runner.github.io/latest/index.html, [Online; accessed 10-Mar-2021].
The JUnit Team. (2019). JUnit. https://junit.org/, [Online; accessed 25-Jan-2019].
The NUnit Team. (2019). NUnit. http://nunit.org/, [Online; accessed 25-Jan-2019].
Toledo, F. (2019). Gatling vs. JMeter . https://dzone.com/articles/gatling-vs-jmeter, [Online; accessed 11-Sep-2019].
Typhoeus Community. (2021). Typhoeus. https://github.com/typhoeus/typhoeus, [Online; accessed 15-Oct-2021].
VCR Development Team. (2019). Vcr. https://github.com/vcr, [Online; accessed 14-Feb-2020].
VCR Development Team. (2021). Relish - Project: Vcr 6.0.0. https://relishapp.com/vcr/vcr/v/2-0-0/docs, [Online; accessed 15-Oct-2021].
VMware, Inc. (2020). RabbitMQ. https://www.rabbitmq.com/, [Online; accessed 06-Apr-2021].
Waseem, M., Liang, P., Márquez, G., Di Salle, A. (2020). Testing microservices architecture-based applications: A systematic mapping study. In: Proceedings of the 27th Asia-Pacific Software Engineering Conference (APSEC).
Watir Development Team. (2019). Watir. http://watir.com/, [Online; accessed 25-Jan-2019].
WebMock Community. (2021). WebMock. https://github.com/bblimke/webmock, [Online; accessed 15-Oct-2021].
WireMock Development Team. (2020). WireMock. http://wiremock.org/, [Online; accessed 14-Feb-2020].
Acknowledgements
The authors would like to thank the Test.ai and Ultimate Kronos Group (UKG) teams for the contributions, feedback, and insight provided in preparing this manuscript. The authors would also like to thank the reviewers for their meaningful comments toward improving the article.
Funding
This work is supported in part by Test.ai and Ultimate Kronos Group (UKG). Any opinions, findings, and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Test.ai and UKG. The authors would like to thank the teams at Test.ai and UKG for the contributions, feedback, and insight provided in preparing this manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Financial interests
The authors Dionny Santiago and Tariq M. King have worked for both companies, but the funding of this project is independent of the work of the companies. The testbed application used in the experiments of this research was developed in collaboration with UKG. The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sotomayor, J.P., Allala, S.C., Santiago, D. et al. Comparison of open-source runtime testing tools for microservices. Software Qual J 31, 55–87 (2023). https://doi.org/10.1007/s11219-022-09583-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-022-09583-4