Ivan do Carmo Machado
Skip Abstract Section
Abstract
The software product line engineering strategy enables the achievement of significant improvements in quality through reuse of carefully crafted software assets across multiple products. However, high levels of quality in the software product line assets, which are used to create products, must be accompanied by effective and efficient test strategies for the products in the software product line. The goal of this study is to understand which strategies for testing products in software product lines have been reported in the literature, enabling discussions on the significant issues, and also pointing out further research directions. A systematic literature review was carried out that identified two hundred seventy-three papers, published from the years 1998 and early in 2012. From such a set of papers, a systematic selection resulted in forty-one relevant papers. The analysis of the reported strategies comprised two important aspects: the selection of products for testing, and the actual test of products. The findings showed a range of strategies, dealing with both aspects, but few empirical evaluations of their effectiveness have been performed, which limits the inferences that can be drawn.
- J. Al-Dallal and P. G. Sorenson. Testing software assets of framework-based product families during application engineering stage. Journal of Software, 3(5):11--25, 2008.Google Scholar
- A. Bertolino, A. Fantechi, S. Gnesi, and G. Lami. Product line use cases: Scenario-based specification and testing of requirements. In Software Product Lines - Research Issues in Engineering and Management, pages 425--445. Springer, 2006.Google ScholarCross Ref
- M. N. Borazjany, L. Yu, Y. Lei, R. Kacker, and R. Kuhn. Combinatorial testing of acts: A case study. In Proc. of the 5th IEEE International Conference on Software Testing, Verification and Validation, ICST, pages 591--600, Montreal, Canada, 2012. Google ScholarDigital Library
- I. Cabral, M. B. Cohen, and G. Rothermel. Improving the testing and testability of software product lines. In Proc. of the 14th International Conference on Software Product Lines, SPLC, pages 241--255, Jeju Island, South Korea, 2010. Springer-Verlag. Google ScholarDigital Library
- H. Cichos, S. Oster, M. Lochau, and A. Schürr. Model-based coverage-driven test suite generation for software product lines. In Proc. of the 14th International Conference on Model Driven Engineering Languages and Systems (MODELS), pages 425--439, Wellington, New Zealand, 2011. Springer-Verlag. Google ScholarDigital Library
- P. Clements and J. D. McGregor. Better, faster, cheaper: Pick any three. Business Horizons, 55(2):201--208, 2012.Google ScholarCross Ref
- E. Engström and P. Runeson. Software product line testing - a systematic mapping study. Information & Software Technology, 53(1):2--13, 2011. Google ScholarDigital Library
- E. Engström, M. Skoglund, and P. Runeson. Empirical evaluations of regression test selection techniques: a systematic review. In 2nd Intl. Symposium on Empirical Software Engineering and Measurement, ESEM, pages 22--31, Kaiserslautern, 2008. ACM. Google ScholarDigital Library
- A. Ensan, E. Bagheri, M. Asadi, D. Gasevic, and Y. Biletskiy. Goal-oriented test case selection and prioritization for product line feature models. In Eighth International Conference on Information Technology: New Generations, ITNG, pages 291--298, Las Vegas, USA, 2011. IEEE Computer Society. Google ScholarDigital Library
- B. Geppert, J. J. Li, F. Rößler, and D. M. Weiss. Towards generating acceptance tests for product lines. In Proc. of 8th Intl. Conference on Software Reuse, ICSR, pages 35--48, Madrid, Spain, 2004. Springer.Google ScholarCross Ref
- T. Gustafsson. An approach for selecting software product line instances for testing. In 11th International Conference on Software Product Lines, SPLC, pages 81--86, Tokyo, Japan, 2007.Google Scholar
- J. Hartmann, M. Vieira, and A. Ruder. A UML-based approach for validating product lines. In Proc. of the Intl. Workshop on Software Product Line Testing, SPLiT, pages 58--65, Boston, MA, USA, 2004.Google Scholar
- A. Hervieu, B. Baudry, and A. Gotlieb. PACOGEN: Automatic generation of pairwise test configurations from feature models. In 22nd IEEE International Symposium on Software Reliability Engineering, ISSRE, pages 120--129, Hiroshima, Japan, 2011. Google ScholarDigital Library
- M. F. Johansen, Ø. Haugen, and F. Fleurey. Properties of realistic feature models make combinatorial testing of product lines feasible. In 14th International Conference on Model Driven Engineering Languages and Systems, MODELS, pages 638--652, Wellington, New Zealand, 2011. Springer. Google ScholarDigital Library
- T. Kahsai, M. Roggenbach, and B.-H. Schlinglo. Specification-based testing for software product lines. In Sixth IEEE International Conference on Software Engineering and Formal Methods, SEFM, pages 149--158, Cape Town, South Africa, 2008. IEEE. Google ScholarDigital Library
- E. Kamsties, K. Pohl, S. Reis, and A. Reuys. Testing variabilities in use case models. In Proc. of 5th Intl. Workshop Software Product-Family Engineering, PFE, pages 6--18, Siena, Italy, 2003. Springer.Google Scholar
- C. H. P. Kim, D. S. Batory, and S. Khurshid. Reducing combinatorics in testing product lines. In Proc. of the 10th Intl. Conference on Aspect-oriented Software Development, AOSD, pages 57--68. ACM, 2011. Google ScholarDigital Library
- B. Kitchenham and S. Charters. Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report, 2007.Google Scholar
- R. Kolb and D. Muthig. Challenges in testing software product lines. In Proceedings of the 7th Conference on Quality Engineering in Software Technology, CONQUEST, pages 81--95. Fraunhofer Publica, 2003.Google Scholar
- B. P. Lamancha, M. Usaola, and I. de Guzman. Model-driven testing in software product lines. In IEEE Intl. Conference on Software Maintenance, ICSM, pages 511--514, Edmonton, Canada, 2009.Google ScholarCross Ref
- B. P. Lamancha and M. P. Usaola. Testing product generation in software product lines using pairwise for features coverage. In Proc. of the 22nd IFIP Intl. Conference on Testing Software and Systems, ICTSS, pages 111--125, Natal, Brazil, 2010. Springer-Verlag. Google ScholarDigital Library
- B. P. Lamancha, M. P. Usaola, and M. P. Velthius. Software product line testing - a systematic review. In Proc. of the 4th International Conference on Software and Data Technologies, ICSOFT, pages 23--30, Soa, Bulgaria, 2009. INSTICC Press.Google Scholar
- K. Lee, K. C. Kang, and J. Lee. Concepts and guidelines of feature modeling for product line software engineering. In Proc. of the 7th Intl. Conference on Software Reuse, ICSR, pages 62--77, London, UK, 2002. Springer-Verlag. Google ScholarDigital Library
- M. Lochau, S. Oster, U. Goltz, and A. Schürr. Model-based pairwise testing for feature interaction coverage in software product line engineering. Software Quality Journal, pages 1--38, 2011. Google ScholarDigital Library
- P. R. Mateo, M. Polo, and B. P. Lamancha. Automatic generation of test cases in software product lines. In Proceedings of the 11th International Conference on Enterprise Information Systems, ICEIS, pages 124--130, Milan, Italy, 2009. Springer.Google ScholarCross Ref
- J. D. McGregor. Testing a software product line. Technical Report TR-022, CMU Software Engineering Institute, 2001.Google Scholar
- S. Mishra. Specification based software product line testing: A case study. In 15th International Workshop on Concurrency, Specification and Programming, CS&P, pages 243--254, Wandlitz, Germany, 2006.Google Scholar
- M. Mohamed Ali and R. Moawad. An approach for requirements based software product line testing. In 7th International Conference on Informatics and Systems, INFOS, pages 1--10, Cairo, Egypt, 2010.Google Scholar
- C. Nebut, S. Pickin, Y. Le Traon, and J.-M. Jézéquel. Automated requirements-based generation of test cases for product families. In Proc. of the 18th IEEE Intl. Conference on Automated Software Engineering, ASE, pages 263--266, Montreal, Canada, 2003.Google ScholarDigital Library
- C. Nebut, S. Pickin, Y. L. Traon, and J. M. Jézéquel. Reusable test requirements for uml-modeled product lines. In Proceedings of the Workshop on Requirements Engineering for Product Lines, REPL, pages 51--56, Essen, Germany, 2002. Avaya Labs.Google Scholar
- C. Nebut, Y. Traon, and J.-M. Jézéquel. System testing of product lines: From requirements to test cases. In T. Käköla and J. C. Duenas, editors, Software Product Lines, pages 447--477. Springer, 2006.Google ScholarCross Ref
- P. A. S. Neto, I. C. Machado, J. D. McGregor, E. S. Almeida, and S. R. L. Meira. A systematic mapping study of software product lines testing. Information & Software Technology, 53(5):407--423, 2011. Google ScholarDigital Library
- E. M. Olimpiew and H. Gomaa. Model-based testing for applications derived from software product lines. In Proc. of the 1st Intl. Workshop on Advances in model-based testing, A-MOST, pages 1--7. ACM, 2005. Google ScholarDigital Library
- E. M. Olimpiew and H. Gomaa. Customizable requirements-based test models for software product lines. In Proc. of the 3rd International Workshop on Software Product Line Testing, SPLiT, pages 17--22, Baltimore, USA, 2006.Google Scholar
- E. M. Olimpiew and H. Gomaa. Reusable model-based testing. In 11th Intl. Conference on Software Reuse, ICSR, pages 76--85, Falls Church, USA, 2009. Springer. Google ScholarDigital Library
- S. Oster, F. Markert, and P. Ritter. Automated incremental pairwise testing of software product lines. In Proc. of the 14th International Conference on Software Product Lines, SPLC, pages 196--210, Jeju Island, South Korea, 2010. Springer-Verlag. Google ScholarDigital Library
- S. Oster, A. Schürr, and I. Weisemöller. Towards software product line testing using story driven modeling. Fakultät Informatik, pages 48--51, 2008.Google Scholar
- S. Oster, M. Zink, M. Lochau, and M. Grechanik. Pairwise feature-interaction testing for spls: potentials and limitations. In Proc. of the 15th International Software Product Line Conference, SPLC, pages 1--8, Munich, Germany, 2011. ACM. Google ScholarDigital Library
- B. Perez Lamancha, M. Polo Usaola, and M. Piattini Velthius. A model based testing approach for model-driven development and software product lines. In Proc. of the 5th Intl. Conference on Evaluation of Novel Approaches to Software Engineering, pages 193--208, Athens, Greece, 2010.Google Scholar
- G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. le Traon. Automated and scalable T-wise test case generation strategies for software product lines. In 3rd Intl. Conference on Software Testing, Verification and Validation, ICST, pages 459--468, Paris, France, 2010. Google ScholarDigital Library
- K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. Systematic mapping studies in software engineering. In 12th Intl. Conf. on Evaluation and Assessment in Software Engineering, EASE, Bari, Italy, 2008. Google ScholarDigital Library
- A. P. K. Reis, S.; Metzger. A reuse technique for performance testing of software product lines. In Proc. of the 3rd Intl. Workshop on Software Product Line Testing, SPLiT, pages 5--10, Baltimore, USA, 2006.Google Scholar
- A. Reuys, S. Reis, E. Kamsties, and K. Pohl. Derivation of domain test scenarios from activity diagrams. In Proc. of the International Workshop on Product Line Engineering, PLEES, pages 35--42, Erfut, Germany, 2003. Fraunhofer IESE.Google Scholar
- A. Reuys, S. Reis, E. Kamsties, and K. Pohl. The scented method for testing software product lines. In T. Käköla and J. C. Duenas, editors, Software Product Lines, pages 479--520. Springer, 2006.Google ScholarCross Ref
- C. Robinson-Mallett, M. Grochtmann, J. Wegener, J. Kohnlein, and S. Kuhn. Modelling requirements to support testing of product lines. In Proc. of the 3rd International Conference on Software Testing, Verification, and Validation Workshops, ICSTW, pages 11--18, Paris, France, 2010. IEEE. Google ScholarDigital Library
- K. Scheidemann. Optimizing the selection of representative configurations in verification of evolving product lines of distributed embedded systems. In Proc. of the 10th Intl. Software Product Line Conference, SPLC, pages 75--84. IEEE, 2006. Google ScholarDigital Library
- S. Segura, J. A. Galindo, D. Benavides, J. A. Parejo, and A. Ruiz-Cortés. Betty: benchmarking and testing on the automated analysis of feature models. In 6th Intl. Workshop on Variability Modeling of Software- Intensive Systems, VaMoS, pages 63--71. ACM, 2012. Google ScholarDigital Library
- S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cortés. Automated metamorphic testing on the analyses of feature models. Information and Software Technology, 53(3):245--258, 2011. Google ScholarDigital Library
- J. Shi, M. Cohen, and M. Dwyer. Integration testing of software product lines using compositional symbolic execution. In 15th International Conference on Fundamental Approaches to Software Engineering, FASE, pages 270--284, Tallinn, Estonia, 2012. Springer. Google ScholarDigital Library
- V. Stricker, A. Metzger, and K. Pohl. Avoiding redundant testing in application engineering. In Proc. of the 14th international conference on Software product lines, SPLC, pages 226--240. Springer, 2010. Google ScholarDigital Library
- A. Tevanlinna, J. Taina, and R. Kauppinen. Product family testing: a survey. ACM SIGSOFT Software Engineering Notes, 29(2):12, 2004. Google ScholarDigital Library
- E. Uzuncaova, S. Khurshid, and D. Batory. Incremental test generation for software product lines. IEEE Transactions on Software Engineering, 36(3):309--322, 2010. Google ScholarDigital Library
Index Terms
- Strategies for testing products in software product lines
Recommendations
On strategies for testing software product lines: A systematic literature review
Context: Testing plays an important role in the quality assurance process for software product line engineering. There are many opportunities for economies of scope and scale in the testing activities, but techniques that can take advantage of these ...
Continuous test suite augmentation in software product lines
SPLC '13: Proceedings of the 17th International Software Product Line ConferenceSoftware Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, to generate test cases for individual products in product lines more efficiently. In this paper we ...
Risk-based integration testing of software product lines
VaMoS '17: Proceedings of the 11th International Workshop on Variability Modelling of Software-Intensive SystemsSoftware product lines (SPL) capture commonalities and variabilities of product families and, thus, enable mass customization of product variants according to customers desired configurations. However, they introduce new challenges to software testing ...
Comments