Abstract
Experience with the development and maintenance of large test suites specified using the Testing and Test Control Notation (TTCN-3) has shown that it is difficult to construct tests that are concise with respect to quality aspects such as maintainability or usability. The ISO/IEC standard 9126 defines a general software quality model that substantiates the term “quality” with characteristics and subcharacteristics. The domain of test specifications, however, requires an adaption of this general model. To apply it to specific languages such as TTCN-3, it needs to be instantiated. In this paper, we present an instantiation of this model as well as an approach to assess and improve test specifications. The assessment is based on metrics and the identification of code smells. The quality improvement is based on refactoring. Example measurements using our TTCN-3 tool TRex demonstrate how this procedure is applied in practise.
Article PDF
Similar content being viewed by others
References
Baker, P., Evans, D., Grabowski, J., Neukirchen, H., Zeiss, B.: TRex—the refactoring and metrics tool for TTCN-3 test specifications. In: Proceedings of TAIC PART 2006 (Testing: Academic & Industrial Conference—Practice And Research Techniques), Cumberland Lodge, Windsor Great Park, UK, 29–31 August 2006. IEEE Computer Society (2006). doi:10.1109/TAIC-PART.2006.35
Baker, P., Loh, S., Weil, F.: Model-driven engineering in a large industrial context—Motorola case study. In: Briand, L., Williams, C. (eds.) Model Driven Engineering Languages and Systems: 8th International Conference, MoDELS 2005, Montego Bay, Jamaica, October 2–7, 2005. Lecture Notes in Computer Science (LNCS), vol. 3713, pp. 476–491. Springer, Berlin (2005). doi:10.1007/11557432_36
Basili, V.R., Weiss, D.M.: A methodology for collecting valid software engineering data. IEEE Trans. Softw. Eng. 10(6), 728–738 (1984)
Bisanz, M.: Pattern-based smell detection in TTCN-3 test suites. Master’s thesis, Center for Computational Sciences, University of Göttingen, Germany, ZFI-BM-2006-44 (2006). http://www.swe.informatik.uni-goettingen.de/publications/MB/bisanz_mastersthesis.pdf
Boehm, B., Brown, J., Kaspar, J., Lipow, M., MacLead, C., Merrit, M.: Characteristics of Software Quality. North-Holland, Amsterdam (1978)
DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Comp. 11(4), 34–43 (1978). doi:10.1109/C-M.1978.218136
van Deursen, A., Moonen, L., van den Bergh, A., Kok, G.: Extreme Programming Perspectives, Chap. Refactoring Test Code, pp. 141–152. Addison-Wesley, Boston (2002)
Dixon-Peugh, D.: PMD (2007). http://pmd.sourceforge.net
Eclipse Foundation: Eclipse (2007). http://www.eclipse.org
Eclipse Foundation: Eclipse Graphical Editing Framework (2007). http://www.eclipse.org/gef
Eclipse Foundation: Eclipse Test & Performance Tools Platform Project (TPTP) (2007). http://www.eclipse.org/tptp
van Emden, E., Moonen, L.: Java quality assurance by detecting code smells. In: Proceedings Ninth Working Conference on Reverse Engineering WCRE 2002, pp. 97–106. IEEE Computer Society Press, New York (2002). doi:10.1109/WCRE.2002.1173068
ETSI: Technical Specification (TS) 102 027-3 V4.1.1 (2006–07): SIP ATS & PIXIT; Part 3: Abstract Test Suite (ATS) and partial Protocol Implementation eXtra Information for Testing (PIXIT). European Telecommunications Standards Institute (ETSI), Sophia-Antipolis, France (2006)
ETSI: Technical Specification (TS) 102 385-3 V2.2.1 (2006–04): Conformance Testing for WiMAX/HiperMAN 1.2.1; Part 3: Abstract Test Suite (ATS). European Telecommunications Standards Institute (ETSI), Sophia-Antipolis, France (2006)
ETSI: Technical Specification (TS) 102 516 V1.1 (2006–04): IPv6 Core Protocol; Conformance Abstract Test Suite (ATS) and partial Protocol Implementation eXtra Information for Testing (PIXIT). European Telecommunications Standards Institute (ETSI), Sophia-Antipolis, France (2006)
ETSI: ETSI Standard (ES) 201 873 V3.2.1: The Testing and Test Control Notation version 3; Parts 1–8. European Telecommunications Standards Institute (ETSI), Sophia-Antipolis, France, also published as ITU-T Recommendation series. Z.140 (2007)
Fan, C.F., Yih, S.: Prescriptive metrics for software quality assurance. In: Proceedings of the First Asia-Pacific Software Engineering Conference, pp. 430–438. IEEE-CS Press, Tokyo (1994). doi:10.1109/APSEC.1994.465237
Fenton, N.E., Pfleeger, S.L.: Software Metrics. PWS Publishing Company, Boston (1997)
Fosdick, L.D., Osterweil, L.J.: Data flow analysis in software reliability. ACM Comput. Surv. 8(3), 305–330 (1976). doi:10.1145/356674.356676
Fowler, M.: Refactoring—Improving the Design of Existing Code. Addison-Wesley, Boston (1999)
Gamma, E., Beck, K.: JUnit (2007). http://junit.sourceforge.net
Grabowski, J., Hogrefe, D., Réthy, G., Schieferdecker, I., Wiles, A., Willcock, C.: An introduction to the testing and test control notation (TTCN-3). Comp. Netw. 42(3), 375–403 (2003). doi:10.1016/S1389-1286(03)00249-4
Huang, J.C.: Detection of data flow anomaly through program instrumentation. IEEE Trans. Softw. Eng. 5(3), 226–236 (1979). doi:10.1109/TSE.1979.234184
Instantiations: CodePro AnalytiX (2007). http://www.instantiations.com/codepro/
ISO/IEC: ISO/IEC Standard No. 14598: Information technology—Software product evaluation; Parts 1–6. International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC), Geneva, Switzerland (1999–2001)
ISO/IEC: ISO/IEC Standard No. 9126: Software engineering—Product quality; Parts 1–4. International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC), Geneva, Switzerland (2001–2004)
JetBrains: IntelliJ IDEA (2007). http://www.jetbrains.com
Johnson, S.: Lint, a C Program Checker. Unix Programmer’s Manual. AT&T Bell Laboratories, (1978)
McCabe, T.J.: A complexity measure. IEEE Trans. Softw. Eng. 2(4), 308–320 (1976)
McCall, J., Richards, P., Walters, G.: Factors in software quality. Technical Report RADC TR-77-369, US Rome Air Development Center (1977)
Meszaros, G.: XUnit Test Patterns. Addison-Wesley, Boston (2007)
Moha, N., Guéhéneuc, Y.G., Leduc, P.: Automatic generation of detection algorithms for design defects. In: 21st IEEE/ACM International Conference on Automated Software Engineering (ASE 2006), 18–22 September 2006, Tokyo, Japan, pp. 297–300. IEEE Computer Society, New york (2006). doi:10.1109/ASE.2006.22
Munro, M.: Product metrics for automatic identification of “Bad Smell” design problems in Java source-code. In: 11th IEEE International Symposium on Software Metrics (METRICS 2005), 19–22 September 2005, Como, Italy. IEEE Computer Society, New York (2005). doi:10.1109/METRICS.2005.38
Myers, G.: The Art of Software Testing. Wiley, New York (1979)
Neukirchen, H., Bisanz, M.: Utilising code smells to detect quality problems in TTCN-3 test suites. In: Petrenko, A., Veanes, M., Tretmans, J., Grieskamp, W. (eds.) Testing of Communicating Systems/Formal Approaches to Testing of Software 2007, Tallinn, Estonia, June 26–29, 2007. Lecture Notes in Computer Science (LNCS), vol. 4581, pp. 228–243. Springer, Berlin (2007). doi:10.1007/978-3-540-73066-8_16
Nödler, J.: An XML-based approach for software analysis—applied to detect bad smells in TTCN-3 test suites. Master’s thesis, Center for Computational Sciences, University of Göttingen, Germany, ZFI-BM-2007-36 (2007). http://www.swe.informatik.uni-goettingen.de/publications/JN/noedler-masters-thesis.pdf
OMG: UML Testing Profile (Version 1.0 formal/05-07-07). Object Management Group (OMG) (2005)
Opdyke, W.: Refactoring object-oriented frameworks. Ph.D. thesis, University of Illinois at Urbana-Champaign, USA (1992)
Parnas, D.: Software aging. In: Proceedings of the 16th International Conference on Software Engineering (ICSE), May 16–21, 1994, Sorrento, Italy, pp. 279–287. IEEE Computer Society/ACM Press, New York (1994)
Parr, T.: ANTLR parser generator v2 (2007). http://www.antlr2.org
Pugh, B.: FindBugs (2007). http://findbugs.sourceforge.net
Roberts, D., Brant, J., Johnson, R.: A refactoring tool for smalltalk. theory and practice of object systems 3(4), 253–263 (1997). doi:10.1002/(SICI)1096-9942(1997)3:4<253::AID-TAPO3>3.3.CO;2-I
van Rompaey, B., du Bois, B., Demeyer, S.: Characterizing the relative significance of a test smell. In: Proceedings of the 22nd IEEE International Conference on Software Maintenance (ICSM 2006), Philadelphia, Pennsylvania, September 25–27, 2006, pp. 391–400. IEEE Computer Society (2006). doi:10.1109/ICSM.2006.18
Sauer, F.: Eclipse Metrics Plugin (2007). http://metrics.sourceforge.net
Sneed, H.M.: Measuring the effectiveness of software testing. In: Beydeda, S., Gruhn, V., Mayer, J., Reussner, R., Schweiggert, F. (eds.) Proceedings of SOQUA 2004 (First International Workshop on Software Quality) and TECOS 2004 (Workshop Testing Component-Based Systems). Lecture Notes in Informatics (LNI), vol. 58. Gesellschaft für Informatik, Köllen Verlag, Bonn (2004)
Telelogic: Logiscope (2007). http://www.telelogic.de/products/logiscope/
Testing Technologies: TTworkbench (2007). http://www.testingtech.de/products_services/ttwb_intro.php
TRex Team: TRex Website (2007). http://www.trex.informatik.uni-goettingen.de
Vega, D.E., Schieferdecker, I.: Towards quality of TTCN-3 tests. In: Proceedings of SAM’06: Fifth Workshop on System Analysis and Modelling, May 31–June 2, 2006, University of Kaiserslautern, Germany. University of Kaiserslautern, Germany (2006)
XQuery 1.0: An XML Query Language. World Wide Web Consortium (W3C) Recommendation 23 January 2007 (2007)
Watson, A.H., McCabe, T.J.: Structured testing: a testing methodology using the cyclomatic complexity metricy. NIST Special Publication 500–235, National Institute of Standards and Technology, Computer Systems Laboratory, Gaithersburg (1996)
Werner, E., Grabowski, J., Neukirchen, H., Röttger, N., Waack, S., Zeiss, B.: TTCN-3 quality engineering: using learning techniques to evaluate metric sets. In: Gaudin, E., Najm, E., Reed, R. (eds.) SDL 2007: Design for Dependable Systems, 13th International SDL Forum, Paris, France, September 18–21, 2007, Proceedings. Lecture Notes in Computer Science (LNCS), vol. 4745, pp. 54–68. Springer, Berlin (2007). doi:10.1007/978-3-540-74984-4_4
Zeiss, B.: A refactoring tool for TTCN-3. Master’s thesis, Center for Computational Sciences, University of Göttingen, Germany, ZFI-BM-2006-05 (2006). http://www.swe.informatik.uni-goettingen.de/publications/BZ/zeiss_mastersthesis.pdf
Zeiss, B., Neukirchen, H., Grabowski, J., Evans, D., Baker, P.: Refactoring and Metrics for TTCN-3 Test Suites. In: Gotzhein, R., Reed, R. (eds.) System Analysis and Modeling: Language Profiles. 5th International Workshop, SAM 2006, Kaiserslautern, Germany, May 31–June 2, 2006, Revised Selected Papers, Lecture Notes in Computer Science (LNCS), vol. 4320, pp. 148–165. Springer, Berlin (2006). doi:10.1007/11951148_10
Zeiss, B., Neukirchen, H., Grabowski, J., Evans, D., Baker, P.: TRex—an open-source tool for quality assurance of TTCN-3 test suites. In: Proceedings of CONQUEST 2006—9th International Conference on Quality Engineering in Software Technology, September 27–29, Berlin, Germany, pp. 117–128. dpunkt.Verlag, Heidelberg (2006)
Zeiss, B., Vega, D., Schieferdecker, I., Neukirchen, H., Grabowski, J.: Applying the ISO 9126 quality model to test specifications—exemplified for TTCN-3 test specifications. In: Bleeck, W.G., Rasch, J. Züllighoven, H. (eds.) Proceedings of Software Engineering 2007 (SE 2007). Lecture Notes in Informatics (LNI), vol. 105, pp. 231–242. Gesellschaft für Informatik, Köllen, Bonn (2007)
Author information
Authors and Affiliations
Corresponding author
Additional information
B. Zeiss is supported by a Ph.D. scholarship from Siemens AG, Corporate Technology.
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License ( https://creativecommons.org/licenses/by-nc/2.0 ), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Neukirchen, H., Zeiss, B. & Grabowski, J. An approach to quality engineering of TTCN-3 test specifications. Int J Softw Tools Technol Transf 10, 309–326 (2008). https://doi.org/10.1007/s10009-008-0075-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10009-008-0075-0