research-article

Feedback-Directed Metamorphic Testing

Authors:
Chang-Ai Sun

University of Science and Technology Beijing, Haidian District, Beijing, China

University of Science and Technology Beijing, Haidian District, Beijing, China

0000-0003-3696-6176
View Profile

,
Hepeng Dai

University of Science and Technology Beijing, Haidian District, Beijing, China

University of Science and Technology Beijing, Haidian District, Beijing, China

0000-0001-5507-3053
View Profile

,
Huai Liu

Swinburne University of Technology, John Street, Hawthorn, Australia

Swinburne University of Technology, John Street, Hawthorn, Australia

0000-0003-3125-4399
View Profile

,
Tsong Yueh Chen

Swinburne University of Technology, John Street, Hawthorn, Australia

Swinburne University of Technology, John Street, Hawthorn, Australia

0000-0003-3578-0994
View Profile

ACM Transactions on Software Engineering and Methodology Volume 32 Issue 1Article No.: 20pp 1–34https://doi.org/10.1145/3533314

Published:13 February 2023Publication History

ACM Transactions on Software Engineering and Methodology

Abstract

Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a set of metamorphic relations among multiple software inputs and their expected outputs, metamorphic testing not only provides a test case generation strategy by constructing new (or follow-up) test cases from some original (or source) test cases, but also a test result verification mechanism through checking the relationship between the outputs of source and follow-up test cases. Many efforts have been made to further improve the cost-effectiveness of metamorphic testing from different perspectives. Some studies attempted to identify “good” metamorphic relations, while other studies were focused on applying effective test case generation strategies especially for source test cases. In this article, we propose improving the cost-effectiveness of metamorphic testing by leveraging the feedback information obtained in the test execution process. Consequently, we develop a new approach, namely feedback-directed metamorphic testing, which makes use of test execution information to dynamically adjust the selection of metamorphic relations and selection of source test cases. We conduct an empirical study to evaluate the proposed approach based on four laboratory programs, one GNU program, and one industry program. The empirical results show that feedback-directed metamorphic testing can use fewer test cases and take less time than the traditional metamorphic testing for detecting the same number of faults. It is clearly demonstrated that the use of feedback information about test execution does help enhance the cost-effectiveness of metamorphic testing. Our work provides a new perspective to improve the efficacy and applicability of metamorphic testing as well as many other software testing techniques.

REFERENCES

[1] Ahlgren John, Berezin Maria Eugenia, Bojarczuk Kinga, Dulskyte Elena, Dvortsova Inna, George Johann, Gucevska Natalija, Harman Mark, Lomeli Maria, Meijer Erik, Sapora Silvia, and Spahr Justin. 2021. Testing web enabled simulation at scale using metamorphic testing. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP’21). 140–149.Google ScholarDigital Library
[2] Arcuri Andrea and Briand Lionel. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (ICSE’11). 1–10.Google ScholarDigital Library
[3] Asrafi Mahmuda, Liu Huai, and Kuo Fei-Ching. 2011. On testing effectiveness of metamorphic relations: A case study. In Proceedings of the 15th International Conference on Secure Software Integration and Reliability Improvement (SSIRI’11). 147–156.Google ScholarDigital Library
[4] Ayerdi Jon, Terragni Valerio, Arrieta Aitor, Tonella Paolo, Sagardui Goiuria, and Arratibel Maite. 2021. Generating metamorphic relations for cyber-physical systems with genetic programming: An industrial case study. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’21). 1264–1274.Google ScholarDigital Library
[5] Barr Earl T., Harman Mark, McMinn Phil, Shahbaz Muzammil, and Yoo Shin. 2015. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering 41, 5 (2015), 507–525.Google ScholarDigital Library
[6] Barus Arlinta Christy, Chen Tsong Yueh, Kuo Fei-Ching, Liu Huai, Merkel Robert, and Rothermel Gregg. 2016. A cost-effective random testing method for programs with non-numeric inputs. IEEE Transactions on Computers 65, 12 (2016), 3509–3523.Google ScholarDigital Library
[7] Barus Arlinta Christy, Chen Tsong Yueh, Kuo Fei-Ching, Liu Huai, and Schmidt Heinz W.. 2016. The impact of source test case selection on the effectiveness of metamorphic testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16), Co-Located with the 38th International Conference on Software Engineering (ICSE’16). 5–11.Google ScholarDigital Library
[8] Bourque Pierre and Fairley Richard E. (Eds.). 2014. SWEBOK: Guide to the Software Engineering Body of Knowledge (version 3.0 ed.). IEEE Computer Society, Los Alamitos, CA.Google Scholar
[9] Brilliant Susan S., Knight John C., and Ammann P. E.. 1990. On the performance of software testing using multiple versions. In Proceedings of the 20th International Symposium on Fault-Tolerant Computing (FTCS’90). 408–415.Google ScholarCross Ref
[10] Cai Kai-Yuan. 2002. Optimal software testing and adaptive software testing in the context of software cybernetics. Information and Software Technology 44, 14 (2002), 841–855.Google ScholarCross Ref
[11] Cai Kai-Yuan, Chen Tsong Yueh, and Tse T. H.. 2002. Towards research on software cybernetics. In Proceedings of the 7th IEEE International Symposium on High Assurance Systems Engineering (HSE’02). 240–241.Google Scholar
[12] Cai Kai-Yuan, Gu Bo, Hu Hai, and Li Yong-Chao. 2007. Adaptive software testing with fixed-memory feedback. Journal of Systems and Software 80, 8 (2007), 1328–1348.Google ScholarDigital Library
[13] Cangussu João W., Cai Kai-Yuan, Miller Scott D., and Mathur Aditya P.. 2007. Software cybernetics. Wiley Encyclopedia of Computer Science and Engineering (2007). Google ScholarCross Ref
[14] Chan Alvin, Ma Lei, Juefei-Xu Felix, Ong Yew-Soon, Xie Xiaofei, Xue Minhui, and Liu Yang. 2021. Breaking neural reasoning architectures with metamorphic relation-based adversarial examples. IEEE Transactions on Neural Networks and Learning Systems (2021). Google ScholarCross Ref
[15] Chen Tsong Yueh, Cheung Shing C., and Yiu Shiu Ming. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report. Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, Tech. Rep. HKUST-CS98-01.Google Scholar
[16] Chen Tsong Yueh, Ho Joshua W. K., Liu Huai, and Xie Xiaoyuan. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10, 1 (2009), 24–32.Google ScholarCross Ref
[17] Chen Tsong Yueh, Huang D. H., Tse T. H., and Zhou Zhi Quan. 2004. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th lberoAmerican Symposium on Software Engineering and Knowledge Engineer-ing (JIISIC’04). 569–583.Google Scholar
[18] Chen Tsong Yueh, Kuo Fei-Ching, Liu Huai, Poon Pak-Lok, Towey Dave, Tse T. H., and Zhou Zhi Quan. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys 51, 1 (2018), 4:1–4:27.Google Scholar
[19] Chen Tsong Yueh, Kuo Fei-Ching, Liu Ying, and Tang Antony. 2004. Metamorphic testing and testing with special values. In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD’04). 128–134.Google Scholar
[20] Chen Tsong Yueh, Kuo Fei-Ching, Ma Wenjuan, Susilo Willy, Towey Dave, Voas Jeffrey, and Zhou Zhi Quan. 2016. Metamorphic testing for cybersecurity. Computer 49, 6 (2016), 48–55.Google ScholarDigital Library
[21] Chen Tsong Yueh, Leung Hing, and Mak Ieng Kei. 2004. Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference (ASIAN’04). 320–329.Google ScholarDigital Library
[22] Chen Tsong Yueh, Poon Pak-Lok, and Xie Xiaoyuan. 2016. METRIC: Metamorphic relation identification based on the category-choice framework. Journal of Systems and Software 116 (2016), 177–190.Google ScholarDigital Library
[23] Chen Tsong Yueh and Tse T. H.. 2021. New visions on metamorphic testing after a quarter of a century of inception. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’21). 1487–1490.Google ScholarDigital Library
[24] Chen Tsong Yueh and Yu Yuen-Tak. 1994. On the relationship between partition and random testing. IEEE Transactions on Software Engineering 20, 12 (1994), 977–980.Google ScholarDigital Library
[25] Coppit David and Haddox-Schatz Jennifer M.. 2005. On the use of specification-based assertions as test oracles. In Proceedings of the 29th IEEE/NASA Software Engineering Workshop (SEW’05). 305–314.Google ScholarDigital Library
[26] Do Hyunsook, Elbaum Sebastian, and Rothermel Gregg. 2005. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering 10, 4 (2005), 405–435.Google ScholarDigital Library
[27] Dong Guowei, Guo Tao, and Zhang Puhan. 2013. Security assurance with program path analysis and metamorphic testing. In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13). 193–197.Google Scholar
[28] Project GNU. 2006. GNU grep. (2006). Retrieved from http://www.gnu.org/software/grep.Google Scholar
[29] Grechanik Mark, Fu Chen, and Xie Qing. 2012. Automatically finding performance problems with feedback-directed learning software testing. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). 156–166.Google ScholarDigital Library
[30] Hamlet Richard. 2002. Random testing. In Proceedings of the Encyclopedia of Software Engineering. John Wiley & Sons, Inc.Google ScholarCross Ref
[31] He Pinjia, Meister Clara, and Su Zhendong. 2020. Structure-invariant testing for machine translation. In Proceedings of the 42nd International Conference on Software Engineering (ICSE’20). 961–973.Google ScholarDigital Library
[32] He Xiao, Wang Xingwei, Shi Jia, and Liu Yi. 2020. Testing high performance numerical simulation programs: Experience, lessons learned, and open issues. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’20). 502–515.Google ScholarDigital Library
[33] Hedges Larry and Olkin Ingram. 2014. Statistical Methods for Meta-Analysis. Academic Press.Google Scholar
[34] Hughes John. 2019. How to specify it!. In Proceedings of the 20th International Symposium on Trends in Functional Programming (TFP’19) (Lecture Notes in Computer Science), Vol. 12053. 58–83.Google Scholar
[35] Hui Zhan-Wei, Huang Song, Chen Tsong Yueh, Lau Man F., and Ng Sebastian. 2017. Identifying failed test cases through metamorphic testing. In Proceedings of the 28th International Symposium on Software Reliability Engineering Workshops (ISSREW’17). 90–91.Google ScholarCross Ref
[36] Just René and Schweiggert Franz. 2010. Automating software tests with partial oracles in integrated environments. In Proceedings of the 5th Workshop on Automation of Software Test (AST’10). 91–94.Google ScholarDigital Library
[37] Lemieux Caroline, Padhye Rohan, Sen Koushik, and Song Dawn. 2018. Perffuzz: Automatically generating pathological inputs. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’18). 254–265.Google ScholarDigital Library
[38] Lemieux Caroline and Sen Koushik. 2018. Fairfuzz: A targeted mutation strategy for increasing greybox fuzz testing coverage. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 475–485.Google ScholarDigital Library
[39] Li Ye, Yin Bei-Bei, Lv Junpeng, and Cai Kai-Yuan. 2015. Approach for test profile optimization in dynamic random testing. In Proceedings of the 39th International Computer Software and Applications Conference (COMPSAC’15), Vol. 3. 466–471.Google ScholarDigital Library
[40] Liu Huai, Kuo Fei-Ching, Towey Dave, and Chen Tsong Yueh. 2014. How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering 40, 1 (2014), 4–22.Google ScholarDigital Library
[41] Liu Huai, Liu Xuan, and Chen Tsong Yueh. 2012. A new method for constructing metamorphic relations. In Proceedings of the 12th International Conference on Quality Software (QSIC’12). 59–68.Google ScholarDigital Library
[42] Lv Junpeng, Hu Hai, and Cai Kai-Yuan. 2011. A sufficient condition for parameters estimation in dynamic random testing. In Proceedings of the 35th IEEE Annual International Computer Software and Applications Conference Workshops (COMPSACW’11). 19–24.Google ScholarDigital Library
[43] Lv Junpeng, Yin Bei-Bei, and Cai Kai-Yuan. 2014. On the asymptotic behavior of adaptive testing strategy for software reliability assessment. IEEE Transactions on Software Engineering 40, 4 (2014), 396–412.Google ScholarDigital Library
[44] Ma Pingchuan, Wang Shuai, and Liu Jin. 2020. Metamorphic testing and certified mitigation of fairness violations in NLP models. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI’20). 458–465.Google ScholarCross Ref
[45] Ma Yu-Seung, Offutt Jeff, and Kwon Yong Rae. 2005. MuJava: An automated class mutation system. Software Testing, Verification and Reliability 15, 2 (2005), 97–133.Google ScholarCross Ref
[46] Mai Phu X., Pastore Fabrizio, Goknil Arda, and Briand Lionel C.. 2020. Metamorphic security testing for web systems. In Proceedings of the 13th IEEE International Conference on Software Testing, Validation and Verification (ICST’20). 186–197.Google ScholarCross Ref
[47] Mansur Muhammad Numair, Christakis Maria, and Wüstholz Valentin. 2021. Metamorphic testing of datalog engines. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’21). 639–650.Google ScholarDigital Library
[48] Marijan Dusica, Gotlieb Arnaud, and Ahuja Mohit Kumar. 2019. Challenges of testing machine learning based systems. In Proceedings of the 1st IEEE International Conference On Artificial Intelligence Testing (AITest’19). 101–102.Google ScholarCross Ref
[49] Mayer Johannes and Guderlei Ralph. 2006. An empirical study on the selection of good metamorphic relations. In Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC’06). 475–484.Google ScholarDigital Library
[50] McNutt Andrew, Kindlmann Gordon, and Correll Michael. 2020. Surfacing visualization mirages. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI’20). 1–16.Google ScholarDigital Library
[51] Ostrand Thomas J. and Balcer Marc J.. 1988. The category-partition method for specifying and generating fuctional tests. Communications of the ACM 31, 6 (1988), 676–686.Google ScholarDigital Library
[52] Pacheco Carlos, Lahiri Shuvendu K, and Ball Thomas. 2008. Finding errors in. net with feedback-directed random testing. In Proceedings of the 7th International Symposium on Software Testing and Analysis (ISSTA’08). 87–96.Google ScholarDigital Library
[53] Pacheco Carlos, Lahiri Shuvendu K., Ernst Michael D., and Ball Thomas. 2007. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07). 75–84.Google ScholarDigital Library
[54] Patel Krishna and Hierons Robert M. 2018. A mapping study on testing non-testable systems. Software Quality Journal 26, 4 (2018), 1373–1413.Google ScholarDigital Library
[55] Qiu Kun, Zheng Zheng, Chen Tsong Yueh, and Poon Pak-Lok. 2022. Theoretical and empirical analyses of the effectiveness of metamorphic relation composition. IEEE Transactions on Software Engineering 48, 3 (2022), 1001–1007.Google ScholarCross Ref
[56] Ribeiro Marco Túlio, Wu Tongshuang, Guestrin Carlos, and Singh Sameer. 2020. Beyond accuracy: Behavioral testing of NLP models with checkList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL’20). 4902–4912.Google ScholarCross Ref
[57] Sawilowsky Shlomo S.. 2009. New effect size rules of thumb. Journal of Modern Applied Statistical Methods 8, 2 (2009), 467–474.Google ScholarCross Ref
[58] Segall Itai and Tzoref-Brill Rachel. 2015. Feedback-driven combinatorial test design and execution. In Proceedings of the 8th ACM International Systems and Storage Conference. 1–6.Google ScholarDigital Library
[59] Segura Sergio, Fraser Gordon, Sanchez Ana B., and Ruiz-Cortés Antonio. 2016. A survey on metamorphic testing. IEEE Transactions on Software Engineering 42, 9 (2016), 805–824.Google ScholarCross Ref
[60] Segura Sergio, Parejo José A., Troya Javier, and Ruiz-Cortés Antonio. 2018. Metamorphic testing of RESTful web APIs. IEEE Transactions on Software Engineering 44, 11 (2018), 1083–1099.Google ScholarCross Ref
[61] Spieker Helge and Gotlieb Arnaud. 2020. Adaptive metamorphic testing with contextual bandits. Journal of Systems and Software 165 (2020), 110574:1–110574:14.Google ScholarCross Ref
[62] Sun Chang-ai, Dai Hepeng, Liu Huai, Chen Tsong Yueh, and Cai Kai-Yuan. 2018. Adaptive partition testing. IEEE Transactions on Computers 68, 2 (2018), 157–169.Google ScholarDigital Library
[63] Sun Chang-ai, Dai Hepeng, Wang Guan, Towey Dave, Chen Tsong Yueh, and Cai Kai-Yuan. 2022. Dynamic random testing of web services: A methodology and evaluation. IEEE Transactions on Services Computing 15, 2 (2022), 736–751.Google ScholarCross Ref
[64] Sun Chang-ai, Fu An, Poon Pak-Lok, Xie Xiaoyuan, Liu Huai, and Chen Tsong Yueh. 2021. METRIC+: A metamorphic relation identification technique based on input plus output domains. IEEE Transactions on Software Engineering 47, 9 (2021), 1765–1786.Google Scholar
[65] Sun Chang-ai, Liu Yiqiang, Wang Zuoyi, and Chan W. K.. 2016. \(\mu\)MT: A data mutation directed metamorphic relation acquisition methodology. In Proceedings of 2016 IEEE/ACM the 1st International Workshop on Metamorphic Testing (MET’16), Co-Located with the 38th International Conference on Software Engineering (ICSE’16). 12–18.Google Scholar
[66] Sun Chang-ai, Wang Zuoyi, and Wang Guan. 2014. A property-based testing framework for encryption programs. Frontiers of Computer Science 8, 3 (2014), 478–489.Google ScholarDigital Library
[67] Tian Yuchi, Pei Kexin, Jana Suman, and Ray Baishakhi. 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). 303–314.Google ScholarDigital Library
[68] Xie Xiaoyuan, Wong W. Eric, Chen Tsong Yueh, and Xu Baowen. 2013. Metamorphic slice: An application in spectrum-based fault localization. Information and Software Technology 55, 5 (2013), 866–879.Google ScholarDigital Library
[69] Yang Hongji, Chen Feng, and Aliyu Suleiman. 2017. Modern software cybernetics: New trends. Journal of System and Software 124 (2017), 157–169.Google ScholarDigital Library
[70] Yang Zijiang, Yin Beibei, Lv Junpeng, Cai Kai-Yuan, Yau Stephen S., and Yu Jia. 2014. Dynamic random testing with parameter adjustment. In Proceedings of the 38th IEEE Annual International Computer Software and Applications Conference Workshops (COMPSACW’14). 37–42.Google ScholarDigital Library
[71] Yuan Yuanyuan, Wang Shuai, Jiang Mingyue, and Chen Tsong Yueh. 2021. Perception matters: Detecting perception failures of VQA models using metamorphic testing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). 16908–16917.Google ScholarCross Ref
[72] Zhang Jie, Chen Junjie, Hao Dan, Xiong Yingfei, Xie Bing, Zhang Lu, and Mei Hong. 2014. Search-based inference of polynomial metamorphic relations. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE’14). 701–712.Google ScholarDigital Library
[73] Zhang Jie Ming, Harman Mark, Ma Lei, and Liu Yang. 2022. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering 48, 1 (2022), 1–36.Google ScholarDigital Library
[74] Zhang Lei, Yin Bei-Bei, Lv Junpeng, Cai Kai-Yuan, Yau Stephen S., and Yu Jia. 2014. A history-based dynamic random software testing. In Proceedings of the 38th International Computer Software and Applications Conference Workshops (COMPSACW’14). 31–36.Google ScholarDigital Library

Index Terms

Feedback-Directed Metamorphic Testing
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Metamorphic Testing: A Review of Challenges and Opportunities

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and ...
Read More
Fault detection effectiveness of source test case generation strategies for metamorphic testing
MET '18: Proceedings of the 3rd International Workshop on Metamorphic Testing

Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is ...
Read More
On Testing Effectiveness of Metamorphic Relations: A Case Study
SSIRI '11: Proceedings of the 2011 Fifth International Conference on Secure Software Integration and Reliability Improvement

One fundamental challenge for software testing is the oracle problem, which means that either there does not exist a mechanism (called oracle) to verify the test output given any possible program input, or it is very expensive, if not impossible, to ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Transactions on Software Engineering and Methodology Volume 32, Issue 1
January 2023
954 pages
ISSN:1049-331X
EISSN:1557-7392
DOI:10.1145/3572890
Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 13 February 2023
- Online AM: 24 May 2022
- Accepted: 22 April 2022
- Revised: 25 February 2022
- Received: 6 August 2021
Published in tosem Volume 32, Issue 1

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Metamorphic testing
metamorphic relation
test execution
feedback control
random testing
adaptive partition testing
Qualifiers
- research-article
- Refereed
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 0
  Total Citations
  View Citations
- 691
  Total Downloads
- Downloads (Last 12 months)407
- Downloads (Last 6 weeks)31
Other Metrics
View Author Metrics
Cited By
This publication has not been cited yet

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

View Full Text

HTML Format

View this article in HTML Format .

View HTML Format

Feedback-Directed Metamorphic Testing

ACM Transactions on Software Engineering and Methodology

Abstract

REFERENCES

Cited By

Index Terms

Recommendations

Metamorphic Testing: A Review of Challenges and Opportunities

Fault detection effectiveness of source test case generation strategies for metamorphic testing

On Testing Effectiveness of Metamorphic Relations: A Case Study