Skip to main content
SpringerLink
Log in

Search-Based Selection and Prioritization of Test Scenarios for Autonomous Driving Systems

  • Conference paper
  • First Online:
Search-Based Software Engineering (SSBSE 2021)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 12914))

Included in the following conference series:

Abstract

Violating the safety of autonomous driving systems (ADSs) could lead to fatal accidents. ADSs are complex, constantly-evolving and software-intensive systems. Testing an individual ADS is challenging and expensive on its own, and consequently testing its multiple versions (due to evolution) becomes much more costly. Thus, it is needed to develop approaches for selecting and prioritizing tests for newer versions of ADSs based on historical test execution data of their previous versions. To this end, we propose a multi-objective search-based approach for Selection and Prioritization of tEst sCenarios for auTonomous dRiving systEms (SPECTRE) to test newer versions of an ADS based on four optimization objectives, e.g., demand of a test scenario put on an ADS. We experimented with five commonly used multi-objective evolutionary algorithms and used a repository of 60,000 test scenarios. Among all the algorithms, IBEA achieved the best performance for solving all the optimization problems of varying complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://apollo.auto/.

  2. 2.

    https://www.svlsimulator.com/.

  3. 3.

    https://www.sfmta.com.

  4. 4.

    https://github.com/ssbse2021/SPECTRE.

References

  1. Ali, S., Arcaini, P., Pradhan, D., Safdar, S.A., Yue, T.: Quality indicators in search-based software engineering: an empirical evaluation. ACM Trans. Softw. Eng. Methodol. (TOSEM) 29(2), 1–29 (2020)

    Article  Google Scholar 

  2. Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of the 33rd International Conference on Software Engineering (ICSE 2011), pp. 1–10 (2011)

    Google Scholar 

  3. Arcuri, A., Fraser, G.: On parameter tuning in search based software engineering. In: Cohen, M.B., Ó Cinnéide, M. (eds.) SSBSE 2011. LNCS, vol. 6956, pp. 33–47. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23716-4_6

    Chapter  Google Scholar 

  4. Ben Abdessalem, R., Nejati, S., Briand, L.C., Stifter, T.: Testing advanced driver assistance systems using multi-objective search and neural networks. In: Proceedings of the Conference on Automated Software Engineering, pp. 63–74. ACM (2016)

    Google Scholar 

  5. Ben Abdessalem, R., Nejati, S., Briand, L.C., Stifter, T.: Testing vision-based control systems using learnable evolutionary algorithms. In: Proceedings of the Conference on Software Engineering, pp. 1016–1026. ACM (2018)

    Google Scholar 

  6. Ben Abdessalem, R., Panichella, A., Nejati, S., Briand, L.C., Stifter, T.: Testing autonomous cars for feature interaction failures using many-objective search. In: Proceedings of the Conference on Automated Software Engineering, pp. 143–154. ACM (2018)

    Google Scholar 

  7. Corso, A., Du, P., Driggs-Campbell, K., Kochenderfer, M.J.: Adaptive stress testing with reward augmentation for autonomous vehicle validatio. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 163–168. IEEE (2019)

    Google Scholar 

  8. Czarnecki, K.: Operational design domain for automated driving systems: Taxonomy of basic terms. Waterloo Intelligent Systems Engineering (WISE) Lab, University of Waterloo, Canada (2018)

    Google Scholar 

  9. Durillo, J.J., Nebro, A.J.: jMetal: a Java framework for multi-objective optimization. Adv. Eng. Softw. 42(10), 760–771 (2011)

    Article  Google Scholar 

  10. Gambi, A., Mueller, M., Fraser, G.: Automatically testing self-driving cars with search-based procedural content generation. In: Proceedings of International Symposium on Software Testing and Analysis, pp. 318–328. ACM (2019)

    Google Scholar 

  11. Greer, D., Ruhe, G.: Software release planning: an evolutionary and iterative approach. Inf. Softw. Technol. 46(4), 243–253 (2004)

    Article  Google Scholar 

  12. Li, G., et al.: AV-FUZZER: finding safety violations in autonomous driving systems. In: International Symposium on Software Reliability Engineering, pp. 25–36. IEEE (2020)

    Google Scholar 

  13. Li, Z., Harman, M., Hierons, R.M.: Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng. 33(4), 225–237 (2007)

    Article  Google Scholar 

  14. Luo, Q., Moran, K., Poshyvanyk, D., Di Penta, M.: Assessing test case prioritization on real faults and mutants. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 240–251. IEEE (2018)

    Google Scholar 

  15. Pradhan, D., Wang, S., Ali, S., Yue, T., Liaaen, M.: STIPI: using search to prioritize test cases based on multi-objectives derived from industrial practice. In: Wotawa, F., Nica, M., Kushik, N. (eds.) ICTSS 2016. LNCS, vol. 9976, pp. 172–190. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47443-4_11

    Chapter  Google Scholar 

  16. Ramirez, A., Romero, J.R., Ventura, S.: A survey of many-objective optimisation in search based software engineering. Syst. Softw. Eng. 149, 382–395 (2019)

    Article  Google Scholar 

  17. Singh, Y., Kaur, A., Suri, B.: Test case prioritization using ant colony optimization. ACM SIGSOFT Softw. Eng. Notes 35(4), 1–7 (2010)

    Article  Google Scholar 

  18. Wang, S., Ali, S., Yue, T., Bakkeli, Ø., Liaaen, M.: Enhancing test case prioritization in an industrial setting with resource awareness and multi-objective search. In: Proceedings of the 38th International Conference on Software Engineering Companion, pp. 182–191 (2016)

    Google Scholar 

  19. Yoo, S., Harman, M.: Pareto efficient multi-objective test case selection. In: Proceedings of the International Symposium on Software Testing and Analysis, pp. 140–150 (2007)

    Google Scholar 

  20. Zhang, H., Zhang, M., Yue, T., Ali, S., Li, Y.: Uncertainty-wise requirements prioritization with search. ACM Trans. Softw. Eng. Methodol. (TOSEM) 30(1), 1–54 (2020)

    Article  Google Scholar 

  21. Zhang, M., Ali, S., Yue, T.: Uncertainty-wise test case generation and minimization for cyber-physical systems. J. Syst. Softw. 153, 1–21 (2019)

    Article  Google Scholar 

  22. Zitzler, E., Künzli, S.: Indicator-based selection in multiobjective search. In: Yao, X., et al. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 832–842. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30217-9_84

    Chapter  Google Scholar 

Download references

Acknowledgements

The work is supported by the National Natural Science Foundation of China under Grant No. 61872182. The work is also partially supported by the Co-evolver project (No. 286898/F20) funded by the Research Council of Norway. Huihui Zhang is supported by the Science and Technology Program of Public Wellbeing (No. 2020KJHM01).

Author information

Authors and Affiliations

  1. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Chengjie Lu & Tao Yue

  2. Weifang University, Weifang, China

    Huihui Zhang

  3. Simula Research Laboratory, Oslo, Norway

    Tao Yue & Shaukat Ali

Authors
  1. Chengjie Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huihui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tao Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shaukat Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chengjie Lu .

Editor information

Editors and Affiliations

  1. Massachusetts Institute of Technology, Cambridge, MA, USA

    Una-May O'Reilly

  2. Delft University of Technology, Delft, The Netherlands

    Xavier Devroey

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lu, C., Zhang, H., Yue, T., Ali, S. (2021). Search-Based Selection and Prioritization of Test Scenarios for Autonomous Driving Systems. In: O'Reilly, UM., Devroey, X. (eds) Search-Based Software Engineering. SSBSE 2021. Lecture Notes in Computer Science(), vol 12914. Springer, Cham. https://doi.org/10.1007/978-3-030-88106-1_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-88106-1_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88105-4

  • Online ISBN: 978-3-030-88106-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation