Abstract
Extensive testing and simulation in different environments has been suggested as one piece of evidence for the safety of autonomous systems, e.g., in the automotive domain. To enable statements on the absolute number or fractions of tested scenarios, methods and tools for computing their coverage are needed. In this paper, we present STARS, a tool for specifying semantic environment features and measuring scenario coverage when testing autonomous systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Note: The data model used for the evaluation of the TSCs is described in [17].
References
Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Standard J3016, SAE International (2021). https://www.sae.org/standards/content/j3016_202104/
Standard for safety for the evaluation of autonomous products. Standard ANSI/UL 4600-2023, UL Standards & Engagement (2023). https://ul.org/UL4600
Amersbach, C., Winner, H.: Defining required and feasible test coverage for scenario-based validation of highly automated vehicles. In: ITSC 2019, IEEE (2019). https://doi.org/10.1109/itsc.2019.8917534
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: PMLR 2017, vol. 78. PMLR (2017). https://proceedings.mlr.press/v78/dosovitskiy17a.html
Duret-Lutz, A., Lewkowicz, A., Fauchille, A., Michaud, T., Renault, É., Xu, L.: Spot 2.0 — a framework for LTL and \(\omega \)-automata manipulation. In: Artho, C., Legay, A., Peled, D. (eds.) ATVA 2016. LNCS, vol. 9938, pp. 122–129. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46520-3_8
Esterle, K., Gressenbuch, L., Knoll, A.C.: Formalizing traffic rules for machine interpretability. In: CAVS 2020, IEEE (2020). https://doi.org/10.1109/CAVS51000.2020.9334599
Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The KITTI dataset. Int. J. Robot. Res. (IJRR) 32(11), 1231–1237 (2013). https://doi.org/10.1177/0278364913491297
Hildebrandt, C., von Stein, M., Elbaum, S.: PhysCov: physical test coverage for autonomous vehicles. In: ISSTA 2023, ACM (2023). https://doi.org/10.1145/3597926.3598069
Huck, T.P., Ledermann, C., Kröger, T.: Simulation-based testing for early safety-validation of robot systems. In: SPCE 2020, IEEE (2020). https://doi.org/10.1109/SPCE50045.2020.9296157
ISO central secretary: road vehicles - safety of the intended functionality. Standard ISO 21448:2022, International Organization for Standardization (2022). https://www.iso.org/standard/77490.html
Jenkins, I.R., Gee, L.O., Knauss, A., Yin, H., Schroeder, J.: Accident scenario generation with recurrent neural networks. In: ITSC 2018, IEEE (2018). https://doi.org/10.1109/itsc.2018.8569661
Li, C., Cheng, C.H., Sun, T., Chen, Y., Yan, R.: ComOpT: combination and optimization for testing autonomous driving systems. In: ICRA 2022, IEEE (2022). https://doi.org/10.1109/icra46639.2022.9811794
Majumdar, R., Mathur, A., Pirron, M., Stegner, L., Zufferey, D.: Paracosm: a language and tool for testing autonomous driving systems. Technical Report arXiv:1902.01084, arXiv (2021). https://doi.org/10.48550/arXiv.1902.01084
Mariani, R.: An overview of autonomous vehicles safety. In: IRPS 2018, IEEE (2018). https://doi.org/10.1109/irps.2018.8353618
Porres, I., Azimi, S., Lilius, J.: Scenario-based testing of a ship collision avoidance system. In: SEAA 2020, IEEE (2020). https://doi.org/10.1109/SEAA51224.2020.00090
Rizaldi, A., et al.: Formalising and monitoring traffic rules for autonomous vehicles in Isabelle/HOL. In: Polikarpova, N., Schneider, S. (eds.) IFM 2017. LNCS, vol. 10510, pp. 50–66. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66845-1_4
Schallau, T., Naujokat, S.: Validating behavioral requirements, conditions, and rules of autonomous systems with scenario-based testing. Electron Communications of the EASST 82 (2023). https://doi.org/10.14279/tuj.eceasst.82.1222
Schallau, T., Naujokat, S., Kullmann, F., Howar, F.: Tree-based scenario classification: a formal framework for coverage analysis on test drives of autonomous vehicles. Technical Report arXiv:2307.05106, arXiv (2023). https://doi.org/10.48550/arXiv.2307.05106, submitted to NFM 2024
Schmidt, T., Hauer, F., Pretschner, A.: Understanding safety for unmanned aerial vehicles in urban environments. In: IV 2021, IEEE (2021). https://doi.org/10.1109/IV48863.2021.9575755
Weber, H., et al.: A framework for definition of logical scenarios for safety assurance of automated driving. Traffic Injury Prev. 20(sup1) (2019). https://doi.org/10.1080/15389588.2019.1630827
Weng, B., Capito, L., Ozguner, U., Redmill, K.: Towards guaranteed safety assurance of automated driving systems with scenario sampling: an invariant set perspective. IEEE Trans. Intell. Veh. 7(3) (2022). https://doi.org/10.1109/tiv.2021.3117049
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Schallau, T., Mäckel, D., Naujokat, S., Howar, F. (2024). STARS: A Tool for Measuring Scenario Coverage When Testing Autonomous Robotic Systems. In: Sangchoolie, B., Adler, R., Hawkins, R., Schleiss, P., Arteconi, A., Mancini, A. (eds) Dependable Computing – EDCC 2024 Workshops. EDCC 2024. Communications in Computer and Information Science, vol 2078. Springer, Cham. https://doi.org/10.1007/978-3-031-56776-6_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-56776-6_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-56775-9
Online ISBN: 978-3-031-56776-6
eBook Packages: Computer ScienceComputer Science (R0)