PhysCov: Physical Test Coverage for Autonomous Vehicles
Published:
Abstract:Adequately covering the behaviors of autonomous vehicles ($AV$) is fundamental in their validation. However, quantifying such coverage is challenging as the $AV$s’ behavior is influenced by its physical environment that is often large and highly complex. This work builds on the insights that data sensed by the $AV$ provides a unique spatial signature of the environment inputs and that inputs which reside outside the $AV$’s physically reachable regions are less relevant. Building on those insights, we introduce a new abstraction, RSR, and corresponding coverage metric, PhysCov. RSR integrates the sensor readings with a physical reachability analysis based on the vehicle’s state and dynamics to determine the input region that may affect the $AV$. It then characterizes that region through a parameterizable geometric approximation that can trade quality for cost. Environments with distinct approximations likely result in distinct behavior and thus will increase PhysCov. This paper also provides a study on two $AV$’s running on two different simulators that shows the applicability of RSR to generate PhysCov. The study also demonstrates PhysCov’s ability to quantify an $AV$’s test suite coverage, showcases its characterization cost and precision, and highlights its value in terms of its high-positive correlation with the number of vehicle crashes found.
Recommended citation: Carl Hildebrandt, Meriel von Stein, and Sebastian Elbaum. 2023. PhysCov: Physical Test Coverage for Autonomous Vehicles. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2023). Association for Computing Machinery, New York, NY, USA, 449–461. https://doi.org/10.1145/3597926.3598069