ON THE SYSTEM-LEVEL EFFECTIVENESS OF PHYSICAL OBJECT-HIDING ADVERSARIAL ATTACK IN AUTONOMOUS DRIVING

Abstract

In Autonomous Driving (AD) systems, perception is crucial for both security and safety. Among the different attacks on AD perception, the physical object-hiding adversarial attacks are especially severe due to their direct impact on road safety. However, we find that all existing works so far only evaluate their attack effect at the targeted AI component level, without any evaluation at the system level, i.e., with the entire system semantics and context such as the full AD system pipeline and closed-loop control. This thus inevitably raise a critical research question: can these existing research efforts actually effectively achieve the desired system-level attack effects (e.g., causing vehicle collisions, traffic rule violations, etc.) in the real-world AD system context? In the paper, we perform the first measurement study on whether and how effective the existing designs can lead to system-level effects, where we take the STOP sign-hiding attack as our target. Our evaluation results show that all the representative prior works cannot achieve any systemlevel effect in a classical closed-loop AD setup in road speeds controlled by common STOP signs. We then point out two limitation hypotheses that appear in all existing works: 1) the unpractical STOP sign size distribution in pixel sampling, and 2) missing particular consideration in system-critical attack range. Our results demonstrate that after overcoming these two limitations, the system-level effects can be further improved, i.e., the violation rate can increase around 70%.

1. INTRODUCTION

Autonomous Driving (AD) vehicles are now a reality in our daily life, where a wide variety of commercial and private AD vehicles are driving on the road. For example, the millions of Tesla cars (Kane, 2021) are equipped with Autopilot (Tesla, 2022) . To ensure safe and correct driving, a fundamental pillar in the AD system is perception, which is designed to detect surrounding objects in real time. Due to the direct impact on safety-critical driving decisions such as collision avoidance, various prior works have studied the security of AD perception, especially the ones that aim at causing the disappearance of critical physical road objects (e.g., STOP signs), or physical objecthiding adversarial attacks (Jia et al., 2022; Xu et al., 2020; Chen et al., 2018; Wu et al., 2020) . Although a plethora of prior works studied such physical object-hiding adversarial attacks in AD settings, we find that all of them only evaluate their attack effect at the targeted AI component level (i.e., judged by per-frame object misdetection rates (Chen et al., 2018; Eykholt et al., 2018; Xu et al., 2020; Zhao et al., 2019; Jia et al., 2022) ), without any evaluation at the system level, i.e., with the full system semantics and context enclosing such AI component (e.g., the remaining AD system pipeline such as object tracking, planning, and control, closed-loop control, and the attack-targeted driving scenario), which we call the system model for such adversarial attacks in this paper ( §2). This thus inevitably raises a critical research question: can these existing works on physical objecthiding adversarial attacks effectively achieve the desired system-level attack effects (e.g., causing vehicle collisions, traffic rule violations, etc.) in the real-world AD system context? To systematically answer this critical research question, we take the necessary first step by performing a measurement study on prior works with regard to their capabilities in causing system-level effects. We select STOP sign-hiding attack as our target considering its high representativeness in physical object-hiding adversarial attack today (Shen et al., 2022) , and its direct impacts on driving correctness and road safety. We first classify the existing STOP sign-hiding adversarial attacks based on targeted object detection model designs, and then for each model design, we select the most effective attack design published so far to perform system-level effect measurement. Due to the availability of source code, we reproduce multiple STOP sign-hiding adversarial attack works. Then, we design a simulation-centric evaluation platform to perform the measurement study. More details will be introduced in §3. Our results show that all the representative existing works, can not cause any STOP sign traffic rule violation against a representative closed-loop control AD system in the common speed range for STOP sign-controlled roads in the real world even if the most effective attack can achieve more than 80% average attack success rate in general on the AI component alone. We further explore the root causes and find that all the existing works have design limitation to achieve effective system-level effects due to lack of consideration of system model in AD context. We propose two design limitation hypotheses: 1) the unpractical STOP sign size distribution in pixel sampling, and 2) missing particular consideration in system-critical attack range, which will be detailed in §4. With that, we propose system model-driven attack design, which can be an addon of the existing attack methodologies to improve system-level effects by overcoming the two limitations. We evaluate our attack improvement in the platform we designed and show that the system-level effect can be significant improved, i.e., the system violation rate can be increased around 70%. Ablation studies are also included in the evaluation, which shows the improvement on both componentand system-level for the setting with anyone of the hypothesis mentioned above and obtains the best results after applying both two hypotheses. Our code will be released after the double-blind review. Note that we do not intend to claim to be the first to point out, analyze, measure, or optimize the gap between AI component errors and their system-level effect in general; there exists a large body of prior works in various other problem contexts (e.g., camera surveillance, video analytics, and control) across academia and industry that have studied the characterization, modeling, and/or optimization of end-to-end system performance with regard to AI/vision component errors (Jain & Binford, 1991; Ramesh et al., 1997; Thacker et al., 2008; Haralick, 1992; Ji & Haralick, 1999; Zhang & Zhu, 2018; Phillips et al., 2021; Greiffenhagen et al., 2000; 2001a; b; Philion et al., 2020; Caesar et al., 2020; Topan et al., 2022; Gog et al., 2021) . Nevertheless, to the best of our knowledge, none of them (1) quantified such gaps in the context of adversarial attacks on autonomous systems, especially those in real-world system setups; and (2) identified novel designs that can systematically address/fill such gaps on autonomous systems, which we believe are our novel and unique contributions. Contributions. To sum up, this paper makes the following contributions: • We are the first to perform a comprehensive measurement study on the system-level effect of the representative prior works with the entire AD system pipeline with closed-loop control on our designed simulation-centric evaluation platform. Our results indicate that all the representative existing works, cannot cause any STOP sign traffic rule violation in common speed range for STOP sign-controlled roads in real world. • We point out two design limitations of the prior works to hinder them in better achieving the system-level effects and propose system model-driven attack designs to overcome these. • We further evaluate the validity of the two design limitations proposed in this paper and show that with our novel designs, the system-level effect can be significantly improved, i.e., the system violation rate can be increased around 70% YOLO v3 (Redmon & Farhadi, 2018), and YOLO v5 (Jocher, 2022) , usually has higher detection speed, while the latter, such as Faster R-CNN (Ren et al., 2015) , usually has higher detection accuracy. Since one-stage object detector processes bounding box (BBox) regression and object classification concurrently without a region proposal stage, it is generally much faster than twostage ones and thus can better meet the real-time requirement in AD context (Carranza-García et al., 2020) . In this paper, we focus on the security aspects of camera-based AD perception, and perform the corresponding experiments on both object detector categories.

