DYNAMIC NEURAL NETWORK IS ALL YOU NEED: UNDERSTANDING THE ROBUSTNESS OF DYNAMIC MECHANISMS IN NEURAL NETWORKS

Abstract

Deep Neural Networks (DNNs) have been used to solve different day-to-day problems. Recently, DNNs have been deployed in real-time systems, and lowering the energy consumption and response time has become the need of the hour. To address this scenario, researchers have proposed incorporating dynamic mechanism to static DNNs (SDNN) to create Dynamic Neural Networks (DyNNs) performing dynamic amount of computation based on the input complexity. Although incorporating dynamic mechanism into SDNNs would be preferable in real-time systems, it also becomes important to evaluate how the introduction of dynamic mechanism impacts the robustness of the models. However, there has not been a significant number of works focusing on the robustness trade-off between SDNNs and DyNNs. To address this issue, we propose to investigate four aspects of including dynamic mechanism into SDNNs. For that purpose, we evaluate four research questions. These evaluations are performed on three models and two datasets. Through the studies, we find that attack transferability from DyNNs to SDNNs is higher than attack transferability from SDNNs to DyNNs. Also, we find that DyNNs can be used to generate adversarial samples more efficiently than SDNNs. We also provide insight into the design choices through research studies. Finally, we propose a novel attack to understand the additional attack surface introduced by the dynamic mechanism.

1. INTRODUCTION

Deep Neural Networks (DNNs) are used in multiple applications such as computer vision and natural language processing. After the rapid growth of IoT and embedded devices, many real-time systems use DNNs in their applications. As the real-time systems require faster response time and low energy consumption, researchers have proposed to incorporate energy-saving dynamic mechanism (Wang et al., 2018; Kaya et al., 2019; Wu et al., 2018) to popular static DNN (SDNN) models like ResNet (He et al., 2015) , VGG (Simonyan & Zisserman, 2014), MobileNet (Howard et al., 2017) etc. Early-exit is one of the dynamic mechanism techniques where multiple exits are included in SDNNs (creating multiple sub-networks), and SDNNs can terminate the operation early if a certain sub-network is confident about the prediction. These types of DNNs are named as early-exit Dynamic Neural Networks (DyNNs). Although the transition from SDNNs to DyNNs is preferred in real time systems because of increased efficiency, whether the use of dynamic mechanism will impact the robustness of the systems is still unknown. Studying the impact of the dynamic mechanisms on the robustness is important for developers or users to understand the trade-offs between DyNN and SDNN. In this work, we propose to investigate four different aspects of including dynamic mechanism through four research questions. These four aspects are: Transferability, Impact on Efficiency, Earlyexits Design and, Added Attack Surface. Transferability. First, we investigate the adversarial attack transferability between SDNNs and DyNNs to evaluate the robustness of the models in black-box scenarios. In the black-box scenarios, adversaries normally assume the target models are always static. However, the target models can be

