HEATING UP DECISION BOUNDARIES: ISOCAPACITORY SATURATION, ADVERSARIAL SCENARIOS AND

Abstract

In the present work we study classifiers' decision boundaries via Brownian motion processes in ambient data space and associated probabilistic techniques. Intuitively, our ideas correspond to placing a heat source at the decision boundary and observing how effectively the sample points warm up. We are largely motivated by the search for a soft measure that sheds further light on the decision boundary's geometry. En route, we bridge aspects of potential theory and geometric analysis (Maz'ya (2011); Grigor'Yan & Saloff-Coste ( 2002)) with active fields of ML research such as adversarial examples and generalization bounds. First, we focus on the geometric behavior of decision boundaries in the light of adversarial attack/defense mechanisms. Experimentally, we observe a certain capacitory trend over different adversarial defense strategies: decision boundaries locally become flatter as measured by isoperimetric inequalities (Ford et al. ( 2019)); however, our more sensitive heat-diffusion metrics extend this analysis and further reveal that some non-trivial geometry invisible to plain distance-based methods is still preserved. Intuitively, we provide evidence that the decision boundaries nevertheless retain many persistent "wiggly and fuzzy" regions on a finer scale. Second, we show how Brownian hitting probabilities translate to soft generalization bounds which are in turn connected to compression and noise stability (Arora et al. ( 2018)), and these bounds are significantly stronger if the decision boundary has controlled geometric features.

1. INTRODUCTION AND BACKGROUND

The endeavor to understand certain geometric aspects of decision problems has lead to intense research in statistical learning. These range from the study of data manifolds, through landscapes of loss functions to the delicate analysis of a classifier's decision boundary. In the present work we focus on the latter. So far, a wealth of studies has analyzed the geometry of decision boundaries of deep neural networks (DNN), reaching profound implications in the fields of adversarial machine learning 2016)), we attempt to provide some new aspects of decision boundary analysis by introducing and studying a corresponding diffusion-inspired approach. In this note the guiding idea is to place a heat source at the classifier's decision boundary and estimate its size/shape in terms of the amount of heat the boundary is able to emit within a given time (Fig. 1 ). The goal is to extract geometric information from the behavior of heat transmission. This technique of heat content seems well-known within capacity/potential theory and has led to a variety of results in spectral analysis relating heat diffusion and geometry, Jorgenson & Lang (2001); Grigor 'Yan & Saloff-Coste (2002); Maz'ya (2011) . However, working with such heat diffusion directly in



(adversarial examples), robustness, margin analysis and generalization. Inspired by recent isoperimetric results and curvature estimates (Ford et al. (2019); Moosavi-Dezfooli et al. (2019); Fawzi et al. (

