Ablation Path Saliency

Abstract

We consider the saliency problem for black-box classification. In image classification, this means highlighting the part of the image that is most relevant for the current decision. We cast the saliency problem as finding an optimal ablation path between two images. An ablation path consists of a sequence of ever smaller masks, joining the current image to a reference image in another decision region. The optimal path will stay as long as possible in the current decision region. This approach extends the ablation tests in Sturmfels et al. (2020). The gradient of the corresponding objective function is closely related to the integrated gradient method Sundararajan et al. ( 2017). In the saturated case (when the classifier outputs a binary value) our method would reduce to the meaningful perturbation approach Fong & Vedaldi (2017), since crossing the decision boundary as late as possible would then be equivalent to finding the smallest possible mask lying on the decision boundary. Our interpretation provides geometric understanding of existing saliency methods, and suggests a novel approach based on ablation path optimisation.

1. Introduction

The basic idea of saliency or attribution is to provide something from which a human can judge how a classifier arrived at its decision of the prediction it gives for a certain input. It is difficult to give a more mathematical definition, but various properties that such a method should fulfill have been proposed. Sundararajan et al. (2017) give axioms, of which sensitivity comes closest to the notion of saliency. Essentially, the features on which the output is most sensitive should be given a higher saliency value. The authors give further axioms to narrow it down -implementation invariance, completeness, linearity and symmetry-preservation -and obtain a corresponding method: the integrated gradient method. Note that we have another way to arrive at a similar method, see §4.1 Fong & Vedaldi ( 2017) is closer to our work: the authors directly compute the saliency of a given pixel by deleting, or altering that pixel, to see how this affects the output of the classifier. Our method is to define a proper maximisation problem as follows. First, we define ablation paths as time dependent smooth masks φ : [0, 1] → C(Ω, R), going a full mask to an empty mask, such that at each pixel the mask value decreases over time (see Figure 1 ). We also impose constant area speed: the area covered by the mask should increase linearly over time (see §3). Let F be the classifier, that outputs a probability between zero and one. We choose a current image of interest x 0 and a baseline image x 1 . The objective function is then P (φ) = ∫ 1 0 F (x 0 + φ(t)(x 1 -x 0 )) dt (see §4 ). Assuming that F (x 0 ) ≃ 1 and F (x 1 ) ≃ 0, maximising the objective function means that we try to find an ablation path that stays as long as possible in the decision region of x 0 . Intuitively, we try to replace as many pixels of x 0 by pixels of x 1 while staying in the same class as x 0 . The main contribution of this paper is to formulate the saliency problem as an optimisation problem on paths. Doing so connects previous notions of saliency, notably integrated gradients Sundararajan et al. ( 2017 2019) (see § 4.3). Our formulation is also resolution invariant (it does not assume that images are defined on pixels); this allows to make a clear difference between images and their duals, for instance, which gives guidance as to where regularisation is needed.

2. Related Work

Simonyan et al. ( 2013) defines a saliency map as the gradient of the network output at the given image. This would appear to be a sensible definition, but the resulting saliency is very noisy because the network output is roughly constant around any particular image. Selvaraju et al. ( 2016) improves the situation by computing the gradient after each layer instead. This is, however, not a black-box method such as the one we propose. Koh & Liang (2017) computes an influence function, that is, a function that measures how the parameters would be changed by a change in the training data. Although it is a black-box method, it is not a saliency method per se. They use the gradient of the network output to find the pixel most likely to have a high saliency. The pixel that have most effect are given a higher saliency. By contrast, Petsiuk et al. (2018) proposes to directly evaluate the saliency by finding out which pixels most affect the output, similarly to Fong & Vedaldi (2017), but without using any gradients. There are a number of meta-studies of saliency methods. 3 Ablation Paths

3.1. Images and Masks

We consider a compact domain Ω. Note that Ω may be discrete or continuous: in fact, we assume that Ω is endowed with a measure which could be the discrete measure (if Ω is a set of pixels) or the Lebesgue measure (if Ω is a domain in R 2 , for instance). In the sequel, ∫ Ω denotes integration with respect to that measure. Without loss of generality, we assume the mass of that measure to be one, i.e., ∫ Ω 1 = 1. We consider a module M of functions on Ω with values in a vector space V (the dimensions of V represent the channels, and elements of M represent images). This module is equipped with a commutative ring R which represents masks. Concretely, in most of the paper we choose M := C(Ω, V ) R := C(Ω, R). The module structure simply means that masks can multiply images, i.e., that the operation θx gives a new image in M when θ ∈ R and x ∈ M , and that this operation is bilinear.

3.2. Ablation Paths

Definition 3.1. We define the set A of ablation paths as the set of functions φ : [0, 1] → R fulfilling the following properties: Boundary conditions φ(0) = 0 and φ(1) = 1



Figure 1: Example of how an ablation path (sequence of masks, top) gives rise to a transition between a current target (image of a house finch) and a baseline (orca whale).

) (see §4.1), the ablation tests Sturmfels et al. (2020) (see §4.2), and meaningful perturbations Fong & Vedaldi (2017); Fong et al. (

Adebayo et al. (2018) lists essential properties, for instance the requirement that the results should depend on the training data in a sense that perturbing model parameters should change the saliency. Kindermans et al. (2017) proposes a number of property that saliency methods should satisfy. Ancona et al. (2017) compares several saliency methods and proposes a method to evaluate them (the sensitiviy-n property).

