POST-HOC CONCEPT BOTTLENECK MODELS

Abstract

Concept Bottleneck Models (CBMs) map the inputs onto a set of interpretable concepts ("the bottleneck") and use the concepts to make predictions. A concept bottleneck enhances interpretability since it can be investigated to understand what concepts the model "sees" in an input and which of these concepts are deemed important. However, CBMs are restrictive in practice as they require dense concept annotations in the training data to learn the bottleneck. Moreover, CBMs often do not match the accuracy of an unrestricted neural network, reducing the incentive to deploy them in practice. In this work, we address these limitations of CBMs by introducing Post-hoc Concept Bottleneck models (PCBMs). We show that we can turn any neural network into a PCBM without sacrificing model performance while still retaining the interpretability benefits. When concept annotations are not available on the training data, we show that PCBM can transfer concepts from other datasets or from natural language descriptions of concepts via multimodal models. A key benefit of PCBM is that it enables users to quickly debug and update the model to reduce spurious correlations and improve generalization to new distributions. PCBM allows for global model edits, which can be more efficient than previous works on local interventions that fix a specific prediction. Through a model-editing user study, we show that editing PCBMs via conceptlevel feedback can provide significant performance gains without using data from the target domain or model retraining. The code for our paper can be found in https://github.com/mertyg/post-hoc-cbm.

1. INTRODUCTION

There is growing interest in developing deep learning models that are interpretable and yet still flexible. One approach is concept analysis (Kim et al., 2018) , where the goal is to understand if and how high-level human-understandable features are "engineered" and used by neural networks. For instance, we may like to probe a skin lesion classifier to understand if the Irregular Streaks concept is encoded in the embedding space of the classifier and used later to make the prediction. Our work builds on the earlier idea of concept bottlenecks, specifically Concept Bottleneck Models (CBMs) (Koh et al., 2020) . Concept bottlenecks are inspired by the idea that we can solve the task of interest by applying a function to an underlying set of human-interpretable concepts. For instance, when trying to classify whether a skin tumor is malignant, dermatologists look for different visual patterns, e.g. existence of Blue-Whitish Veils can be a useful indicator of melanoma (Menzies et al., 1996; Lucieri et al., 2020) . CBMs train an entire model in an end-to-end fashion by first predicting concepts (e.g. the presence of Blue-Whitish Veils), then using these concepts to predict the label.

By constraining the model to only rely on a set of concepts and an interpretable predictor, we can:

explain what information the model is using when classifying an input by looking at the weights/rules in the interpretable predictor and understand when the model made a particular mistake due to incorrect concept predictions. While CBMs provide several of the benefits mentioned above, they have several key limitations: 1. Data: CBMs require access to concept labels during model training, i.e. training data should be annotated with which concepts are present. Even though there are a number of densely annotated datasets such as CUB (Wah et al., 2011) , this is particularly restrictive for real-world use cases, where training datasets rarely have concept annotations.

