BATCH MULTIVALID CONFORMAL PREDICTION

Abstract

We develop fast distribution-free conformal prediction algorithms for obtaining multivalid coverage on exchangeable data in the batch setting. Multivalid coverage guarantees are stronger than marginal coverage guarantees in two ways: (1) They hold even conditional on group membership-that is, the target coverage level 1 -α holds conditionally on membership in each of an arbitrary (potentially intersecting) group in a finite collection G of regions in the feature space. (2) They hold even conditional on the value of the threshold used to produce the prediction set on a given example. In fact multivalid coverage guarantees hold even when conditioning on group membership and threshold value simultaneously. We give two algorithms: both take as input an arbitrary non-conformity score and an arbitrary collection of possibly intersecting groups G, and then can equip arbitrary black-box predictors with prediction sets. Our first algorithm BatchGCP is a direct extension of quantile regression, needs to solve only a single convex minimization problem, and produces an estimator which has group-conditional guarantees for each group in G. Our second algorithm BatchMVP is iterative, and gives the full guarantees of multivalid conformal prediction: prediction sets that are valid conditionally both on group membership and non-conformity threshold. We evaluate the performance of both of our algorithms in an extensive set of experiments.

1. INTRODUCTION

Consider an arbitrary distribution D over a labeled data domain Z = X × Y. A model is any function h : X → Y for making point predictions. The traditional goal of conformal prediction in the "batch" setting is to take a small calibration dataset consisting of labeled examples sampled from D and use it to endow an arbitrary model h : X → Y with prediction sets T h (x) ⊆ Y that have the property that these prediction sets cover the true label with probability 1 -α marginally for some target miscoverage rate α: Pr (x,y)∼D [y ∈ T h (x)] = 1 -α. This is a marginal coverage guarantee because the probability is taken over the randomness of both x and y, without conditioning on anything. In the batch setting (unlike in the sequential setting), labels are not available when the prediction sets are deployed. Our goal in this paper is to give simple, practical algorithms in the batch setting that can give stronger than marginal guarantees -the kinds of multivalid guarantees introduced by Gupta et al. ( 2022 Following the literature on conformal prediction (Shafer and Vovk, 2008) , our prediction sets are parameterized by an arbitrary non-conformity score s h : Z → R defined as a function of the model h. Informally, smaller values of s h (x, y) should mean that the label y "conforms" more closely to the prediction h(x) made by the model. For example, in a regression setting in which Y = R, the simplest non-conformity score is s h (x, y) = |h(x) -y|. By now there is a large literature giving more sophisticated non-conformity scores for both regression and classification problemssee Angelopoulos and Bates (2021) for an excellent recent survey. A non-conformity score function s h (x, y) induces a distribution over non-conformity scores, and if τ is the 1 -α quantile of this distribution (i.e. Pr (x,y)∼D [s h (x, y) ≤ τ ] = 1 -α), then defining prediction sets as T τ h (x) = {y : s h (x, y) ≤ τ ] gives 1 -α marginal coverage. Split conformal prediction (Papadopoulos et al., 2002; Lei et al., 2018) simply finds a threshold τ that is an empirical 1 -α quantile on the calibration set, and then uses this to deploy the prediction sets T τ h (x) defined above. Our goal is to give stronger coverage guarantees, and to do so, rather than learning a single threshold τ from the calibration set,



);Bastani et al. (2022)  in the sequential prediction setting.

