SELECTING TREATMENT EFFECTS MODELS FOR DOMAIN ADAPTATION USING CAUSAL KNOWLEDGE Anonymous authors Paper under double-blind review

Abstract

Selecting causal inference models for estimating individualized treatment effects (ITE) from observational data presents a unique challenge since the counterfactual outcomes are never observed. The problem is challenged further in the unsupervised domain adaptation (UDA) setting where we only have access to labeled samples in the source domain, but desire selecting a model that achieves good performance on a target domain for which only unlabeled samples are available. Existing techniques for UDA model selection are designed for the predictive setting. These methods examine discriminative density ratios between the input covariates in the source and target domain and do not factor in the model's predictions in the target domain. Because of this, two models with identical performance on the source domain would receive the same risk score by existing methods, but in reality, have significantly different performance on the test domain. We leverage the invariance of causal structures across domains to introduce a novel model selection metric specifically designed for ITE models under the UDA setting. In particular, we propose selecting models whose predictions of the effects of interventions satisfy known causal structures in the target domain. Experimentally, our method selects ITE models that are more robust to covariate shifts on several synthetic and real healthcare datasets, including on estimating the effect of ventilation in COVID-19 patients from different geographic locations.

1. INTRODUCTION

Causal inference models for estimating individualized treatment effects (ITE) are designed to provide actionable intelligence as part of decision support systems and, when deployed on mission-critical domains, such as healthcare, require safety and robustness above all (Shalit et al., 2017; Alaa & van der Schaar, 2017) . In healthcare, it is often the case that the observational data used to train an ITE model may come from a setting where the distribution of patient features is different from the one in the deployment (target) environment, for example, when transferring models across hospitals or countries. Because of this, it is imperative to select ITE models that are robust to these covariate shifts across disparate patient populations. In this paper, we address the problem of ITE model selection in the unsupervised domain adaptation (UDA) setting where we have access to the response to treatments for patients on a source domain, and we desire to select ITE models that can reliably estimate treatment effects on a target domain containing only unlabeled data, i.e., patient features. UDA has been successfully studied in the predictive setting to transfer knowledge from existing labeled data in the source domain to unlabeled target data (Ganin et al., 2016; Tzeng et al., 2017) . In this context, several model selection scores have been proposed to select predictive models that are most robust to the covariate shifts between domains (Sugiyama et al., 2007; You et al., 2019) . These methods approximate the performance of a model on the target domain (target risk) by weighting the performance on the validation set (source risk) with known (or estimated) density ratios. However, ITE model selection for UDA differs significantly in comparison to selecting predictive models for UDA (Stuart et al., 2013) . Notably, we can only approximate the estimated counterfactual error (Alaa & van der Schaar, 2019), since we only observe the factual outcome for the received treatment and cannot observe the counterfactual outcomes under other treatment options (Spirtes et al., 2000) . Consequently, existing methods for selecting predictive models for UDA that compute a weighted sum of the validation error as a proxy of the target risk (You et al., 2019) is suboptimal for

