UNCERTAINTY PREDICTION FOR DEEP SEQUENTIAL REGRESSION USING META MODELS Anonymous authors Paper under double-blind review

Abstract

Generating high quality uncertainty estimates for sequential regression, particularly deep recurrent networks, remains a challenging and open problem. Existing approaches often make restrictive assumptions (such as stationarity) yet still perform poorly in practice, particularly in presence of real world non-stationary signals and drift. This paper describes a flexible method that can generate symmetric and asymmetric uncertainty estimates, makes no assumptions about stationarity, and outperforms competitive baselines on both drift and non drift scenarios. This work helps make sequential regression more effective and practical for use in real-world applications, and is a powerful new addition to the modeling toolbox for sequential uncertainty quantification in general.

1. INTRODUCTION

The ability to quantify the uncertainty of a model is one of the fundamental requirements in trusted, safe, and actionable AI (Arnold et al., 2019; Jiang et al., 2018; Begoli et al., 2019) . This paper focuses on uncertainty quantification in regression tasks, particularly in the context of deep neural networks (DNN). We define a sequential task as one involving an ordered series of input elements, represented by features, and an ordered series of outputs. In sequential regression tasks (SRT), the output elements are (possibly multivariate) real-valued variables. SRT occur in numerous applications, among others, in weather modeling, environmental modeling, energy optimization, and medical applications. When the cost of making an incorrect prediction is particularly high, such as in human safety, models without a reliable uncertainty estimation are perceived high risk and may not be adopted. Uncertainty prediction in DNNs has been subject to active research, in particular, spurred by what has become known as the "Overconfidence Problem" of DNNs Guo et al. (2017) , and by their susceptibility to adversarial attacks Madry et al. (2017) . However, the bulk of work is concerned with non-sequential, classification tasks (see Section 2) leaving a noticeable gap for SRT. In this paper we introduce a meta-modeling concept as an approach to achieving high-quality uncertainty quantification in DNNs for SRT. We demonstrate that it not only outperforms competitive baselines but also provides consistent results across a variety of drift scenarios. We believe the approach represents a new powerful addition to the modeling toolbox in general. The novel contributions of this paper are summarized as follows: (1) Application of the meta-modeling concept to SRT, (2) Developing a joint base-meta model along with a comparison to white-and black-box alternatives, (3) Generating asymmetric uncertainty bounds in DNNs, and (4) Proposing a new evaluation methodology for SRT.

2. RELATED WORK

Classical statistics on time series offers an abundance of work dealing with uncertainty quantification (Papoulis & Saunders, 1989) . Most notably in econometrics, a variety of heteroskedastic variance models lead to highly successful application in financial market volatility analyses (Engle, 1982; Bollerslev, 1986; Mills, 1991) . An Autoregressive Conditional Heteroskedastic, or ARCH, model

