TEMPORAL DEPENDENCIES IN FEATURE IMPORTANCE FOR TIME SERIES PREDICTION

Abstract

Time series data introduces two key challenges for explainability methods: firstly, observations of the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance to model predictions over time. In this paper, we propose Windowed Feature Importance in Time (WinIT), a feature removal based explainability approach to address these issues. Unlike existing feature removal explanation methods, WinIT explicitly accounts for the temporal dependence between different observations of the same feature in the construction of its importance score. Furthermore, WinIT captures the varying importance of a feature over time, by summarizing its importance over a window of past time steps. We conduct an extensive empirical study on synthetic and real-world data, compare against a wide range of leading explainability methods, and explore the impact of various evaluation strategies. Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics.

1. INTRODUCTION

Reliably explaining predictions of machine learning models is important given their wide-spread use. Explanations provide transparency and aid reliable decision making, especially in domains such as finance and healthcare, where explainability is often an ethical and legal requirement (Amann et al., 2020; Prenio & Yong, 2021) . Multivariate time series data is ubiquitous in these sensitive domains, however explaining time series models has been relatively under explored. In this work we focus on saliency methods, a common approach to explainability that provides explanations by highlighting the importance of input features to model predictions (Baehrens et al., 2010; Mohseni et al., 2020) . It has been shown that standard saliency methods underperform on deep learning models used in the time series domain (Ismail et al., 2020) . In time series data, observations of the same feature at different points in time are typically related and their order matters. Methods that aim to highlight important observations but treat them as independent face significant limitations. Furthermore, it is important to note that the same observation of a feature can vary in importance to predictions at different times, which we refer to as the temporal dependency in feature importance. For example, there can be a delay between important feature shifts and a change in the model's predictions. These temporal dynamics can be difficult for current explainability methods to capture. To address these challenges, we propose the Windowed Feature Importance in Time (WinIT), a feature removal based method which determines the importance of a given observation to the predictions over a time window. Feature removal-based methods generate importance by measuring the change in outcomes when a particular feature is removed. However, removing a feature at a particular time does not account for the dependence between current and future observations of the same feature. WinIT addresses this by assigning the importance of a feature at a specific time based on a difference of two scores, that each take the temporal dependence of the subsequent observations into account. To capture the varying importance of the same feature observation to predictions at different times, WinIT aggregates the importance to predictions over a window of time steps to generate its final score. This allows our approach to identify important features that have delayed impact on the predictive distribution, thus better capturing temporal dynamics in feature importance. It is well known that the evaluation of explainability methods is challenging because no ground truth identification of important features exists for real world data (Doshi-Velez & Kim, 2017) . A common approach is to evaluate the impact of removing features that are highlighted as important by the explainability method (Lundberg et al., 2020) by masking them with a prior value. For time series data, an additional challenge for evaluation is that since observations of the same feature are related, removing the information from an important observation is non-trivial. This fact is under-explored in prior work. We present a detailed investigation of masking strategies to remove important observations and demonstrate their large impact on explainability performance evaluation. We present several masking strategies with complementary properties that we propose should be part of a comprehensive evaluation scheme. In summary, our main contributions are: • We propose a new time series feature importance method that accounts for the temporal dependence between different observations of the same feature and computes the impact of a feature observation to predictions over a window of time steps. • We conduct a detailed investigation of evaluation approaches for time series model explainability and present several complementary masking strategies that we propose should be part of a robust evaluation scheme. • We expand synthetic datasets from Tonekaboni et al. ( 2020) to explicitly evaluate the ability of explainability methods to capture shifted temporal dependencies. • We conduct extensive experiments and demonstrate that our approach leads to a significant improvement in explanation performance on real-world data, and is more stable under different evaluation settings.

2. RELATED WORK

A wide range of saliency methods have been proposed in the literature. 2019) for attention-based models, use the model architecture, in this case the attention layers, to generate importance scores. However, when applied to time series models, these explainability methods do not directly consider the temporal nature of the problem and have been shown to underperform (Ismail et al., 2020) . In contrast there has been little work on saliency methods for time series. TSR (Ismail et al., 2020) separates the importance calculation along the time and feature input dimensions to improve performance. FIT (Tonekaboni et al., 2020) measures each observation's importance to the prediction change at the same time step using a KL-divergence based score. Dynamask (Crabbé & van der Schaar, 2021) is a perturbation method that learns a mask of the input feature matrix to highlight important observations. We propose a new feature removal based saliency method that explicitly accounts for the temporal nature of the data in a novel way. et al., 2019) measure the change in the loss instead. The summarization method of an explainability approach describes how the importance score is generated. Feature occlusion uses the L1 norm of the difference in model predictions, which does not account for the temporal nature of the problem, while FIT uses the KL-divergence between the prediction with and without a feature at the same time step. Our summarization method differs from these approaches by accounting for the importance of a feature observation over multiple time steps, improving our ability to capture temporal patterns in feature importance.



To understand how our work relates to other feature removal based methods we can utilize the unified framework presented in Covert et al. (2021) that categorizes feature removal based methods into three dimensions: feature removal method, model behaviour explained and summarization method. Approaches such as feature occlusion (Zeiler & Fergus, 2014), feature ablation (Suresh et al., 2017) and RISE (Petsiuk et al., 2019) replace the removed features with a specific baseline value. Like us, approaches such as FIT (Tonekaboni et al., 2020) and FIDO-CA (Chang et al., 2019) use a generator to replace the removed features. In terms of model behaviour explained, like most methods we aim to explain the model prediction before and after feature removal, while methods such as INVASE (Yoon

