SEQSHAP: SUBSEQUENCE LEVEL SHAPLEY VALUE EXPLANATIONS FOR SEQUENTIAL PREDICTIONS

Abstract

With the increasing demands of interpretability in real-world applications, various methods for explainable artificial intelligence (XAI) have been proposed. However, most of them overlook the interpretability in sequential scenarios, which have a wide range of applications, e.g., online transactions and sequential recommendations. In this paper, we propose a Shapley value based explainer named SeqSHAP to explain the model predictions in sequential scenarios. Compared to existing methods, SeqSHAP provides more intuitive explanations at a subsequence level, which explicitly models the effect of contextual information among the related elements in a sequence. We propose to calculate subsequence-level feature attributions instead of element-wise attributions to utilize the information embedded in sequence structure, and provide a distribution-based segmentation method to obtain reasonable subsequences. Extensive experiments on two online transaction datasets from a real-world e-commerce platform show that the proposed method could provide valid and reliable explanations for sequential predictions.

1. INTRODUCTION

Sequential prediction tasks have a wide range of applications in real-world, e.g., Online Transaction (Wang et al., 2017; Zhang et al., 2018; Weber et al., 2018; Tam et al., 2019; Zhu et al., 2020; Chen & Lai, 2021) and Sequential Recommendation (Quadrana et al., 2017; Tang & Wang, 2018; Sun et al., 2019; Shen et al., 2021; Cui et al., 2022) , since sequences contain continuous signals which are important for model predictions. With the development of deep learning technique, sequence-based models have achieved a desirable performance in recent years (Hidasi et al., 2015; Quadrana et al., 2017; Wang et al., 2017; Tang & Wang, 2018; Zhang et al., 2018; Sun et al., 2019; Zhu et al., 2020; Qiao & Wang, 2022) . However, the complicated sequential data and increased model complexity make it hard for humans to understand the prediction of models. Indeed, for security and trust considerations, it is essential to develop effective explainable artificial intelligence (XAI) methods for sequence-based models in scenarios like fraud detection and medical care, so that end-users could understand how model predictions are produced with these complicated sequential data and models. In recent years, considerable efforts have been made on the model explanation algorithms (Ribeiro et al., 2016; Shrikumar et al., 2017; Lundberg & Lee, 2017; Selvaraju et al., 2017; Wachter et al., 2017; Alvarez-Melis & Jaakkola, 2018; Mothilal et al., 2020; Slack et al., 2021; Ghalebikesabi et al., 2021; Ali et al., 2022) . Among these works, feature attribution methods (Ribeiro et al., 2016; Shrikumar et al., 2016; 2017; Lundberg & Lee, 2017) are a popular family of post-hoc XAI methods. They calculate an attribution score for each feature to capture those important features for model predictions. However, most existing methods mainly pay attention to explain tabular data or images. And when dealing with the data and models in sequential scenarios, the complex input sequences make the element-wise explanations produced by these methods less explainable. The high-dimensional features and abundant interactions bring difficulty to existing element-wise XAI methods to provide explanations. Separately assigning attribution scores to individual feature cells in the sequence is not informative enough for users to understand the predictions. In addition, the great amount of features in a sequence could bring an extensive execution cost for existing methods, since the time complexity of them are mostly related to the number of features to be explained. In this paper, we propose SeqSHAP, a Shapley value based method to explain model predictions in sequential scenarios. SeqSHAP provides explanations at a unique subsequence level, which is more intuitive in sequential scenarios for humans compared to the element-wise explanations. Meanwhile, we propose a distribution-based segmentation method to split the sequence into reasonable subsequences which utilizes the distribution information of sequential features. With obtained subsequences, we group the feature elements under each subsequence as independent units. Then Shapley value estimations for feature units are calculated, to capture the important features that strongly influence the model prediction. Extensive experiments on two large-scale online transaction datasets collected from real-world are carried out. We analyze the local explanations produced by SeqSHAP and prove that our method provides intuitive explanations with meaningful subsequences, compared to existing feature attribution methods in sequential scenarios. Our contribution could be summarized as follows: • We propose an effective XAI method to explain sequential predictions at a subsequence level, which is a unique and intuitive view in sequential scenarios. • We propose a distribution-based segmentation method characterizing the distribution information of sequential features to capture the context information and obtain reasonable subsequences. • Extensive experiments on two real-world transaction datasets are provided to evaluate the validity of our segmentation method and subsequence-level explanations produced by Se-qSHAP.

2. BACKGROUND

In this section, we firstly introduce the task of explaining model predictions with sequential inputs. Then we introduce the background of SHAP (Lundberg & Lee, 2017), a popular interpretable framework based on Shapley values in game theory.

2.1. EXPLAINING PREDICTIONS WITH SEQUENTIAL DATA

Machine learning (ML) models for sequential prediction tasks have been widely applied in realworld applications (Hidasi et al., 2015; Tang & Wang, 2018; Sun et al., 2019; Zhu et al., 2020) , since the historical behaviour records in a sequence contain valuable information for the prediction task. However, while different models with desirable performance are proposed, predictions are getting particularly difficult to explain due to the increasing model complexity, which blocks the application of new techniques in some scenarios requiring a high degree of interpretability. As a result, the demand of XAI methods in sequential domains is growing rapidly, as existing methods mostly focus on tabular data and are not suitable for data with sequence structure. Task Description In this paper, our task is building an interpreter g to explain model predictions in sequential scenarios. Specifically, given a classifier f and a sequence X which could be formed as: X = {e 1 , e 2 , ..., e T }, where e t = {x t 1 , x t 2 , ..., x t M }, where T is the length of sequence and M is the number of features, e t ∈ R M represents the t-th element of sequence which has M feature fields to describe it. The interpreter g is expected to generate an explanation for the model prediction ŷ = f (x) ∈ [0, 1]. For the family of additive feature attribution methods, an element-wise explanation ϕ ∈ R T ×M assigns an importance score ϕ i,j (1 ≤ i ≤ T, 1 ≤ j ≤ M ) to the corresponding feature cell x i j in the sequence X, which represents the influence of features on the model prediction.

2.2. SHAPLEY VALUE BASED EXPLANATIONS

SHapley Additive exPlanation, termed as SHAP (Lundberg & Lee, 2017) , is a popular framework to explain model predictions based on the Shapley value in game theory. Through summarizing previous methods (Lipovetsky & Conklin, 2001; Štrumbelj & Kononenko, 2014; Bach et al., 2015; Datta et al., 2016; Ribeiro et al., 2016; Shrikumar et al., 2017) , SHAP builds an additive explanation model g as:

