DT+GNN: A FULLY EXPLAINABLE GRAPH NEURAL NETWORK USING DECISION TREES

Abstract

We propose a new Decision Tree Graph Neural Network (DT+GNN) architecture for Graph Neural Network (GNN) explanation. Existing post-hoc explanation methods highlight important inputs, but fail to reveal how a GNN uses these inputs. In contrast DT+GNN is fully explainable: Humans can inspect and understand the decision making of DT+GNN at every step. DT+GNN internally uses a novel GNN layer that is restricted to categorical state spaces for nodes and messages. After training with gradient descent, we can distill these layers into decision trees. These trees are further pruned using our newly proposed method to ensure they are small and easy to interpret. DT+GNN can also compute node-level importance scores like the existing explanation methods. We demonstrate on real-world GNN benchmarks that DT+GNN has competitive classification accuracy and computes competitive explanations. Furthermore, we leverage DT+GNN's full explainability to inspect the decision processes in synthetic and real-world datasets with surprising results. We make these inspections accessible through an interactive web tool.

1. INTRODUCTION

Graph Neural Networks (GNNs) have been successful in applying machine learning techniques to many graph-based domains [5; 20; 56; 28] . However, current GNNs are black-box models, and their lack of human interpretability limits their use in many application areas. This motivated the adoption of existing deep learning explanation methods [4; 25; 41] to GNNs, and creating new 51] . Such methods allow us to understand what parts of the input were important for making a prediction. However, it usually remains difficult to impossible for users to understand why these parts were important. In this dataset nodes are secondary structural elements of amino acids. Each node has one of three types: helix (input 0), sheet (input 1), or turn (input 2). Figure (a) shows an example graph consisting of mostly helixes (green) and two sheets (yellow). The final layer (b) learns to decide whether a protein is an enzyme based on how many sheet respectively "no sheet" nodes the graph has in the input layer. Whether nodes are helixes or turns does not seem to matter, and consequently the input layer (c) learns to only distinguish between sheet and no sheet. Both trees (b,c) together then imply: A protein is an enzyme if it has at most 8 nodes that are not sheets, or if it has at most 3 sheets. This DT+GNN explanation is consistent with previous human explanations [16] . 1



Figure1: (a) An example graph in the PROTEINS dataset. In this dataset nodes are secondary structural elements of amino acids. Each node has one of three types: helix (input 0), sheet (input 1), or turn (input 2). Figure (a) shows an example graph consisting of mostly helixes (green) and two sheets (yellow). The final layer (b) learns to decide whether a protein is an enzyme based on how many sheet respectively "no sheet" nodes the graph has in the input layer. Whether nodes are helixes or turns does not seem to matter, and consequently the input layer (c) learns to only distinguish between sheet and no sheet. Both trees (b,c) together then imply: A protein is an enzyme if it has at most 8 nodes that are not sheets, or if it has at most 3 sheets. This DT+GNN explanation is consistent with previous human explanations[16].

