SHAPLEY EXPLANATION NETWORKS

Abstract

Shapley values have become one of the most popular feature attribution explanation methods. However, most prior work has focused on post-hoc Shapley explanations, which can be computationally demanding due to its exponential time complexity and preclude model regularization based on Shapley explanations during training. Thus, we propose to incorporate Shapley values themselves as latent representations in deep models-thereby making Shapley explanations first-class citizens in the modeling paradigm. This intrinsic explanation approach enables layer-wise explanations, explanation regularization of the model during training, and fast explanation computation at test time. We define the Shapley transform that transforms the input into a Shapley representation given a specific function. We operationalize the Shapley transform as a neural network module and construct both shallow and deep networks, called SHAPNETs, by composing Shapley modules. We prove that our Shallow SHAPNETs compute the exact Shapley values and our Deep SHAPNETs maintain the missingness and accuracy properties of Shapley values. We demonstrate on synthetic and real-world datasets that our SHAPNETs enable layer-wise Shapley explanations, novel Shapley regularizations during training, and fast computation while maintaining reasonable performance. Code is available at https: //github.com/inouye-lab/ShapleyExplanationNetworks.



< l a t e x i t s h a 1 _ b a s e 6 4 = " L + s b l l x o 6 1 5 u e Q 9 H e A 5 H U 0 g t Q j U = " > A A A G h H i c p V T L b t N Q E J 0 G M K W 8 U l i y s Y g q F Z E G O 1 D R D a i C D Q s W B T V t p T 4 i 2 7 l p r f h V P y B V l C 2 f w o o t / A t / A H / B m b H T p E 3 i V s J X T e f O n X P m z u v a k e c m q W H 8 X q j c u H l L u 7 1 4 Z + n u v f s P H l a X H + 0 k Y R Y 7 q u W E X h j v 2 V a i P D d Q r d R N P b U X x c r y b U / t 2 r 3 3 f L 7 7 R c W J G w b b 6 V m k D n 3 r O H C 7 r m O l U L W r e n e 1 3 z b r e r / d f P b m I D m N 0 w H 2 R 8 3 n U B w 1 h 2 t m u 1 o z G o Z 8 + r R g F k K N i m 8 r X K 5 8 p w P q U E g O Z e S T o o B S y B 5 Z l G D t k 0 k G R d A d 0 g C 6 G J I r 5 4 q G t A R s B i s F C w v a H n 6 P s d s v t A H 2 z J k I 2 o E X D 3 8 x k D q t A B P C L o b M 3 n Q 5 z 4 S Z t f O 4 B 8 L J d z v D f 7 v g 8 q F N 6 Q T a q 3 A j y + v j b F i V x 5 p S l z Y k R h c x R 6 L h 6 J 3 C S y Z Z 4 8 j 0 i a h T M E T Q s d z B e Q z Z E e S o D r p g E s k N 5 9 6 S 8 z 9 i y V r e O 4 V t R n 9 L o 2 C f n s T x P 5 G w T 0 + q + B V y b r 8 G h D 5 h P 5 9 / G 9 3 E 3 F 3 g g w v M + V o R z k T y t A F e G 6 y c B c b p 5 6 i y K E / Q F X l O O U f d C e 6 x h 7 G N J 7 f u X S t v + p z F j A l 4 f C D z C r f o M 3 2 U + u Z + m D W V D g j A W d Z n I V Z P + s K G x a y 7 R 7 h N i L i U T I A r G e G K r N E p W C 2 J i P 3 q B U v e P 1 f N x D i z s 3 z a M p + M 4 h t y H 3 a g H U 2 O f j 6 J P N 1 l v g J 5 O b g q e S f P 8 s V v g F 8 g 0 y n u v A d 4 X k x 6 Q U 2 q Q 1 L y X j V K / P r i l / M 2 q s h w y u 9 l m x D Y W O a T u 2 U + N 8 8 t y 3 z D P g 3 x / p q X X 9 t p Y a f Z M N c b x q d X t c 1 3 x U u 8 S E / o K a 0 i r t e 0 S R 9 o C 1 3 k 0 D f 6 Q T / p l 6 Z p d e 2 l t p 6 b V h Y K z G O 6 8 G l v / w F X a T v j < / l a t e x i t > f (x 1 , x 2 ) = q x 2 1 + x 2 2 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " a 9 K O P 9 W f S f B x 8 1 8 h x z 6 G t x e E C u Y = " > A A A G g 3 i c p V T L b t N Q E J 0 G a k p 5 p b B k g U V U q Y g 2 2 F E Q 3 V S q Y M O C R U F N W 6 m p I t u 5 a a 3 4 h R + Q K s q S X 2 H D F j 6 G P 4 C / 4 M z Y a d I m c S v h q z j j u X P O 3 H l d O / L c J D W M 3 0 u V W 7 e X t T s r d 1 f v 3 X / w 8 F F 1 7 f F B E m a x o 1 p O 6 I X x k W 0 l y n M D 1 U r d 1 F N H U a w s 3 / b U o d 1 / x / u H X 1 S c u G G w n 5 5 H 6 s S 3 T g O 3 5 z p W C l W n + q y 3 M e i Y m / q g 0 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " E z P 2 8  i x A 6 m d u r 5 K 2 o k b D J v Q j T r V m l E 3 5 N F n B b M Q a l Q 8 e + F a 5 T u 1 q U s h O Z S R T 4 o C S i F 7 Z F G C d U w m G R R B d 0 J D 6 G J I r u w r G t E q s B m s F C w s a P t 4 n + L r u N A G + G b O R N A O v H j 4 x U D q t A 5 M C L s Y M n v T Z T 8 T Z t Y u 4 h 4 K J 5 / t H P 9 2 w e V D m 9 I Z t N f h x p Y 3 x 9 m w K o 8 1 p R 5 t S 4 w u Y o 5 E w 9 E 7 h Z d M s s a R 6 V N R p 2 C I o G O 5 i / 0 Y s i P I c R 1 0 w S S S G 8 6 9 J f t / x J K 1 / O 0 U t h n 9 L Y 2 C f X o S x / 9 E w j 4 9 q e J X y L n 9 F h D 6 l P 1 i / n 1 0 E 3 P 3 g A 8 u M e d r X T g T y d M 2 e G 2 w c h Y Y p 1 + g y q I 8 Q 1 f k O e U c 9 a a 4 J x 4 m N p 6 c u n + j v O k L F j M m 4 P G B z C v c o k / 0 Q e q b + 2 H W V D o g A G d Z n 4 V Y f e k L G x b z z h 7 h N C H i U j I B r m S E K 7 J F n 8 F q S U T s V y 9 Y 8 v 6 5 b i Y m m Z 3 n 0 5 b 5 Z B S f k P u w C + 1 4 c v S L S e T p L v M V y M 3 B V c k 7 e Z 4 v v g P 8 A p n O c O c 9 w P N i 0 i t q 0 C Y k J f d V v c S v L 3 4 5 b + O K j G b 8 X r U J g Y 1 l P r l b F n P z 3 L L M J x w Q 3 7 / m 1 d t 2 V j h o 1 M 3 X d e N j s 7 b 7 t r i J V + g p P a c N x P W G d u k 9 7 a G L H P p G P + g n / d K W t Z d a Q 2 v m p p W l A v O E L j 3 a z j 9 a e j x i < / l a t e x i t > f (x 1 , x 2 ) = x 1 ⇥ sin 4x 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " f + i I V 9 N k U 0 R k 1 F N j i X y Q w w z 7 r c U = " > A A A G Z X i c p V T N b t N A E J 4 G a k q h 0 A L i w g G L q B I H G u w I R I 8 V X D h w K N C 0 l U p V 2 c 6 m t b L + w V 5 D q y q P w B X e g H f i C Y C 3 4 J u x 0 6 R N 4 l b C q z j j 2 f m + 2 f l b P 9 V h b h z n 1 1 z j 2 v V 5 6 8 b C z c V b t 5 f u 3 F 1 e u b e d J 0 U W q E 6 Q 6 C T b 9 b 1 c 6 T B W H R M a r X b T T H m R r 9 W O 3 3 / D + z t f V J a H S b x l T l K 1 H 3 m H c d g L A 8 9 A 9 f H 4 o H 2 w 3 H R a j j z 2 p O B W Q p O q Z z N Z a f y k T 9 S l h A I q K C J F M R n I m j z K s f b I J Y d S 6 P b p F L o M U i j 7 i g a 0 C G w B K w U L D 9 o + 3 o f 4 2 q u 0 M b 6 Z M x d 0 A C 8 a v w x I m 1 a B S W C X Q W Z v t u w X w s z a W d y n w s l n O 8 G / X 3 F F 0 B o 6 g v Y y 3 N D y 6 j g f V v W x G u r R u s Q Y I u Z U N B x 9 U H k p J G s c m T 0 W t Q F D C h 3 L X e 1 3 i n J Z 9 g c R f 6 v d t s A y 3 G N Q = " > A A A G Z X i c p V T N T t R Q F D 6 M U h F F Q Y 0 b F z Z O S F z I 2 B K I L I l u X L h A Z Y A E C W k 7 d 6 C Z 2 x / b W 4 W Q e Q S 3 q + 0 M b 6 Z M x d 0 A C 8 a v w x I m 5 a B S W C X Q W Z v t u w X w s z a a d x n w s l n O 8 W / X 3 F F 0 B o 6 h v Y q 3 M D y + j g f V v W x G u r S h s Q Y I u Z U N B x 9 U H k p J G s c m T 0 S t Q F D C h 3 L H e x n k A N B D u p g C y a X 3 H D u P d n / I 5 a s 5 e + g s i 3 o b 2 0 U 7 F N L H P 8 T C f v U U s W v k E v 7 F S D s E f v p / N v o J u b u A h 9 f Y C 7 X W I x 0 Q g 7 O u z x K s n v S F D 4 t J Z 0 9 x m g R x K Z m A U D L C F V m h z 2 D 1 J C L 2 a 1 c s Z f 9 c N R P D z E 7 y 6 c t 8 M o p P y H 3 Y g X Y w O f b 5 J P J 0 1 / m K 5 e b g q p S d P M k X 3 w F R h T R j 3 G U P 8 L y 4 9 J J W 6 Q U k J f d V q 8 Z v J H 4 5 b 4 O K 9 M f 8 X r Z J g M 1 k P r l b p n P z 3 L L M J z y h P u 5 f 9 / J t O y 7 s r L b c 9 Z b z f q 2 5 + b q 6 i e f o C T 2 j 5 4 j r F W 3 S W 9 p C F w X g / U b f 6 c f s b 2 v B e m Q 9 L k 0 b M x X m I V 1 4 r K f / A G j k M m 4 = < / l a t e x i t > x 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " x 2 + P Q N w Z w M P R q f 1 d a d H 5 0 k + p N Y O r 0 = " > A A A G d H i c p V T L b t N Q E J 0 G G k p 5 p b C E h U U U q U h t a l c g u q x g w 4 J F Q U 1 b q a k i 2 7 l p r P i F f V 1 a R f k T t v S f + A P 4 A d a c G T t N 2 i R u J X w V Z z x 3 z p k 7 r + v E v p d q 0 / y 1 V L l 3 f 7 n 6 Y O X h 6 q P H T 5 4 + q 6 0 9 P 0 i j L H F V y 4 3 8 K D l y 7 F T 5 X q h a 2 t O + O o o T Z Q e O r w 6 d w U f e P z x T S e p F 4 b 6 + i N V J Y J + G X s 9 z b Q 1 V p 1 Z r x 3 2 v Y 6 3 3 N t p O M D w f v e n U 6 m b T l M e Y F a x C q F P x 7 E V r l U t q U 5 c i c i m j g B S F p C H 7 Z F O K d U w W m R R D d 0 J D 6 B J I n u w r G t E q s B m s F C x s a A d 4 n + L r u N C G + G b O V N A u v P j 4 J U A a 1 A A m g l 0 C m b 0 Z s p 8 J M 2 s X c Q + F k 8 9 2 g X + n 4 A q g 1 d S H 9 j b c 2 P L u O A d W 5 b F q 6 t G O x O g h 5 l g 0 H L 1 b e M k k a x y Z M R W 1 B k M M H c t d 7 C e Q X U G O 6 2 A I J p X c c O 5 t 2 f 8 t l q z l b 7 e w z e h P a R T s 0 5 c 4 / i c S 9 u l L F b 9 D z u 0 3 g T C m 7 B f z 7 6 O b m L s H f H i N O V 8 N 4 U w l T z v g d c D K W W C c c Y U q i 7 K P r s h z y j n q T X F P P E x s f D n 1 4 E 5 5 M x Y s Z k z B E w C Z V 7 h F X + m z 1 D f 3 w 6 x a O i A E Z 1 m f R V g D 6 Q s H F v P O H u M 0 E e J S M g G e Z I Q r s k n f w G p L R O z X K F j y / r l t J i a Z n e f T k f l k F J + Q + 7 A L 7 X h y j K t J 5 O k u 8 x X K z c F V y T t 5 n i + + A 4 I C q W e 4 8 x 7 g e b F o i 7 Z p A 5 K S + 6 p Z 4 j c Q v 5 y 3 c U V G M 3 5 v 2 k T A J j K f 3 C 2 L u X l u W e Y T n t M I 9 6 9 1 8 7 a d F Q 6 2 m 9 a 7 p v n l b X 3 3 Q 3 E T r 9 B L e k 3 r i O s 9 7 d I n 2 k M X u X R G P + g n X S 7 / r b 6 q 1 q u N 3 L S y V G B e 0 L W n 2 v w H f k Y 3 x A = = < / l a t e x i t > 1 (f, x) < l a t e x i t s h a 1 _ b a s e 6 4 = " f + i I V 9 N k U 0 R k 1 F N j i X y Q w w z 7 r c U = " > A A A G Z X i c p V T N b t N A E J 4 G a k q h 0 A L i w g G L q B I H G u w I R I 8 V X D h w K N C 0 l U p V 2 c 6 m t b L + w V 5 D q y q P w B X e g H f i C Y C 3 4 J u x 0 6 R N 4 l b C q z j j 2 f m + 2 f l b P 9 V h b h z n 1 1 z j 2 v V 5 6 8 b C z c V b t 5 f u 3 F 1 e u b e d J 0 U W q E 6 Q 6 C T b 9 b 1 c 6 T B W H R M a r X b T T H m R r 9 W O 3 3 / D + z t f V J a H S b x l T l K 1 H 3 m H c d g L A 8 9 A 9 f H 4 o H 2 w 3 H R a j j z 2 p O B W Q p O q Z z N Z a f y k T 9 S l h A I q K C J F M R n I m j z K s f b I J Y d S 6 P b p F L o M U i j 7 i g a 0 C G w B K w U L D 9 o + 3 o f 4 2 q u 0 M b 6 Z M x d 0 A C 8 a v w x I m 1 a B S W C X Q W Z v t u w X w s z a W d y n w s l n O 8 G / X 3 F F 0 B o 6 g v Y y 3 N D y 6 j g f V v W x G u r R u s Q Y I u Z U N B x 9 U H k p J G s c m T 0 W t Q F D C h 3 L X e x n k A N B D u t g C y a X 3 H D u P d n / I 5 a s 5 e + g s i 3 o b 2 0 U 7 F N L H P 8 T C f v U U s W v k E v 7 N S D s M f v Z / F v o J u b u A R + f Y y 7 X q n D m k q d 1 8 P p g 5 S w w z j 5 D 1 U V 5 h K 4 o c 8 o 5 6 o 1 x j z y M b L S c u n + l v N k z F j P m 4 I m A L C v c o Q / 0 T u p b + m F W I x 0 Q g 7 O u z x K s v v S F D 4 t p Z 0 9 x m g R x K Z m A U D L C F V m j z 2 D 1 J C L 2 a 1 c s Z f 9 c N h O j z E 7 z 6 c t 8 M o p P y H 3 Y h X Y 4 O f b Z J P J 0 1 / m K 5 e b g q p S d P M 0 X 3 w F R h T Q T 3 G U P 8 L y 4 9 J z a 9 A y S k v u q V e M 3 E < l a t e x i t s h a 1 _ b a s e 6 4 = " E z P 2 8 x 1 1 3 i n J Z 9 g c R f 6 v d t s A y 3 G N Q = " > A A A G Z X i c p V T N T t R Q F D 6 M U h F F Q Y 0 b F z Z O S F z I 2 B K I L I l u X L h A Z Y A E C W k 7 d 6 C Z 2 x / b W 4 W Q e Q S 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " X e g F C 4 q 7 L l c H k M h 7 6 A K  V 9 F R g F V o = " > A A A G d H i c p V T L T t t Q E B 3 S k l L 6 C u 2 y X V i N I l E J g h M V w R K 1 m y 6 6 o B U B J I I i 2 7 k h V v y q f U 1 B U f = " > A A A G d H i c p V T L b t N Q E J 0 G G k p 5 p b C E h U U U q U h t a l c g u q x g w 4 J F Q U 1 b q a k i 2 7 l p r P i F f V 1 a R f k T t v S f + A P 4 A d a c G T t N 2 i R u J X w V Z z x 3 z p k 7 r + v E v p d q 0 / y 1 V L l 3 f 7 n 6 Y O X h 6 q P H T 5 4 + q 6 0 9 P 0 i j L H F V y 4 3 8 K D l y 7 F T 5 X q h a 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " X e g F C 4 q 7 L l c H k M h 7 6 A K < l a t e x i t s h a 1 _ b a s e 6 4 = " f + i I V 9 F R g F V o = " > A A A G d H i c p V T L T t t Q E B 3 S k l L 6 C u 2 y X V i N I l E J g h M V w R K 1 m y 6 6 o B U B J I I i 2 7 k h V v y q f U 1 B U f V 0 K y a G U f F I U k I b s k U U J 1 g k 1 y K Q I u l M a Q h d D c m V f 0 Y h W g U 1 h p W B h Q T v A + w x f J 7 k 2 w D d z J o J 2 4 M X V 9 N k U 0 R k 1 F N j i X y Q w w z 7 r c U = " > A A A G Z X i c p V T N b t N A E J 4 G a k q h 0 A L i w g G L q B I H G u w I R I 8 V X D h w K N C 0 l U p V 2 c 6 m t b L + w V 5 D q y q P w B X e g H f i C Y C 3 4 J u x 0 6 R N 4 l b C q z j j 2 f m + 2 f l b P 9 V h b h z n 1 1 z j 2 v V 5 6 8 b C z c V b t 5 f u 3 F 1 e u b e d J 0 U W q E 6 Q 6 C T b 9 b 1 c 6 T B W H R M a r X b T T H m R r 9 W O 3 3 / D + z t f V J a H S b x l T l K 1 H 3 m H c d g L A 8 9 A 9 f H 4 o H 2 w 3 H R a j j z 2 p O B W Q p O q Z z N Z a f y k T 9 S l h A I q K C J F M R n I m j z K s f b I J Y d S 6 P b p F L o M U i j 7 i g a 0 C G w B K w U L D 9 o + 3 o f 4 2 q u 0 M b 6 Z M x d 0 A C 8 a v w x I m 1 a B S W C X Q W Z v t u w X w s z a W d y n w s l n O 8 G / X 3 F F 0 B o 6 g v Y y 3 N D y 6 j g f V v W x G u r R u s Q Y I u Z U N B x 9 U H k p J G s c m T 0 W t Q F D C h 3 L X e x n k A N B D u t g C y a X 3 H D u P d n / I 5 a s 5 e + g s i 3 o b 2 0 U 7 F N L H P 8 T C f v U U s W v k E v 7 N S D s M f v Z / F v o J u b u A R + f Y y 7 X q n D m k q d 1 8 P p g 5 S w w z j 5 D 1 U V 5 h K 4 o c 8 o 5 6 o 1 x j z y M b L S c u n + l v N k z F j P m 4 I m A L C v c o Q / 0 T u p b + m F W I x 0 Q g 7 O u z x K s v v S F D 4 t p Z 0 9 x m g R x K Z m A U D L C F V m j z 2 D 1 J C L 2 a 1 c s Z f 9 c N h O j z E 7 z 6 c t 8 M o p P y H 3 Y h X Y 4 O f b Z J P J 0 1 / m K 5 e b g q p S d P M 0 X 3 w F R h T Q T 3 G U P 8 L y 4 9 J z a 9 A y S k v u q V e M 3 E r + c t 2 F F B h N + L 9 o k w G Y y n 9 w t s 7 l 5 b l n m E x 7 T A P e v e / G 2 n R S 2 2 y 3 3 Z c t 5 / 6 K 5 8 b q 6 i R f o E T 2 h p 4 j r F W 3 Q W 9 p E F w X g / U b f 6 c f 8 b 2 v J e m A 9 L E 0 b c x X m P p 1 7 r M f / A G 7 j M m 8 = < / l a t e x i t > x 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " d O z J < l a t e x i t s h a 1 _ b a s e 6 4 = " F i 2 q o C A x S a d K q u 9 L v F F L L 5 G g w that can be arbitrarily complex. Interestingly, the Shapley explanation values, often denoted φ s (x, f ), for a GAM are exactly the values of the independent function, i.e., ∀s, φ s (x, f GAM ) = f s (x s ). Hence, the prediction and the corresponding exact SHAP explanation can be computed simultaneously for GAM models. However, GAM models are inherently limited in their representational power, particularly for perceptual data such as images in which deep networks are state-of-the-art. Thus, prior work is either post-hoc (which precludes leveraging the method during training) or limited in its representational power (e.g., GAMs). U I A h w d t Y 7 m t a V s X Q V G o L N y Y = " > A A A G h n i c p V R L T x N R F D 6 g 1 o o P i i 7 d 3 N C Q Y A J 1 h k g g r h r d u H C B h A I J E D I z v Y V J 5 + X M H Y Q 0 3 f t L X L r V v + I / x g = " > A A A G h n i c p V T L T t t Q E B 1 o m 6 b 0 Q W i X 3 V h E S F Q C 1 0 F F o K 6 i d t N F F x Q R Q A K E b O c a r P h V + 5 q C o u z 7 J V To overcome these drawbacks, we propose to incorporate Shapley values themselves as learned latent representations (as opposed to post-hoc) in deep models-thereby making Shapley explanations first-class citizens in the modeling paradigm. Intuitive illustrations of such representation are provided in Fig. 1 and detailed discussion in subsection 2.1. We summarize our core contributions as follows: • We formally define the Shapley transform and prove a simple but useful linearity property for constructing networks. • We develop a novel network architecture, the SHAPNETs, that includes Shallow and Deep SHAPNETs and that intrinsically provides layer-wise explanations (i.e., explanations at every layer of the network) in the same forward pass as the prediction. • We prove that Shallow SHAPNET explanations are the exact Shapley values-thus satisfying all three SHAP properties-and prove that Deep SHAPNET explanations maintain the missingness and local accuracy properties. • To reduce computation, we propose an instance-specific dynamic pruning method for Deep SHAPNETs that can skip unnecessary computation. • We enable explanation regularization based on Shapley values during training because the explanation is a latent representation in our model. • We demonstrate empirically that our SHAPNETs can provide these new capabilities while maintaining comparable performance to other deep models. Dedicated Related Works Section We present related works above and in the text where appropriate. Due to space limit, we refer to Appendix D for a dedicated literature review section.

2. SHAPLEY EXPLANATION NETWORKS

Background We give a short introduction to SHAP explanations and their properties as originally introduced in Lundberg & Lee (2017). Given a model f : R d → R that is not inherently interpretable (e.g., neural nets), additive feature-attribution methods form a linear approximation of the function over simplified binary inputs, denoted z ∈ {0, 1} d , indicating the "presence" and "absence" of each feature, respectively: i.e., a local linear approximation η(z) = a 0 + d i=1 a i z i . While there are different ways to model "absence" and "presence", in this work, we take a simplified viewpoint: "presence" means that we keep the original value whereas "absence" means we replace the original value with a reference value, which has been validated in Sundararajan & Najmi (2020) as Baseline Shapley. If we denote the reference vector for all features by r, then we can define a simple mapping



x n k A N B D u t g C y a X 3 H D u P d n / I 5 a s 5 e + g s i 3 o b 2 0U 7 F N L H P 8 T C f v U U s W v k E v 7 N S D s M f v Z / F v o J u b u A R + f Y y 7 X q n D m k q d 1 8 P p g 5 S w w z j 5 D 1 U V 5 h K 4 o c 8 o 5 6 o 1 x j z y M b L S c u n + l v N k z F j P m 4 I m A L C v c o Q / 0 T u p b + m F W I x 0 Q g 7 O u z x K s v v S F D 4 t p Z 0 9 x m g R x K Z m A U D L C F V m j z 2 D 1 J C L 2 a 1 c s Z f 9 c N h O j z E 7 z 6 c t 8 M o p P y H 3 Y h X Y 4 O f b Z J P J 0 1 / m K5 e b g q p S d P M 0 X 3 w F R h T Q T 3 G U P 8 L y 4 9 J z a 9 A y S k v u q V e M 3 E r + c t 2 F F B h N + L 9 o k w G Y y n 9 w t s 7 l 5 b l n m E x 7 T A P e v e / G 2 n R S 2 2 y 3 3 Z c t 5 / 6 K 5 8 b q 6 i R f o E T 2 h p 4 j r F W 3 Q W 9 p E F w X g / U b f 6 c f 8 b 2 v J e m A 9 L E 0 b c x X m P p 1 7 r M f / A G 7 j M m 8 = < / l a t e x i t >

+ g a + k 0 + g v o X f O e 0 w A z N T S O z N d E 7 P P d 9 3 7 v m 7 f q r D 3 D j O r 5 n G j Z u z 1 q 2 5 2 / N 3 7 i 7 c u 7 + 4 9 G A n T 4 o s U O 0 g 0 U m 2 5 3 u 5 0 m G s 2 i Y 0 W u 2 l m f I i X 6 t d v / e G 9 3 e / q C w P k 3 j b n K b q I P K O 4 r A b B p 6 B 6 u P J o X u 4 2 H R a j j z 2 u O B W Q p O q Z y t Z a v y k T 9 S h h A I q K C J F M R n I m j z K s f b J J Y d S 6 A 7 o D L o M U i j 7 i v o 0 D 2 w B K w U L D 9 o e 3 k f 4 2

s n D m k q c N 8 P p g 5 S w w z j 5 H 1 U V 5 j K 4 o c 8 o 5 6 o 5 w D z 0 M b b S c u n e t v N l T F j P m 4 I m A L C v c p g / 0 T u p b + m F

r + c t 2 F F B h N + L 9 o k w G Y y n 9 w t s 7 l 5 b l n m E x 7 T A P e v e / G 2 n R S 2 2 y 3 3 Z c t 5 / 6 K 5 8 b q 6 i R f o E T 2 h p 4 j r F W 3 Q W 9 p E F w X g / U b f 6 c f 8 b 2 v J e m A 9 L E 0 b c x X m P p 1 7 r M f / A G 7 j M m 8 = < / l a t e x i t >

+ g a + k 0 + g v o X f O e 0 w A z N T S O z N d E 7 P P d 9 3 7 v m 7 f q r D 3 D j O r 5 n G j Z u z 1 q 2 5 2 / N 3 7 i 7 c u 7 + 4 9 G A n T 4 o s U O 0 g 0 U m 2 5 3 u 5 0 m G s 2 i Y 0 W u 2 l m f I i X 6 t d v / e G 9 3 e / q C w P k 3 j b n K b q I P K O 4 r A b B p 6 B 6 u P J o X u 4 2 H R a j j z 2 u O B W Q p O q Z y t Z a v y k T 9 S h hA I q K C J F M R n I m j z K s f b J J Y d S 6 A 7 o D L o M U i j 7 i v o 0 D 2 w B K w U L D 9 o e 3 k f 4 2 q + 0 M b 6 Z M x d 0 A C 8 a v w x I m 5 a B S W C X Q W Z v t u w X w s z a a d x n w s l n O 8 W / X 3 F F 0 B o 6 h v Y q 3 M D y + j g f V v W x G u r S h s Q Y I u Z U N B x 9 U H k p J G s c m T 0 S t Q F D C h 3 LH e x n k A N B D u p g C y a X 3 H D u P d n / I 5 a s 5 e + g s i 3 o b 2 0 U 7 F N L H P 8 T C f v U U s W v k E v 7 F S D s E f v p / N v o J u b u A h 9 f Y C 7 X s n D m k q c N 8 P p g 5 S w w z j 5 H 1 U V 5 j K 4 o c 8 o 5 6 o 5 w D z 0 M b b S c u n e t v N l T F j P m 4 I m A L C v c p g / 0 T u p b + m F W I x 0 Q g 7 O u z x K s n v S F D 4 t J Z 0 9 x m g R x K Z m A U D L C F V m h z 2 D 1 J C L 2 a 1 c s Z f 9 c N R P D z E 7 y 6 c t 8 M o p P y H 3 Y g X Y w O f b 5 J P J 0 1 / m K 5 e b g q p S d P M k X 3 w F R h T R j 3 G U P 8 L y 4 9 J J W 6 Q U k J f d V q 8 Z v J H 4 5 b 4 O K 9 M f 8 X r Z J g M 1 k P r l b p n P z 3 L L M J z y h P u 5 f 9 / J t O y 7 s r L b c 9 Z b z f q 2 5 + b q 6 i e f o C T 2 j 5 4 j r F W 3 S W 9 p C F w X g / U b f 6 c f s b 2 v B e m Q 9 L k 0 b M x X m I V 1 4 r K f / A G j k M m 4 = < / l a t e x i t >

6 k 2 / J P / Y P 2 B 7 r u m b F D A k k M U n 0 V Z z x 3 z p k 7 r 2 t H n p t o 0 / y 1 V H r w c L n 8 a O X x 6 p O n z 5 6 / q K y 9 P E z C N H Z U y w m 9 M D 6 2 r U R 5 b q B a 2 t W e O o 5 i Z f m 2 p 4 7 s w U f e P z p X c e K G w Y G + j N S p b 5 0 F b s 9 1 L A 1 V p 1 J p R 3 2 3 0 1 z v b b R t f 3 g x e t e p V M 2 6 K Y 8 x K z R y o U r 5 s x + u l a 6 o TV 0 K y a G U f F I U k I b s k U U J 1 g k 1 y K Q I u l M a Q h d D c m V f 0 Y h W g U 1 h p W B h Q T v A + w x f J 7 k 2 w D d z J o J 2 4 M X D L w b S o B o w I e x i y O z N k P 1 U m F m 7 i H s o n H y 2 S / z b O Z c P r a Y + t H f h x p b 3 x 9 m w K o 5 V U 4 9 2 J U Y X M U e i 4 e i d 3 E s q W e P I j K m o N R g i 6 F j u Y j + G 7 A h y X A d D M I n k h n N v y f 5 v s W Q t f z u 5 b U p / C q N g n 5 7 E 8 T + R s E 9 P q v g d c m a / C Y Q x Z b + Y / w D d x N w 9 4 I M b z N m q C W c i e d o F r w 1 W z g L j j G t U U Z R 9 d E W W U 8 5 R b 4 p 7 4 m F i 4 8 m p B / f K m 7 F g M W M C H h / I r M I t + k q f p b 6 Z H 2 b V 0 g E B O I v 6 L M Q a S F / Y s J h 3 9 g i n C R G X k g l w J S N c k U 3 6 B l Z L I m K / Rs 6 S 9 c 9 d M z H J 7 D y f t s w n o / i E 3 I d d a M e T Y 1 x P I k 9 3 k a 9 A b g 6 u S t b J 8 3 z x H e D n S D 3 D n f U A z 0 u D t q h J G 5 C U 3 F f 1 A r + + + O W 8 j S s y m v F 7 2 y Y E N p b 5 5 G 5 Z z M 1 z y z K f 8 I J G u H 8 b t 2 / b W e G w W W 9 s 1 8 0 v 7 6 t 7 H / K b e I V e 0 1 t a R 1 w 7 t E e f a B 9 d 5 N A 5 / a C f d L X 8 t / y m X C 3 X M t P S U o 5 5 R T e e c v 0 f h E 8 3 x Q = = < / l a t e x i t > 2 (f, x) < l a t e x i t s h a 1 _ b a s e 6 4 = " + P Q N w Z w M P R q f 1 d a d H 5 0 k + p N Y O r 0

t O + O o o T Z Q e O r w 6 d w U f e P z x T S e p F 4 b 6+ i N V J Y J + G X s 9 z b Q 1 V p 1 Z r x 3 2 v Y 6 3 3 N t p O M D w f v e n U 6 m b T l M e Y F a x C q F P x 7 E V r l U t q U 5 c i c i m j g B S F p C H 7 Z F O K d U w W m R R D d 0 J D 6 B J I n u w r G t E q s B m s F C x s a A d 4 n + L r u N C G + G b O V N A u v P j 4 J U A a 1 A A m g l 0 C m b 0 Z s p 8 J M 2 s X c Q + F k 8 9 2 g X + n 4 A q g 1 d S H 9 j b c 2 P L u O A d W 5 b F q 6 t G O x O g h 5 l g 0 H L 1 b e M k k a x y Z M R W 1 B k M M H c t d 7 C e Q X U G O 6 2 A I J p X cc O 5 t 2 f 8 t l q z l b 7 e w z e h P a R T s 0 5 c 4 / i c S 9 u l L F b 9 D z u 0 3 g T C m 7 B f z 7 6 O bm L s H f H i N O V 8 N 4 U w l T z v g d c D K W W C c c Y U q i 7 K P r s h z y j n q T X F P P E x s f D n 1 4 E 5 5 M x Y s Z k z B E w C Z V 7 h F X + m z 1 D f 3 w 6 x a O i A E Z 1 m f R V g D 6 Q s H F v P O H u M 0 E e J S M g G e Z I Q r s k n f w G p L R O z X K F j y / r l t J i a Z n e f T k f l k F J + Q + 7 A L 7 X h y j K t J 5 O k u 8 x X K z c F Vy T t 5 n i + + A 4 I C q W e 4 8 x 7 g e b F o i 7 Z p A 5 K S + 6 p Z 4 j c Q v 5 y 3 c U V G M 3 5 v 2 k T A J j K f 3 C 2 L u X l u W e Y T n t M I 9 6 9 1 8 7 a d F Q 6 2 m 9 a 7 p v n l b X 3 3 Q 3 E T r 9 B L e k 3 r i O s 9 7 d I n 2 k M X u X R G P + g n X S 7 / r b 6 q 1 q u N 3 L S y V G B e 0 L W n 2 v w H f k Y 3 x A = = < / l a t e x i t > 1 (f, x)

6 k 2 / J P / Y P 2 B 7 r u m b F D A k k M U n 0 V Z z x 3 z p k 7 r 2 t H n p t o 0 / y 1 V H r w c L n 8 a O X x 6 p O n z 5 6 / q K y 9 P E z C N H Z U y w m 9 M D 6 2 r U R 5 b q B a 2 t W e O o 5 i Z f m 2 p 4 7 s w U f e P z p X c e K G w Y G + j N S p b 5 0 F b s 9 1 L A 1 V p 1 J p R 3 2 3 0 1 z v b b R t f 3 g x e t e p V M 2 6 K Y 8 x K z R y o U r 5 s x + u l a 6 o T

D L w b S o B o w I e x i y O z N k P 1 U m F m 7 i H s o n H y 2 S / z b O Z c P r a Y + t H f h x p b 3 x 9 m w K o 5 V U 4 9 2 J U Y X M U e i 4 e i d 3 E s q W e P I j Km o N R g i 6 F j u Y j + G 7 A h y X A d D M I n k h n N v y f 5 v s W Q t f z u 5 b U p / C q N g n 5 7 E 8 T + R s E 9 P q v g d c m a / C Y Q x Z b + Y / w D d x N w 9 4 I M b z N m q C W c i e d o F r w 1 W z g L j j G t U U Z R 9 d E W W U 8 5 R b 4 p 7 4 m F i 4 8 m p B / f K m 7 F g M W M C H h / I r M I t + k q f p b 6 Z H 2 b V 0 g E B O I v 6 L M Q a S F / Y s J h 3 9 g i n C R G X k g l w J S N c k U 3 6 B l Z L I m K / Rs 6 S 9 c 9 d M z H J 7 D y f t s w n o / i E 3 I d d a M e T Y 1 x P I k 9 3 k a 9 A b g 6 u S t b J 8 3 z x H e D n S D 3 D n f U A z 0 u D t q h J G 5 C U 3 F f 1 A r + + + O W 8 j S s y m v F 7 2 y Y E N p b 5 5 G 5 Z z M 1 z y z K f 8 I J G u H 8 b t 2 / b W e G w W W 9 s 1 8 0 v 7 6 t 7 H / K b e I V e 0 1 t a R 1 w 7 t E e f a B 9 d 5 N A 5 / a C f d L X 8 t / y m X C 3 X M t P S U o 5 5 R T e e c v 0 f h E 8 3 x Q = = < / l a t e x i t > 2 (f, x)

0 H / h d 8 5 M a b E P S J y b T s + c e 7 7 v 3 P O 6 b h L 4 m b G s X 3 P z 9 + 4 / q D y s P l p 4 / O T p s 8 X a 0 v P 9 L M 5 T T 7 e 8 O I j TQ 9 f J d O B H u m V 8 E + j D J N V O 6 A b 6 w O 2 + 5 / 2 D C 5 1 m f h z t m a t E n 4 T O W e R 3 f M 8 x U J 3 W l o + N v j Q 9 e b u d 3 q 7 u N N S F E + Q 6 6 7 9 V / V V 7 z X 5 1 W q v b D U s e Z Y 0 J g 6 0 6 l c 9 O v D T / j Y 6 p T T F 5 l F N I m i I y k A N y K M M 6 I p s s S q A 7 o R 5 0 K S R f 9 j X 1 a Q H Y H F Y a F g 6 0 X b z P 8 H V U a i N 8 M 2 c m a A 9 e A v x S I B W t A B P D L o X M 3 p T s 5 8 L M 2 m n c P e H k s 1 3 h 3 y 2 5 Q m g N n U N 7 G 2 5 g e X e c C 6 v Z s R r q 0 L b E 6 C P m R D Q c v V d 6 y S V r H J k a i d q A I Y G O 5 T b 2 U 8 i e I A d 1 U I L J J D e c e 0 f 2 f 4 s l a / n b K 2 1 z + j M z C v Y Z S B z / E w n 7 D K S K X y A X 9 u t A q B H 7 6 f x 7 6 C b m 7 g A f 3 W A u 1 o p w Z p K n b f C 6 Y O U s M E 5 d o 2 Z F e Y 6 u K H L K O e q M c A 8 9 D G 0 C O X X 3 T n l T U x Y z Z u A J g S w q 3 K J d + i j 1 L f w w q 5 E O i M A 5 q 8 9 i r K 7 0 h Q u L S W d P c J o Y c W m Z A F 8 y w h V Z p 8 9 g d S Q i 9 q t K l q J / b p u J Y W Y n + X R l P h n F J + Q + b E M 7 m Bx 1 P Y k 8 3 b N 8 R X J z c F W K T p 7 k i + + A s E S a M e 6 i B 3 h e b H p N G 7 Q G S c t 9 1 Z j h N x S / n L d B R f p j f v + 1 i Y F N Z T 6 5 W 6 Z z 8 9 y y z C e 8 p P 7 o / T t d 2 N 9 o 2 J s N 6 9 O b e v N d e R N X 6 S U t 0 y r i 2 q I m f a A d d J F H X + k 7 / a C f l W q l U d m s b B W m 8 3 M l 5 g X d e C r N v w 6 u P f 0 = < / l a t e x i t > Ref. values: (1, 1)

Figure 1: Shapley representations span beyond one-dimensional manifolds and depend on both the inner function and the reference values. In both groups, the gray-scale background indicates the respective function value while the rainbow-scale color indicates correspondence between input (left) and Shapley representation (right) along with function values-red means highest and means lowest function values. The red cross represents the reference values. More details in subsection 2.1.

