ROUTE, INTERPRET, REPEAT: BLURRING THE LINE BETWEEN POST HOC EXPLAIN-ABILITY AND INTERPRETABLE MODELS

Abstract

The current approach to ML model design is either to choose a flexible Blackbox model and explain it post hoc or to start with an interpretable model. Blackbox models are flexible but difficult to explain, whereas interpretable models are designed to be explainable. However, developing interpretable models necessitates extensive ML knowledge, and the resulting models tend to be less flexible, offering potentially subpar performance compared to their Blackbox equivalents. This paper aims to blur the distinction between a post hoc explanation of a Black-Box and constructing interpretable models. We propose beginning with a flexible BlackBox model and gradually carving out a mixture of interpretable models and a residual network. Our design identifies a subset of samples and routes them through the interpretable models. The remaining samples are routed through a flexible residual network. We adopt First Order Logic (FOL) as the interpretable model's backbone, which provides basic reasoning on concepts retrieved from the BlackBox model. On the residual network, we repeat the method until the proportion of data explained by the residual network falls below a desired threshold. Our approach offers several advantages. First, the mixture of interpretable and flexible residual networks results in almost no compromise in performance. Second, the route, interpret, and repeat approach yields a highly flexible interpretable model. Our extensive experiment demonstrates the performance of the model on various datasets. We show that by editing the FOL model, we can fix the shortcut learned by the original BlackBox model. Finally, our method provides a framework for a hybrid symbolic-connectionist network that is simple to train and adaptable to many applications.

1. INTRODUCTION

Model explainability is essential in high-stakes applications of AI, such as healthcare. While BlackBox models (e.g., Deep Learning) offer flexibility and modular design, post hoc explanation is prone to confirmation bias Wan et al. (2022) , lack of fidelity to the original model Adebayo et al. (2018) , and insufficient mechanistic explanation of the decision-making process Rudin (2019) . Interpretable-by-design models do not suffer from those issues but tend to be less flexible than Blackbox models and demand substantial expertise to design and fine-tune. Using post hoc explanation or adopting an interpretable model is a mutually exclusive decision to be made at the initial phase of AI model design. This paper aims to blur the line on that dichotomous model design. The literature on post hoc explainable AI is extensive. The methods such as model attribution (e.g., Saliency Map Simonyan et al. (2013) ; Selvaraju et al. (2017) ), counterfactual approach Abid et al. (2021) ; Singla et al. (2019) , and distillation methods Alharbi et al. (2021) ; Cheng et al. (2020) are examples of post hoc explainability approaches. Those methods either identify important features of input that contribute the most to the network's output Shrikumar et al. (2016) , generate perturbation to the input that flips the network's output Samek et al. (2016) , Montavon et al. (2018) , or estimate simpler functions that locally approximate the network output. The advantage of the post hoc explainability methods is that they do not compromise the flexibility and performance of the BlackBox. However, the post hoc explainability method suffers from several undesirable significant drawbacks, such as a lack of fidelity and mechanistic explanation of the network output Rudin (2019) . Without a mechanistic explanation, recourse to a model's undesirable behavior is unclear. Interpretable models are alternative designs to the BlackBox model that do not suffer from many of those drawbacks. Interpretable models also have a long history in statistics and machine learning Letham et al. (2015) ; Breiman et al. (1984) . Several families of interpretable models exist, such as the rule-based approach and generalized additive models Hastie & Tibshirani (1987) . Many methods focus on tabular or categorical data and less on high-dimensional structured data such as images. Interpretable models for structured data rely mostly on projecting to a lower dimensional concept or symbolic space that is understandable for humans Koh et al. (2020) . Aside from a few exceptions Ciravegna et al. (2021) ; Barbiero et al. (2022) , the current State-Of-The-Art (SOTA) design does not model the interaction between concepts and symbols, hence offering limited reasoning capabilities and less robustness. Furthermore, current designs are not as flexible as the Blackbox model, which may compromise the performance of such models. We aim to achieve the best of both worlds: the flexibility of the BlackBox and the mechanistic explainability of the interpretable models. The general idea is that a single interpretable model may not be sufficiently powerful to explain all samples, and several interpretable models might be hidden inside the Blackbox model. We construct a hybrid neuro-symbolic model by progressively carving out a mixture of interpretable model and a residual network. Our design identifies a subset of samples and routes them through the interpretable models. The remaining samples are routed through a flexible residual network. We adopt First Order Logic (FOL) as the interpretable model's backbone, which provides basic reasoning on concepts retrieved from the BlackBox model. FOL is a logical function that accepts predicates (concept presence/absent) as input and returns a True/False output being a logical expression of the predicates. The logical expression, which is a set of AND, OR, Negative, and parenthesis, can be written in the so-called Disjunctive Normal Form (DNF). DNF is a FOL logical formula composed of a disjunction (OR) of conjunctions (AND), known as the "sum of products." On the residual network, we repeat the method until the proportion of data explained by the residual network falls below a desired threshold. The experimental results across various computer vision and medical imaging datasets reveal that our method accounts for the diversity of the explanation space and has minimal impact on the Blackbox's performance. Additionally, we apply our method's explanations to detect shortcuts in computer vision and successfully eliminate the bias from the Blackbox's representation.

2. RELATED WORK

Post hoc explanations Simonyan et al. (2013) ; Selvaraju et al. (2017) ; Smilkov et al. (2017) discuss the post hoc based explanation method using saliency maps to explain a convolution neural network. This method aims to highlight the pixels in the input images that contributed to the network's prediction. Adebayo et al. (2018) ; Kindermans et al. (2019) demonstrates that the saliency maps highlight the correct regions in the image even though the backbone's representation was arbitrarily perturbed. Additionally, in LIME Ribeiro et al. (2016) , given a superpixel, a surrogate linear function attempts to learn the prediction of the Blackbox surrounding that superpixel. SHAP Lundberg & Lee (2017) utilizes a prominent game-theoretic strategy called SHAPLY values to estimate the Blackbox's prediction by considering all the permutations of adding and removing a specific feature to determine its importance in the final prediction. Having the explainations in terms of pixel intensities, they do not correspond to the high-level interpretable attributes (concept), understood by humans. In this paper, we aim to provide the post hoc explanation of the Blackbox in terms of the interpretable concepts, rather than the pixel intensities. Interpretable models In this class, the researchers try to design inherently interpretable models to eliminate the requirement for post hoc explanations. In the literature, we find interpretable models in Generalized Additive Models (GAM) Hastie & Tibshirani (1987) , or on logic formulas, as in Decision Trees Breiman et al. (1984) or Bayesian Rule Lists (BRL) Letham et al. (2015) . However, most of these methods work well in categorical datasets rather than continuous data such as images. Additionally, Chen et al. (2019) ; Nauta et al. (2021) introduce a "case-based reasoning" technique where the authors first dissect an image in Prototypical parts and then classifies by combining evidence from the pre-defined prototypes. This method is highly sensitive to the choice of prototypes and the distance metric used to compare with the prototype. So instead of using any prototypes, our proposed method uses human-understandable concepts to build a mixture of interpretable models. Concept-based interpretable models Recently, Concept bottleneck models (CBM) Koh et al. (2020) broaden the initial aim Lampert et al. (2009) ; Kumar et al. (2009) to leverage the weak annotations of visual attributes as high-level human-comprehensible concepts Kim et al. (2017) in a fine-grained image classification dataset to predict the concepts from input images. Subsequently, the downstream classifier predicts the class labels from the discovered concepts. Chen et al. (2020) aims to align the axes of the latent space with the concepts of interest. Margeloiu et al. (2021) ; Mahinpei et al. (2021) demonstrated how the information is encoded in the concept representation. However, they do not explain how the final classifier composes those concepts for prediction. LENs Ciravegna et al. (2021) utilizes a constraint-based approach from Ciravegna et al. (2020a) , Ciravegna et al. (2020b) to address this gap by providing explanations in terms of first-order logic (FOL) using the concepts. However, a strong pruning strategy and L1 regularization hinder the learning capability of LENs. E-Lens Barbiero et al. (2022) solves this problem by introducing an attention-driven entropy layer that focuses on the relevant concept for downstream classification. Later, Yuksekgonul et al. (2022) introduces the Post hoc-concept-based model (PCBM) to learn the concepts from the backbone of a trained Blackbox and utilize an interpretable classifier to learn the class labels from the concepts. In addition, they propose the Hybrid-Post hoc-concept-based model (PCBM-h) to replicate the performance of the original Blackbox by fitting the unexplained portion of the Blackbox. However, all these methods either employ a single interpretable by-design model or a surrogate model to explain a Blackbox, failing to apprehend different nuances of explanations per sample. Instead of a single interpretable model for all samples, this paper introduces a mixture of interpretable models that utilize the Blackbox's flexibility and offers local explanation.

3. METHOD

Notation Assume we have a dataset {X , Y, C}, where X , Y, and C are the input images, class labels, and human interpretable attributes, respectively. f 0 : X ! Y, is a pre-trained deep neural network (our initial Blackbox model), trained only with supervision from the labels Y. We assume that f 0 is a composition h 0 , where : X ! R l is the image embeddings and h 0 : R l ! Y is a transformation from the embeddings, , to the class labels. We denote the learnable function t : R l ! C, projecting the image embeddings to the concept space. The concept space is the space spanned by the attributes C. Per this definition, function t outputs a scalar value representing a concept for each input image. Method Overview Figure 1 shows an overview of our approach. We aim to iteratively carve out an interpretable model from the given Blackbox model. Each iteration results in an interpretable model (the downward grey paths in Figure 1 ) and a residual (the straightforward black paths in Figure 1 ). We start this process with f 0 -the initial Blackbox given to us. At iteration k, we distill the Blackbox of the previous iteration f k 1 with a neuro-symbolic interpretable model, g k : C ! Y. We generate explanations in terms of First Order Logic (FOL) from g k as suggested by Barbiero et al. (2022) . We coin the term residual r k to refer to the portion of f k 1 that g k cannot explain, specifically r k = f k 1 g k . We then approximate r k with f k = h k ( (.)). f k will be the Blackbox for the subsequent iterations and be explained by the respective interpretable models. A learnable gating mechanism, denoted by ⇡ k : C ! {0, 1} (shown as the selector in Figure 1 ) is responsible for routing an input image towards either g k or r k . The thickness of the lines in Figure 1 represents the samples covered by the interpretable models (grey line) and the residuals (black line). With every iteration, the cumulative coverage of the interpretable models increases, but the residual decreases. We name our method route, interpret and repeat.

3.1. NEURO-SYMBOLIC KNOWLEDGE DISTILLATION

Neuro-symbolic AI is an area of study that encompasses deep neural networks with symbolic approaches to computing and AI to complement the strengths and weaknesses of each, resulting in a robust AI capable of reasoning and cognitive modeling Belle (2020). Neuro-symbolic systems are hybrid models that leverage the robustness of connectionist methods and the soundness of symbolic reasoning to effectively integrate learning and reasoning Garcez et al. (2015) ; Besold et al. (2017) . Neuro-Symbolic knowledge distillation in our method involves three parts. (1) a series of trainable soft attention scores that routes each sample through the interpretable models and the residual networks, (2) A sequence of learnable neuro-symbolic interpretable models, each providing FOL explanations to interpret the Blackbox, and (3) repeating with Residual networks for the samples that cannot be explained with their interpretable counterparts. We explain each component in detail in what follows.

3.1.1. THE SELECTOR FUNCTION

The first step of our method is to route an input sample through the interpretable model g k and residuals r k using a series of soft attention scores ⇡ k for k 2 [0, K], where K is the number of iterations. We denote this as the selector in Figure 1 . Specifically, at each iteration k, the specific selector routes a sample {x j , y j , c j } towards g k and r k with probability ⇡ k (c j ) and 1 ⇡ k (c j ) respectively. We define the empirical coverage of the k th iteration as, ⇣(⇡ k ) = 1 m m X j=1 ⇡ k (c j ) , the empirical mean of the samples selected by the selector for the corresponding interpretable model g k , with m being the total number of samples in the training set. Thus the total selective risk is, R k (⇡ k , g k ) = 1 m P m j=1 L k (g k ,⇡ k ) (x j , c j ) ⇣(⇡ k ) , where L k (g k ,⇡ k ) is the optimization loss used to learn g k and ⇡ k together, discussed in section 3.1.2. For a given coverage of ⌧ k 2 (0, 1], we solve the following optimization problem ✓ ⇤ s k , ✓ ⇤ g k = min ✓ s k ,✓ g k R k ⇡ k (.; ✓ s k ), g k (.; ✓ g k ) s.t. ⇣ ⇡ k (.; ✓ s k ) ⌧ k For k th iteration, ✓ ⇤ s k , ✓ ⇤ g k are the optimal parameters for the selector ⇡ k and the interpretable model g k respectively. In this work, ⇡ k 's are neural networks with sigmoid activation. At inference time, the concept vector c j of sample x j is fed to g k if and only if ⇡ k (c j ) 0.5 for k 2 [0, K].

3.1.2. NEURO-SYMBOLIC INTERPRETABLE MODELS

In this stage for the iteration k, we design interpretable model g k to interpret the Blackbox f k 1 from the previous iteration k 1 by optimizing the following loss function, L k (g k ,⇡ k ) (x j , c j ) = ` f k 1 (x j ), g k (c j ) ⇡ k (c j ) | {z } trainable component for current iteration k k 1 Y i=1 1 ⇡ i (c j ) | {z } fixed component trained in the previous iterations The term multiplied by the loss function `in equation 3, is the probability of sample x j reaching the interpretable model g k , which is the probability of going through the residual for iterations 0 through k 1 i.e. Q k 1 i=1 1 ⇡ i (c j ) times the probability of going through the interpretable model at iteration k i.e. ⇡ k (c j ) . Refer to Figure 1 for an illustration. Note that ⇡ 1 , . . . ⇡ k 1 are learned in the previous iterations and are not trainable at iteration k. As each interpretable model g k specializes in explaining a specific subset of samples (denoted by coverage ⌧ ), we refer to it as an expert moving forward. We adopted the optimization strategy, discussed in Geifman & El-Yaniv (2019) to optimize equation 2. We discuss the loss function in equation 3 in detail in Appendix A.3. After training, we refer to the interpretable experts of all the iterations as a "Mixture of Interpretable Experts" (MoIE) cumulatively.

3.1.3. THE RESIDUALS

The last step is to repeat with the residual r k , as r k (x j , c j ) = f k 1 (x j ) g k (c j ), to generate f k , effectively making a new Blackbox for the next iteration k +1. Specifically, we train f k = h k (.) Residual (! ! ) Residual (! " ) Fix < l a t e x i t s h a 1 _ b a s e 6 4 = " r 4 / t A 6 W p F  I / D + A t N Q X q T t G 5 T u B s = " > A A A B 9 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l K q R 6 L X j x W s B / Q x L L Z b p q l m 0 3 c 3 R R K 6 O / w 4 k E R V C w W D 3 q a U C / C I 8 E C R r A 2 k u c m I X M J k w S F j 7 V B u W J X 7 Q X Q O n F y U o E c r U H 5 y x 3 G J I 2 o 0 I R j p f q O n W g v w 1 I z w u m s 5 K a K J p i M 8 Y j 2 D R U 4 o s r L F k f P 0 I V R h i i I p S m h 0 U L 9 P Z H h S K l p 5 J v O C O t Q r X p z 8 T + v n + r g 2 s u Y S F J N B V k u C l K O d I z m C a A h k 5 R o P j U E E 8 n M r Y i E W G K i T U 4 l E 4 K z + v I 6 6 d S q T q N a v 6 9 X m j d 5 H E U 4 g 3 O 4 B A e u o A l 3 0 I I 2 E H i C Z 3 i F N 2 t i v V j v 1 s e y t W D l M 6 f w B 9 b n D y s K k b c = < / l a t e x i t > h 2 Residual (! # ) < l a t e x i t s h a 1 _ b a s e 6 4 = " d C l y 3 G q I J d W p V I 7 1 l q q  C i O d x M 2 U = " > A A A B 9 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m 0 q M e i F 4 8 V 7 A c 0 s W y 2 m 2 b p Z p P u b g o l 9 H d 4 8 a C I V 3 + M N / + N 2 z Y H b X 0 w 8 H h v h p l 5 f s K Z 0 r b 9 b R X W 1 j c 2 t 4 r b p Z 3 d v f 2 D 8 u F R S 8 W p J L R J Y h 7 L j o 8 V 5 U z Q p m a a 0 0 4 i K Y 5 8 T t v + 8 G 7 m t 8 d U K h a L R z 1 J q B f h g W A B I 1 g b y X O T k L m E S Y L C p 8 t e u W J X 7 T n Q K n F y U o E c j V 7 5 y + 3 H J I 2 o 0 I R j p b q O n W g v w 1 I z w u m 0 5 K a K J p g M 8 Y B 2 D R U 4 o s r L 5 k d P 0 Z l R + i i I p S m h 0 V z 9 P Z H h S K l J 5 J v O C O t Q L X s z 8 T + v m + r g x s u Y S F J N B V k s C l K O d I x m C a A + k 5 R o P j E E E 8 n M r Y i E W G K i T U 4 l E 4 K z / P I q a V 1 U n a t q 7 a F W q d / m c R T h B E 7 h H B y 4 h j r c Q w O a Q G A E z / A v 6 B f X j U V k k m C W 2 R h C e y G 2 B F O R O 0 p Z n m t J t K i u O A 0 0 4 w u p 3 5 n T G V i i X i Q U 9 S 6 s d 4 K F j I C N Z G 8 r 0 0 Y h 5 h k q D o 0 e 3 b V a f m z I F W i V u Q K h R o 9 u 0 v b 5 C Q L K Z C E 4 6 V 6 r l O q v 0 c S 8 0 I p 9 O K l y m a Y j L C Q 9 o z V O C Y K j + f H z 1 F Z 0 Y Z o D C R p o R G c / X 3 R I 5 j p S Z x Y D p j r C O 1 7 M 3 E / 7 x e p s N r P 2 c i z T Q V Z L E o z D j S C Z o l g A Z M U q L 5 x B B M J D O 3 I h J h i Y k 2 O V V M C O 7 y y 6 u k f V F z L 2 v 1 + 3 q 1 c V P E U Y Y T O I V z c O E K G n A H T W g B g S d 4 h l d 4 s 8 b W i / V u f S x a S 1 Y x c w x / Y H 3 + A C m G k b Y = < / l a t e x i t > h 1 < l a t e x i t s h a _ b a s e = " R O w B f t Q O Y c R U Z J + Y d G b T g B E = " > A A A B H i c b V B N S N A E J U r q / o h L B b B U m k q M e i F V A c s W y m b p Z h N N U S + j u e F D E q z / G m / / G b Z u D t j Y e L w w y I O V M a c f t k p r x u b W + X t y s u v B f X j U V k k m C W R h C e y G B F O R O p Z n m t J t K i u O A w u p n T G V i i X i Q U S s d K F j I C N Z G r Y h h k q D o e n b V a f m z I F W i V u Q K h R o u v b C Q L K Z C E V r l O q v c S I p O K l y m a Y j L C Q o z V O C Y K j + f H z F Z Y Z o D C R p o R G c / X R I j p S Z x Y D p j r C O M E / x e p s N r P c i z T Q V Z L E o z D j S C Z o l g A Z M U q L x B B M J D O I h J h i Y k O V V M C O y y u k f V F z L v + q c V P E U Y Y T O I V z c O E K G n A H T W g B g S d h l d s b W i / V u f S x 8 G H u / N M D M v i A X X x n W / n Z X V t f W N z c J W c X t n d 2 + / d H D Y 1 F G i G D Z Y J C L V D q h G w S U 2 D D c C 2 7 F C G g Y C W 8 H 4 J v N b T 6 g 0 j + S 9 m c T o h 3 Q o + Y A z a q z 0 4 J F z 0 o 3 5 o 1 f s l c p u x Z 2 B L B M v J 2 X I U e + V v r r 9 i C U h S s M E 1 b r j u b H x U 6 o M Z w K n x W 6 i M a Z s T I f Y s V T S E L W f z i 6 e k l O r 9 M k g U r a k I T P 1 9 0 R K Q 6 0 n Y W A 7 Q 2 p G e t H L x P + 8 T m I G 1 3 7 K Z Z w Y l G y + a J A I Y i K S v U / 6 X C E z Y m I J Z Y r b W w k b U U W Z s S F l I X i L L y + T 5 k X F u 6 x U 7 6 r l W j W P o w D H c A J n 4 M E V 1 O A W 6 t A A B h K e 4 R X W i C G 0 T y a X q B V h T z g R t G 2 Y 4 7 c W K 4 i j g t B t M b 3 K / + 0 S V Z l L c m z S m f o T H g o W M Y G O l z i B m D 1 5 l W K 2 5 d X c O t E q 8 g t S g Q G t Y / R q M J E k i K g z h W O u + 5 8 b G z 7 A y j H A 6 q w w S T W N M p n h M + 5 Y K H F H t Z / N r Z + j M K i M U S m V L G D R X f 0 9 k O N I 6 j Q L b G W E z 0 c t e L v 7 n 9 R M T X v s Z E 3 F i q C C L R W H C k Z E o f x 2 N m K L E 8 N Q S T B S z t y I y w Q o T Y w P K Q / C W X 1 4 l n Y u 6 d 1 l v 3 D V q z U Y R R x l O 4 B T O w Y M r a M I t t K A N B B 7 h G V 7 h z Z H O i / P u f C x J C a / 9 l A m Z G C r I f F G Y c G R i l L 2 P B k x R Y v j E E k w U s 7 c i M s I K E 2 N D y k L w F l 9 e J q 3 z q n d Z r d 3 V K v V a H k c R j u A Y T s G D K 6 j D L T S g C Q Q E P M M r v D n a e X H e n Y 9 5 a 8 H J Z w 7 h D 5 z P H z T K j 0 c = < / l a t e x i t > 1 ⇡ 3 < l a t e x i t s h a 1 _ b a s e 6 4 = "  f x n 0 Q L B n 0 R L o E h C q K o 3 X f j 7 f e h 8 = " > A A A B 7 X i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 n V U j 0 W v H i s Y D + g X U s 2 z b a x 2 W R J s k J Z + h + 8 e F D E q / / H m / / G b L s H b X 0 w 8 H h v h p l 5 Q c y Z N q 7 7 7 R T W 1 j c 2 t 4 r b p Z 3 d v f 2 D 8 u F R W 8 t E E d o i k k v V D b C m n A n a M s x w 2 o 0 V x V H A a S e Y 3 G R + 5 4 k q z a S 4 N 9 O Y + h E e C R Y y g o 2 V 2 v 2 Y P V y W B u W K W 3 X n Q K v E y 0 k F c j Q H 5 a / + U J I k o s I Q j r G 6 G U / g Q v H h T x 6 i / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A X X x n W / n c L K 6 t r 6 R n G z t L W 9 s 7 t X 3 j 9 o 6 i R T D B s s E Y l q B 1 S j 4 B I b h h u B 7 V Q h j Q O B r W B 4 M / V b T 6 g 0 T + S D G a X o x z S S P O S M G i v d R 4 9 e r 1 x x q + 4 M Z J l 4 O a l A j n q v / N X t J y y L U R o m q N Y d z 0 2 N P 6 b K c C Z w U u p m G l P K h j T C j q W S x q j 9 8 e z U C T m x S p + E i b I l D Z m p v y f G N N Z 6 F A e 2 M 6 Z m o B e 9 q f i f 1 8 l M e O W P u U w z g 5 L N F 4 W Z I C Y h 0 7 9 J n y t k R o w s o U x x e y t h A 6 o o M z a d k g 3 B W 3 x 5 m T T P q t 5 F 9 f z u v F K 7 z u M o w h E c w y l 4 c A k 1 u I U 6 N I B B B M / w C m + O c F 6 c d + d j 3 l p w 8 p l D + A P n 8 w f z J I 2 X < / l a t e x i t > g 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " + D 9 J j F Z w z q 5 Y X c I 7 p U s b D R z n W I g = " > A A A B 6 n i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 k t R T 0 W v X i s a D + g X U s 2 z W 5 D s 8 m S Z I W y 9 C d 4 8 a C I V 3 + R N / + N a b s H b X 0 w 8 H h v h p l 5 Q c K Z N q 7 7 7 R T W 1 j c 2 t 4 r b p Z 3 d v f 2 D 8 u F R W 8 t U E d o i k k v V D b C m n A n a M s x w 2 k 0 U x X H A a S c Y 3 8 z 8 z h N V m k n x Y C Y J 9 W M c C R Y y g o 2 V 7 q P H 2 q B c c a v u H G i V e D m p Q I 7 m o P z V H 0 q S x l Q Y w r H W P c 9 N j J 9 h Z R j h d F r q p 5 o m m I x x R H u W C h x T 7 W f z U 6 f o z C p D F E p l S x g 0 V 3 9 P Z D j W e h I H t j P G Z q S X v Z n 4 n 9 d L T X j l Z 0 w k q a G C L B a F K U d G o t n f a M g U J Y Z P L M F E M X s r I i O s M D E 2 n Z I N w V t + e Z W 0 a 1 X v o l q / q 1 c a 1 3 k c R T i B U z g H D y 6 h A b f Q h B Y Q i O A Z X u H N 4 c 6 L 8 + 5 8 L F o L T j 5 z D H / g f P 4 A 9 K i N m A = = < / l a t e x i t > g 2 Selector < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 T 4 P X G u V N r 2 F A l 1 a g K K A h T O V K + o = " > A A A B 6 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e C F 4 8 V 7 A e 0 o W y 2 m 2 b p 7 i b s b o Q S + h e 8 e F D E q 3 / I m / / G T Z q D t j 4 Y e L w 3 w 8 y 8 I O F M G 9 f 9 d i o b m 1 v b O 9 X d 2 t 7 + w e F R / f i k p + N U E d o l M Y / V I M C a c i Z p 1 z D D 6 S B R F I u A 0 3 4 w u 8 v 9 / h N V m s X y 0 c w T 6 g s 8 l S x k B J t c G i U R G 9 c b b t M t g N a J V 5 I G l O i M 6 1 + j S U x S Q a U h H G s 9 9 N z E + B l W h h F O F 7 V R q m m C y Q x P 6 d B S i Q X V f l b c u k A X V p m g M F a 2 p E G F + n s i w 0 L r u Q h s p 8 A m 0 q t e L v 7 n D V M T 3 v o Z k 0 l q q C T L R W H K k Y l R / j i a M E W J 4 X N L M F H M 3 o p I h B U m x s Z T s y F 4 q y + v k 9 5 V 0 7 t u t h 5 a j X a r j K M K Z 3 A O l + D B D b T h H j r Q B Q I R P M M r v D n C e X H e n Y 9 l a 8 U p Z 0 7 h D 5 z P H x J A j j o = < / l a t e x i t > Figure 1 : Schematic view of route, interpret and repeat. At step k, the selector routes each sample either towards the interpretable model g k (to interpret) with probability ⇡ k or the residual (to repeat in the further iterations) r k = f k 1 g k with probability 1 ⇡ k . Here, f k 1 is the Blackbox of the previous iteration, k 1. If the sample goes through g k , we obtain the explanations for that sample in terms of First Order Logic (FOL). Otherwise, the sample goes through the next step until it either goes through a subsequent interpretable model or reaches the last residual. Note that components in black and grey indicate the fixed and trainable modules in our model. to match the residual r k . This step is crucial to make f k specialize over those samples not covered by g k . For the k th iteration, we optimize the following loss function to obtain f k : L k f (x j , c j ) = ` r k (x j , c j ), f k (x j ) | {z } trainable component for iteration k k Y i=1 1 ⇡ i (c j ) | {z } non-trainable component for iteration k Notice that the embedding (.) is fixed for all the iterations. Due to computational reasons, we only update the last few layers of the Blackbox (h k ) in order to train f k . At the final iteraion K, our method yeilds a MoIE, explaining the interpretable component of the initial Blackbox f 0 and the final residual for the uninterpretable component of f 0 . Algorithm 1 in Appendix A. For CUB-200, we employ six experts each for both VIT and ResNet-based Blackboxes. For Awa2, we utilize 4 and experts for ResNet-101 and VIT, respectively. For HAM10000 and MIMIC-CXR (for both Cardiomegaly and Effusion), we use 5 and 3 experts, respectively. Refer to Appendix A.2 and A.7 for details about the dataset and hyperparameters. Furthermore, we only include concepts as input to g if their validation accuracy or auroc exceeds a certain threshold (in all of our experiments, we fix 0.7 or 70% as the threshold of validation auroc or accuracy). For HAM10000, we use the identical "concept bank" as proposed in Yuksekgonul et al. (2022) . 

4.1. BASELINE

Our baseline is based on the interpretable by design principle, consisting of two parts: 1) a concept extractor : X ! C, a mapping from the image to the concept space; and 2) g : C ! Y, a mapping from the concepts to the prediction. We train and g sequentially from scratch with the supervision of the human attributes C and class labels Y respectively. The concept extractor for Resnet, Inception, and Densenet121 includes all layers until the last convolution block. The VITbased consists of the transformer encoder block (excluding the classification head). We use the identical configurations for g as described in Appendix A.7 for each dataset. Figure 2 displays the performance of our method using a heldout test set. "MoIE" and "MoIE + Residual" denote the mixture of interpretable experts (g) excluding and including the final residual, respectively. Coverage (Cov in the figure) represents the proportion of samples in the test set covered by all the experts throughout iterations. For CUB-200, Awa2, and HAM10000, MoIE achieves similar performance to the Blackbox. For CUB-200 and HAM10000, our method achieves a significant performance gain compared to the baseline. For Awa2, the baseline performs at par with our model. (ResNet-based baseline exceeds ours). Note that all the datasets include noisy concepts except Awa2, making it appropriate for the interpretable by design models. In general, VIT-derived experts performed better than their ResNet-based counterparts. The MIMIC-CXR dataset is highly imbalanced, with the positive samples being too few compared to the negative ones. As a result, the baselines for them attain similar accuracy scores as our MoIE model (Figure 2a ). Therefore, precision and recall are better metrics for assessing the models' performance on these datasets. To 

4.2.1. QUANTITATIVE ANALYSIS OF MOIE WITH THE BLACKBOX AND BASELINE

Expert6 (a) Baseline (Interpretable by design) Expert4 < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p 5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p 5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C C o f H z S t m F s G G + x U I a m G 1 D L p d C 8 h Q I l 7 0 a G U x V I 3 g m m d 5 n f e e L G i l A / 4 i z i v q J j L U a C U c y k R q d d G p Q r b t V d g K w T L y c V y N E c l L / 6 w 5 D F i m t k k l r b 8 9 w I / Y Q a F E z y e a k f W x 5 R N q V j 3 k u p p o p b P 1 n c O i c X q T I k o 9 C k p Z E s 1 N 8 T C V X W z l S Q d i q K E 7 v q Z e J / X i / G 0 a 2 f C B 3 F y D V b L h r F k m B I s s f J U B j O U M 5 S Q p k R 6 a 2 E T a i h D N N 4 s h C 8 1 Z f X S f u q 6 l 1 X a w + 1 S r 2 R x 1 G E M z i H S / D g B u p w D 0 1 o A Y M J P M M r v D n K e X H e n Y 9 l a 8 H J Z 0 7 h D 5 z P H x 1 F j a Q = < / l a t e x i t > BW V < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p 5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z A C 1 R z J O 8 y E i 9 P X K Q a y a Q D n i L 8 s = " > A A A B / n i c b V D L S s N A F L 3 x W e s r K q 7 c D B a L q 5 J I U Z d F N 7 q r a B / Q h D K Z T t q h k 0 m Y m Q g l F P w V N y 4 U c e t 3 u P N v n L Z Z a O u B g c M 5 9 3 D v n C D h T G n H + b a W l l d W 1 9 Y L G 8 X N r e 2 d X X t v v 6 n i V B L a I D G P Z T v A i n I m a E M z z W k 7 k R R H A a e t Y H g 9 8 V u P V C o W i w c 9 S q g f 4 b 5 g I S N Y G 6 l r H 5 Z v p a T 9 l G P p e a h 8 r 0 1 4 q L p 2 y a k 4 U 6 B F 4 u a k B D n q X f v L 6 8 U k j a j Q h G O l O q 6 T a D / D U j P C 6 b j o p Y o m m A x x n 3 Y M F T i i y s + m 5 4 / R i V F 6 K I y l e U K j q f o 7 k e F I q V E U m M k I 6 4 G a 9 y b i f 1 4 n 1 e G l n z G R p J o K M l s U p h z p G E 2 6 Q D 0 m K d F 8 Z A g m k p l b E R l g i Y k 2 j R V N C L Q p r Q x S I K L D g D y C V u f O x c = " > A A A C C H i c b V C 7 T g J B F J 3 F F + I L t b R w I p F Y k V 1 D 1 J J o Y 4 k P H g l L y O x w g Q m z s 5 u Z u y a E U N r 4 K z Y W G m P r J 9 j 5 N w 6 P Q s G T T H J y z r m 5 c 0 8 Q S 2 H Q d b + d 1 N L y y u p a e j 2 z s b m 1 v Z P d 3 a u a K N E c K j y S k a 4 H z I A U C i o o U E I 9 1 s D C Q E I t 6 F + N / d o D a C M i d Y + D G J o h 6 y r R E Z y h l V r Z Q 1 9 B l + Z v o a v B j F P U 9 2 n + D n X C M b F S K 5 t z C + 4 E d J F 4 M 5 I j M 5 R b 2 S + / H f E k B I V c M m M a n h t j c 8 g 0 C i 5 h l P E T A z H j f d a F h q W K h W C a w 8 k h I 3 p s l T b t R N o + h X S i / p 4 Y s t C Y Q R j Y Z M i w Z + a 9 s f i f 1 0 i w c 9 E c C h U n C I p P F 3 U S S T G i 4 1 Z o W 2 j g K A e W M K d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p Q M / + + j X F c 4 = " > A A A B 8 n i c b V B N S 8 N A E N 3 U r x q / q h 6 9 L B b B U 0 m k q M d S L x 4 r 2 A 9 I Q 9 l s p + 3 S z W 7 Y 3 Q g l 9 G d 4 8 a C I V 3 + N N / + N m z Y H b X 0 w 8 H h v h p l 5 U c K Z N p 7 3 7 Z Q 2 N r e 2 d 8 q 7 7 t 7 + w e F R 5 f i k o 2 W q K L S p 5 F L 1 I q K B M w F t w w y H X q K A x B G H b j S 9 y / 3 u E y j N p H g 0 s w T C m I w F G z F K j J W C v o A x b n Y 7 r u s O K l W v 5 i 2 A 1 4 l f k C o q 0 B p U v v p D S d M Y h K G c a B 3 4 X m L C j C j D K I e 5 2 0 8 1 J I R O y R g C S w W J Q Y f Z 4 u Q 5 v r D K E I + k s i U M X q i / J z I S a z 2 L I 9 s Z E z P R q 1 4 u / u c F q R n d h h k T S W p A 0 O W i U c q x k T j / H w + Z A m r 4 z B J C F b O 3 Y j o h i l B j U 8 p D 8 F d f X i e d q 5 p / X a s / 1 K u N Z h F H G Z 2 h c 3 S J f H S D G u g e t V A b U S T R M 3 p F b 4 5 x X p x 3 5 2 P Z W n K K m V P 0 B 8 7 n D + Z V j 7 Q = < / l a t e x i t > ¬BW V < l a t e x i t s h a 1 _ b a s e 6 4 = " T Q W G U Q P s F Z y J W b Y U T R S x q u F 9 W j w = " > A A A B + X i c b V B N S 8 N A E N 3 U r x q / o h 6 9 L B b B U 0 l E 1 G N R Q Y 9 V 7 A e 0 o W y 2 k 3 b p Z h N 2 N 4 U S + k + 8 e F D E q / / E m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 I O F M a d f 9 t k o r q 2 v r G + V N e 2 t 7 Z 3 f P 2 T 9 o q j i V F B o 0 5 r F s B 0 Q B Z w I a m m k O 7 U Q C i Q I O r W B 0 k / u t M U j F Y v G k J w n 4 E R k I F j J K t J F 6 j t M V M M C P M E g 5 k b d 3 t t 1 z K m 7 V n Q E v E 6 8 g F V S g 3 n O + u v 2 Y p h E I T T l R q u O 5 i f Y z I j W j H K Z 2 N 1 W Q E D o i A + g Y K k g E y s 9 m l 0 / x i V H 6 O I y l K a H x T P 0 9 k Z F I q U k U m M 6 I 6 K F a 9 H L x P 6 + T 6 v D K z 5 h I U g 2 C z h e F K c c 6 x n k M u M 8 k U M 0 n h h A q m b k V 0 y G R h G o T V h 6 C t / j y M m m e V b 2 L 6 v n D e a V 2 X c R R R k f o G J 0 i D 1 2 i G r p H d d R A F I 3 R M 3 p F b 1 Z m v V j v G J B Q 1 H V T X d X k H C m t O t + O 4 W V 1 b X 1 j e J m a W t 7 Z 3 e v v H / Q U r G R h D Z J z G P Z C b C i n A n a 1 E x z 2 k k k x V H A a T s Y X 2 d + + 4 l K x W L x q C c J 9 S M 8 F C x k B G s r 9 e 6 l p E P D s b y 5 L f X L F b f q z o C W i Z e T C u R o 9 M t f v U F M T E S F J h w r 1 f X c R P s p l p o R T q e l n l E 0 w W S M h 7 R r q c A R V X 4 6 u 3 m K T q w y Q G E s b Q m N Z u r v i R R H S k 2 i w H Z G W I / U o p e J / 3 l d o 8 N L P 2 U i M Z o K M l 8 U G o 5 0 j L I A 0 I B J S j S f W I K J Z P Z W R E Z Y Y q J t T F k I 3 u L L y 6 R 1 V v X O q 7 W H W q V + l c d R h C M v 0 3 Z / f R D k j n 1 h 2 Y v E 0 = " > A A A B / 3 i c b V D L S s N A F J 3 U V 6 2 v q O D G z W A R X J V E i r o s u t F d R f u A N p T J 9 C Y d O p m E m Y l Q a h f + i h s X i J U U G j T m s W z 7 R A F n A h q a a Q 7 t R A K J f A 4 t f 3 i V + a 0 H k I r F 4 l 6 P E v A i E g o W M E q 0 k X r 2 Q V d A i G + k h D D l R N 5 p E x 6 q U s 8 u O x V n C r x I 3 J y U U Y 5 6 z / 7 q 9 m O a R i A 0 5 U S p j u s k 2 h s T q R n l M C l 1 U w U J o U M S Q s d Q Q S J Q 3 n h 6 / w Q f G 6 W P g 1 i a J z j i V F B o 0 5 r F s B 0 Q B Z w I a m m k O 7 U Q C i Q I O r W B 0 k / u t M U j F Y v G k J w n 4 E R k I F j J K t J F 6 j t M V M M C P M E g 5 k b d 3 t t 1 z K m 7 V n Q E v E 6 8 g F V S g 3 n O + u v 2 Y p h E I T T l R q u O 5 i f Y z I j W j H K Z 2 N 1 W Q E D o i A + g Y K k g E y s 9 m l 0 / x i V H 6 O I y l K a H x T P 0 9 k Z F I q U k U m M 6 I 6 K F a 9 H L x P 6 + T 6 v D K z 5 h I U g 2 C z h e F K c c 6 x n k M u M 8 k U M 0 n h h A q m b k V 0 y G R h G o T V h 6 C t / j y M m m e V b 2 L 6 v n D e a V 2 X c R R R k f o G J 0 i D 1 2 i G r p H d d R A F I 3 R M 3 p F b 1 Z m v V j v O i v + i X p 9 e p f v 0 / q H m B P 1 M S z E = " > A A A C C H i c b V C 7 T s M w F H V 4 l v I K M D J g U S E x V Q m q g L H A w o S K R B 9 S E 1 W O e 9 N a d e L I d k B V 1 Z G F X 2 F h A C F W P o G N v 8 F p M 0 D L k S w d n 3 O v f e 8 J E s 6 U d p x v a 2 F x a X l l t b B W X N / Y 3 N q 2 d 3 Y b S q S S Q p 0 K L m Q r I A o 4 i 6 G u m e b Q S i S Q K O D Q D A Z X m d + 8 B 6 m Y i O / 0 M A E / I r 2 Y h Y w S b a S O f X B h V H P j 2 P N w j f U i i H V G b 0 A / C D k o d u y S U 3 Y m w P P E z U k J 5 a h 1 7 C + v K 2 i a v U M 5 U a r t O o n 2 R 0 R q R j m M i 1 6 q I C F 0 Q H r Q N j Q m E S h / N F l k j I + M 0 s W h k O a Y O S b q 7 4 4 R i Z Q a R o G p j I j u q 1 k v E / / z 2 q k O z / 0 R i 5 N U Q 0 y n H 4 U p x 1 r g L B X c Z R K o 5 k N D C J X M z I p p n 0 h C t c k u C 8 G d X X m e N E 7 K 7 m m 5 c l s p V S / z O A p o H x 2 i Y + S i M 1 R F 1 6 i G 6 o i i R / S M X t G b 9 W S 9 W O / W x 7 R 0 w c p 7 9 t A f W J 8 / P c O Y 2 Q = = < / l a t e x i t > Atypical P igment Network < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p 5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t A 2 Z A C g U N F C i h n W h g U S C h F Y x u M r / 1 C N q I W N 3 j O A E / Y g M l Q s E Z W u k h S P k I A X B Y y t A r V 9 y q O w N d J l 5 O K i R H v V f + 6 v Z j n k a g k E t m T M d z E / Q n T K P g E q a l b m o g Y X z E B t C x V L E I j D + Z X T 2 l J 1 b p 0 z D W t h T S m f p 7 Y s I i Y 8 Z R Y D s j h k O z 6 G X i f 1 4 n x f D K n w i V p A i K z x e F q a Q Y 0 y w C 2 h c a O M q x J Y x r Y W + l f M g 0 4 2 i D y k L w F l 9 e J s 2 z q n d R P b 8 7 r 9 S u 8 z i K 5 I g c k 1 P i k U t S I 7 e k T h q E E 0 2 e y S t 5 c 5 6 c F + f d + Z i 3 F p x 8 5 p D 8 g f P 5 A y 8 7 k Q A = < / l a t e x i t > bu ck tee th < l a t e x i t s h a 1 _ b a s e 6 4 = " F 0 6 + g 9 O 8 v M T b 1 L 0 u r O G A i P p K I i s = " > A A A B 9 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 c K 9 g P a W D b b T b t 0 s x t 2 N 0 o J / R 9 e P C j i 1 f / i z X / j J s 1 B W x 8 M P N 6 b Y W Z e E H O m j e t + O 6 W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q 1 j J R h L a I 5 F J 1 A 6 w p Z 4 K 2 D D O c d m N F c R R w 2 g k m N 5 n f e a R K M y n u z T S m f o R H g o W M Y G O l B y 7 F S F A y q e Q Y V G t u 3 c 2 B l o l X k B o U a A 6 q X / 2 h J E l E h S E c a 9 3 z 3 N j 4 K V a G E U 5 n l X 6 i a Y z J B I 9 o z 1 K B I 6 r 9 N L 9 6 h k 6 s M k S h V L a E Q b n 6 e y L F k d b T K L C d E T Z j v e h l 4 n 9 e L z H h l Z 8 y E S e G C j J f F C Y c G Y m y C N C Q K U o M n 1 q C i W L 2 V k T G W G F i b F B Z C N 7 i y 8 u k f V b 3 L u r n d + e 1 x n U R R x m O 4 B h O w Y N L a M A t N K E F B B Q 8 w y u 8 O U / O i / P u f M x b S 0 4 x c w h / 4 H z + A J X X k J w = < / l a t e x i t > lon gne ck < l a t e x i t s h a 1 _ b a s e 6 4 = " J L 4 D 6 g x C B W b 8 k P 2 1 l q n O r V i h i u g = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 c K 9 g P a U D b b S b t 0 N w m 7 G y G U / g g v H h T x 6 u / x 5 r 9 x 0 + a g r Q 8 G H u / N M D M v S A T X x n W / n d L a + s b m V n m 7 s r O 7 t 3 9 Q P T x q 6 z h V D F s s F r H q B l S j 4 B G 2 D D c C u 4 l C K g O B n W B y l / u d J 1 S a x 9 G j y R L 0 J R 1 F P O S M G i t 1 t E Q h s s q g W n P r 7 h x k l X g F q U G B 5 q D 6 1 R / G L J U Y G S a o 1 j 3 P T Y w / p c p w J n B W 6 a c a E 8 o m d I Q 9 S y M q U f v T + b k z c m a V I Q l j Z S s y Z K 7 + n p h S q X U m A 9 s p q R n r Z S 8 X / / N 6 q Q l v / C m P k t R g x B a L w l Q Q E 5 P 8 d z L k C p k R m S W U K W 5 v J W x M F W X G J p S H 4 C 2 / v E r a F 3 X v q n 7 5 c F l r 3 B Z x l O E E T u E c P L i G B t x D E 1 r A Y A L P 8 A p v T u K 8 O O / O x 6 K 1 5 B Q z x / A H z u c P F I S P a Q = = < / l a t e x i t > sm ell y < l a t e x i t s h a 1 _ b a s e 6 4 = " e M s j H O j p 2 7 V 1 Y j I Q m 2 5 B T u w H E G s = " > A A A B 8 H i c b V B N S 8 N A E N 3 U r x q / q h 6 9 L B b B U 0 m k q M e i F 4 8 V 7 I e 0 o W y 2 k 3 T p Z j f s b o R S + i u 8 e F D E q z / H m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 M O V M G 8 / 7 d k p r 6 x u b W + V t d 2 d 3 b / + g c n j U 1 j J T F F p U c q m 6 I d H A m Y C W Y Y Z D N 1 V A k p B D J x z f 5 n 7 n C Z R m U j y Y S Q p B Q m L B I k a J s d I j l y L m E L v u o F L 1 a t 4 c e J X 4 B a m i A s 1 B 5 a s / l D R L Q B j K i d Y 9 3 0 t N M C X K M M p h 5 v Y z D S m h Y x J D z 1 J B E t D B d H 7 w D J 9 Z Z Y g j q W w J g + f q 7 4 k p S b S e J K H t T I g Z 6 W U v F / / z e p m J r o M p E 2 l m Q N D F o i j j 2 E i c f 4 + H T A E 1 f G I J o Y r Z W z E d E U W o s R n l I f j L L 6 + S 9 k X N v 6 z V 7 + v V x k 0 R R x m d o F N 0 j n x 0 h R r o D j V R C 1 G U o G f 0 i t 4 c 5 b w 4 7 8 7 H o r X k F D P H 6 A + c z x / v k Y / Z < / l a t e x i t > lo n gl eg  q Q h s k r F q B h T z i R t G G Y b S e K Y h F w g q G V n e u q d K s j e m l F C f Y H k k W M Y G O t u y H M i x Z e u W K W U n o E X h z U T l v M p n e K w i k g k p D O N a m J T O s D C O c j k v d V N M E k y H u V E g u q / W w y B g d W S d E U a z s k w Z N N d G R Z a j R g K w U A z f e Z / W k 0 W i 1 h 1 Q q p R c I l N w 4 3 A T q K Q R q H A d j i 5 y / 3 2 E y r N Y / l o p g k G E R 1 J P u S M G i u 1 e x J H r u v 2 K 1 W v 5 s 1 B V o l f k C o U a P Q r X 7 1 B z N I I p W G C a t 3 1 v c Q E G V W G M 4 E z t 5 d q T C i b 0 B F 2 L Z U 0 Q h 1 k 8 3 N n 5 M w q A z K M l S 1 p y F z 9 P Z H R S O t p F N r O i J q x X v Z y 8 T + v m 5 r h T Z B x m a Q G J V s s G q a C m J j k v 5 M B V 8 i M m F p C m e L 2 V s L G V F F m b E J 5 C P 7 y y 6 u k d V H z r 2 q X D 5 f V + m 0 R R x l O 4 B T O w Y d r q M M 9 N K A J D C b w D K / w 5 i T O i / P u f C x a S 0 4 x c w x / 4 H z + A K z v j n 0 = < / l a t e x i t > ¬ < l a t e x i t s h a 1 _ b a s e 6 4 = " J L 4 D 6 g x C B W b 8 k P 2 1 l q n O r V i h i u g = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 c K 9 g P a U D b b S b t 0 N w m 7 G y G U / g g v H h T x 6 u / x 5 r 9 x 0 + a g r Q 8 G H u / N M D M v S A T X x n W / n d L a + s b m V n m 7 s r O 7 t 3 9 Q P T x q 6 z h V D F s s F r H q B l S j 4 B G 2 D D c C u 4 l C K g O B n W B y l / u d J 1 S a x 9 G j y R L 0 J R 1 F P O S M G i t 1 t E Q h s s q g W n P r 7 h x k l X g F q U G B 5 q D 6 1 R / G L J U Y G S a o 1 j 3 P T Y w / p c p w J n B W 6 a c a E 8 o m d I Q 9 S y M q U f v T + b k z c m a V I Q l j Z S s y Z K 7 + n p h S q X U m A 9 s p q R n r Z S 8 X / / N 6 q Q l v / C m P k t R g x B a L w l Q  Q E 5 P 8 d z L k C p k R m S W U K W 5 v J W x M F W X G J p S H 4 C 2 / v E r a F 3 X v q n 7 5 c F l r 3 B Z x l O E E T u E c P L i G B t x D E 1 r A Y A L P 8 A p v T u K 8 O O / O x 6 K 1 5 B Q z x / D i V F N o 0 5 r H s + U Q B Z w L a m m k O v U Q C i X w O X X 9 y k / v d B 5 C K x e J e T x P w I h I K F j B K t J G G d n U g I M Q 8 F q E A O q k Y D O 2 a U 3 f m w K v E L U g N F W g N 7 a / B K K Z p B E J T T p T q u 0 6 i v Y x I z S i H W W W Q K k g I n Z A Q + o Y K E o H y s v n p M 3 x q l B E O Y m l K a D x X f 0 9 k J F J q G v m m M y J 6 r J a 9 X P z P 6 6 c 6 u P I y J p J U g 6 ¬ lo ng ne ck < l a t e x i t s h a 1 _ b a s e 6 4 = " a w 3 o T j m v D I X 7 R H a I H Y k / C t c v K V c = " > A A A B 7 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 B I v g q S R S 1 G P R i 8 c K 9 g P a U D a b S b t 2 s x t 2 N 0 I p / Q 9 e P C j i 1 f / j z X / j t s 1 B W x 8 M P N 6 b Y W Z e m H K m j e d 9 O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 j J T F J t U c q k 6 I d H I m c C m Y Y Z j J 1 V I k p B j O x z d z v z 2 E y r N p H g w 4 x S D h A w E i x k l x k q t m C G P d L 9 c 8 a r e H O 4 q 8 X N S g R y N f v m r F 0 m a J S g M 5 U T r r u + l J p g Q Z R j l O C 3 1 M o 0 p o S M y w K 6 l g i S o g 8 n 8 2 q l 7 Z p X I j a W y J Y w 7 V 3 9 P T E i i 9 T g J b W d C z F A v e z P x P 6 + b m f g 6 m D C R Z g Y F X S y K M + 4 a 6 c 5 e d y O m k B o + t o R Q x e y t L h 0 S R a i x A Z V s C P 7 y y 6 u k d V H 1 L 6 u 1 + 1 q l f p P H U Y Q T O I V z 8 O E K 6 n A H D W g C h U d 4 h l d 4 c 6 T z 4 r w 7 H 4 v W g p P P H M M f O J 8 / r n S P N g = = < / l a t e x i t > f ie ld s demonstrate this, we plot the auroc in Figure 2b . As it can be seen, MoIE significantly outperforms the baseline in terms of auroc. C L R U H K s Y 5 x n g M e M Q l U 8 6 k h h E p m b s V 0 T C S h 2 q S V h + A u v 7 x K O u d 1 9 6 L e u G v U m t d F H G V 0 j E 7 Q G X L R J W q i W 9 R

4.2.2. LOCAL EXPLANATIONS BY MOIE

As shown in E-Lens Barbiero et al. (2022) , each expert constructs local FOL per sample, composing the concepts based on their attention weights. Due to space constraints, we report the analysis for the FOL explanation of HAM10000 and CUB-200 datasets here; Appendices A.9.1 and A.8.5 include the results of Awa2 and MIMIC-CXR datasets, respectively. Figures 3a-b exhibit a quantitative analysis of the relative frequency of the concepts used by various experts to correctly classify the "Malignant" and "Benign" skin lesions, respectively. As per Figure 3a , local FOLs' of all samples analyzed by experts 2, 3, and 5 include the concept AtypicalPigmentNetwork, proving it to be an essential concept for predicting malignancy. Also, expert4 (the red bars in the Figure 3a ) classifies "Malignancy," using multiple concepts, such as IrregularStreaks, IrregularDG, and Blue Whitish Veil (BWV), whereas expert2 (the blue bar in the Figure 3a ) just utilizes AtypicalPigmentNetwork.

Bay breasted warbler

< l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5  E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p M v A q F o N q z a 2 7 c 6 B V 4 h W k B g W a g + p X f x i T V F B p C M d a 9 z w 3 M U G G l W G E 0 1 m l n 2 q a Y D L B I 9 q z V G J B d Z D N D 5 6 h M 6 s M U R Q r W 9 K g u f p 7 I s N C 6 6 k I b a f A Z q y X v V z 8 z + u l J r o J M i a T 1 F B J F o u i l C M T o / x 7 N G S K E s O n l m C i m L 0 V k T F W m B i b U R 6 C t / z y K m l f 1 r 2 r u n / v 1 x p + E U c Z T u A U L s C D a 2 j A H T S h B Q Q E P M M r v D n K e X H e n Y 9 F a 8 k p Z o 7 h D 5 z P H 3 T X j j A = < / l a t e x i t > c 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " X 0 q j z S Q y j 3 6  V t B I C R F r I l p D u R I 0 = " > A A A B 8 H i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b F U 9 k t p X o 9 U a R b J B z O L q S / w W L K Q E W y s 9 H i J y L B W s h i W K 2 7 V X Q C t E y 8 n F c j R G p a / B q O I J I J K Q z j W u u + 5 s f F T r A w j n M 5 L g 0 T T G J M p H t O + p R I L q v 1 0 c f A c X V h l h M J I 2 Z I G L d T f E y k W W s 9 E Y D s F N h O 9 6 m X i f 1 4 / M e G N n z I Z J 4 Z K s l w U J h y Z C G X f o x F T l B g + s w Q T x e y t i E y w w s T Y j L I Q v N W X 1 0 m n V v U a 1 f p 9 v d K s 5 3 E U 4 Q z O 4 Q o 8 u I Y m 3 E E L 2 k B A w D O 8 w p u j n B f n 3 f l Y t h a c f O Y U / s D 5 / A F 2 X 4 4 x < / l a t e x i t > c 2 Expert4 Expert2 < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p 5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W q M j d / C r E e 8 9 V D u 4 k U Q a 2 1 4 9 i 7 I = " > A A A B 8 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b F U 0 k k q M e C F 4 8 V 7 I e 0 o W y 2 m 3 b p 7 i b s b o Q S + i u 8 e F D E q z / H m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 M O F M G 9 f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f C o r e N U E d o i M Y 9 V N 8 S a c i Z p y z D D a T d R F I u Q 0 0 4 4 u c 3 9 z h N V m s X y w U w T G g g 8 k i x i B B s r P Z 4 j M v A q F o N q z a 2 7 c 6 B V 4 h W k B g W a g + p X f x i T V F B p C M d a 9 z w 3 M U G G l W G E 0 1 m l n 2 q a Y D L B I 9 q z V G J B d Z D N D 5 6 h M 6 s M U R Q r W 9 K g u f p 7 I s N C 6 6 k I b a f A Z q y X v V z 8 z + u l J r o J M i a T 1 F B J F o u i l C M T o / x 7 N G S K E s O n l m C i m L 0 V k T F W m B i b U R 6 C t / z y K m l f 1 r 2 r u n / v 1 x p + E U c Z T u A U L s C D a 2 j A H T S h B Q Q E P M M r v D n K e X H e n Y 9 F a 8 k p Z o 7 h D 5 z P H 3 T X j j A = < / l a t e x i t > c 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " r + R w J U P k T h h k y q x u S p s z x N 6 G J l k = " > A A A B 8 H i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b F U 9 m 1 R T 0 W v H i s Y D + k X U o 2 z b a h S X Z J s k J Z + i u 8 e F D E q z / H m / / G b L s H b X 0 w 8 H h v h p l 5 Q c y Z N q 7 7 7 R T W 1 j c 2 t 4 r b p Z 3 d v f 2 D 8 u F R W 0 e J I r R F I h 6 p b o A 1 5 U z S l m G G 0 2 6 s K B Y B p 5 1 g c p v 5 n S e q N I v k g 5 n G 1 B d 4 J F n I C D Z W e j x H Z F A r W Q z K F b f q z o F W i Z e T C u R o D s p f / W F E E k G l I R x r 3 f P c 2 P g p V o Y R T m e l f q J p j M k E j 2 j P U o k F 1 X 4 6 P 3 i G z q w y R G G k b E m D 5 u r v i R Q L r a c i s J 0 C m 7 F e 9 j L x P 6 + X m P D G T 5 m M E 0 M l W S w K E 4 5 M h L L v 0 Z A p S g y f W o K J Y v Z W R M Z Y Y W J s R l k I 3 v L L q 6 R 9 W f W u q v X 7 e q V R z + M o w g m c w g V 4 c A 0 N u I M m t I C A g G d 4 h T d H O S / O u / O x a C 0 4 + c w x / I H z + Q N 3 5 4 4 y < / l a t e x i t > c 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " k z / G m K w 8 4 y 0 1 C p X F f Y + / s b F 7 e H 8 = " > A A A B 8 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b F U 0 k k q M e C F 4 8 V 7 I e 0 o W y 2 m 3 b p 7 i b s b o Q S + i u 8 e F D E q z / H m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 M O F M G 9 f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f C o r e N U E d o i M Y 9 V N 8 S a c i Z p y z D D a T d R F I u Q 0 0 4 4 u c 3 9 z h N V m s X y w U w T G g g 8 k i x i B B s r P Z 4 j M v A r F o N q z a 2 7 c 6 B V 4 h W k B g W a g + p X f x i T V F B p C M d a 9 z w 3 M U G G l W G E 0 1 m l n 2 q a Y D L B I 9 q z V G J B d Z D N D 5 6 h M 6 s M U R Q r W 9 K g u f p 7 I s N C 6 6 k I b a f A Z q y X v V z 8 z + u l J r o J M i a T 1 F B J F o u i l C M T o / x 7 N G S K E s O n l m C i m L 0 V k T F W m B i b U R 6 C t / z y K m l f 1 r 2 r u n / v 1 x p + E U c Z T u A U L s C D a 2 j A H T S h B Q Q E P M M r v D n K e X H e n Y 9 F a 8 k p Z o 7 h D 5 z P H 3 l v j j M = < / l a t e x i t > c 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 / p e n 8 N 8 u S w W 3 j N H u c O d f i q F j k s = " > A A A B 9 X i c b V D L T g J B E O z F F + I L 9 e h l I t F 4 I r s G H 0 c S L x 4 x k U c C K 5 k d e m H C 7 O x m Z l Z D C P / h x Y P G e P V f v P k 3 D r A H B S v p p F L V n e 6 u I B F c G 9 f 9 d n I r q 2 v r G / n N w t b 2 z u 5 e c f + g o e N U M a y z W M S q F V C N g k u s G 2 4 E t h K F N A o E N o P h z d R v P q L S P J b 3 Z p S g H 9 G + 5 C F n 1 F j p 4 Z R 0 J P Y J 6 1 4 U L L r F k l t 2 Z y D L x M t I C T L U u s W v T i 9 m a Y T S M E G 1 b n t u Y v w x V Y Y z g Z N C J 9 W Y U D a k f W x b K m m E 2 h / P r p 6 Q E 6 v 0 S B g r W 9 K Q m f p 7 Y k w j r U d R Y D s j a g Z 6 0 Z u K / 3 n t 1 I T X / p j L J D U o 2 X x R m A p i Y j K N g P S 4 Q m b E y B L K F L e 3 E j a g i j J j g 5 q G 4 C 2 + v E w a 5 2 X v s l y 5 q 5 S q l S y O P B z B M Z y B B 1 d Q h V u o Q R 0 Y K H i G V 3 h z n p w X 5 9 3 5 m L f m n G z m E P 7 A + f w B 3 R e Q H A = = < / l a t e x i t > ¬c 5 < l a t e x i t s h a 1 _ b a s e 6 4 = " R t e V R m U M T N Y m d 0 Z r V a f l S T Z T 5 R 0 = " > A A A B 9 X i c b V D L T g J B E O z F F + I L 9 e h l I t F 4 I r u G o E c S L x 4 x k U c C K 5 k d G p g w O 7 u Z m d U Q w n 9 4 8 a A x X v 0 X b / 6 N s 7 A H B S v p p F L V n e 6 u I B Z c G 9 f 9 d n J r 6 x u b W / n t w s 7 u 3 v 5 B 8 f C o q a N E M W y w S E S q H V C N g k t s G G 4 E t m O F N A w E t o L x T e q 3 H l F p H s l 7 M 4 n R D + l Q 8 g F n 1 F j p 4 Z x 0 J Q 4 J 6 1 U L F r 1 i y S 2 7 c 5 B V 4 m W k B B n q v e J X t x + x J E R p m K B a d z w 3 N v 6 U K s O Z w F m h m 2 i M K R v T I X Y s l T R E 7 U / n V 8 / I m V X 6 Z B A p W 9 K Q u f p 7 Y k p D r S d h Y D t D a k Z 6 2 U v F / 7 x O Y g b X / p T L O D E o 2 W L R I B H E R C S N g P S 5 Q m b E x B L K F L e 3 E j a i i j J j g 0 p D 8 J Z f X i X N y 7 J X L V f u K q V a J Y s j D y d w C h f g w R X U 4 B b q 0 A A G C p 7 h F d 6 c J + f F e X c + F q 0 5 J 5 s 5 h j 9 w P n 8 A 3 p + Q H Q = = < / l a t e x i t > ¬c 6 < l a t e x i t s h a 1 _ b a s e 6 4 = " u a 8 A i j 0 y z i 1 U 2 G Y e I f m R L f w e q e g = " > A A A B 9 X i c b V D L T g J B E O z F F + I L 9 e h l I t F 4 I r u G i E c S L x 4 x k U c C K 5 k d G p g w O 7 u Z m d U Q w n 9 4 8 a A x X v 0 X b / 6 N s 7 A H B S v p p F L V n e 6 u I B Z c G 9 f 9 d n J r 6 x u b W / n t w s 7 u 3 v 5 B 8 f C o q a N E M W y w S E S q H V C N g k t s G G 4 E t m O F N A w E t o L x T e q 3 H l F p H s l 7 M 4 n R D + l Q 8 g F n 1 F j p 4 Z x 0 J Q 4 J 6 1 U L F r 1 i y S 2 7 c 5 B V 4 m W k B B n q v e J X t x + x J E R p m K B a d z w 3 N v 6 U K s O Z w F m h m 2 i M K R v T I X Y s l T R E 7 U / n V 8 / I m V X 6 Z B A p W 9 K Q u f p 7 Y k p D r S d h Y D t D a k Z 6 2 U v F / 7 x O Y g b X / p T L O D E o 2 W L R I B H E R C S N g P S 5 Q m b E x B L K F L e 3 E j a i i j J j g 0 p D 8 J Z f X i X N y 7 J 3 V a 7 c V U q 1 S h Z H H k 7 g F C 7 A g y r U 4 B b q 0 A A G C p 7 h F d 6 c J + f F e X c + F q 0 5 J 5 s 5 h j 9 w P n 8 A 4 C e Q H g = = < / l a t e x i t > ¬c 7 < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 b P U e e X 5 W y Z x d 3 k A U w 2 n 7 q u n n 8 o = " > A A A B 9 X i c b V D L T g J B E O z F F + I L 9 e h l I t F 4 I r u G K E c S L x 4 x k U c C K 5 k d G p g w O 7 u Z m d U Q w n 9 4 8 a A x X v 0 X b / 6 N s 7 A H B S v p p F L V n e 6 u I B Z c G 9 f 9 d n J r 6 x u b W / n t w s 7 u 3 v 5 B 8 f C o q a N E M W y w S E S q H V C N g k t s G G 4 E t m O F N A w E t o L x T e q 3 H l F p H s l 7 M 4 n R D + l Q 8 g F n 1 F j p 4 Z x 0 J Q 4 J 6 1 U L F r 1 i y S 2 7 c 5 B V 4 m W k B B n q v e J X t x + x J E R p m K B a d z w 3 N v 6 U K s O Z w F m h m 2 i M K R v T I X Y s l T R E 7 U / n V 8 / I m V X 6 Z B A p W 9 K Q u f p 7 Y k p D r S d h Y D t D a k Z 6 2 U v F / 7 x O Y g Z V f 8 p l n B i U b L F o k A h i I p J G Q P p c I T N i Y g l l i t t b C R t R R Z m x Q a U h e M s v r 5 L m Z d m 7 K l f u K q V a J Y s j D y d w C h f g w T X U 4 B b q 0 A A G C p 7 h F d 6 c J + f F e X c + F q 0 5 J 5 s 5 h j 9 w P n 8 A 4 a + Q H w = = < / l a t e x i t > ¬c 8 < l a t e x i t s h a 1 _ b a s e 6 4 = " H i C m N f F v 4 e d r h G r 8 p q z u O Z H k k e A = " > A A A B 9 X i c b V D L S g N B E O y N r x h f U Y 9 e B o P i K e x K 8 H E L e P E Y w T w g W c P s p D c Z M j u 7 z M w q I e Q / v H h Q x K v / 4 s 2 / c Z L s Q R M L G o q q b r q 7 g k R w b V z 3 2 8 m t r K 6 t b + Q 3 C 1 v b O 7 t 7 x f 2 D h o 5 T x b D O Y h G r V k A 1 C i 6 x b r g R 2 E o U 0 i g Q 2 A y G N 1 O / + Y h K 8 1 j e m 1 G C f k T 7 k o e c U W O l h 1 P S k d g n r H t d s O g W S 2 7 Z n Y E s E y 8 j J c h Q 6 x a / O r 2 Y p R F K w w T V u u 2 5 i f H H V B n O B E 4 K n V R j Q t m Q 9 r F t q a Q R a n 8 8 u 3 p C T q z S I 2 G s b E l D Z u r v i T G N t B 5 F g e 2 M q B n o R W 8 q / u e 1 U x N e + W M u k 9 S g Z P N F Y S q I i c k 0 A t L j C p k R I 0 s o U 9 z e S t i A K s q M D W o a g r f 4 8 j J p n J e 9 i 3 L l r l K q V r I 4 8 n A E x 3 A G H l x C F W 6 h B n V g o O A Z X u H N e X J e n H f n Y 9 6 a c 7 K Z Q / g D 5 / M H 4 z e Q I A = = < / l a t e x i t > ¬c 9 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 W D q y D 0 f r h p i l o v 6 7 Y G S u c c s + 3 E = " > A A A B + n i c b V B N S 8 N A E N 3 U r 1 q / U j 1 6 W S y K p 5 J I U Y 8 F L x 4 r 2 A 9 o Q 9 h s J + 3 S z S b s b p Q S + 1 O 8 e F D E q 7 / E m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 I O F M a c f 5 t k p r 6 x u b W + X t y s 7 u 3 v 6 B X T 3 s q D i V F N o 0 5 r H s B U Q B Z w L a m m k O v U Q C i Q I O 3 W B y k / v d B 5 C K x e J e T x P w I j I S L G S U a C P 5 d v U M D w S M M P U z 1 5 1 V D H y 7 5 t S d O f A q c Q t S Q w V a v v 0 1 G M Y 0 j U B o y o l S f d d J t J c R q R n l M K s M U g U J o R M y g r 6 h g k S g v G x + + g y f G m W I w 1 i a E h r P 1 d 8 T G Y m U m k a B 6 Y y I H q t l L x f / 8 / q p D q + 9 j I k k 1 S D o Y l G Y c q x j n O e A h 0 w C 1 X x q C K G S m V s x H R N J q D Z p 5 S G 4 y y + v k s 5 F 3 b 2 s N + 4 a t W a j i K O M j t E J O k c u u k J N d I t a q I 0 o e k T P 6 B W 9 W U / W i / V u f S x a S 1 Y x c 4 T + w P r 8 A Y 8 d k Z A = < / l a t e x i t > ¬c 11 Expert1 < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p 5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p 5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " i C 8 P q t V p g H d z u z Z o 4 j I j T 3 y p  Q k 8 = " > A A A B 8 3 i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x a J 4 K o k U 9 V j w 4 r G C / Y A 2 l M 1 2 0 i 7 d b M L u R i i h f 8 O L B 0 W 8 + m e 8 + W / c t D l o 6 4 O B x 3 s z z M w L E s G 1 c d 1 v Z 2 1 9 Y 3 N r u 7 R T 3 t 3 b P z i s H B 2 3 d Z w q h i 0 W i 1 h 1 A 6 p R c I k t w 4 3 A b q K Q R o H A T j C 5 y / 3 O E y r N Y / l o p g n 6 E R 1 J H n J G j Z X 6 F 4 S w Q e b N y h a D S t W t u X O Q V e I V p A o F m o P K V 3 8 Y s z R C a Z i g W v c 8 N z F + R p X h T O C s 3 E 8 1 J p R N 6 A h 7 l k o a o f a z + c 0 z c m 6 V I Q l j Z U s a M l d / T 2 Q 0 0 n o a B b Y z o m a s l 7 1 c / M / r p S a 8 9 T M u k 9 S g Z I t F Y S q I i U k e A B l y h c y I q S W U K W 5 v J W x M F W X G x p S H 4 C 2 / v E r a V z X v u l Z / q F c b 9 S K O E p z C G V y C B z f Q g H t o Q g s Y J P A M v 6 B f V j v q i S T F D o 0 4 Y n s h U Q B Z w I 6 m m k O v V Q C i U M O D + H k p v A f H k E q l o h 7 P U 3 B j 8 l I s I h R o o 0 U 2 P U z j A c C R p g G u e v N a g a B 3 X C a z h x 4 l b g l a a A S 7 c D + G g w T m s U g N O V E q b 7 r p N r P i d S M c p j V B p m C l N A J G U H f U E F i U H 4 + v 3 2 G T 4 0 y x F E i T Q m N 5 + r v i Z z E S k 3 j 0 H T G R I / V s l e I / 3 n 9 T E f X f s 5 E m m k Q d L E o y j j W C S 6 C w E M m g W o + N Y R Q y c y t m I 6 J J F S b u I o Q 3 O W X V v 6 B f V j t q C i R F N o 0 4 p H s B U Q B Z w L a m m k O v V g C C Q M O 3 W B 6 m / v d R 5 C K R e J B z 2 L w Q j I W b M Q o 0 U b y 7 e o Z x g M B Y 0 z 9 1 L 3 M K g a + X X P q z h x 4 l b g F q a E C L d / + G g w j m o Q g N O V E q b 7 r x N p L i d S M c s g q g 0 R B T O i U j K F v q C A h K C + d 3 5 7 h U 6 M M 8 S i S p o T G c / X 3 R E p C p W Z h Y D p D o i d q 2 c v F / 7 x + o k c 3 X s p E n G g Q d L F o l H C s I 5 w H g Y d M A t V 8 Z g i h k p l b M Z 0 Q S a g 2 c e U h u M s v r 5 L O R d 2 9 q j f u G 7 V m o 4 i j j I 7 R C T p H L r p G T X S H W q i N K H p C z + g V v V m Z 9 W K 9 W x + L 1 p J V z B y h P 7 A + f w D p + J G 8 < / l a t e x i t > ¬c 13 < l a t e x i t s h a 1 _ b a s e 6 4 = " o B m N O H T J C 9 P 5 / g X K M H f  J P U N g 2 o Y = " > A A A B 9 H i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b F U 9 n V o h 4 L X j x W s B / Q L i W b Z t v Q b L I m 2 U J Z + j u 8 e F D E q z / G m / / G b L s H b X 0 w 8 H h v h p l 5 Q c y Z N q 7 7 7 R T W 1 j c 2 t 4 r b p Z 3 d v f 2 D 8 u F R S 8 t E E d o k k k v V C b C m n A n a N M x w 2 o k V x V H A a T s Y 3 2 V + e 0 K V Z l I 8 m m l M / Q g P B Q s Z w c Z K / j l C i P T T q 1 n J o l + u u F V 3 D r R K v J x U I E e j X / 7 q D S R J I i o M 4 V j r r u f G x k + x M o x w O i v 1 E k 1 j T M Z 4 S L u W C h x R 7 a f z o 2 f o z C o D F E p l S x g 0 V 3 9 P p D j S e h o F t j P C Z q S X v U z 8 z + s m J r z 1 U y b i x F B B F o v C h C M j U Z Y A G j B F i e F T S z B R z N 6 K y A g r T I z N K Q v B W 3 5 5 l b Q u q 9 5 1 t f Z Q q 9 R 8 M j + 7 j a V X E q K X R o z G P Z D 4 g C z g R 0 N N M c + o k E E g U c e s H 0 N v d 7 j y A V i 8 W D n i X g R W Q s W M g o 0 U b y 7 e o F x n g o Y I y p n z X n F Q P f r j l 1 Z w G 8 T t y C 1 F C B t m 9 / D U c x T S M Q m n K i 1 M B 1 E u 1 l R G p G O c w r w 1 R B Q u i U j G F g q C A R K C 9 b 3 D 7 H 5 0 Y Z 4 T C W p o T G C / X 3 R E Y i p W Z R Y D o j o i d q 1 c v F / 7 x B q s M b L 2 M i S T U I u l w U p h z r G O d B 4 B G T Q D W f G U K o Z O Z W T C d E E q p N X H k I 7 u r L 6 6 R 7 V X e b 9 c Z 9 o 9 Z q F H G U 0 S k 6 Q 5 f I R d e o h e 5 Q G 3 U Q R U / o G b 2 i N 2 t u v V j v 1 s e y t W Q V M y f o D 6 z P H 9 J 2 k a 4 = < / l a t e x i t > ¬c 6 < l a t e x i t s h a 1 _ b a s e 6 4 = " q p z l F E X Q 6 g V a M a E Y X Z F v 2 h l T y 6 0 = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v i q i R S H 8 u C G 5 c V 7 A P a E C b T 2 3 b o Z B J m J k I I 9 V f c u F D E r R / i z r 9 x 0 m a h r Q c u H M 6 5 l 3 v v C W L O l H a c b 6 u 0 t r 6 x u V X e r u z s 7 u 0 f 2 I d H H R U l k k K b R j y S v Y A o 4 E x A W z P N o R d L I G H A o R t M b 3 O / + w h S s U g 8 6 D Q G L y R j w U a M E m 0 k 3 6 6 e Y Y w H A s a Y + p l 7 O a s Y + H b N q T t z 4 F X i F q S G C r R 8 + 2 s w j G g S g t C U E 6 X 6 r h N r L y N S M 8 p h V h k k C m J C p 2 Q M f U M F C U F 5 2 f z 4 G T 4 1 y h C P I m l K a D x X f 0 9 k J F Q q D Q P T G R I 9 U c t e L v 7 n 9 R M 9 u v E y J u J E g 6 C L R a O E Y x 3 h P A k 8 Z B K o 5 q k h h E p m b s V 0 Q i S h 2 u S V h + A u v 7 x K O h d 1 9 6 r e u G / U m o 0 i j j I 6 R i f o H L n o G j X R H W q h N q I o R c / o F b 1 Z T 9 a L 9 W 5 9 L F p L V j F T R X 9 g f f 4 A R Q S R 6 A = = < / l a t e x i t > N I v k o 5 n F 1 B d 4 J F n I C D Z W 8 i 8 R I o P U q 8 9 L F o N y x a 2 6 C 6 B 1 4 u W k A j m a g / J X f x i R R F B p C M d a 9 z w 3 N n 6 K l W G E 0 3 m p n 2 g a Y z L B I 9 q z V G J B t Z 8 u j p 6 j C 6 s M U R g p W 9 K g h f p 7 I s V C 6 5 k I b K f A Z q x X v U z 8 z + s l J r z 1 U y b j x F B J l o v C h C M T o S w B N G S K E s N n l m C i m L 0 V k T F W m B i b U x a C t / r y O m l f V 7 1 6 t f Z Q q z R q e R I i 5 Y u A G E O o 1 O d o i 1 o 6 g P 9 2 l A = " > A A A B 9 H i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b F U 9 m V Y j 0 W v H i s Y D + g X U o 2 z b a h S X Z N s o W y 9 H d 4 8 a C I V 3 + M N / + N 2 X Y P 2 v p g 4 P H e D D P z g p g z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t o 4 S R W i L R D x S 3 Q B r y p m k L c M M p 9 1 Y U S w C T j v B 5 C 7 z O 1 O q N I v k o 5 n F 1 B d 4 J F n I C D Z W 8 i 8 R I o P U q 8 9 L F o N y x a 2 6 C 6 B 1 4 u W k A j m a g / J X f x i R R F B p C M d a 9 z w 3 N n 6 K l W G E 0 3 m p n 2 g a Y z L B I 9 q z V G J B t Z 8 u j p 6 j C 6 s M U R g p W 9 K g h f p 7 I s V C 6 5 k I b K f A Z q x X v U z 8 z + s l J r z 1 U y b j x F B J l o v C h C M T o S w B N G S K E s N n l m C i m L 0 V k T F W m B i b U x a C t / r y O m l f V 7 2 b a u 2 h V m n U 8 j i K c A b n c A U e 1 K E B 9 9 C E F h B v 6 B f V j t q C i R F N o 0 4 p H s B U Q B Z w L a m m k O v V g C C Q M O 3 W B 6 m / v d R 5 C K R e J B z 2 L w Q j I W b M Q o 0 U b y 7 e o Z x g M B Y 0 z 9 1 L 3 M K g a + X X P q z h x 4 l b g F q a E C L d / + G g w j m o Q g N O V E q b 7 r x N p L i d S M c s g q g 0 R B T O i U j K F v q C A h K C + d 3 5 7 h U 6 M M 8 S i S p o T G c / X 3 R E p C p W Z h Y D p D o i d q 2 c v F / 7 x + o k c 3 X s p E n G g Q d L F o l H C s I 5 w H g Y d M A t V 8 Z g i h k p l b M Z 0 Q S a g 2 c e U h u M s v r 5 L O R d 2 9 q j f u G 7 V m o 4 i j j I 7 R C T p H L r p G T X S H W q i N K H p C z + g V v V m Z 9 W K 9 W x + L 1 p J V z B y h P 7 A + f w D p + J G 8 < / l a t e x i t > ¬c 13 < l a t e x i t s h a 1 _ b a s e 6 4 = " S D a G 4 I O X / k g 2 t P Y k r c 5 Z V 5 0 5 e g E = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 c K 9 g P a U D b b T b t 0 s w m 7 E 6 G E / g g v H h T x 6 u / x 5 r 9 x 0 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T u 9 z v P H F t R K w e c Z p w P 6 I j J U L B K F q p w w a Z 5 8 4 q g 2 r N r b t z k F X i F a Q G B Z q D 6 l d / G L M 0 4 g q Z p M b 0 P D d B P 6 M a B Z N 8 V u m n h i e U T e i I 9 y x V N O L G z + b n z s i Z V Y Y k j L U t h W S u / p 7 I a G T M N A p s Z 0 R x b J a 9 X P z P 6 6 U Y 3 v i Z U E m K X L H F o j C V B G O S / 0 6 G Q n O G c m o J Z V r Y W w k b U 0 0 Z 2 o T y E L z l l 1 d J + 6 L u X d U v H y 5 r j d s i j j K c w C m c g w f X 0 I B 7 a E I L G E z g G V 7 h z U m c F + f d + V i 0 l p x i 5 h j + w P n 8 A V i W j u 4 = < / l a t e x i t > c 10 < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 e 9 x 4 N a Y 2 V a 6 U X G D O B x 7 J s G Q i l Y = " > A A A B 9 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b F U 0 k k q M e C F 4 8 V b C 2 0 o W y 2 m 3 b p 7 i b u b g o l 9 H d 4 8 a C I V 3 + M N / + N m z Y H b X 0 w 8 H h v h p l 5 Y c K Z N q 7 7 7 Z T W 1 j c 2 t 8 r b l Z 3 d v f 2 D 6 u F R W 8 e p I r R F Y h 6 r T o g 1 5 U z S l m G G 0 0 6 i K B Y h p 4 / h + D b 3 H y d U a R b L B z N N a C D w U L K I E W y s F J w j h E g / 8 2 c V i 3 6 1 5 t b d O d A q 8 Q p S g w L N f v W r N 4 h J K q g 0 h G O t u 5 6 b m C D D y j D C 6 a z S S z V N M B n j I e 1 a K r G g O s j m R 8 / Q m V U G K I q V L W n Q X P 0 9 k W G h 9 V S E t l N g M 9 L L X i 7 + 5 3 V T E 9 0 E G Z N J a q g k i 0 V R y p G J U Z 4 A G j B F i e F T S z B R z N 6 K y A g r T I z N K Q / B W 3 5 5 l b Q v 6 9 5 V 3 b / 3 a w 2 / i K M M J 3 A K F + D B N T T g D p r Q A g J P 8 A y v 8 O Z M n B f n 3 f l Y t J a c Y u Y Y / s D 5 / A H w T 4 + T < / l a t e x i t > c 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " n a c D 9 X q M y A e y 8 7 Q t 7 Y j G i c 1 O v U E = " > A A A B + 3 i c b V D L S s N A F J 3 U V 6 2 v W J d u B o v i q i R S H 8 u C G 5 c V 7 A P a E C b T 2 3 b o Z B J m J m I J + R U 3 L h R x 6 4 + 4 8 2 + c t F l o 6 4 E L h 3 P u 5 d 5 7 g p g z p R 3 n 2 y q t r W 9 s b p W 3 K z u 7 e / s H 9 m G 1 o 6 J E U m j T i E e y F x A F n A l o a 6 Y 5 9 G I J J A w 4 d I P p b e 5 3 H 0 We perform the same analysis for CUB-200 to formulate a global explanation of a class by combining the different local FOLs per sample using conjunctive normal form (CNF) as in E-Lens. Figure 5 depicts such global explanations for the species "Bay-breasted warbler" by three distinct experts. E q F o k H P Y v B C 8 l Y s B G j R B v J t 6 t n G O O B g D G m f n q Z V Q x 8 u + b U n T n w K n E L U k M F W r 7 9 N R h G N A l B a M q J U n 3 X i b W X E q k Z 5 Z B V B o m C m N A p G U P f U E F C U F 4 6 v z 3 D p 0 Y Z 4 l E k T Q m N 5 + r v i Z S E S s 3 C w H S G R E / U s p e L / 3 n 9 R I 9 u v J S J O N E g 6 G L R K O F Y R z g P A g + Z B K r 5 z B B C J T O 3 Y j o h k l B t 4 s p D c J d f X i W d i 7 p 7 V W / c N 2 r N R h F H G R 2 j E 3 S O X H S N m u g O t V A b U f S E n t E r e r M y 6 8 V 6 t z 4 W r S W r m D l C f 2 B 9 / g D Q 7 Z G t < / l a t e x i t > Here we capture the heterogeneity in class-level explanations since each FOL corresponding to an expert has an amalgamation of distinct and shared concepts. For instance leg_color_grey is unique to expert4, but belly_pattern_solid and back_pattern_multicolored are unique to experts 1 and 2, respectively.

4.2.3. PERFORMANCE OF EACH EXPERT AND THE RESIDUAL

Figure 6 displays the proportional accuracy of MoIE and the residual for the CUB-200 and Awa2 datasets. Proportional accuracy of each model (experts and/or residuals) is defined as the accuracy of that model times its coverage. Recall that the model's coverage is the empirical mean of the samples selected by the selector. The dashed line in each of the plots indicates the performance of the blackbox, f 0 , denoting the upper bound of our MoIE models. As shown in Figure 6a , the VITderived expert and residual cumulatively achieve an accuracy of approximately 91% for the CUB-200 dataset in iteration 1, where the residual's contribution (black bar) is greater than the expert1's (blue bar). Later iterations reveal that the MoIE's performances cumulatively increase, and the residual worsens. In the final iteration, the MoIE contributes most to performance as they carve out all the interpretable segments from the Blackbox. We observe that the final uninterpretable samples that are routed towards the final residual are "harder" samples for the model to predict, resulting in a lower residual accuracy. Tracing these samples back to f 0 , we observe that the original blackbox has a similar low performance for predicting these samples. We have shown some of these "hard" images side by side with the ones explained by the experts for CUB-200 in Appendix A.8.8. Also, Table 7 verfies this claim, Per our investigation and as it can be in the images, "hard" examples tend to be blurry or distorted images or ones with birds not fully centered or hard to tell from the background. As shown in the coverage plot, this experiment reinforces Figure 1 , where the flow through the experts gradually becomes thicker compared to the narrower flow of the residual with every iteration. We observe similar trends for the other architectures and datasets. For medical imaging (HAM and MIMIC-CXR) datasets and ResNet-derived experts of vision datasets, refer to figures 12 and 11 in the Appendix A.8.4 and A.8.2, respectively. 

4.3. APPLICATION IN REMOVAL OF SHORTCUTS

To conduct this experiment, we create the Waterbirds dataset as in Sagawa et al. (2019) by spuriously correlating birds' images from the Caltech-UCSD Birds-200-2011 dataset to the images of land and water background from the Places dataset. Specifically, we use forest and bamboo as the spurious land concepts for landbirds; ocean and lakes as the spurious water concepts for waterbirds. Thus, we have 112 visual concepts (108 concepts from CUB-200 and four new concepts for the background). We aim to identify each bird as a Waterbird or a Landbird. We use ResNet50 (He et al. (2016) ) as the Blackbox f 0 . The Blackbox quickly latches on the spurious background to classify the birds. As a result, the black box's accuracy varies for land-based subsets versus aquatic subsets of the bird species, as shown in Figure 7a . The Waterbird on water is more accurate than on land ((96% vs. 67 % in the orange bar in the Figure 7a ). We train t to retrieve the concepts from the biased Blackbox. Next, we train our MoIE to explain the bird species using FOL. As per Figure 7c (middle row), the FOL of a waterbird misclassified as a landbird captures the spurious concept forest. Assuming the background concepts as metadata, we aim to reduce the background bias from the representation of the Blackbox using metadata normalization (MDN) Lu et al. (2021) . Therefore, we leverage the MDN layers between two successive layers of the convolutional backbone to fine-tune the biased Blackbox. Next, we train t, using the embedding of the robust Blackbox. Figures 7d-e compare the auroc and accuracy scores of the spurious background concepts from the embedding of the biased and robust Blackboxes, respectively. The validation auroc (accuracy) of all the spurious concepts retrieved from the robust Blackbox fell well short of the predefined threshold 0.7 (70 %) compared to the biased Blackbox. Finally, we re-train the MoIE distilling from the new robust Blackbox. Figure 7b illustrates that the accuracies of Waterbird on water vs. Waterbird on land become pretty similar (91% -88 %). The FOL from the robust Blackbox does not include any background concepts (Figure 7c , bottom row). Refer to the Figure 9 in Appendix A.6 for the flow diagram of this experiment.

5. DISCUSSION & CONCLUSIONS

This paper proposes a novel method to iteratively extract a mixture of interpretable models from a trained Blackbox by utilizing the flexibility of the Blackbox to bridge the gap between post hoc explanation and the construction of interpretable models. The comprehensive experiments on a various datasets demonstrate that 1) our model maintains a similar performance compared to the Black box while making it explainable, 2) the local explanations by our model account for the diversity of explanations of the various samples in a dataset, 3) the coverage of the residuals gradually declines over the iterations. Finally, we employ our method to eliminate the shortcuts in images effectively. We identify the concepts from the representation of the blackbox based on correlation, not causation. So, it is still unclear if the blackbox uses those concepts for prediction. A possible future direction can be the discovering only those concepts causally, used by the blackbox for prediction. In the future we also aim to apply our method to other modalities such as text or video.



r / 4 Y b / 4 b t 2 0 O 2 v p g 4 P H e D D P z / I Q z p W 3 7 2 y p s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T j o p T S W i b x D y W P R 8 r y p m g b c 0 0 p 7 1 E U h z 5 n H b 9 8 e 3 c 7 0 6 o

K b 9 b Y e r H e r Y 9 F a 8 H K Z 4 7 h D 6 z P H y y O k b g = < / l a t e x i t > h 3 Residual (! $ ) < l a t e x i t s h a 1 _ b a s e 6 4 = " V L P n 9 G c W 8 b l 3 Z 1 v p c Z D 9 M L 5 D w O I = " > A A A B 9 H i c b V B N S 8 N A E J 3 U r 1 q / o h 6 9 L B b B U 0 m k q M e i F 4 8 V 7 A c 0 s W y 2 m 2 b p Z h N 3 N 4 U S + j u 8 e F D E q z / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I O V M a c f 5 t k p r 6 x u b W + X t y s 7 u 3

a S Y x c w x / Y H + A C g C k b U = < / l a t e x i t > h 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " + g D 4 Z J H y j B 5 t 4 k p t H P Z 9 y V / 4 y 2 o = " > A A A B 8 X i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 s S R S 1 G P B i 8 c K 9 g P b W D b b a b t 0 s w m 7 G 6 G E / g s v H h T x 6 r / x 5 r 9 x 0 + a g r Q

e H O 2 8 O O / O x 7 x 1 x c l n j u A P n M 8 f M c C P R Q = = < / l a t e x i t > 1 ⇡ 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 F e 4 l 0 U W G p y f 3 5 h v J k N m / 7 a I 9 Q M = " > A A A B 7 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e C F 4 8 V 7 A e 0 s W y 2 m 3 b t Z j f s b o Q Q + h + 8 e F D E q / / H m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 I O Z M G 9 f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f C o o 2

a S 0 4 x c w x / 4 H z + A K m y j n 8 = < / l a t e x i t > ⇡ 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " n 3 F 6 D u l c y o u w e k X r m 8 K N E o m U L O c = " > A A A B 7 X i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 k t p X o s e P F Y w X 5 A u 5 Z s m m 1 j s 8 m S Z I W y 9 D 9 4 8 a C I V / + P N / + N 2 X Y P 2 v p g 4 P H e D D P z g p g z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T j p aJ I r R N J J e q F 2 B N O R O 0 b Z j h t B c r i q O A 0 2 4 w v c n 8 7 h N V m k l x b 2 Y x 9 S M 8 F i x k B B s r d Q Y x e 6 i V h u W K W 3 U X Q O v E y 0 k F c r S G 5 a / B S J I k o s I Q j r X u e 2 5 s / B Q r w w i n 8 9 I g 0 T T G Z I r H t G + p w B H V f r q 4 d o 4 u r D J C o V S 2 h E E L 9 f d E i i O t Z 1 F g O y N s J n r V y 8 T / v H 5 i w m s / Z S J O D B V k u S h M O D I S Z a + j E V O U G D 6 z B B P F 7 K 2 I T L D C x N i A s h C 8 1 Z f X S a d W 9 R r V + l 2 9 0 q z n c R T h D M 7 h E j y 4 g i b c Q g v a Q O A R n u E V 3 h z p v D j v z s e y t e D k M 6 f w B 8 7 n D 6 s 3 j o A = < / l a t e x i t > ⇡ 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " o M s 7 u B K E k R 2 2 V b o X 4 o j 4 s y m T L C c = " > A A A B 8 X i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B i 2 W 3 F P V Y 8 O K x g v 3 A d i 3 Z N N u G Z r M h y Q p l 6 b / w 4 k E Rr / 4 b b / 4 b s + 0 e t P X B w O O 9 G W b m B Z I z b V z 3 2 y m s r W 9 s b h W 3 S z u 7 e / s H 5 c O j t o 4 T R W i L x D x W 3 Q B r y p m g L c M M p 1 2 p K I 4 C T j v B 5 C b z O 0 9 U a R a L e z O V 1 I / w S L C Q E W y s 9 O C h C 9 S X 7 L F W G p Q r b t W d A 6 0 S L y c V y N E c l L / 6 w 5 g k E R W G c K x 1 z 3 O l 8 V O s D C O c z k r 9 R F O J y Q S P a M 9 S g S O q / X R + 8 Q y d W W W I w l j Z E g b N 1 d 8 T K Y 6 0 n k a B 7 Y y w G e t l L x P / 8 3 q J C a / 9 l A m Z G C r I Y l G Y c G R i l L 2 P h k x R Y v j U E k w U s 7 c i M s Y K E 2 N D y k L w l l 9 e J e 1 a 1 b u s 1 u / q l U Y 9 j 6 M I J 3 A K 5 + D B F T T g F p r Q A g I C n u E V 3 h z t v D j v z s e i t e D k M 8 f w B 8 7 n D z N F j 0 Y = < / l a t e x i t > 1 ⇡ 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " h 4 p k r b C + h 2 T K p e C P 6 u U x h g / L p o M = " > A A A B 8 X i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B i 2 V X i 3 o s e P F Y w X 5 g u 5 Z s m m 1 D s 9 m Q Z I W y 9 F 9 4 8 a C I V / + N N / + N 2 X Y P 2 v p g 4 P H e D D P z A s m Z N q 7 7 7 R R W V t f W N 4 q b p a 3 t n d 2 9 8 v 5 B S 8 e J I r R J Y h 6 r T o A 1 5 U z Q p m G G 0 4 5 U F E c B p + 1 g f J P 5 7 S e q N I v F v Z l I 6 k d 4 K F j I C D Z W e v D Q G e p J 9 n h R 6 p c r b t W d A S 0 T L y c V y N H o l 7 9 6 g 5 g k E R W G c K x 1 1 3 O l 8 V O s D C O c T k u 9 R F O J y R g P a d d S g S O q / X R 2 8 R S d W G W A w l j Z E g b N 1 N 8 T K Y 6 0 n k S B 7 Y y w G e l F L x P / 8 7 q

X u e W 5 s / B Q r w w i n s 1 I / 0 T T G Z I J H t G e p w B H V f j q / d o b O r D J E o V S 2 h E F z 9 f d E i i O t p 1 F g O y N s x n r Z y 8 T / v F 5 i w m s / Z S J O D B V k s S h M O D I S Z a + j I V O U G D 6 1 B B P F 7 K 2 I j L H C x N i A s h C 8 5 Z d X S f u i 6 t W r t b t a p V H L 4 y j C C Z z C O X h w B Q 2 4 h S a 0 g M A j P M M r v D n S e X H e n Y 9 F a 8 H J Z 4 7 h D 5 z P H 6 y 8 j o E = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " A z N u c 9 0E e R k 7 4 7 R 0 X 2 a o e Y E 0 G + E = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a W D b b S b p 0 s w m 7 G 6 G U / g Q v H h T x 6 i / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A X X x n W / n c L K 6 t r 6 R n G z t L W 9 s 7 t X 3 j 9 o 6 i R T D B s s E Y l q B 1 S j 4 B I b h h u B 7 V Q h j Q O B r W B 4 M / V b T 6 g 0 T + S D G a X o x z S S P O S M G i v d R 4 9 e r 1 x x q + 4 M Z J l 4 O a l A j n q v / N X t J y y L U R o m q N Y d z 0 2 N P 6 b K c C Z w U u p m G l P K h j T C j q W S xq j 9 8 e z U C T m x S p + E i b I l D Z m p v y f G N N Z 6 F A e 2 M 6 Z m o B e 9 q f i f 1 8 l M e O W P u U w z g 5 L N F 4 W Z I C Y h 0 7 9 J n y t k R o w s o U x x e y t h A 6 o o M z a d k g 3 B W 3 x 5 m T T P q t 5 F 9 f z u v F K 7 z u M o w h E c w y l 4 c A k 1 u I U 6 N I B B B M / w C m + O c F 6 c d + d j 3 l p w 8 p l D + A P n 8 w f z J I 2 X < / l a t e x i t > g 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " A z N u c 9 0 E e R k 7 4 7 R 0 X 2 a o e Y E 0 G + E = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a W D b b S b p 0 s w m 7

4 explains the training procedure of our model and extraction of FOL to explain samples locally. Figure 8 in Appendix A.4 shows the architecture of our model at inference.4 EXPERIMENTSTo validate our method, we utilize four datasets -1) bird species classification usingCUB-200 Wah  et al. (2011)  dataset, 4) animals species classification using Animals with Attributes2 (Awa2) datasetXian et al. (2018), 3) skin lesion classification using HAM10000 datasetTschandl et al. (2018), 4) Effusion and Cardiomegaly classification from radiology images using MIMIC-CXR dataset Johnson et al.. First, we demonstrate that MoIE does not hinder the performance of the Blackbox. Second, we conduct qualitative analysis to illustrate the effectiveness of local explanations. Third, we validate that the cumulative coverage of the residuals decreases over successive iterations. Finally, we leverage the Waterbirds datasetSagawa et al. (2019) to eliminate shortcut bias in vision datasets.Training configurations As a Blackbox, we use ResNet-101He et al. (2016) and Vision Transformer (VIT)Wang et al. (2021) for the CUB-200 and Awa2 datasets. For HAM10000 and MIMIC-CXR, we utilize InceptionSzegedy et al. (2015) andDensenet-121 Huang et al. (2017)  as the same.

Figure 2: MoIE does not hurt the performance of the Blackbox. From left to right, we compare the (a) accuracy and (b) auroc metrics respectively. For the MoIE (ours), we specify the empirical coverage as cov in the figure w.r.t each datasets.

Figure 3: From left to right, for each expert, bar plot showing the importance of a concept for predicting a skin lesion as Malignant and Benign respectively. For example, atypicalpigmentnetwork concept plays a crucial role for predicting "Malignant" skin lesion by expert5 as is present in all the local explanations for expert5.

r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " l p a 0 O c j n J T A q 2 0 a P b 0 o T e H k 1 l Y U = " > A A A B 6 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M d S L x 4 r 2 A 9 o Q 9 l s t + 3 S 3 U 3 Y n Q g l 9 C 9 4 8 a C I V / + Q N / + N S Z u D t j 4 Y e L w 3 w 8 y 8 I J L C o u t + O 4 W N z a 3 t n e J u a W / / 4 P

e 7 8 l x d J 8 6 z i n l e q d 9 V S 7 S q v o w B H c A y n 4 M I F 1 O A G 6 t A A A h k 8 w y u 8 W U / W i / V u f c x G l 6 w 8 c w B / Y H 3 + A H 5 5 l T Q = < / l a t e x i t > Irregular Streaks < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 e c 3 0 Q

6 F / S v l P a Y Z R 9 t d x p b g z Z + 8 S K q n B e + s U L w p 5 k q X s z r S 5 I A c k R P i k X N S I t e k T C q E k 0 f y T F 7 J m / P k v D j v z s c 0 m n J m M / v k D 5 z P H + S o m U k = < / l a t e x i t > ¬Regression Structures < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1

5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 C a I s P c B 2 h A T B A N K H

1 s e 8 t W Q V M 4 f o D 6 z P H 9 B 9 k n s = < / l a t e x i t > ¬RegularDG < l a t e x i t s h a 1 _ b a s e 6 4 = " K Q k E M j m v k e M G / Q B a x 5 R / h Q 4 g G l 4 = " > A A A B 8 3 i c b V D L S g N B E O y N r x h f U Y 9 e B o P g K e x K U I 9 B B f U W w T w g W c L s Z D Y Z M j u 7 z E M I S 3 7 D i w d F v P o z 3 v w b Z 5 M 9 a

4 h l P w 4 A L q c A c N a A K B B J 7 h F d 4 c 4 7 w 4 7 8 7 H v L X g 5 D O H 8 A f O 5 w 9 Y F p E 9 < / l a t e x i t > IrregularDG < l a t e x i t s h a 1 _ b a s e 6 4 = " c 6 o K C S b A h

r j 1 N 9 z 5 N 0 7 a L L T 1 w M D h n H u 4 d 4 6 f c K a 0 4 3 x b h a X l l d W 1 4 n p p Y 3 N r e 8 f e 3 W u q O

S e q r 8 T Y x I p N Y p 8 M x k R P V D z X i b + 5 3V S H V x 4 Y y a S V I O g s 0 V B y r G O c V Y G 7 j M J V P O R I Y R K Z m 7 F d E A k o d p U l p X g z n 9 5 k T R P K + 5 Z p X p b L d c u 8 z q K 6 B A d o R P k o n N U Q 9 e o j h q I o k f 0 j F 7 R m / V k v V j v 1 s d s t G D l m X 3 0 B 9 b n D 5 0 k l d o = < / l a t e x i t > ¬IrregularStreaks < l a t e x i t s h a 1 _ b a s e 6 4 = " T Q W G U Q P s F Z y J W b Y U T R S x q u F 9 W j w = " > A A A B + X i c b V B N S 8 N A E N 3 U r x q / o h 6 9 L B b B U 0 l E 1 G N R Q Y 9 V7 A e 0 o W y 2 k 3 b p Z h N 2 N 4 U S + k + 8 e F D E q / / E m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 I O F M a d f 9 t k o r q 2 v r G + V N e 2 t 7 Z 3 f P 2 T 9 o q

1 s e 8 t W Q V M 4 f o D 6 z P H 9 B 9 k n s = < / l a t e x i t > ¬RegularDG < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " z m 5

> < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " m H u d Y G 7 O p 4 4 2 R N 1 Z E h 9 1 1 o i Y B m 0 = " > A A A B 9 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 c K 9 g P a W D b b S b t 0 s w m 7 E 6 W U / g 8 v H h T x 6 n / x 5 r 9 x 0 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f T m F l d W 1 9 o 7 h Z 2 t r e 2 d 0 r 7 x 8 0 T Z x q D g 0 e y 1 i 3

l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5 E T E 3 h W W b / 6 X b q I 8 f T k w o = " > AA A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a _ b a s e = " d a F H Q E T E h W W b / X b q I f T k w o = " > A A A B H i c b Z D L S g M x F I b P F u t t p L N E i u C o z I u r O g h u X F e x F q F k M p k N M k M S U Y o Q / C j Y o i b n D X O d b O m U J b P w j / P J w T J J x p r f T m F p e W V r b h e t j c t p + d Z w

S c Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D yA h P F K y I D L D C x N g b U f w l d e F M T q n d W P b x K V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r y H p a W n B m P f v w B + f j B B Y k = < /l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " s O 3 N x + C X T l d + P 8 G i 4 I I / N H 8 n D G E = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r x q / q h 6 9 L B b B U 0 l E 1 G P R i 8 c K 9 g P a U D b b a b t 0 s w m 7 G 6 G E / g g v H h T x 6 u / x 5 r 9 x 0 + a g r Q 8 G H u / N M D M v T A T X x v O + n d L a + s b m V n n b 3 d n d 2 z + o H B 6 1 d J w q h

A H z u c P F I S P a Q = = < / l a t e x i t > smelly < l a t e x i t s h a 1 _ b a s e 6 4 = " q 9 Y U E / 0 P / I y g b 8 m k X U f l u G I F Z o E = " > A A A B + n i c b V B N S 8 N A E N 3 U r 1 q / U j 1 6 W S y C p 5 J I U Y 9 F L x 4 r 2 A 9 o Q 9 l s J + n S z S b s b p Q S + 1 O 8 e F D E q 7 / E m / / G T Z u D t j 4 Y e L w 3 w 8 w 8 P + F M a c f 5 t k p r 6 x u b W + X t y s 7 u 3 v 6 B X T 3 s q

C b U T R I 3 p G r + j N e r J e r H f r Y 9 F a s o q Z I / Q H 1 u c P C A W S j Q = = < / l a t e x i t >

l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X hz U T l 8 v M p 5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t

Figure 4: Illustration of flexibility of FOL explanations by MoIE. We report the FOL for different experts and the baseline to identify (a) Malignant lesion and (b) Horse. For example, in Figure (a), the FOL by the baseline composes BWV, RegressionStructures and IrregularStreaks concepts to distinguish all the samples of the skin lesion "Malignant". However in the same Figure (a), expert6 identifies the instances of "Malignant" in the first two rows with AtypicalPigmentNetwork and RegularDG as the identifying concept. Similarly expert4 classifies the samples of "Malignant" in the third row of Figure (a) using other BWV, IrregularStreaks, RegularDG and IrregularDG.

5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j kv d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A Cr 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W q M j d / C r E e 8 9 V D u 4 k U Q a 2 1 4 9 i 7 I = " > A A A B 8 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b F U 0 k k q M e C F 4 8 V 7 I e 0 o W y 2 m 3 b p 7 i b s b o Q S + i u 8 e F D E q z / H m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 M O F M G 9 f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f C o r e N U E d o i M Y 9 V N 8 S a c i Z p y z D D a T d R F I u Q 0 0 4 4 u c 3 9 z h N V m s X y w U w T G g g 8 k i x i B B s r P Z 4 j

s e P F Y w X 5 I u 5 R s m m 1 D k + y S Z I W y 9 F d 4 8 a C I V 3 + O N / + N 2 X Y P 2 v p g 4 P H e D D P z g p g z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T j o 4 S R W i b R D x S v Q B r y p m k b c M M p 7 1 Y U S w C T r v B 9 D b z u 0

r / D m p M 6 L 8 + 5 8 L F r X n G L m B P 7 A + f w B l L S P Z g = = < / l a t e x i t > c 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " d a F H 8 Q 5E T E 3 h W W b / 6 X b q I 8 f T k w o = " > A A A B 8 H i c b Z D L S g M x F I b P 1 F u t t 6 p L N 8 E i u C o z I u r O g h u X F e x F 2 q F k M p k 2 N M k M S U Y o Q 5 / C j Y o i b n 0 D X 8 O d b 2 O m 7 U J b P w j 8 / P 8 5 5 J w T J J x p 4 7 r f T m F p e W V 1 r b h e 2 t j c 2 t 4 p 7 + 4 1 d Z w q Q h s k 5 r F q B 1 h T z i R t G G Y 4 b S e K Y h F w 2 g q G V 3 n e u q d K s 1 j e m l F C f Y H 7 k k W M Y G O t u y 7 H M i x Z e u W K W 3 U n o E X h z U T l 8 v M p5 7 n e K 3 9 1 w 5 i k g k p D O N a 6 4 7 m J 8 T O s D C O c j k v d V N M E k y H u 0 4 6 V E g u q / W w y 8 B g d W S d E U a z s k w Z N 3 N 8 d G R Z a j 0 R g K w U 2 A z 2 f 5 e Z / W S c 1 0 Y W f M Z m k h k o y / S h K O T I x y r d H I V O U G D 6 y A h P F 7 K y I D L D C x N g b 5 U f w 5 l d e F M 2 T q n d W P b 1 x K 7 V T m F K E A z i E Y / D g H G p w D X V o A A E B D / A C r 4 5 y H p 0 3 5 3 1 a W n B m P f v w B + f j B 8 B Y k 3 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " J U j j k 0 p J 1 O l 4 T l 0 w z W 7 z 2 H T E z C A = " > A A A B + 3 i c b V B N S 8 N A E N 3 U r 1 q / Y j 1 6 W S y K p 5 J I U I 8 F L x 4 r 2 F p o Q 9 h s J + 3 S z S b s b s Q S + l e 8 e F D E q 3 / E m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 M O V M a c f 5 t i p r 6 x u b W 9 X t 2 s 7 u 3

0 n 3 o u l e N r 0 7 r 9 H y y j i q 6 B i d o HP k o i v U Q r e o j T q I o i f 0 j F 7 R m z W z X q x 3 6 2 P R W r H K m S P 0 B 9 b n D + u B k b 0 = < / l a t e x i t >¬c 14 < l a t e x i t s h a 1 _ b a s e 6 4 = " i v R 5 K C 5 O 5 B i n d A 5 s Z Q 8 N f d Q l / p w = " > A A A B + 3 i c b V B N S 8 N A E N 3 U r 1 q / Y j 1 6 W S y K p 5 J o U Y 8 F L x 4 r 2 A 9 o Q 9 h s p + 3 S z S b s b s Q S 8 l e 8 e F D E q 3 / E m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 I O Z M a c f 5 t k p r 6 x u b W + X t y s 7 u 3

r e R x F O I F T u A A P b q A O 9 9 C A J h B 4 g m d 4 h T d n 4 r w 4 7 8 7 H o r X g 5 D P H 8 A f O 5 w / u x o + S < / l a t e x i t > c 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 dF V d x 9 k Z q Y x H f j 8 v F C s s 6 5 B 5 n U = " > A A A B + 3 i c b V B N S 8 N A E N 3 U r 1 q / Y j 1 6 W S y K p 5 J I q R 4 L X j x W s B / Q h r D Z T t q l m 0 3 Y 3 Y g l 9 K9 4 8 a C I V / + I N / + N m z Y H b X 0 w 8 H h v h p l 5 Q c K Z 0 o 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g

¬c 15 < l a t e x i t s h a 1 _ b a s e 6 4 = " T b 3 H B 1 p e Y C 9 Y s B D 2 5 8 t j v 0 d f /B o = " > A A A B 9 H i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b F U 9 m V U j 0 W v H i s Y D + g X U o2 z b a h S X Z N s o W y 9 H d 4 8 a C I V 3 + M N / + N 2 X Y P 2 v p g 4 P H e D D P z g p g z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t o 4 S R W i L R D x S 3 Q B r y p m k L c M M p 9 1 Y U S w C T j v B 5 C 7 z O 1 O q

x F O I N z u A I P b q A B 9 9 C E F h B 4 g m d 4 h T d n 6 r w 4 7 8 7 H s r X g 5 D O n 8 A f O 5 w 8 O d 4 + m < / l a t e x i t > c 16 < l a t e x i t s h a 1 _ b a s e 6 4 = " j r Y

4 g m d 4 h T d n 6 r w 4 7 8 7 H s r X g 5 D O n 8 A f O 5 w 8 Q A I + n < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " i v R 5 K C 5 O 5 B i n d A 5 s Z Q 8 N f d Q l / p w = " > A A A B + 3 i c b V B N S 8 N A E N 3 U r 1 q / Y j 1 6 W S y K p 5 J o U Y 8 F L x 4 r 2 A 9 o Q 9 h s p + 3 S z S b s b s Q S 8 l e 8 e F D E q 3 / E m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 I O Z M a c f 5 t k p r 6 x u b W + X t y s 7 u 3

Figure 5: Illustration of class-level or global explanations. We report the global explanation of "Bay breasted warbler" using the VIT-based blackbox by combining (using DNF) the local explanations. (a) Explanations by expert1 (top row), expert2 (middle row) and expert4 (bottom row). Shared concepts across all the experts are highlighted in blue. We highlight the same for experts and 2 in orange. (b) Symbols used in Figure 3a and their corresponding true concept labels.

Figure 6: Coverage and performance of each VIT-derived expert and residual for (a) CUB-200 (b) Awa2

t e x i t s h a 1 _ b a s e 6 4 = " J G C f p K W P v O O d T + I G S q 0 B n e W z q / E = " > A A A D l H i c h V J d b 9 M w F H U T P k b 5 a k H i h R e L C s T T 1 E y D w c R D P z S x h w k V t V 0 n N V X l O D e N V c f O b A d W o v 4 h f g 5 v / B u c N I j R s X E l y 1 f n n n O u 7 e s g 5 U y b d v t n z X F v 3 b 5 z d + d e / f 6 D h 4 8 e N 5 p P T r X M FI U x l V y q s 4 B o 4 E z A 2 D D D 4 S x V Q J K A w y R Y 9 o v 6 5 A s o z a Q Y m V U K s 4 Q s B I s Y J c Z C 8 2 b t + y v f w I X J P y q Z i d C o z M S H e E I M q B 5 T 4 R r 7 f r 1 i D B S E j B Y y f A 3 j 6 C L l R J T O 6 0 O 8 w S 4 T W R R V a I 9 x f g J i Y e J u I D M z i m F I E u j q Y y A F M d Y p o Z C 3 d 9 t l e K m x o L U O y 2 b Y P z / P S L i 9 C V h s m w 9 j q W z 7 U U x E 4 Y z / a 1 1 5 X W s 9 Z N 9 g m B D O 3 / h z f / 6 e i Z s d / + h i k s I A F I 2 Z W J y w J d y s K z V 9 J b + K z Q R j Z m B d n z d a v 8 n 4 a u J V S Q t V M Z g 3 f v i h p F k C w l B O t J 5 6 7 d T M c q I M o 9 w a + p k G e 4 Y l W c D U p s K O Q M / y 8 l O t 8 U u L h D i S y i 5 h c I l e V u Q k 0 X q V B J a Z E B P r 7 V o B / q s 2 z U z 0 b p Y z k W Y G B N 0 0 i j K O j c T F D 8 U h U 0 A N X 9 m E U M X s W T G N i S L UT l I X j + B t X / l q c r q 3 6 7 3 d 3 f + 8 3 + r 0 q u f Y Q c / R C / Q a e e g A d d A x G q A x o k 7 T O X A 6 T t d 9 5 n 5 w + + 7 R h u r U K s 1 T 9 F e 4 n 3 4 B G U U k O A = = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " P C e 3 V M e t u S k O o C J m 9 X F 5 6 + q N r 0Q = " > A A A C p X i c b V H J b t s w E K X U L X U 3 t z 3 2 Q s T o c j K k I G g L n 4 I E S X N o A S e N 7 Q C W Y V D U y C J C U Q Q 5 a m M I + r N + R W 7 9 m 1 K y C q R O B y D x + O b N w p l Y S 2 E x C H 5 7 / r 3 7 D x 4 + 2 n n c e / L 0 2 f M X / Z e v p r Y o D Y c J L 2 R h L m N m Q Q o F E x Q o 4 V I b Y H k s Y R Z f H T X + 2 Q 8 w V h T q A t c a F j l b K Z E K z t B R y / 6 v d x H C N V Z f T F G q B E 2 J 2 Y j O G I I 5 F C a p a R T 1 O s X Y Q C J 4 E 0 V H 9 C t T y b b g + F p L p t q 8 9 Y h u u F s 6 k a Y d O R N q 9 T 1 j G s 6 b m p D 8 d G / r F J n V j E M V D I P W Q o 2 O d C m T L q 6 9 4 7 Q 6 K Q x Y r G u 6 7 A / + i u l d E H Z g Q D o b L / s 3 U V L w M g e F X D J r 5 2 G g c V E x g 4 J L q H t R a c H 1 c M V W M H d Q s R z s o m q n X N O 3 j k l o W h h 3 F N K W v R 1 Rs d z a d R 4 7 Z c 4 w s 9 u + h v y f b 1 5 i + n l R C a V L B M U 3 h d J S U i x o s z K a C A M c 5 d o B x o 1 w v V K e M c O 4 2 5 L t u S G E 2 1 + + C 6 Z 7 w / D j c P 9 s f 3 B w 2 I 1 j h 7 w h u + Q D C c k n c k B O y Z h M C P d 2 v V P v z D v 3 3 / v f / A t / u p H 6 X h f z m v x j / v I P / O L R R w = = < / l a t e x i t >

Figure 7: Applying MoIE to eliminate shortcuts. (a) Performance of the biased Blackbox. (b) Performance of final MoIE, trained using the supervision of the robust Blackbox after removing the shortcuts using MDN. (c) Examples of samples and their explanations by the biased (top-row) and robust Blackboxes(bottom-row). (d) auroc and (e) accuracy metrics of the biased Blackbox (when the shortcuts are present) and the robust Blackbox (when the shortcuts are removed using MDN).

