DON'T THROW YOUR OLD POLICIES AWAY: KNOWLEDGE-BASED POLICY RECYCLING PROTECTS AGAINST ADVERSARIAL ATTACKS

Abstract

Recent work has shown that Deep Reinforcement Learning (DRL) is vulnerable to adversarial attacks, in which minor perturbations of input signals cause agents to behave inappropriately and unexpectedly. Humans, on the other hand, appear robust to these particular sorts of input variations. We posit that this part of robustness stems from accumulated knowledge about the world. In this work, we propose to leverage prior knowledge to defend against adversarial attacks in RL settings using a framework we call Knowledge-based Policy Recycling (KPR). Different from previous defense methods such as adversarial training and robust learning, KPR incorporates domain knowledge over a set of auxiliary tasks policies and learns relations among them from interactions with the environment via a Graph Neural Network (GNN). KPR can use any relevant policy as an auxiliary policy and, importantly, does not assume access or information regarding the adversarial attack. Empirically, KPR results in policies that are more robust to various adversarial attacks in Atari games and a simulated Robot Foodcourt environment.

1. INTRODUCTION

Despite significant performance breakthroughs in recent years (Mnih et al., 2015; Silver et al., 2016; Berner et al., 2019, e.g.,) , Deep Reinforcement Learning (DRL) policies can be brittle. Specifically, recent works have shown that DRL policies are vulnerable to adversarial attacks -adversarially manipulated inputs (e.g., images) of small magnitude can cause RL agents to take incorrect actions (Ilahi et al., 2022; Chen et al., 2019; Behzadan & Munir, 2017; Oikarinen et al., 2021; Lee et al., 2021; Chan et al., 2020; Bai et al., 2018) . To counter such attacks, recent work has proposed a range of defense strategies including adversarial training (Oikarinen et al., 2021; Behzadan & Munir, 2018; Han et al., 2018) , robust learning (Mandlekar et al., 2017; Smirnova et al., 2019; Pan et al., 2019) , defensive distillation (Rusu et al., 2016) , and adversarial detection (Gallego et al., 2019a; Havens et al., 2018; Gallego et al., 2019a) . While these defense methods can be effective, each has its limitations; adversarial training and adversarial detection require specific knowledge about the attacker. Robust learning adds noise during agent training, which can degrade performance (Tsipras et al., 2019; Yang et al., 2020) . Defensive distillation is typically unable to protect against diverse adversarial attacks (Carlini & Wagner, 2016; Soll et al., 2019) . In this work, we explore an alternative defense strategy that exploits existing knowledge encoded in auxiliary task policies and known relationships between the policies. The key intuition underlying our approach is that existing task policies encode learnt low-level knowledge regarding the environment (e.g., possible observations, dynamics), whilst high-level specifications can provide guidance for transfer or generalization. Our approach is to leverage known and learnt relations between different policies as structural priors for an ensemble of policies; our hypothesis is that while a single task policy can be attacked, perturbing inputs such that multiple policies are negatively affected in a consistent manner is more difficult. Our framework, which we call Knowledge-based Policy Fusion (KPR), is partially inspired by the use of domain knowledge to address vulnerabilities to adversarial attacks in supervised learning (Melacci et al., 2021; Gürel et al., 2021; Zhang et al., 2022) . In these works, domain knowledge is encoded as logical formulae over predicted labels and a set of features. A soft satisfiability score between  V c B Q N s k m v w o s X G B p y f X T C L u I U = " > A A A B / n i c b Z D L S s N A F I Z P v N Z 6 i 4 o r N 4 N F c F W S W t R l 0 Y 3 L C r 1 B G 8 J k M m 2 H T i Z h Z i K U U P B V 3 L h Q x K 3 P 4 c 6 3 c d p G 0 N Y f B j 7 + c w 7 n z B 8 k n C n t O F / W y u r a + s Z m Y a u 4 v b O 7 t 2 8 f H L Z U n E p C m y T m s e w E W F H O B G 1 q p j n t J J L i K O C 0 H Y x u p / X 2 A 5 W K x a K h x w n 1 I j w Q r M 8 I 1 s b y 7 e O G 7 / Y 4 F i F q + J U f u P D t k l N 2 Z k L L 4 O Z Q g l x 1 3 / 7 s h T F J I y o 0 4 V i p r u s k 2 s u w 1 I x w O i n 2 U k U T T E Z 4 Q L s G B Y 6 o 8 r L Z + R N 0 Z p w Q 9 W N p n t B o 5 v 6 e y H C k 1 D g K T G e E 9 V A t 1 q b m f 7 V u q v v X X s Z E k m o q y H x R P + V I x 2 i a B Q q Z p E T z s Q F M J D O 3 I j L E E h N t E i u a E N z F L y 9 D q 1 J 2 L 8 v V + 2 q p d p P H U Y A T O I V z c O E K a n A H d W g C g Q y e 4 A V e r U f h V D J s s F r H q B F S j 4 B K b h h u B n U Q h j Q K B 7 W B 8 N / f b T 6 g 0 j 2 X D T B L 0 I z q U P O S M G i s 9 N v q V f r H k l t 0 F y D r x M l K C D P V + 8 a s 3 i F k a o T R M U K 2 7 n p s Y f 0 q V 4 U z g r N B L N S a U j e k Q u 5 Z K G q H 2 p 4 t T Z + T C K g M S x s q W N G S h / p 6 Y 0 k j r S R T Y z o i a k V 7 1 5 u J / X j c 1 4 Y 0 / 5 T J J D U q 2 X B S m g p i Y z P 8 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 C j Y E b / X l d d K q l L 2 r c v W h W q r d Z n H k 4 Q z O V c C s W b K F D y T q I 5 j Q L J 2 8 H 4 b u a 3 n 7 g 2 I l Y N n C T c j + h Q i V A w i l Z 6 b P S 9 f r n i V t 0 5 y C r x c l K B H P V + + a s 3 i F k a c Y V M U m O 6 n p u g n 1 G N g k k + L f V S w x P K x n T I u 5 Y q G n H j Z / N T p + T M K g M S x t q W Q j J X f 0 9 k N D J m E g W 2 M 6 I 4 M s v e T P z P 6 6 Y Y 3 v i Z U E m K X L H F o j C V B G M y + 5 s M h O Y M 5 c Q S y r S w t x I 2 o p o y t O m U b A j e 8 s u r p H V R 9 a 6 q l w + X l d p t H k c R T u A U z s G D a 6 j B P d S h C Q y G 8 A y v 8 O Z I 5 8 V 5 d z 4 W r Q U n n z m G P 3 A + f w D X t 4 2 F < / l a t e x i t > T 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 A 3 E h a 3 D P o W J H e t Z z A I r X W n 7 1 1 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K / Y I 2 l M 1 2 0 i 7 d b M L u R i i h P 8 G L B 0 W 8 + o u 8 + W / c t j l o 6 4 O B x 3 s z z M w L E s G 1 c d 1 v p 7 C 2 v r G 5 V d w u 7 e z u 7 R + U D 4 9 a O k 4 V w y a L R a w 6 A d U o u M S m 4 U Z g J 1 F I o 0 B g O x j f z f z 2 E y r N Y 9 k w k w T 9 i A 4 l D z m j x k q P j b 7 s l y t u 1 Z 2 D r B I v J x X I U e + X v 3 q D m K U R S s M E 1 b r r u Y n x M 6 o M Z w K n p V 6 q M a F s T I f Y t V T S C L W f z U + d k j O r D E g Y K 1 v S k L n 6 e y K j k d a T K L C d E T U j v e z N x P + 8 b m r C G z / j M k k N S r Z Y F K a C m J j M / i Y D r p A Z M b G E M s X t r Y S N q K L M 2 H R K N g R v + e V V 0 r q o e l f V y 4 f L S u 0 2 j 6 M I J 3 A K 5 + D B N d T g H u r Q B A Z D e I Z X e H O E 8 + K 8 O x + L 1 o K T z x z D H z i f P z Q 6 j c I = < / l a t e x i t > T n < l a t e x i t s h a 1 _ b a s e 6 4 = " N h T W a 8 R n 2 s 9 M 1 u 1 W m D T 1 F r z T b x E = " > A A A B 6 n i c b V D L S g N B E O y N r x h f U Y 9 e B o P g K e x K U I 9 B L 1 6 E i H l B s o T Z S W 8 y Z H Z 2 m Z k V Q s g n e P G g i F e / y J t / 4 y T Z g y Y W N B R V 3 X R 3 B Y n g 2 r j u t 5 N b W 9 / Y 3 M p v F 3 Z 2 9 / Y P i o d H T R 2 n i m G D x S J W 7 Y B q F F x i w 3 A j s J 0 o p F E g s B W M b m d + 6 w m V 5 r G s m 3 G C f k Q H k o e c U W O l x 3 r v v l c s u W V 3 D r J K v I y U I E O t V / z q 9 m O W R i g N E 1 T r j u c m x p 9 Q Z T g T O C 1 0 U 4 0 J Z S M 6 w I 6 l k k a o / c n 8 1 C k 5 s 0 q f h L G y J Q 2 Z q 7 8 n J j T S e h w F t j O i Z q i X v Z n 4 n 9 d J T X j t T 7 h M U o O S L R a F q S A m J r O / S Z 8 r Z E a M L a F M c X s r Y U O q K D M 2 n Y I N w V t + e Z U i M s / P M 5 V k J M Z i l g L 5 Q O T a N Q S 8 = " > A A A B 6 n i c b V C 7 S g N B F L 0 b X z G + o o K N z W A Q r M K u i F q G 2 F g m m B c k S 5 i d z C Z D Z m e W m V k h L P k E G w t F b G 3 9 C 7 / A z s Z v c f I o N P H A h c M 5 9 3 L v P U H M m T a u + + V k V l b X 1 j e y m 7 m t 7 Z 3 d v f z + Q U P L R B F a J 5 J L 1 Q q w p p w J W j f M c N q K F c V R w G k z G N 5 M / O Y 9 V Z p J U T O j m P o R 7 g s W M o K N l e 5 q X a + b L 7 h F d w q 0 T L w 5 K Z S O q t / s v f x R 6 e Y / O z 1 J k o g K Q z j W u u 2 5 s f F T r A w j n I 5 z n U T T G J M h 7 t O 2 p Q J H V P v p 9 N Q x O r V K D 4 V S 2 R I G T d X f E y m O t B i M s / P M 5 V k J M Z i l g L 5 Q O T a N Q S 8 = " > A A A B 6 n i c b V C 7 S g N B F L 0 b X z G + o o K N z W A Q r M K u i F q G 2 F g m m B c k S 5 i d z C Z D Z m e W m V k h L P k E G w t F b G 3 9 C 7 / A z s Z v c f I o N P H A h c M 5 9 3 L v P U H M m T a u + + V k V l b X 1 j e y m 7 m t 7 Z 3 d v f z + Q U P L R B F a J 5 J L 1 Q q w p p w J W j f M c N q K F c V R w G k z G N 5 M / O Y 9 V Z p J U T O j m P o R 7 g s W M o K N l e 5 q X a + b L 7 h F d w q 0 T L w 5 K Z S O q t / s v f x R 6 e Y / O z 1 J k o g K Q z j W u u 2 5 s f F T r A w j n I 5 z n U T T G J M h 7 t O 2 p Q J H V P v p 9 N Q x O r V K D 4 V S 2 R I G T d X f E y m O t B 5 F g e 2 M s B n o R W 8 i / u e 1 E x N e + y k T c W K o I L N F Y c K R k W j y N + o x R Y n h I 0 s w U c z e i s g A K 0 y M T S d n Q / A W X 1 4 m j f O i d 1 m 8 q N o 0 y j B D F o 7 h B M 7 A g y s o w S 1 U o A 4 E + v A A T / D s c O f R e X F e Z 6 0 Z Z z 5 z C H / g v P 0 A t l + R I g = = < / l a t e x i t > T1 < l a t e x i t s h a 1 _ b a s e 6 4 = " G Q 2 o 0 K c J q v L i N u k U P P F Q n O m 8 / r M = " > A A A B 6 n i c b V C 7 S g N B F L 3 j M 8 Z X V L C x G Q y C V d g N o p Y h N p Y J 5 g X J E m Y n s 8 m Q 2 d l l Z l Y I S z 7 B x k I R W 1 v / w i + w s / F b n D w K T T x w 4 X D O v d x 7 j x 8 L r o 3 j f K G V 1 b X 1 j c 3 M V n Z 7 Z 3 d v P 3 d w 2 N B R o i i r 0 0 h E q u U T z Q S X r G 6 4 E a w V K 0 Z C X 7 C m P 7 y Z + M 1 7 p j S P Z M 2 M Y u a F p C 9 5 w C k x V r q r d Y v d X N 4 p O F P g Z e L O S b 5 0 X P 3 m 7 + W P S j f 3 2 e l F N A m Z N F Q Q r d u u E x s v J c p w K t g 4 2 0 k 0 i w k d k j 5 r W y p J y L S X T k 8 d 4 z O r 9 H A Q K V v S 4 K n 6 e y I l o d a j 0 L e d I T E D v e h N x P + 8 d m K C a y / l M k 4 M k 3 S 2 K E g E N h G e / I 1 7 X D F q x M g S Q h W 3 t 2 I 6 I I p Q Y 9 P J 2 h D c x Z e X S a N Y c C 8 L F 1 W b R h l m y M A J n M I 5 u H A F J b i F C t S B Q h 8 e 4 A m e k U C P 6 A W 9 z l p X 0 H z m C P 4 A v f 0 A t + O R I w = = < / l a t e x i t > T2 < l a t e x i t s h a 1 _ b a s e 6 4 = " G Q 2 o 0 K c J q v L i N u k U P P F Q n O m 8 / r M = " > A A A B 6 n i c b V C 7 S g N B F L 3 j M 8 Z X V L C x G Q y C V d g N o p Y h N p Y J 5 g X J E m Y n s 8 m Q 2 d l l Z l Y I S z 7 B x k I R W 1 v / w i + w s / F b n D w K T T x w 4 X D O v d x 7 j x 8 L r o 3 j f K G V 1 b X 1 j c 3 M V n Z 7 Z 3 d v P 3 d w 2 N B R o i i r 0 0 h E q u U T z Q S X r G 6 4 E a w V K 0 Z C X 7 C m P 7 y Z + M 1 7 p j S P Z M 2 M Y u a F p C 9 5 w C k x V r q r d Y v d X N 4 p O F P g Z e L O S b 5 0 X P 3 m 7 + W P S j f 3 2 e l F N A m Z N F Q Q r d u u E x s v J c p w K t g 4 2 0 k 0 i w k d k j 5 r W y p J y L S X T k 8 d 4 z O r 9 H A Q K V v S 4 K n 6 e y I l o d a j 0 L e d I T E D v e h N x P + 8 d m K C a y / l M k 4 M k 3 S 2 K E g E N h G e / I 1 7 X D F q x M g S Q h W 3 t 2 I 6 I I p Q Y 9 P J 2 h D c x Z e X S a N Y c C 8 L F 1 W b R h l m y M A J n M I 5 u H A F J b i F C t S B Q h 8 e 4 A m e k U C P 6 A W 9 z l p X 0 H z m C P 4 A v f 0 A t + O R I w = = < / l a t e x i t > T2 < l a t e x i t s h a 1 _ b a s e 6 4 = " B k V U r d O 1 K P H F 2 Y 9 0 P C J I a 3 l A p E Q = " > A A A B 7 H i c b Z D L S s N A F I Z P 6 q 3 G W 9 W l m 8 E i u C q J S H U j F t 2 4 r G D a Q h v K Z D J p h 0 4 m Y W Y i l N B n c O N C E V e C r + L e j f g 2 T i 8 L b f 1 h 4 O P / z 2 H O O U H K m d K O 8 2 0 V l p Z X V t e K 6 / b G 5 t b 2 T m l 3 r 6 G S T B L q k Y Q n s h V g R T k T 1 N N M c 9 p K J c V x w G k z G F y P 8 + Y 9 l Y o l 4 k 4 P U + r H u C d Y x A j W x v I 6 H I u w W y o 7 F W c i t A j u D M q X H / Z F + v Z l 1 7 u l z 0 6 Y k C y m Q h O O l W q 7 T q r 9 H E v N C K c j u 5 M p m m I y w D 3 a N i h w T J W f T 4 Y d o S P j h C h K p H l C o 4 n 7 u y P H s V L D O D C V M d Z 9 N Z + N z f + y d q a j c z 9 n I s 0 0 F W T 6 U Z R x p B M 0 3 h y F T F K i + d A A J p K Z W R H p Y 4 m J N v e x z R H c + Z U X o X F S c a u V 0 1 u n X L u C q Y p w A I d w D C 6 c Q Q 1 u o A 4 e E G D w A E / w b A n r 0 X q x X q e l B W v W s w 9 / Z L 3 / A C r 2 k e 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B k V U r d O 1 K P H F 2 Y 9 0 P C J I a 3 l A p E Q = " > A A A B 7 H i c b Z D L S s N A F I Z P 6 q 3 G W 9 W l m 8 E i u C q J S H U j F t 2 4 r G D a Q h v K Z D J p h 0 4 m Y W Y i l N B n c O N C E V e C r + L e j f g 2 T i 8 L b f 1 h 4 O P / z 2 H O O U H K m d K O 8 2 0 V l p Z X V t e K 6 / b G 5 t b 2 T m l 3 r 6 G S T B L q k Y Q n s h V g R T k T 1 N N M c 9 p K J c V x w G k z G F y P 8 + Y 9 l Y o l 4 k 4 P U + r H u C d Y x A j W x v I 6 H I u w W y o 7 F W c i t A j u D M q X H / Z F + v Z l 1 7 u l z 0 6 Y k C y m Q h O O l W q 7 T q r 9 H E v N C K c j u 5 M p m m I y w D 3 a N i h w T J W f T 4 Y d o S P j h C h K p H l C o 4 n 7 u y P H s V L D O D C V M d Z 9 N Z + N z f + y d q a j c z 9 n I s 0 0 F W T 6 U Z R x p B M 0 3 h y F T F K i + d A A J p K Z W R H p Y 4 m J N v e x z R H c + Z U X o X F S c a u V 0 1 u n X L u C q Y p w A I d w D C 6 c Q Q 1 u o A 4 e E G D w A E / w b A n r 0 X q x X q e l B W v W s w 9 / Z L 3 / A C r 2 k e 0 = < / l a t e x i t > ^< l a t e x i t s h a 1 _ b a s e 6 4 = " X l z X + g U 0 4 5 8 n 4 n u T T l U 0 j U x p O G 0 = " > A A A B 6 H i c b Z D J S g N B E I Z r 4 h b H L e r R S 2 M Q P I U Z E f U i B r 1 4 T M A s k A y h p 1 O T t O l Z 6 O 4 R Y s g T e P G g i F d 9 G O 9 e x L e x s x w 0 8 Y e G j / + v o q v K T w R X 2 n G + r c z C 4 t L y S n b V X l v f 2 N z K b e 9 U V Z x K h h U W i 1 j W f a p Q 8 A g r m m u B 9 U Q i D X 2 B N b 9 3 N c p r d y g V j 6 M b 3 U / Q C 2 k n 4 g F n V B u r f N / K 5 Z 2 C M x a Z B 3 c K + Y s P + z x 5 / 7 J L r d x n s x 2 z N M R I M 0 G V a r h O o r 0 B l Z o z g U O 7 m S p M K O v R D j Y M R j R E 5 Q 3 G g w 7 J g X H a J I i l e Z E m Y / d 3 x 4 C G S v V D 3 1 S G V H f V b D Y y / 8 s a q Q 7 O v A G P k l R j x C Y f B a k g O i a j r U m b S 2 R a 9 A 1 Q J r m Z l b A u l Z R p c x v b H M G d X X k e q k c F 9 6 R w X H b y x U u Y K A t 7 s A + H 4 M I p F O E a S l A B B g g P 8 A T P 1 q 3 1 a L 1 Y r 5 P S j D X t 2 Y U / s t 5 + A E v F k E Q = < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " X l z X + g U 0 4 5 8 n 4 n u T T l U 0 j U x p O G 0 = " > A A A B 6 H i c b Z D J S g N B E I Z r 4 h b H L e r R S 2 M Q P I U Z E f U i B r 1 4 T M A s k A y h p 1 O T t O l Z 6 O 4 R Y s g T e P G g i F d 9 G O 9 e x L e x s x w 0 8 Y e G j / + v o q v K T w R X 2 n G + r c z C 4 t L y S n b V X l v f 2 N z K b e 9 U V Z x K h h U W i 1 j W f a p Q 8 A g r m m u B 9 U Q i D X 2 B N b 9 3 N c p r d y g V j 6 M b 3 U / Q C 2 k n 4 g F n V B u r f N / K 5 Z 2 C M x a Z B 3 c K + Y s P + z x 5 / 7 J L r d x n s x 2 z N M R I M 0 G V a r h O o r 0 B l Z o z g U O 7 m S p M K O v R D j Y M R j R E 5 Q 3 G g w 7 J g X H a J I i l e Z E m Y / d 3 x 4 C G S v V D 3 1 S G V H f V b D Y y / 8 s a q Q 7 O v A G P k l R j x C Y f B a k g O i a j r U m b S 2 R a 9 A 1 Q J r m Z l b A u l Z R p c x v b H M G d X X k e q k c F 9 6 R w X H b y x U u Y K A t 7 s A + H 4 M I p F O E a S l A B B g g P 8 A T P 1 q 3 1 a L 1 Y r 5 P S j D X t 2 Y U / s t 5 + A E v F k E Q = < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " a c K w p a q 3 H w 7 N n d / b N g D r t a q y 5 p c = " > A A A B 6 n i c b V D L S g N B E O y N r 5 j 4 i H r 0 M h g F T 2 F X R D 0 G v X i M m B c k S 5 i d z C Z j Z m e X m d l A W P I J X j w o 4 t V v 8 A f 8 A 2 9 + i J 6 d T X L Q x I K G o q q b 7 i 4 v 4 k x p 2 / 6 0 M k v L K 6 t r 2 f V c f m N z a 7 u w s 1 t X Y S w J r Z G Q h 7 L p Y U U 5 E 7 S m m e a 0 G U m K A 4 / T h j e 4 S v 3 G k E r F Q l H V o 4 i 6 A e 4 J 5 j O C t Z F u q 5 2 7 T q F o l + w J 0 C J x Z q R Y P v x 6 e x / m v y u d w k e 7 G 5 I 4 o E I T j p V q O X a k 3 Q R L z Q i n 4 1 w 7 V j T C Z I B 7 t G W o w A F V b j I 5 d Y y O j N J F f i h N C Y 0 m 6 u + J B A d K j Q L P d A Z Y 9 9 W 8 l 4 r / e a 1 Y + x d u w k Q U a y r I d J E f c 6 R D l P 6 N u k x S o v n I E E w k M 7 c i 0 s c S E 2 3 S y Z k Q n P m X F 0 n 9 p O S c l U 5 v T B q X M E U W 9 u E A j s G B c y j D N V S g B g R 6 c A + P 8 G R x 6 8 F 6 t l 6 m r R l r N r M H f 2 C 9 / g D v 6 5 I D < / l a t e x i t >

Tj

< l a t e x i t s h a 1 _ b a s e 6 4 = " a c K w p a q 3 H w 7 N n d / b N g D r t a q y 5 p c = "  > A A A B 6 n i c b V D L S g N B E O y N r 5 j 4 i H r 0 M h g F T 2 F X R D 0 G v X i M m B c k S 5 i d z C Z j Z m e X m d l A W P I J X j w o 4 t V v 8 A f 8 A 2 9 + i J 6 d T X L Q x I K G o q q b 7 i 4 v 4 k x p 2 / 6 0 M k v L K 6 t r 2 f V c f m N z a 7 u w s 1 t X Y S w J r Z G Q h 7 L p Y U U 5 E 7 S m m e a 0 G U m K A 4 / T h j e 4 S v 3 G k E r F Q l H V o 4 i 6 A e 4 J 5 j O C t Z F u q 5 2 7 T q F o l + w J 0 C J x Z q R Y P v x 6 e x / m v y u d w k e 7 G 5 I 4 o E I T j p V q O X a k 3 Q R L z Q i n 4 1 w 7 V j T C Z I B 7 t G W o w A F V b j I 5 d Y y O j N J F f i h N C Y 0 m 6 u + J B A d K j Q L P d A Z Y 9 9 W 8 l 4 r / e a 1 Y + x d u w k Q U a y r I d J E f c 6 R D l P 6 N u k x S o v n I E E w k M 7 c i 0 s c S E 2 3 S y Z k Q n P m X F 0 n 9 p O S c l U 5 v T B q X M E U W 9 u E A j s G B c y j D N V S g B g R 6 c A + P 8 G R x 6 8 F 6 t l 6 m r R l r N r M H f 2 C 9 / g D v 6 5 I D < / l a t e x i t > Tj < l a t e x i t s h a 1 _ b a s e 6 4 = " A D g W F w Y G U I B e 9 R / M / D y Z / / R / 3 / Q = " > A A A B 6 n i c b V C 7 S g N B F L 0 b X z G + o o K N z W A Q r M K u i F q G 2 F g m m B c k S 5 i d z C Z D Z m e W m V k h L P k E G w t F b G 3 9 C 7 / A z s Z v c f I o N P H A h c M 5 9 3 L v P U H M m T a u + + V k V l b X 1 j e y m 7 m t 7 Z 3 d v f z + Q U P L R B F a J 5 J L 1 Q q w p p w J W j f M c N q K F c V R w G k z G N 5 M / O Y 9 V Z p J U T O j m P o R 7 g s W M o K N l e 5 q X d H N F 9 y i O w V a J t 6 c F E p H 1 W / 2 X v 6 o d P O f n Z 4 k S U S F I R x r 3 f b c 2 P g p V o Y R T s e 5 T q J p j M k Q 9 2 n b U o E j q v 1 0 e u o Y n V q l h 0 K p b A m D p u r v i R R H W o + i w H Z G 2 A z 0 o j c R / / P a i Q m v / Z S J O D F U k N m i M O H I S D T 5 G / W Y o s T w k S W Y K G Z v R W S A F S b G p p O z I X i L L y + T x n n R u y x e V G 0 a Z Z g h C 8 d w A m f g w R W U 4 B Y q U A c C f X i A J Q = " > A A A B 6 n i c b V C 7 S g N B F L 0 b X z G + o o K N z W A Q r M K u i F q G 2 F g m m B c k S 5 i d z C Z D Z m e W m V k h L P k E G w t F b G 3 9 C 7 / A z s Z v c f I o N P H A h c M 5 9 3 L v P U H M m T a u + + V k V l b X 1 j e y m 7 m t 7 Z 3 d v f z + Q U P L R B F a J 5 J L 1 Q q w p p w J W j f M c N q K F c V R w G k z G N 5 M / O Y 9 V Z p J U T O j m P o R 7 g s W M o K N l e 5 q X d H N F 9 y i O w V a J t 6 c F E p H 1 W / 2 X v 6 o d P O f n Z 4 k S U S F I R x r 3 f b c 2 P g p V o Y R T s e 5 T q J p j M k Q 9 2 n b U o E j q v 1 0 e u o Y n V q l h 0 K p b A m D p u r v i R R H W o + i w H Z G 2 A z 0 o j c R / / P a i Q m v / Z S J O D F U k N m i M O H I S D T 5 G / W Y o s T w k S W Y K G Z v R W S A F S b G p p O z I X i L L y + T x n n R u y x e V G 0 a Z Z g h C 8 d w A m f g w R W U 4 B Y q U A c C f X i A J V C 7 S g N B F L 0 b X z G + o o K N z W A Q r M K u i F q G 2 F g m m B c k S 5 i d z C Z D Z m e W m V k h L P k E G w t F b G 3 9 C 7 / A z s Z v c f I o N P H A h c M 5 9 3 L v P U H M m T a u + + V k V l b X 1 j e y m 7 m t 7 Z 3 d v f z + Q U P L R B F a J 5 J L 1 Q q w p p w J W j f M c N q K F c V R w G k z G N 5 M / O Y 9 V Z p J U T O j m P o R 7 g s W M o K N l e 5 q X d b N F 9 y i O w V a J t 6 c F E p H 1 W / 2 X v 6 o d P O f n Z 4 k S U S F I R x r 3 f b c 2 P g p V o Y R T s e 5 T q J p j M k Q 9 2 n b U o E j q v 1 0 e u o Y n V q l h 0 K p b A m D p u r v i R R H W o + i w H Z G 2 A z 0 o j c R / / P a i Q m v / Z S J O D F U k N m i M O H I S D T 5 G / W Y o s T w k S W Y K G Z v R W S A F S b G p p O z I X i L L y + T x n n R u y x e V G 0 a Z Z g h C 8 d w A m f g w R W U 4 B Y q U A c C f X i A J V C 7 S g N B F L 0 b X z G + o o K N z W A Q r M K u i F q G 2 F g m m B c k S 5 i d z C Z D Z m e W m V k h L P k E G w t F b G 3 9 C 7 / A z s Z v c f I o N P H A h c M 5 9 3 L v P U H M m T a u + + V k V l b X 1 j e y m 7 m t 7 Z 3 d v f z + Q U P L R B F a J 5 J L 1 Q q w p p w J W j f M c N q K F c V R w G k z G N 5 M / O Y 9 V Z p J U T O j m P o R 7 g s W M o K N l e 5 q X d b N F 9 y i O w V a J t 6 c F E p H 1 W / 2 X v 6 o d P O f n Z 4 k S U S F I R x r 3 f b c 2 P g p V o Y R T s e 5 T q J p j M k Q 9 2 n b U o E j q v 1 0 e u o Y n V q l h 0 K p b A m D p u r v i R R H W o + i w H Z G 2 A z 0 o j c R / / P a i Q m v / Z S J O D F U k N m i M O H I S D T 5 G / W Y o s T w k S W Y K G Z v R W S A F S b G p p O z I X i L L y + T x n n R u y x e V G 0 a Z Z g h C 8 d w A m f g w R W U 4 B Y q U A c C f X i A J G W 9 W l m 8 E i u C q J S H U j F t 2 4 r G D a Q h v K Z D J p h 0 4 m Y W Y i l N B n c O N C E V e C r + L e j f g 2 T i 8 L b f 1 h 4 O P / z 2 H O O U H K m d K O 8 2 0 V l p Z X V t e K 6 / b G 5 t b 2 T m l 3 r 6 G S T B L q k Y Q n s h V g R T k T 1 N N M c 9 p K J c V x w G k z G F y P 8 + Y 9 l Y o l 4 k 4 P U + r H u C d Y x A j W x v I 6 H I u w W y o 7 F W c i t A j u D M q X H / Z F + v Z l 1 7 u l z 0 6 Y k C y m Q h O O l W q 7 T q r 9 H E v N C K c j u 5 M p m m I y w D 3 a N i h w T J W f T 4 Y d o S P j h C h K p H l C o 4 n 7 u y P H s V L D O D C V M d Z 9 N Z + N z f + y d q a j c z 9 n I s 0 0 F W T 6 U Z R x p B M 0 3 h y F T F K i + d A A J p K Z W R H p Y 4 m J N v e x z R H c + Z U X o X F S c a u V 0 1 u n X L u C q Y p w A I d w D C 6 c Q Q 1 u o A 4 e E G D w A E / w b A n r 0 X q x X q e l B W v W s w 9 / Z L 3 / A C r 2 k e 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "  B k V U r d O 1 K P H F 2 Y 9 0 P C J I a 3 l A p E Q = " > A A A B 7 H i c b Z D L S s N A F I Z P 6 q 3 G W 9 W l m 8 E i u C q J S H U j F t 2 4 r G D a Q h v K Z D J p h 0 4 m Y W Y i l N B n c O N C E V e C r + L e j f g 2 T i 8 L b f 1 h 4 O P / z 2 H O O U H K m d K O 8 2 0 V l p Z X V t e K 6 / b G 5 t b 2 T m l 3 r 6 G S T B L q k Y Q n s h V g R T k T 1 N N M c 9 p K J c V x w G k z G F y P 8 + Y 9 l Y o l 4 k 4 P U + r H u C d Y x A j W x v I 6 H I u w W y o 7 F W c i t A j u D M q X H / Z F + v Z l 1 7 u l z 0 6 Y k C y m Q h O O l W q 7 T q r 9 H E v N C K c j u 5 M p m m I y w D 3 a N i h w T J W f T 4 Y d o S P j h C h K p H l C o 4 n 7 u y P H s V L D O D C V M d Z 9 N Z + N z f + y d q a j c z 9 n I s 0 0 F W T 6 U Z R x p B M 0 3 h y F T F K i + d A A J p K Z W R H p Y 4 m J N v e x z R H c + Z U X o X F S c a u V 0 1 u n X L u C q Y p w A I d w D C 6 c Q Q 1 u o A 4 e E G D w A E / w b A n r 0 X q x X q e l B W v K T w R X 2 n G + r c z C 4 t L y S n b V X l v f 2 N z K b e 9 U V Z x K h h U W i 1 j W f a p Q 8 A g r m m u B 9 U Q i D X 2 B N b 9 3 N c p r d y g V j 6 M b 3 U / Q C 2 k n 4 g F n V B u r f N / K 5 Z 2 C M x a Z B 3 c K + Y s P + z x 5 / 7 J L r d x n s x 2 z N M R I M 0 G V a r h O o r 0 B l Z o z g U O 7 m S p M K O v R D j Y M R j R E 5 Q 3 G g w 7 J g X H a J I i l e Z E m Y / d 3 x 4 C G S v V D 3 1 S G V H f V b D Y y / 8 s a q Q 7 O v A G P k l R j x C Y f B a k g O i a j r U m b S 2 R a 9 A 1 Q J r m Z l b A u l Z R p c x v b H M G d X X k e q k c F 9 6 R w X H b y x U u Y K A t 7 s A + H 4 M I p F O E a S l A B B g g P 8 A T P 1 q 3 1 a L 1 Y r 5 P S j D X t 2 Y U / s t 5 + A E v F k E Q = < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " X l z X + g U 0 4 5 8 n 4 n u T T l U 0 j U x p O G 0 = " > A A A B 6 H i c b Z D J S g N B E I Z r 4 h b H L e r R S 2 M Q P I U Z E f U i B r 1 4 T M A s k A y h p 1 O T t O l Z 6 O 4 R Y s g T e P G g i F d 9 G O 9 e x L e x s x w 0 8 Y e G j / + v o q v K T w R X 2 n G + r c z C 4 t L y S n b V X l v f 2 N z K b e 9 U V Z x K h h U W i 1 j W f a p Q 8 A g r m m u B 9 U Q i D X 2 B N b 9 3 N c p r d y g V j 6 M b 3 U / Q C 2 k n 4 g F n V B u r f N / K 5 Z 2 C M x a Z B 3 c K + Y s P + z x 5 / 7 J L r d x n s x 2 z N M R I M 0 G V a r h O o r 0 B l Z o z g U O 7 m S p M K O v R D j Y M R j R E 5 Q 3 G g w 7 J g X H a J I i l e Z E m Y / d 3 x 4 C G S v V D 3 1 S G V H f V b D Y y / 8 s a q Q 7 O v A G P k l R j x C Y f B a k g O i a j r U m b S 2 R a 9 A 1 Q J r m Z l b A u l Z R p c x v b H M G d X X k e q k c F 9 6 R w X H b y x U u Y K A t 7 s A + H 4 M I p F O E a S l A B B g g P 8 A T P 1 q 3 1 a L 1 Y r 5 P S j D X t 2 Y U / s t 5 + A E v F k E Q = < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " X l z X + g U 0  v K T w R X 2 n G + r c z C 4 t L y S n b V X l v f 2 N z K b e 9 U V Z x K h h U W i 1 j W f a p Q 8 A g r m m u B 9 U Q i D X 2 B N b 9 3 N c p r d y g V j 6 M b 3 U / Q C 2 k n 4 g F n V B u r f N / K 5 Z 2 C M x a Z B 3 c K + Y s P + z x 5 / 7 J L r d x n s x 2 z N M R I M 0 G V a r h O o r 0 B l Z o z g U O 7 m S p M K O v R D j Y M R j R E 5 Q 3 G g w 7 J g X H a J I i l e Z E m Y / d 3 x 4 C G S v V D 3 1 S G V H f V b D Y y / 8 s a q Q 7 O v A G P k l R j x C Y f B a k g O i a j r U m b S 2 R a 9 A 1 Q J r m Z l b A u l Z R p c x v b H M G d X X k e q k c F 9 6 R w X H b y x U u Y K A t 7 s A + H 4 M I p F O E a S l A B B g g P 8 A T P 1 q 3 1 a L 1 Y r 5 P S j D X t 2 Y U / s t 5 + A E v F k E Q = < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " X l z X + g U 0 4 5 8 n 4 n u T T l U 0 j U x p O G 0 = " > A A A B 6 H i c b Z D J S g N B E I Z r 4 h b H L e r R S 2 M Q P I U Z E f U i B r 1 4 T M A s k A y h p 1 O T t O l Z 6 O 4 R Y s g T e P G g i F d 9 G O 9 e x L e x s x w 0 8 Y e G j / + v o q v K T w R X 2 n G + r c z C 4 t L y S n b V X l v f 2 N z K b e 9 U V Z x K h h U W i 1 j W f a p Q 8 A g r m m u B 9 U Q i D X 2 B N b 9 3 N c p r d y g V j 6 M b 3 U / Q C 2 k n 4 g F n V B u r f N / K 5 Z 2 C M x a Z B 3 c K + Y s P + z x 5 / 7 J L r d x n s x 2 z N M R I M 0 G V a r h O o r 0 B l Z o z g U O 7 m S p M K O v R D j Y M R j R E 5 Q 3 G g w 7 J g X H a J I i l e Z E m Y / d 3 x 4 C G S v V D 3 1 S G V H f V b D Y y / 8 s a q Q 7 O v A G P k l R j x C Y f B a k g O i a j r U m b S 2 R a 9 A 1 Q J r m Z l b A u l Z R p c x v b H M G d X X k e q k c F 9 6 R w X H b y x U u Y K A t 7 s A + H 4 M I p F O E a S l A B B g g P 8 A T P 1 q 3 1 a L 1 Y r 5 P S j D X t 2 Y U / s t 5 + A E v F k E Q = < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " I z D W F j l g  V g n 8 a o r Y M u f g 3 f y C Q 5 Q = " > A A A B 6 n i c b V C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d g V X 2 W I j W W C e U G y h N n J b D J k d m a Z m R X C k k + w s V D E 1 t a / 8 A v V o Y R T k f Z d q J p j M k A 9 2 j L U o E j q v 1 0 c u o I n V i l i 0 K p b A m D J u r v i R R H W g + j w H Z G 2 P T 1 v D c W / / N a i Q m v / Z S J O D F U k O m i M O H I S D T + G 3 W Z o s T w o S W Y K G Z v R a S P F S b G p p O 1 I X j z L y + S + l n B u y y c V 2 w a J Z g i A 0 d w D K f g w R U U 4 R b K U A M C P X i A J 3 h 2 u P P o v D i v 0 9 Y l Z z Z z A H / g v P 0 A v G + R J g = = < / l a t e x i t >

T5

< l a t e x i t s h a 1 _ b a s e 6 4 = " c e F 9 4 f P T I + t J U P 4 L s s q t m o i 6 5 M Q = " > A A A B 6 n i c b V C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d i V o J Y h N p Y J 5 g X J E m Y n s 8 m Q 2 Z l l Z l Y I S z 7 B x k I R W 1 v / w i + w s / F b n D w K T T x w 4 X D O v d x 7 T x B z p o 3 r f j k r q 2 v r G 5 u Z r e z 2 z u 7 e f u 7 g s K F l o g i t E 8 m l a g V Y U 8 4 E r R t m O G 3 F i u I o 4 L Q Z D G 8 m f v O e K s 2 k q J l R T P 0 I 9 w U L G c H G S n e 1 b r G b y 7 s F d w q 0 T L w 5 y Z e O q 9 / s v f x R 6 e Y + O z 1 J k o g K Q z j W u u 2 5 s f F T r A w j n I 6 z n U T T G J M h 7 t O 2 p Q J H V P v p 9 N Q x O r N K D 4 V S 2 R I G T d X f E y m O t B 5 F g e 2 M s B n o R W 8 i / u e 1 E x N e + y k T c W K o I L N F Y c K R k W j y N + o x R Y n h I 0 s w U c z e i s g A K 0 y M T S d r Q / A W X 1 4 m j Y u C d 1 k o V m 0 a Z Z g h A y d w C u f g w R W U 4 B Y q U A c C f X i A J 3 h 2 u P P o v D i v s 9 Y V Z z 5 z B H / g v P 0 A u u u R J Q = = < / l a t e x i t >

T4

< l a t e x i t s h a 1 _ b a s e 6 4 = " c e F 9 4 f P T I + t J U P 4 L s s q t m o i 6 5 ¬ < l a t e x i t s h a 1 _ b a s e 6 4 = " E u u 5 N C K 3 n C 0 P j B 8 5 i y K L H J / f p e s = " > A A A B 6 3 i c b Z D L S g M x F I b P e K 3 j r e r S T b A I r s q M i L o R i 2 5 c V r A X a I e S S T N t a J I Z k o x Q h r 6 C G x e K u B O f x b 0 b 8 W 3 M t F 1 o 6 w + B j / 8 / h 5 x z w o Q z b T z v 2 1 l Y X F p e W S 2 s u e s b m 1 v b x Z 3 d u o 5 T R W i N x D x W z R B r y p m k N c M M p 8 1 E U S x C T h v h 4 D r P G / d U a R b L O z N M a C B w T 7 K I E W x y q y 1 p r 1 M s e W V v L D Q P / h R K l x / u R f L 2 5 V Y 7 x c 9 2 N y a p o N I Q j r V u + V 5 i g g w r w w i n I 7 e d a p p g M s A 9 2 r I o s a A 6 y M a z j t C h d b o o i p V 9 0 q C x + 7 s j w 0 L r o Q h t p c C m r 2 e z 3 P w v a 6 U m O g 8 y J p P U U E k m H 0 U p R y Z G + e K o y x Q l h g 8 t Y K K Y n R W R P l a Y G H s e 1 x 7 B n 1 1 5 H u r H Z f + 0 f H L r l S p X M F E B 9 u E A j s C H M 6 j A D V S h B g T 6 8 A B P 8 O w I 5 9 F 5 c V 4 n p Q v O t G c P / s h 5 / w F r W p F + < / l a t e x i t > ¬ < l a t e x i t s h a 1 _ b a s e 6 4 = " Finally, we combine/fuse these distributions (Section 3.3) to obtain a final action distribution. In summary, KPR uses the specified logical formula graph as prior structure and then learns flexible relations over auxiliary tasks from interaction data. M Q = " > A A A B 6 n i c b V C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d i V o J Y h N p Y J 5 g X J E m Y n s 8 m Q 2 Z l l Z l Y I S z 7 B x k I R W 1 v / w i + w s / F b n D w K T T x w 4 X D O v d x 7 T x B z p o 3 r f j k r q 2 v r G 5 u Z r e z 2 z u 7 e f u 7 g s K F l o g i t E 8 m l a g V Y U 8 4 E r R t m O G 3 F i u I o 4 L Q Z D G 8 m f v O e K s 2 k q J l R T P 0 I 9 w U L G c H G S n e 1 b r G b y 7 s F d w q 0 T L w 5 y Z e O q 9 / s v f x R 6 e Y + O z 1 J k o g K Q z j W u u 2 5 s f F T r A w j n I 6 z n U T T G J M h 7 t O 2 p Q J H V P v p 9 N Q x O r N K D 4 V S 2 R I G T d X f E y m O t B 5 F g e 2 M s B n o R W 8 i / u e 1 E x N e + y k T c W K o I L N F Y c K R k W j y N + o x R Y n h I 0 s w U c z e i s g A K 0 y M T S d r Q / A W X 1 4 m j Y u C d 1 k o V m 0 a Z Z g h A y d w C u f g w R W U 4 B Y q U A c C f X i A J 3 h 2 u P P o v D i v s 9 Y V Z z 5 z B H / g v P 0 A u u u R J Q = = < B k V U r d O 1 K P H F 2 Y 9 0 P C J I a 3 l A p E Q = " > A A A B 7 H i c b Z D L S s N A F I Z P 6 q 3 G W 9 W l m 8 E i u C q J S H U j F t 2 4 r G D a Q h v K Z D J p h 0 4 m Y W Y i l N B n c O N C E V e C r + L e j f g 2 T i 8 L b f 1 h 4 O P / z 2 H O O U H K m d K O 8 2 0 V l p Z X V t e K 6 / b G 5 t b 2 T m l 3 r 6 G S T B L q k Y Q n s h V g R T k T 1 N N M c 9 p K J c V x w G k z G F y P 8 + Y 9 l Y o l 4 k 4 P U + r H u C d Y x A j W x v I 6 H I u w W y o 7 F W c i t A j u D M q X H / Z F + v Z l 1 7 u l z 0 6 Y k C y m Q h O O l W q 7 T q r 9 H E v N C K c j u 5 M p m m I y w D 3 a N i h w T J W f T 4 Y d o S P j h C h K p H l C o 4 n 7 u y P H s V L D O D C V M d Z 9 N Z + N z f + y d q a j c z 9 n I s 0 0 F W T 6 U Z R x p B M 0 3 h y F T F K i + d A A J p K Z W R H p Y 4 m J N v e x z R H c + Z U X o X F S c a u V 0 1 u n X L u C q Y p w A I d w D C 6 c Q Q 1 u o A 4 e E G N U 0 E g T j p X q u U 6 i v Q x L z Q i n 0 0 o / V T T B Z I y H t G d o h A V V X j a L P k X H R h m g M J b m R R r N 1 N 8 b G R Z K T U R g J v O g a t H L x f + 8 X q r D S y 9 j U Z J q G p H 5 o T D l S M c o 7 w E N m K R E 8 4 k h m E h m s i I y w h I T b d q q m B L c x S 8 v k / Z p 3 T 2 v n 9 2 d 1 R p X R R 1 l O I Q j O A E X L q A B t 9 C E F h B 4 h G d 4 h T f m + 2 G s = " > A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s q u F P V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v m + 2 G s = " > A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s q u F P V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v 0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1 I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J + 6 L m X d b q 9 / V K 4 y a P o w g n c A p V 8 O A K G n A H T W g B A Q 7 P 8 A p v z q P z 4 r w 7 H 4 v W g p P P H M M f O J 8 / Z f y P k A = = < / l a t e x i t > f (•) < l a t e x i t s h a 1 _ b a s e 6 4 = " M R P O Q b i v e a e o q B t c R U l f q n m + 2 G s = " > A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s q u F P V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v 0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1 I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J + 6 L m X d b q 9 / V K 4 y a P o w g n c A p V 8 O A K G n A H T W g B A Q 7 P 8 A p v z q P z 4 r w 7 H 4 v W g p P P H M M f O J 8 / Z f y P k A = = < / l a t e x i t > f (•) < l a t e x i t s h a 1 _ b a s e 6 4 = " M R P O Q b i v e a e o q B t c R U l f q n m + 2 G s = " > A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s q u F P V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v 0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1 I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J + 6 L m X d b q 9 / V K the predictions and given logic formulae is added to the objective to encourage the predictions to comply with the logical formulae. Even if part of the sample is corrupted by the adversary, the final output can be corrected by enforcing the domain knowledge rules. KPR extends this line of research to RL settings. Note that this extension is nontrivial; the auxiliary feature detectors used in supervised learning (Gürel et al., 2021; Zhang et al., 2022) do not capture temporal features, and more importantly, the consistency between the predictions and actions is not directly computable since the optimal actions for each state are unknown. We address these issues by using auxiliary task policies and their relationships. In practice, these policies could be the intermediate by-products of curriculum/hierarchical learning or obtained via direct training with sub-goals. By combining these policies with specified and learnt relations, we construct an ensemble of policies on the target task. KPR uses graph neural networks (GNNs) as a backbone, which enables natural incorporation of graph-based domain knowledge while retaining the flexibility to learn from interaction data. The ensemble is then fused in a simple parameter-less manner to obtain a new robust task policy (Figure 1 ). From a practical perspective, KPR has a key advantage: it is both policy and attack agnostic. Specifically, KPR can utilize any type of policy representation (e.g., neural networks, rule-based policies) as either the main task policy or an auxiliary task policy. In addition, KPR doesn't require knowledge of the specific attack or access to the adversarial environment. Our empirical results show that KPR results in more robust policies across multiple attacks compared to baselines in a representative selection of Atari Games (Bellemare et al., 2013) and a high-dimensional Robot Food Court Environment (RoFoCo). To summarize, this paper contributes Knowledge-based Policy Recycling (KPR), which leverages domain knowledge to defend against adversarial attacks in RL. Different from prior defense methods in reinforcement learning, such as adversarial training and robust learning, KPR is able to incorporate domain knowledge as structural prior and then learn flexible relations from interaction data. To the best of our knowledge, this is the first work to demonstrate that domain knowledge in the form of policies can be used to defend against adversarial attacks.



t e x i t s h a 1 _ b a s e 6 4 = " L G H 8h W 9 O G 2 1 g N n G V 7 6 Z R Q U i E 0 r s = " > A A A B / 3 i c b V C 7 T s N A E F y H V w i v A B I N z Y k I K T S R j S K g j K C h D F J e U m x Z 5 / M 5 O X I + W 7 4 z U m R S 8 C s 0 F C B E y 2 / Q 8 T d c E h e Q M N V o Z l a 7 O 1 7 M m V S m + W 0 U V l b X 1 j e K m 6 W t 7 Z 3 d v f L + Q U d G a U J o m 0 Q 8 S n o e l p Q z Q d u K K U 5 7 c U J x 6 H H a 9 U Y 3 U 7 / 7 Q B P J I t F S 4 5 g 6 I R 4 I F j C C l Z b c 8 l G 1 5 T K b Y + G j l n t / Z u u 8 J t q o m D V z B r R M r J x U I E f T L X / Z f k T S k A p F O J a y b 5 m x c j K c K E Y 4 n Z T s V N I Y k x E e 0 L 6 m A o d U O t n s / g k 6 1 Y q P A r 0 6 i I R C M / X 3 R I Z D K c e h p 5 M h V k O 5 6 E 3 F / 7 x + q o I r J 2 M i T h U V Z L 4 o S D l S E Z q W g X y W U K L 4 W B N M E q Z v R W S I E 0 y U r q y k S 7 A W X 1 4 m n f O a d V G r 3 9 U r j e u 8 j i I c w w l U w Y J L a M A t N K E N B B 7 h G V 7 h z X g y X o x 3 4 2 M e L R j 5 z C H 8 g f H 5 A 3 f u l R 8 = < / l a t e x i t > (Ti ^Tj) _ Tn < l a t e x i t s h a 1 _ b a s e 6 4 = " Q a 3 T A C 9 k s w 7 E H i s 2 8 g Q s 0 A r 3 f + 4 = " > A A A B / H i c b Z D L S s N A F I Y n X m u 9 R b t 0 M 1 i E u i m J 1 M u y 6 M Z l h d 6 g K W U y O W 2 H T i Z h Z i K E U F / F j Q t F 3 P o g 7 n w b p 2 0 W 2 v r D w M d / z u G c + f 2 Y M 6 U d 5 9 t a W 9 / Y 3 N o u 7 B R 3 9 / Y P D u 2 j 4 7 a K E k m h R S M e y a 5 P F H A m o K W Z 5 t C N J Z D Q 5 9 D x J 3 e z e u c R p G K R a O o 0 h n 5 I R o I N G S X a W A O 7 V P E E j H B z U D v 3 O B G B o c u B X X a q z l x 4 F d w c y i h X Y 2 B / e U F E k x C E p p w o 1 X O d W P c z I j W j H K Z F L 1 E Q E z o h I + g Z F C Q E 1 c / m x 0 / x m X E C P I y k e U L j u f t 7 I i O h U m n o m 8 6 Q 6 L F a r s 3 M / 2 q 9 R A 9 v + h k T c a J B 0 M W i Y c K x j v A s C R w w C V T z 1 A C h k p l b M R 0 T S a g 2 e R V N C O 7 y l 1 e h f V F 1 r 6 q 1 h 1 q 5 f p v H U U A n 6 B R V k I u u U R3 d o w Z q I Y p S 9 I x e 0 Z v 1 Z L 1 Y 7 9 b H o n X N y m d K 6 I + s z x 9 8 W J N j < / l a t e x i t > (¬T4) ^T5 < l a t e x i t s h a 1 _ b a s e 6 4 = " m 1

r 2 X q z 3 u e t K 1 Y + c w R / Z H 1 8 A 0 l l l G s = < / l a t e x i t > T1 ^T2 ^T3 < l a t e x i t s h a 1 _ b a s e 6 4 = " Y w u n i X O V B u G h 2 h p g i N 7 U g 9 U 6 0 6 U = " > A A A B 6 n i c b V D L S g N B E O y N r x h f U Y 9 e B o P g K e y G o B 6 D X j x G z A u S J c x O e p M h s 7 P L z K w Q Q j 7 B i w d F v P p F 3 v w b J 8 k e N L G g o a j q p r s r S A T X x n W / n d z G 5 t b 2 T n 6 3 s L d / c H h U P D 5 p 6 T

4 R I 8 u I Y a 3 E M d m s B g C M / w C m + O c F 6 c d + d j 2 Z p z s p l T + A P n 8 w f Z O 4 2 G < / l a t e x i t > T 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " f G z o j A l c k 5 e G l X 9 z d L 4 6 8 y 9 X e U E = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K / Y I 2 l M 1 2 0 y 7 d b M L u R C i h P 8 G L B 0 W 8 + o u 8 + W / c t j l o 6 4 O B x 3 s z z M w L E i k M u u 6 3 U 1 h b 3 9 j c K m 6 X d n b 3 9 g / K h 0 c t E 6 e a 8 S a L Z a w 7 A T

0 L 8 r e Z b n y U C l V b 7 I 4 8 n A C p 3 A O H l x B F e 6 g B g 1 g M I B n e I U 3 R z g v z r v z s W j N O d n M M f y B 8 / k D A j a N o Q = = < / l a t e x i t > T M < l a t e x i t s h a 1 _ b a s e 6 4 = "M f R T g p i 6 z C h f o 8 v O W Y o a r Q / V m Z 0 = " > A A A B 6 n i c b V C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d h V U c s Q G 8 s E 8 4 J k C b O T 2 W T I 7 M w y M y u E J Z 9 g Y 6 G I r a 1 / 4 R f Y 2 f g t T h 6 F J h 6 4 c D j n X u 6 9 J 4 g 5 0 8 Z 1 v 5 y l 5 Z X V t f X M R n Z z a 3 t n N 7 e 3 X 9 c y U Y T W i O R S N Q O s K W e C 1 g w z n D Z j R X E U c N o I B j d j v 3 F P l W Z S V M 0 w p n 6 E e 4 K F j G B j p b t q 5 7 y T y 7 s F d w K 0 S L w Z y R c P K 9 / s v f R R 7 u Q + 2 1 1 J k o g K Q z j W u u W 5 s f F T r A w j n I 6 y 7 U T T G J M B 7 t G W p Q J H V P v p 5 N Q R O r F K F 4 V S 2 R I G T d T f E y m O t B5 G g e 2 M s O n r e W 8 s / u e 1 E h N e + y k T c W K o I N N F Y c K R k W j 8 N + o y R Y n h Q 0 s w U c z e i k g f K 0 y M T S d r Q / D m X 1 4 k 9 b O C d 1 m 4 q N g 0 S j B F B o 7 g G E 7 B g y s o w i 2 U o Q Y E e v A A T / D s c O f R e X F e p 6 1 L z m z m A P 7 A e f s B u W e R J A = = < / l a t e x i t > T3 < l a t e x i t s h a 1 _ b a s e 6 4 = "M f R T g p i 6 z C h f o 8 v O W Y o a r Q / V m Z 0 = " > A A A B 6 n i c b V C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d h V U c s Q G 8 s E 8 4 J k C b O T 2 W T I 7 M w y M y u E J Z 9 g Y 6 G I r a 1 / 4 R f Y 2 f g t T h 6 F J h 6 4 c D j n X u 6 9 J 4 g 5 0 8 Z 1 v 5 y l 5 Z X V t f X M R n Z z a 3 t n N 7 e 3 X 9 c y U Y T W i O R S N Q O s K W e C 1 g w z n D Z j R X E U c N o I B j d j v 3 F P l W Z S V M 0 w p n 6 E e 4 K F j G B j p b t q 5 7 y T y 7 s F d w K 0 S L w Z y R c P K 9 / s v f R R 7 u Q + 2 1 1 J k o g K Q z j W u u W 5 s f F T r A w j n I 6 y 7 U T T G J M B 7 t G W p Q J H V P v p 5 N Q R O r F K F 4 V S 2 R I G T d T f E y m O t B5 G g e 2 M s O n r e W 8 s / u e 1 E h N e + y k T c W K o I N N F Y c K R k W j 8 N + o y R Y n h Q 0 s w U c z e i k g f K 0 y M T S d r Q / D m X 1 4 k 9 b O C d 1 m 4 q N g 0 S j B F B o 7 g G E 7 B g y s o w i 2 U o Q Y E e v A A T / D s c O f R e X F e p 6 1 L z m z m A P 7 A e f s B u W e R J A = = < / l a t e x i t > T3 < l a t e x i t s h a 1 _ b a s e 6 4 = " Q T

5 F g e 2 M s B n o R W 8 i / u e 1 E x N e + y k T c W K o I L N F Y c K R k W j y N + o x R Y n h I 0 s w U c z e i s g A K 0 y M T S d n Q / A W X 1 4 m j f O i d 1 m 8 q N o 0 y j B D F o 7 h B M 7 A g y s o w S 1 U o A 4 E + v A A T / D s c O f R e X F e Z 6 0 Z Z z 5 z C H / g v P 0 A t l + R I g = = < / l a t e x i t > T1 < l a t e x i t s h a 1 _ b a s e 6 4 = " Q T

3 h 2 u P P o v D i v s 9 a M M 5 8 5 h D 9 w 3 n 4 A E u K R X w = = < / l a t e x i t > Tn < l a t e x i t s h a 1 _ b a s e 6 4 = " A D g W F w Y G U I B e 9 R / M / D y Z / / R / 3 /

3 h 2 u P P o v D i v s 9 a M M 5 8 5 h D 9 w 3 n 4 A E u K R X w = = < / l a t e x i t > Tn < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 / b A n A a m l y v g S w v a Z 1 U n 5 0 r N a c U = " > A A A B 6 n i c b

3 h 2 u P P o v D i v s 9 a M M 5 8 5 h D 9 w 3 n 4 A C 0 6 R W g = = < / l a t e x i t > Ti < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 / b A n A a m l y v g S w v a Z 1 U n 5 0 r N a c U = " > A A A B 6 n i c b

3 h 2 u P P o v D i v s 9 a M M 5 8 5 h D 9 w 3 n 4 A C 0 6 R W g = = < / l a t e x i t > Ti < l a t e x i t s h a 1 _ b a s e 6 4 = " l 0 b X M y 3 s O + M p U v q l e w B L F E x R + B Q = " > A A A B 6 3 i c b Z D L S g M x F I b P e K 3 j r e r S T b A I r s qM i L o R i 2 5 c V r A X a I e S S T N t a J I Z k o x Q h r 6 C G x e K u B O f x b 0 b 8 W 3 M t F 1 o 6 w + B j / 8 / h 5 x z w o Q z b T z v 2 1 l Y X F p e W S 2 s u e s b m 1 v b x Z 3 d u o 5 T R W i N x D x W z R B r y p m k N c M M p 8 1 E U S x C T h v h 4 D r P G / d U a R b L O z N M a C B w T 7 K I E W x y q 2 3 z T r H k l b 2 x 0 D z 4 U y h d f r g X y d u X W + 0 U P 9 v d m K S C S k M 4 1 r r l e 4 k J M q w M I 5 y O 3 H a q a Y L J A P d o y 6 L E g u o g G 8 8 6 Q o f W 6 a I o V v Z J g 8 b u 7 4 4 M C 6 2 H I r S V A p u + n s 1 y 8 7 + s l Z r o P M i Y T F J D J Z l 8 F K U c m R j l i 6 M u U 5 Q Y P r S A i W J 2 V k T 6 W G F i 7 H l c e w R / d u V 5 q B + X / d P y y a 1 X q l z B R A X Y h w M 4 A h / O o A I 3 U I U a E O j D A z z B s y O c R + f F e Z 2 UL j j T n j 3 4 I + f 9 B 4 g s k Z E = < / l a t e x i t > _ < l a t e x i t s h a 1 _ b a s e 6 4 = " l 0 b X M y 3 s O + M p U v q l e w B L F E x R + B Q = " > A A A B 6 3 i c b Z D L S g M x F I b P e K 3 j r e r S T b A I r s q M i L o R i 2 5 c V r A X a I e S S T N t a J I Z k o x Q h r 6 C G x e K u B O f x b 0 b 8 W 3 M t F 1 o 6 w + B j / 8 / h 5 x z w o Q z b T z v 2 1 l Y X F p e W S 2 s u e s b m 1 v b x Z 3 d u o 5 T R W i N x D x W z R B r y p m k N c M M p 8 1 E U S x C T h v h 4 D r P G / d U a R b L O z N M a C B w T 7 K I E W x y q 2 3 z T r H k l b 2 x 0 D z 4 U y h d f r g X y d u X W + 0 U P 9 v d m K S C S k M 4 1 r r l e 4 k J M q w M I 5 y O 3 H a q a Y L J A P d o y 6 L E g u o g G 8 8 6 Q o f W 6 a I o V v Z J g 8 b u 7 4 4 M C 6 2 H I r S V A p u + n s 1 y 8 7 + s l Z r o P M i Y T F J D J Z l 8 F K U c m R j l i 6 M u U 5 Q Y P r S A i W J 2 V k T 6 W G F i 7 H l c e w R / d u V 5 q B + X / d P y y a 1 X q l z B R A X Y h w M 4 A h / O o A I 3 U I U a E O j D A z z B s y O c R + f F e Z 2 U L j j T n j 3 4 I + f 9 B 4 g s k Z E = < / l a t e x i t > _ < l a t e x i t s h a 1 _ b a s e 6 4 = " B k V U r d O 1 K P H F 2 Y 9 0 P C J I a 3 l A p E Q = " > A A A B 7 H i c b Z D L S s N A F I Z P 6 q 3

v W s w 9 / Z L 3 / A C r 2 k e 0 = < / l a t e x i t > ^< l a t e x i t s h a 1 _ b a s e 6 4 = " X l z X + g U 0 4 5 8 n 4 n u T T l U 0 j U x p O G 0 = " > A A A B 6 H i c b Z D J S g N B E I Z r 4 h b H L e r R S 2 M Q P I U Z E f U i B r 1 4 T M A s k A y h p 1 O T t O l Z 6 O 4 R Y s g T e P G g i F d 9 G O 9 e x L e x s x w 0 8 Y e G j / + v o q

4 5 8 n 4 n u T T l U 0 j U x p O G 0 = " > A A A B 6 H i c b Z D J S g N B E I Z r 4 h b H L e r R S 2 M Q P I U Z E f U i B r 1 4 T M A s k A y h p 1 O T t O l Z 6 O 4 R Y s g T e P G g i F d 9 G O 9 e x L e x s x w 0 8 Y e G j / + v o q

s b P w W J 4 9 C E w 9 c O J x z L / f e E 8 S c a e O 6 X 8 7 S 8 s r q 2 n p m I 7 u 5 t b 2 z m 9 v b r 2 uZ K E J r R H K p m g H W l D N B a 4 Y Z T p u x o j g K O G 0 E g 5 u x 3 7 i n S j M p q m Y Y U z / C P c F C R r C x 0 l 2 1 c 9 H J 5 d 2 C O w F a J N 6 M 5 I u H l W / 2 X v o o d 3 K f 7 a 4 k S U S F I R x r 3 f L c 2 P g p V o Y R T k f Z d q J p j M k A 9 2 j L U o E j q v 1 0 c u o I n V i l i 0 K p b A m D J u r v i R R H W g + j w H Z G 2 P T 1 v D c W / / N a i Q m v / Z S J O D F U k O m i M O H I S D T + G 3 W Z o s T w o S W Y K G Z v R a S P F S b G p p O 1 I X j z L y + S + l n B u y y c V 2 w a J Z g i A 0 d w D K f g w R U U 4 R b K U A M C P X i A J 3 h 2 u P P o v D i v 0 9 Y l Z z Z z A H / g v P 0 A v G + R J g = = < / l a t e x i t > T5 < l a t e x i t s h a 1 _ b a s e 6 4 = " I z D W F j l g V g n 8 a o r Y M u f g 3 f y C Q 5 Q = " > A A A B 6 n i c b V C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d g V X 2 W I j W W C e U G y h N n J b D J k d m a Z m R X C k k + w s V D E 1 t a / 8 A vs b P w W J 4 9 C E w 9 c O J x z L / f e E 8 S c a e O 6 X 8 7 S 8 s r q 2 n p m I 7 u 5 t b 2 z m 9 v b r 2 u Z K E J r R H K p m g H W l D N B a 4 Y Z T p u x o j g K O G 0 E g 5 u x 3 7 i n S j M p q m Y Y U z / C P c F C R r C x 0 l 2 1 c 9 H J 5 d 2 C O w F a J N 6 M 5 I u H l W / 2 X v o o d 3 K f 7 a 4 k S U S F I R x r 3 f L c 2 P g p

/ l a t e x i t > T4 < l a t e x i t s h a 1 _ b a s e 6 4 = " E u u 5 N C K 3 n C 0 P j B 8 5 i y K L H J / f p e s = " > A A A B 6 3 i c b Z D L S g M x F I b P e K 3 j r e r S T b A I r s q M i L o R i 2 5 c V r A X a I e S S T N t a J I Z k o x Q h r 6 C G x e K u B O f x b 0 b 8 W 3 M t F 1 o 6 w + B j / 8 / h 5 x z w o Q z b T z v 2 1 l Y X F p e W S 2 s u e s b m 1 v b x Z 3 d u o 5 T R W i N x D x W z R B r y p m k N c M M p 8 1 E U S x C T h v h 4 D r P G / d U a R b L O z N M a C B w T 7 K I E W x y q y 1 p r 1 M s e W V v L D Q P / h R K l x / u R f L 2 5 V Y 7 x c 9 2 N y a p o N I Q j r V u + V 5 i g g w r w w i n I 7 e d a p p g M s A 9 2 r I o s a A 6 y M a z j t C h d b o o i p V 9 0 q C x + 7 s j w 0 L r o Q h t p c C m r 2 e z 3 P w v a 6 U m O g 8 y J p P U U E k m H 0 U p R y Z G + e K o y x Q l h g 8 t Y K K Y n R W R P l a Y G H s e 1 x 7 B n 1 1 5 H u r H Z f + 0 f H L r l S p X M F E B 9 u E A j s C H M 6 j A D V S h B g T 6 8 A B P 8 O w I 5 9 F 5 c V 4 n p Q v O t G c P / s h 5 / w F r W p F + < / l a t e x i t >

D w A E / w b A n r 0 X q x X q e l B W v W s w 9 / Z L 3 / A C r 2 k e 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B k V U r d O 1 K P H F 2 Y 9 0 P C J I a 3 l A p E Q = " > A A A B 7 H i c b Z D L S s N A F I Z P 6 q 3 G W 9 W l m 8 E i u C q J S H U j F t 2 4 r G D a Q hv K Z D J p h 0 4 m Y W Y i l N B n c O N C E V e C r + L e j f g 2 T i 8 L b f 1 h 4 O P / z 2 H O O U H K m d K O 8 2 0 V l p Z X V t e K 6 / b G 5 t b 2 T m l 3 r 6 G S T B L q k Y Q n s h V g R T k T 1 N N M c 9 p K J c V x w G k z G F y P 8 + Y 9 l Y o l 4 k 4 P U + r H u C d Y x A j W x v I 6 H I u w W y o 7 F W c i t A j u D M q X H / Z F + v Z l 1 7 u l z 0 6 Y k C y m Q h O O l W q 7 T q r 9 H E v N C K c j u 5 M p m m I y w D 3 a N i h w T J W f T 4 Y d o S P j h C h K p H l C o 4 n 7 u y P H s V L D O D C V M d Z 9 N Z + N z f + y d q a j c z 9 n I s 0 0 F W T 6 U Z R x p B M 0 3 h y F T F K i + d A A J p K Z W R H p Y 4 m J N v e x z R H c + Z U X o X F S c a u V 0 1 u n X L u C q Y p w A I d w D C 6 c Q Q 1 u o A 4 e E G D w A E / w b A n r 0 X q x X q e l B W v W s w 9 / Z L 3 / A C r 2 k e 0 = < / l a t e x i t > ^< l a t e x i t s h a 1 _ b a s e 6 4 = " J a E o q O C f t + u x a Y 8 O + 0 Q H M b 1 / I x A = " > A A A B + n i c b V D L S s N A F L 2 p r 1 p f q S 7 d D B b B V U l E 1 G X R h S 4 r 2 A e 0 I U y m k 3 b o Z B J m J k q J / R Q 3 L h R x 6 5 e 4 8 2 + c t F l o 6 4 G B w z n 3 c s + c I O F M a c f 5 t k o r q 2 v r G + X N y t b 2 z u 6 e X d 1 v q z i V h L Z I z G P Z D b C i n A n a 0 k x z 2 k 0 k x V H A a S c Y X + d + 5 4 F K x W J x r y c J 9 S I 8 F C x k B G s j + X a 1 H 2 E 9 I p h n N 1 M / G / v u 1 L d r T t 2 Z A S 0 T t y A 1 K N D 0 7 a / + I C Z p R I U m H C v V c 5 1 E e x m W m h F O p 5 V + q m i C y R g P a c 9 Q g S O q v G w W f Y q O j T J A Y S z N E x r N 1 N 8 b G Y 6 U m k S B m c y D q k U v F / / z e q k O L 7 2 M i S T V V J D 5 o T D l S M c o 7 w E N m K R E 8 4 k h m E h m s i I y w h I T b d q q m B L c x S 8 v k / Z p 3 T 2 v n 9 2 d 1 R p X R R 1 l O I Q j O A E X L q A B t 9 C E F h B 4 h G d 4 h T fr y X q x 3 q 2 P + W j J K n Y O 4 A + s z x 9 3 g 5 Q l < / l a t e x i t > G k1 < l a t e x i t s h a 1 _ b a s e 6 4 = " / z J g T o 4 n Y E 0 F S P 6 7 4 y 6 f n x P c w o s = " > A A A B + n i c b V D L S s N A F L 2 p r 1 p f q S 7 d B I v g q i S l q M u i C 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M T J Q S 8 y l u X C j i 1 i 9 x 5 9 8 4 a b P Q 1 g M D h 3 P u 5 Z 4 5 f s y oV L b 9 b Z T W 1 j c 2 t 8 r b l Z 3 d v f 0 D s 3 r Y l V E i M O n g i E W i 7 y N J G O W k o 6 h i p B 8 L g k K f k Z 4 / v c 7 9 3 g M R k k b 8 X s 1 i 4 o Z o z G l A M V J a 8 s z q M E R q g h F L b z I v n X q N z D N r d t 2 e w 1 o l T k F q U K D t m V / D U Y S T k H C F G Z J y 4 N i x c l M k F M W M Z J V h I k m M8 B S N y U B T j k I i 3 X Q e P b N O t T K y g k j o x 5 U 1 V 3 9 v p C i U c h b 6 e j I P K p e 9 X P z P G y Q q u H R T y u N E E Y 4 X h 4 K E W S q y 8 h 6 s E R U E K z b T B G F B d V Y L T 5 B A W O m 2 K r o E Z / n L q 6 T b q D v n 9 e Z d s 9 a 6 K u o o w z G c w B k 4 c A E t u I U 2 d A D D I z z D K 7 w Z T 8 a L 8 W 5 8 L E Z L R r F z B H 9 g f P 4 A e Q i U J g = = < / l a t e x i t > G k2 < l a t e x i t s h a 1 _ b a s e 6 4 = " f S b h yO o g T l v U d F v c Y u j Z Y r R e K F E = " > A A A B + n i c b V D L S s N A F L 2 p r 1 p f q S 7 d D B b B V U l E 1 G X R h S 4 r 2 A e 0 I U y m k 3 b o T B J m J k q J / R Q 3 L h Rx 6 5 e 4 8 2 + c t F l o 6 4 G B w z n 3 c s + c I O F M a c f 5 t k o r q 2 v r G + X N y t b 2 z u 6 e X d 1 v q z i V h L Z I z G P Z D b C i n E W 0 p Z n m t J t I i k X A a S c Y X + d + 5 4 F K x e L o X k 8 S 6 g k 8 j F j I C N Z G 8 u 1 q X 2 A 9 I p h n N 1 M / G / t i 6 t s 1 p + 7 M g J a J W 5 A a F G j 6 9 l d / E J

r y X q x 3 q 2 P + W j J K n Y O 4 A + s z x / S r 5 R h < / l a t e x i t > G km < l a t e x i t s h a 1 _ b a s e 6 4 = "M R P O Q b i v e a e o q B t c R U l f q n m + 2 G s = " > A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s q u F P V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v 0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1 I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C Ch N j I y r Z E L z l l 1 d J + 6 L m X d b q 9 / V K 4 y a P o w g n c A p V 8 O A K G n A H T W g B A Q 7 P 8 A p v z q P z 4 r w 7 H 4 v W g p P P H M M f O J 8 / Z f y P k A = = < / l a t e x i t > f (•) < l a t e x i t s h a 1 _ b a s e 6 4 = " M R P O Q b i v e a e o q B t c R U l f q n

0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J + 6 L m X d b q 9 / V K4 y a P o w g n c A p V 8 O A K G n A H T W g B A Q 7 P 8 A p v z q P z 4 r w 7 H 4 v W g p P P H M M f O J 8 / Z f y P k A = = < / l a t e x i t > f (•) …… < l a t e x i t s h a 1 _ b a s e 6 4 = " M R P O Q b i v e a e o q B t c R U l f q n

Figure1: Knowledge-based Policy Recycling (KPR) Overview. Prior domain knowledge comprises a set of auxiliary tasks and a set of logical formulae defined over the auxiliary tasks. KPR combines these auxiliary tasks policies to obtain a new robust policy for the main task. At every time step, we obtain a state-conditioned action distribution for each auxiliary task, which are realizations of the variables in logical formulae. We represent logical formulae as graphs whose nodes are either auxiliary tasks or logical operators, e.g., ¬, ∧, ∨, =⇒ . The instantiated logical formula graphs with node features are processed by a GNN pooling function (Section 3.2) to yield m different action distributions (one for each graph). In other words, each logical graph results in a different policy. Finally, we combine/fuse these distributions (Section 3.3) to obtain a final action distribution. In summary, KPR uses the specified logical formula graph as prior structure and then learns flexible relations over auxiliary tasks from interaction data.

