MEGA: MOVING AVERAGE EQUIPPED GATED ATTENTION

Abstract

The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences. In this paper, we introduce MEGA, a simple, theoretically grounded, single-head gated attention mechanism equipped with (exponential) moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. We further propose a variant of MEGA that offers linear time and space complexity yet yields only minimal quality loss, by efficiently splitting the whole sequence into multiple chunks with fixed length. Extensive experiments on a wide range of sequence modeling benchmarks, including the Long Range Arena, neural machine translation, auto-regressive language modeling, and image and speech classification, show that MEGA achieves significant improvements over other sequence models, including variants of Transformers and recent state space models.

1. INTRODUCTION

Designing a single unified model to capture long range dependencies in sequential data is a central and challenging problem in sequence modeling. A number of different archtectures have been developed, including convolutional neural networks (CNNs) (Kim, 2014; Strubell et al., 2017) , recurrent neural networks (RNNs) (Goller & Kuchler, 1996; Hochreiter & Schmidhuber, 1997) , Transformers (Vaswani et al., 2017) and recent state space models (SSMs) (Gu et al., 2022a; Mehta et al., 2022) . Among these models, Transformer (Vaswani et al., 2017) has stood out for its impressive empirical success on a wide range of language and vision tasks, including machine translation (Vaswani et al., 2017; Ott et al., 2018) , language understanding (Devlin et al., 2019; Liu et al., 2019) , image recognition (Dosovitskiy et al., 2020; Touvron et al., 2021) and genetic sequence modeling (Madani et al., 2020; Jumper et al., 2021) , mainly because of the conceptually attractive attention mechanism (Bahdanau et al., 2015) which directly models interactions between each pair of input tokens. However, there are two common drawbacks in this design: i) weak inductive bias; and ii) quadratic computational complexity. First, the attention mechanism does not assume prior knowledge of the patterns of dependencies between tokens (e.g. positional inductive bias), instead learning to predict the pairwise attention weights directly from data. Second, the cost to compute and store the attention weights is quadratic in the input length. Recent studies have shown the limitations of applying Transformers to longer sequences, both with respect to accuracy and efficiency (Tay et al., 2020) . In this work, we propose a moving average equipped gated attention mechanism (MEGA) to solve the two weaknesses simultaneously. The key idea is to incorporate inductive biases into the attention mechanism across the timestep dimension, by leveraging the classic exponential moving average (EMA) approach (Hunter, 1986) . EMA captures local dependencies that exponentially decay over time (see Figure 1 ), and has been widely used in time series data modeling ( §2). We introduce a multi-dimensional damped form of EMA with learnable coefficients ( §3.1), and subsequently develop the moving average equipped gated attention mechanism by integrating the EMA with a variant of the single-head gated attention (Hua et al., 2022) ( §3.2) . Theoretically, we show that the single-head gated attention is as expressive as the most commonly used multi-head attention ( §3.5). Benefiting Table 1 : Experimental results of Transformer (XFM), S4 and MEGA on five sequence modeling benchmarks, including long range arena (LRA), machine translation (WMT16 en-de), language modeling (WikiText-103), image classification (ImageNet-1k), raw speech classification (SC-Raw). from the incorporated moving average mechanism, we further propose a variant of MEGA with linear complexity, named MEGA-chunk, with minimal loss of contextual information ( §3.4).

LRA (Acc

Experimentally, on five sequence modeling tasks across various data types, including long-context sequence modeling, neural machine translation, auto-regressive language modeling, and image and speech classification, MEGA significantly outperforms a variety of strong baseline models, in terms of both effectiveness and efficiency ( §4) (see Table 1 ).

2. BACKGROUND

We use X = {x 1 , x 2 , . . . , x n } ∈ R n×d to denote a sequence of input representations with length n. Let Y = {y 1 , y 2 , . . . , y n } ∈ R n×d be the sequence of output representations of each layer with the same length n as the input X. In this paper, we assume the representations of the input and output sequences have the same dimension d.

2.1. SELF-ATTENTION MECHANISM

The traditional self-attention mechanism is a function: Y = Attn(X) = f QK T τ (X) V where Attn : R n×d → R n×d is the self-attention function. Q = XW q + b q , K = XW k + b k , and V = XW v + b v are the sequences of queries, keys and values, with learnable parameters W q , W k , W v ∈ R d×d , and b q , b k , b v ∈ R d . f (•) is an attention function, e.g. the softmax function f softmax (•) (Bahdanau et al., 2015) , or the recently proposed squared ReLU function f relu 2 (•) (So et al., 2021; Hua et al., 2022) . τ (X) is a scaling term, which is commonly set to τ (X) = √ d for f softmax (•), or τ (X) = n for f relu 2 (•). The commonly used multi-head variant of attention performs the attention function h times in parallel. We can define a matrix A = f ( QK T τ (X) ) ∈ R n×n following (1), which is called the attention matrix. Since it specifies pairwise dependency weights, the matrix A in principle delivers a flexible and powerful mechanism to learn long-distance dependencies with minimal inductive biases. However, it is in practice a challenging task to recognize all the dependency patterns directly from data, particularly when processing long sequences. Moreover, calculating A with h attention heads takes O(hn 2 ) time and space, which becomes a significant bottleneck.

2.2. EXPONENTIAL MOVING AVERAGE (EMA)

The moving average is a classic approach for sequential data modeling, which has been widely used in time series data to smooth out short-term fluctuations and highlight long-term trends or cycles. The Exponential Moving Average (EMA) (Winters, 1960; Hunter, 1986) , a special case of moving average, applies weighting factors that decrease exponentially: y t = α ⊙ x t + (1 -α) ⊙ y t-1 , where α ∈ (0, 1) d is the EMA coefficient representing the degree of weighting decrease, and ⊙ is the element-wise product. A higher α discounts older observations faster (see Figure 1 ). Using an EMA places a strong inductive bias on the learning of pairwise dependencies: the dependency weight between two tokens decreases exponentially over time with an input-agnostic decay factor α. This property favors local dependencies, and limits long-distance dependencies. Despite the recurrent formulation in (2), the computation of EMA can be represented as n individual convolutions, which can be computed efficiently using fast Fourier transforms (FFTs) (see Appendix A for details). < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 o B k I A 9 h 0 5 M H b V r D / C 0 A h u U O o o 0 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I o / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m T j V j D d Z L G P d C a j h U i j e R I G S d x L N a R R I 3 g 7 G t z O / / c S 1 E b F 6 x C z h f k S H S o S C U b T S Q 9 b H f r n i V t 0 5 y C r x c l K B H I 1 + + a s 3 i F k a c Y V M U m O 6 n p u g P 6 E a B Z N 8 W u q l h i e U j e m Q d y 1 V N O L G n 8 x P n Z I z q w x I G G t b C s l c / T 0 x o Z E x W R T Y z o j i y C x 7 M / E / r 5 t i e O 1 P h E p S 5 I o t F o W p J B i T 2 d 9 k I D R n K D N L K N P C 3 k r Y i G r K 0 K Z T s i F 4 y y + v k t Z F 1 b u s 1 u 5 r l f p N H k c R T u A U z s G D K 6 j D H T S g C Q y G 8 A y v 8 O Z I 5 8 V 5 d z 4 W r Q U n n z m G P 3 A + f w B 1 s I 3 t < / l a t e x i t > y t < l a t e x i t s h a 1 _ b a s e 6 4 = " M b T S 1 h + M 8 t S U v b D J a A P r p e 2 j a B U = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a U D b b T b t 0 s w m 7 E 7 G E / g Q v H h T x 6 i / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f T m F l d W 1 9 o 7 h Z 2 t r e 2 d 0 r 7 x 8 0 T Z x q x h s s l r F u B 9 R w K R R v o E D J 2 4 n m N A o k b w W j m 6 n f e u T a i F g 9 4 D j h f k Q H S o S C U b T S / V M P e + W K W 3 V n I M v E y 0 k F c t R 7 5 a 9 u P 2 Z p x B U y S Y 3 p e G 6 C f k Y 1 C i b 5 p N R N D U 8 o G 9 E B 7 1 i q a M S N n 8 1 O n Z A T q / R J G G t b C s l M / T 2 R 0 c i Y c R T Y z o j i 0 C x 6 U / E / r 5 N i e O V n Q i U p c s X m i 8 J U E o z J 9 G / S F 5 o z l G N L K N P C 3 k r Y k G r K 0 K Z T s i F 4 i y 8 v k + Z Z 1 b u o n t + d V 2 r X e R x F O I J j O A U P L q E G t 1 C H B j A Y w D O 8 w p s j n R f n 3 f m Y t x a c f O Y Q / s D 5 / A F 0 K o 3 s < / l a t e x i t > x t < l a t e x i t s h a 1 _ b a s e 6 4 = " y E j H X y r A 7 H A s E R B P k D N K 2 0 K G 6 f k = " > A A A B 7 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 s S R S 1 G P R i 8 c K 9 g P a U D b b T b t 0 s w m 7 E 7 G E / g g v H h T x 6 u / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f z s r q 2 v r G Z m G r u L 2 z u 7 d f O j h s m j j V j D d Y L G P d D q j h U i j e Q I G S t x P N a R R I 3 g p G t 1 O / 9 c i 1 E b F 6 w H H C / Y g O l A g F o 2 i l 1 l M v w 3 N v 0 i u V 3 Y o 7 A 1 k m X k 7 K k K P e K 3 1 1 + z F L I 6 6 Q S W p M x 3 M T 9 D O q U T D J J 8 V u a n h C 2 Y g O e M d S R S N u / G x 2 7 o S c W q V P w l j b U k h m 6 u + J j E b G j K P A d k Y U h 2 b R m 4 r / e Z 0 U w 2 s / E y p J k S s 2 X x S m k m B M p r + T v t C c o R x b Q p k W 9 l b C h l R T h j a h o g 3 B W 3 x 5 m T Q v K t 5 l p X p f L d d u 8 j g K c A w n c A Y e X E E N 7 q A O D W A w g m d 4 h T c n c V 6 c d + d j 3 r r i 5 D N H 8 A f O 5 w 8 V i o 9 q < / l a t e x i t > x t 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " e 8 v d t X z g G 7 N X j G S G C e t 8 z K e v s 3 o = " > A A A B 7 n i c b V D L S g N B E O y N r x h f U Y 9 e B o P g x b A b g n o M e v E Y w T w g W c L s Z J I M m Z 1 d Z n r F s O Q j v H h Q x K v f 4 8 2 / c Z L s Q R M L G o q q b r q 7 g l g K g 6 7 7 7 e T W 1 j c 2 t / L b h Z 3 d v f 2 D 4 u F R 0 0 S J Z r z B I h n p d k A N l 0 L x B g q U v B 1 r T s N A 8 l Y w v p 3 5 r U e u j Y j U A 0 5 i 7 o d 0 q M R A M I p W a j 3 1 U r y o T H v F k l t 2 5 y C r x M t I C T L U e 8 W v b j 9 i S c g V M k m N 6 X h u j H 5 K N Q o m + b T Q T Q y P K R v T I e 9 Y q m j I j Z / O z 5 2 S M 6 v 0 y S D S t h S S u f p 7 I q W h M Z M w s J 0 h x Z F Z 9 m b i f 1 4 n w c G 1 n w o V J 8 g V W y w a J J J g R G a / k 7 7 Q n K G c W E K Z F v Z W w k Z U U 4 Y 2 o Y I N w V t + e Z U 0 K 2 X v s l y 9 r 5 Z q N 1 k c e T i B U z g H D 6 6 g B n d Q h w Y w G M M z v M K b E z s v z r v z s W j N O d n M M f y B 8 / k D F w + P a w = = < / l a t e x i t > x t 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 r i i Q U G J 4 G g 0 / O 9 Z F X V 8 Q 7 6 h a h 4 = " > A A A B 7 n i c b V D L S g N B E O y N r x h f U Y 9 e B o P g x b C r Q T 0 G v X i M Y B 6 Q h D A 7 m U 2 G z M 4 u M 7 1 i W P I R X j w o 4 t X v 8 e b f O E n 2 o I k F D U V V N 9 1 d f i y F Q d f 9 d n I r q 2 v r G / n N w t b 2 z u 5 e c f + g Y a J E M 1 5 n k Y x 0 y 6 e G S 6 F 4 H Q V K 3 o o 1 p 6 E v e d M f 3 U 7 9 5 i P X R k T q A c c x 7 4 Z 0 o E Q g G E U r N Z 9 6 K Z 5 d T H r F k l t 2 Z y D L x M t I C T L U e s W v T j 9 i S c g V M k m N a X t u j N 2 U a h R M 8 k m h k x g e U z a i A 9 6 2 V N G Q m 2 4 6 O 3 d C T q z S J 0 G k b S k k M / X 3 R E p D Y 8 a h b z t D i k O z 6 E 3 F / 7 x 2 g s F 1 N x U q T p A r N l 8 U J J J g R K a / k 7 7 Q n K E c W 0 K Z F v Z W w o Z U U 4 Y 2 o Y I N w V t 8 e Z k 0 z s v e Z b l y X y l V b 7 I 4 8 n A E x 3 A K H l x B F e 6 g B n V g M I J n e I U 3 J 3 Z e n H f n Y 9 6 a c 7 K Z Q / g D 5 / M H G J S P b A = = < / l a t e x i t > x t 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " a Q 2 c F U 2 h f Q s L p S X T d Y 4 D r i y n 0 f I = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B i y W R o h 6 L X j x W s B / Q h r L Z b t u l m 0 3 Y n Q g h 9 E d 4 8 a C I V 3 + P N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Q S y F Q d f 9 d g p r 6 x u b W 8 X t 0 s 7 u 3 v 5 B + f C o Z a J E M 9 5 k k Y x 0 J 6 C G S 6 F 4 E w V K 3 o k 1 p 2 E g e T u Y 3 M 3 8 9 h P X R k T q E d O Y + y E d K T E U j K K V 2 m k / w w t v 2 i 9 X 3 K o 7 B 1 k l X k 4 q k K P R L 3 / 1 B h F L Q q 6 Q S W p M 1 3 N j 9 D O q U T D J p 6 V e Y n h M 2 Y S O e N d S R U N u / G x + 7 p S c W W V A h p G 2 p Z D M 1 d 8 T G Q 2 N S c P A d o Y U x 2 b Z m 4 n / e d 0 E h z d + J l S c I F d s s W i Y S I I R m f 1 O B k J z h j K 1 h D I t 7 K 2 E j a m m D G 1 C J R u C t / z y K m l d V r 2 r a u 2 h V q n f 5 n E U 4 Q R O 4 R w 8 u I Y 6 3 E M D m s B g A s / w C m 9 O 7 L w 4 7 8 7 H o r X g 5 D P H 8 A f O 5 w 8 X F I 9 r < / l a t e x i t > y t 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " M b T S 1 h + M 8 t S U v b D J a A P r p e 2 j a B U = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a U D b b T b t 0 s w m 7 E 7 G E / g Q v H h T x 6 i / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f T m F l d W 1 9 o 7 h Z 2 t r e 2 d 0 r 7 x 8 0 T Z x q x h s s l r F u B 9 R w K R R v o E D J 2 4 n m N A o k b w W j m 6 n f e u T a i F g 9 4 D j h f k Q H S o S C U b T S / V M P e + W K W 3 V n I M v E y 0 k F c t R 7 5 a 9 u P 2 Z p x B U y S Y 3 p e G 6 C f k Y 1 C i b 5 p N R N D U 8 o G 9 E B 7 1 i q a M S N n 8 1 O n Z A T q / R J G G t b C s l M / T 2 R 0 c i Y c R T Y z o j i 0 C x 6 U / E / r 5 N i e O V n Q i U p c s X m i 8 J U E o z J 9 G / S F 5 o z l G N L K N P C 3 k r Y k G r K 0 K Z T s i F 4 i y 8 v k + Z Z 1 b u o n t + d V 2 r X e R x F O I J j O A U P L q E G t 1 C H B j A Y w D O 8 w p s j n R f n 3 f m Y t x a c f O Y Q / s D 5 / A F 0 K o 3 s < / l a t e x i t > x t < l a t e x i t s h a 1 _ b a s e 6 4 = " y E j H X y r A 7 H A s E R B P k D N K 2 0 K G 6 f k = " > A A A B 7 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 s S R S 1 G P R i 8 c K 9 g P a U D b b T b t 0 s w m 7 E 7 G E / g g v H h T x 6 u / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f z s r q 2 v r G Z m G r u L 2 z u 7 d f O j h s m j j V j D d Y L G P d D q j h U i j e Q I G S t x P N a R R I 3 g p G t 1 O / 9 c i 1 E b F 6 w H H C / Y g O l A g F o 2 i l 1 l M v w 3 N v 0 i u V 3 Y o 7 A 1 k m X k 7 K k K P e K 3 1 1 + z F L I 6 6 Q S W p M x 3 M T 9 D O q U T D J J 8 V u a n h C 2 Y g O e M d S R S N u / G x 2 7 o S c W q V P w l j b U k h m 6 u + J j E b G j K P A d k Y U h 2 b R m 4 r / e Z 0 U w 2 s / E y p J k S s 2 X x S m k m B M p r + T v t C c o R x b Q p k W 9 l b C h l R T h j a h o g 3 B W 3 x 5 m T Q v K t 5 l p X p f L d d u 8 j g K c A w n c A Y e X E E N 7 q A O D W A w g m d 4 h T c n c V 6 c d + d j 3 r r i 5 D N H 8 A f O 5 w 8 V i o 9 q < / l a t e x i t > x t 1 < l a t e x i t s h a _ b a s e = " e v d t X z g G N X j G S G C e t z K e v s o = " > A A A B n i c b V D L S g N B E O y N r x h f U Y e B o P g x b A b g n o M e v E Y w T w g W c L s Z J I M m Z d Z n r F s O Q j v H h Q x K v f / c Z L s Q R M L G o q q b r q g l g K g e T W j c t / L b h Z d v f D u F R S J Z r z B I h n p d k A N l L x B g q U v B r T s N A l Y w v p r U e u j Y j U A i o d q M R A M I p W a j U r y o T H v F k l t y C r x M t I C T L U e W v b j i S c g V M k m N X h u j H K N Q o m + b T Q T Q y P K R v T I e Y q m j I j Z / O z S M v y S D S t h S S u f p I q W h M Z M w s J h x Z F Z m b i f n w c G n w o V J g V W y w a J J J g R G a / k Q n K G c W E K Z F v Z W w k Z U U Y o Y I N w V t + e Z U K X v s l y r Z q N k c e T i B U z g H D g B n d Q h w Y w G M M z v M K b E z s v z r v z s W j N O d n M M f y B / k D F w + P a w = = < / l a t e x i t > x t 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 r i i Q U G J 4 G g 0 / O 9 Z F X V 8 Q 7 6 h a h 4 = " > A A A B 7 n i c b V D L S g N B E O y N r x h f U Y 9 e B o P g x b C r Q T 0 G v X i M Y B 6 Q h D A 7 m U 2 G z M 4 u M 7 1 i W P I R X j w o 4 t X v 8 e b f O E n 2 o I k F D U V V N 9 1 d f i y F Q d f 9 d n I r q 2 v r G / n N w t b 2 z u 5 e c f + g Y a J E M 1 5 n k Y x 0 y 6 e G S 6 F 4 H Q V K 3 o o 1 p 6 E v e d M f 3 U 7 9 5 i P X R k T q A c c x 7 4 Z 0 o E Q g G E U r N Z 9 6 K Z 5 d T H r F k l t 2 Z y D L x M t I C T L U e s W v T j 9 i S c g V M k m N a X t u j N 2 U a h R M 8 k m h k x g e U z a i A 9 6 2 V N G Q m 2 4 6 O 3 d C T q z S J 0 G k b S k k M / X 3 R E p D Y 8 a h b z t D i k O z 6 E 3 F / 7 x 2 g s F 1 N x U q T p A r N l 8 U J J J g R K a / k 7 7 Q n K E c W 0 K Z F v Z W w o Z U U 4 Y 2 o Y I N w V t 8 e Z k 0 z s v e Z b l y X y l V b 7 I 4 8 n A E x 3 A K H l x B F e 6 g B n V g M I J n e I U 3 J 3 Z e n H f n Y 9 6 a c 7 K Z Q / g D 5 / M H G J S P b A = = < / l a t e x i t > x t 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 H o B Z h L T l L F n a U O O C P k K 4 a H r 8 y Y = " > A A A B + 3 i c b Z D L S s N A F I Z P 6 q 3 W W 6 x L N 4 N F q A t L U o q 6 L L p x W c F e o I 1 l M p 2 0 Q y e T M D M R S + i r u H G h i F t f x J 1 v 4 7 T N Q l t / G P j 4 z z m c M 7 8 f c 6 a 0 4 3 x b u b X 1 j c 2 t / H Z h Z 3 d v / 8 A + L L Z U l E h C m y T i k e z 4 W F H O B G 1 q p j n t x J L i 0 O e 0 7 Y 9 v Z v X 2 I 5 W K R e J e T 2 L q h X g o W M A I 1 s b q 2 8 U e 5 v E I l 9 3 z B Z w 9 V P t 2 y a k 4 c 6 F V c D M o Q a Z G 3 / 7 q D S K S h F R o w r F S X d e J t Z d i q R n h d F r o J Y r G m I z x k H Y N C h x S 5 a X z 2 6 f o 1 D g D F E T S P K H R 3 P 0 9 k e J Q q U n o m 8 4 Q 6 5 F a r s 3 M / 2 r d R A d X X s p E n G g q y G J R k H C k I z Q L A g 2 Y p E T z i Q F M J D O 3 I j L C E h N t 4 i q Y E N z l L 6 9 C q 1 p x L y q 1 u 1 q p f p 3 F k Y d j O I E y u H A J d b i F B j S B w B M 8 w y u 8 W V P r x X q 3 P h a t O S u b O Y I / s j 5 / A H I q k 2 s = < / l a t e x i t >

↵(1 ↵)

< l a t e x i t s h a 1 _ b a s e 6 4 = " e O z M A 8 I g h 4 T U 5 j p g D z F g g 5 s t Y g A = " > A A A B + 3 i c b Z D L S s N A F I Y n 9 V b r L d a l m 8 E i 1 I U l 0 a I u i 2 5 c V r A X a G M 5 m U 7 a o Z N J m J m I J f R V 3 L h Q x K 0 v 4 s 6 3 c d p m o a 0 / D H z 8 5 x z O m d + P O V P a c b 6 t 3 M r q 2 v p G f r O w t b 2 z u 2 f v F 5 s q S i S h D R L x S L Z 9 U J Q z Q R u a a U 7 b s a Q Q + p y 2 / N H N t N 5 6 p F K x S N z r c U y 9 E A a C B Y y A N l b P L n a B x 0 M o u 6 d z O H k 4 7 9 k l p + L M h J f B z a C E M t V 7 9 l e 3 H 5 E k p E I T D k p 1 X C f W X g p S M 8 L p p N B N F I 2 B j G B A O w Y F h F R 5 6 e z 2 C T 4 2 T h 8 H k T R P a D x z f 0 + k E C o 1 D n 3 T G Y I e q s X a 1 P y v 1 k l 0 c O W l T M S J p o L M F w U J x z r C 0 y B w n 0 l K N B 8 b A C K Z u R W T I U g g 2 s R V M C G 4 i 1 9 e h u Z Z x b 2 o V O + q p d p 1 F k c e H a I j V E Y u u k Q 1 d I v q q I E I e k L P 6 B W 9 W R P r x X q 3 P u a t O S u b O U B / Z H 3 + A H O u k 2 w = < / l a t e x i t > ↵(1 ↵) 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " d B F 4 U m D n W 4 6 s N Z C v W + a L g Z 3 3 I y Q = " > A A A B + X i c b Z B N S 8 N A E I Y n f t b 6 F f X o J V i E e r A k U t R j 0 Y v H C v Y D 2 l A m 2 0 2 7 d L M J u 5 t C C f 0 n X j w o 4 t V / 4 s 1 / 4 7 b N Q V t f W H h 4 Z 4 a Z f Y O E M 6 V d 9 9 t a W 9 / Y 3 N o u 7 B R 3 9 / Y P D u 2 j 4 6 a K U 0 l o g 8 Q 8 l u 0 A F e V M 0 I Z m m t N 2 I i l G A a e t Y H Q / q 7 f G V C o W i y c 9 S a g f 4 U C w k B H U x u r Z d h d 5 M s S y d 7 m A i 5 5 d c i v u X M 4 q e D m U I F e 9 Z 3 9 1 + z F J I y o 0 4 a h U x 3 M T 7 W c o N S O c T o v d V N E E y Q g H t G N Q Y E S V n 8 0 v n z r n x u k 7 Y S z N E 9 q Z u 7 8 n M o y U m k S B 6 Y x Q D 9 V y b W b + V + u k O r z 1 M y a S V F N B F o v C l D s 6 d m Y x O H 0 m K d F 8 Y g C J Z O Z W h w x R I t E m r K I J w V v + 8 i o 0 r y r e d a X 6 W C 3 V 7 v I 4 C n A K Z 1 A G D 2 6 g B g 9 Q h w Y Q G M M z v M K b l V k v 1 r v 1 s W h d s / K Z E / g j 6 / M H Q l O S x w = = < / l a t e x i t > ↵(1 ↵) < l a t e x i t s h a 1 _ b a s e 6 4 = " w E I 8 J z t g P f 6 N 9 2 x 6 C u s V k C / 0 L Z g = " > A A A B 7 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e i F 4 8 V 7 A e 0 o U y 2 m 3 b t Z h N 2 N 0 I J / Q 9 e P C j i 1 f / j z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q K G v S W M S q E 6 B m g k v W N N w I 1 k k U w y g Q r B 2 M b 2 d + + 4 k p z W P 5 Y C Y J 8 y M c S h 5 y i s Z K r R 6 K Z I T 9 c s W t u n O Q V e L l p A I 5 G v 3 y V 2 8 Q 0 z R i 0 l C B W n c 9 N z F + h s p w K t i 0 1 E s 1 S 5 C O c c i 6 l k q M m P a z + b V T c m a V A Q l j Z U s a M l d / T 2 Q Y a T 2 J A t s Z o R n p Z W 8 m / u d 1 U x N e + x m X S W q Y p I t F Y S q I i c n s d T L g i l E j J p Y g V d z e S u g I F V J j A y r Z E L z l l 1 d J 6 6 L q X V Z r 9 7 V K / S a P o w g n c A r n 4 M E V 1 O E O G t A E C o / w D K / w 5 s T O i / P u f C x a C 0 4 + c w x / 4 H z + A I 5 v j y E = < / l a t e x i t > ↵ < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 H o B Z h L T l L F n a U O O C P k K 4 a H r 8 y Y = " > A A A B + 3 i c b Z D L S s N A F I Z P 6 q 3 W W 6 x L N 4 N F q A t L U o q 6 L L p x W c F e o I 1 l M p 2 0 Q y e T M D M R S + i r u H G h i F t f x J 1 v 4 7 T N Q l t / G P j 4 z z m c M 7 8 f c 6 a 0 4 3 x b u b X 1 j c 2 t / H Z h Z 3 d v / 8 A + L L Z U l E h C m y T i k e z 4 W F H O B G 1 q p j n t x J L i 0 O e 0 7 Y 9 v Z v X 2 I 5 W K R e J e T 2 L q h X g o W M A I 1 s b q 2 8 U e 5 v E I l 9 3 z B Z w 9 V P t 2 y a k 4 c 6 F V c D M o Q a Z G 3 / 7 q D S K S h F R o w r F S X d e J t Z d i q R n h d F r o J Y r G m I z x k H Y N C h x S 5 a X z 2 6 f o 1 D g D F E T S P K H R 3 P 0 9 k e J Q q U n o m 8 4 Q 6 5 F a r s 3 M / 2 r d R A d X X s p E n G g q y G J R k H C k I z Q L A g 2 Y p E T z i Q F M J D O 3 I j L C E h N t 4 i q Y E N z l L 6 9 C q 1 p x L y q 1 u 1 q p f p 3 F k Y d j O I E y u H A J d b i F B j S B w B M 8 w y u 8 W V P r x X q 3 P h a t O S u b O Y I / s j 5 / A H I q k 2 s = < / l a t e x i t > ↵(1 ↵) 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " d B F 4 U m D n W 4 6 s N Z C v W + a L g Z 3 3 I y Q = " > A A A B + X i c b Z B N S 8 N A E I Y n f t b 6 F f X o J V i E e r A k U t R j 0 Y v H C v Y D 2 l A m 2 0 2 7 d L M J u 5 t C C f 0 n X j w o 4 t V / 4 s 1 / 4 7 b N Q V t f W H h 4 Z 4 a Z f Y O E M 6 V d 9 9 t a W 9 / Y 3 N o u 7 B R 3 9 / Y P D u 2 j 4 6 a K U 0 l o g 8 Q 8 l u 0 A F e V M 0 I Z m m t N 2 I i l G A a e t Y H Q / q 7 f G V C o W i y c 9 S a g f 4 U C w k B H U x u r Z d h d 5 M s S y d 7 m A i 5 5 d c i v u X M 4 q e D m U I F e 9 Z 3 9 1 + z F J I y o 0 4 a h U x 3 M T 7 W c o N S O c T o v d V N E E y Q g H t G N Q Y E S V n 8 0 v n z r n x u k 7 Y S z N E 9 q Z u 7 8 n M o y U m k S B 6 Y x Q D 9 V y b W b + V + u k O r z 1 M y a S V F N B F o v C l D s 6 d m Y x O H 0 m K d F 8 Y g C J Z O Z W h w x R I t E m r K I J w V v + 8 i o 0 r y r e d a X 6 W C 3 V 7 v I 4 C n A K Z 1 A G D 2 6 g B g 9 Q h w Y Q G M M z v M K b l V k v 1 r v 1 s W h d s / K Z E / g j 6 / M H Q l O S x w = = < / l a t e x i t > ↵(1 ↵) < l a t e x i t s h a 1 _ b a s e 6 4 = " w E I 8 J z t g P f 6 N 9 2 x 6 C u s V k C / 0 L Z g = " > A A A B 7 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e i F 4 8 V 7 A e 0 o U y 2 m 3 b t Z h N 2 N 0 I J / Q 9 e P C j i 1 f / j z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q K G v S W M S q E 6 B m g k v W N N w I 1 k k U w y g Q r B 2 M b 2 d + + 4 k p z W P 5 Y C Y J 8 y M c S h 5 y i s Z K r R 6 K Z I T 9 c s W t u n O Q V e L l p A I 5 G v 3 y V 2 8 Q 0 z R i 0 l C B W n c 9 N z F + h s p w K t i 0 1 E s 1 S 5 C O c c i 6 l k q M m P a z + b V T c m a V A Q l j Z U s a M l d / T 2 Q Y a T 2 J A t s Z o R n p Z W 8 m / u d 1 U x N e + x m X S W q Y p I t F Y S q I i c n s d T L g i l E j J p Y g V d z e S u g I F V J j A y r Z E L z l l 1 d J 6 6 L q X V Z r 9 7 V K / S a P o w g n c A r n 4 M E V 1 O E O G t A E C o / w D K / w 5 s T O i / P u f C x a C 0 4 + c w x / 4 H z + A I 5 v j y E = < / l a t e x i t >

↵

Figure 1 : Illustration of the exponential moving average (EMA) approach, which averages the input values X with weights decaying exponentially over timesteps.

2.3. WHY COMBINE ATTENTION WITH EMA?

As discussed in Sections 2.1 and 2.2, EMA and attention mechanisms each have their own limitations, despite their wide applications and impressive successes in sequence modeling. By leveraging their properties to complement each other, we propose to embed an EMA into the calculation of the attention matrix A. The resulting model enjoys the benefit from strong inductive bias, while maintaining the capacity to learn complex dependency patterns. Moreover, this integration enables the design of a computationally efficient chunk-wise attention mechanism with linear complexity w.r.t sequence length ( §3.4).

3. MOVING AVERAGE EQUIPPED GATED ATTENTION (MEGA)

In this section, we describe in detail our proposed method, moving average equipped gated attention (MEGA). We first introduce multi-dimensional damped EMA ( §3.1), which is a key component combined with the single-head gated attention in MEGA ( §3.2), and discuss the relationship between MEGA and three closely related models: GRU (Cho et al., 2014) , Flash (Hua et al., 2022) and S4 (Gu et al., 2022a) . Then, we describe the detailed architecture of each MEGA block, including feed-forward and normalization layers ( §3.3), and present MEGA-chunk, a variant of MEGA that splits input sequences into fixed chunks, reducing time and space complexity from quadratic to linear ( §3.4). At last, we provide theoretical justification for the design of single-head gated attention ( §3.5).

3.1. MULTI-DIMENSIONAL DAMPED EMA

Damped EMA. Previous studies (McKenzie & Gardner Jr, 2010; Svetunkov, 2016) have shown that relaxing the coupled weights of the previous and current observations (α vs. 1α in (2)) produces robust dependency modeling. Inspired by this, MEGA allows the damping of the influence of the previous time step: y t = α ⊙ x t + (1 -α ⊙ δ) ⊙ y t-1 , where δ ∈ (0, 1) d is the damping factor. Multi-dimensional Damped EMA. To further improve the expressiveness of EMA, we introduce a multi-dimensional variant of EMA. Concretely, we first expand each dimension j ∈ {1, 2, . . . , d} of the input sequence X individually into h dimensions via an expansion matrix β ∈ R d×h : u (j) t = β j x t,j where β j ∈ R h is the j-th row of β, u (j) t ∈ R h is the expanded h-dimensional vector for the j-th dimension at timestep t. Correspondingly, we extend the shape of α and δ from a one-dimensional vector to a two-dimensional matrix, i.e. α, δ ∈ R d×h , where α j , δ j ∈ R h denote the j-th row of α and δ, respectively. Then, for each dimension j, the damped EMA is applied to the h-dimensional hidden space: h (j) t = α j ⊙ u (j) t + (1 -α j ⊙ δ j ) ⊙ h (j) t-1 y t,j = η T j h (j) t (5) where h

(j) t

∈ R h is the EMA hidden state for the j-th dimension at timestep t. η ∈ R d×h is the projection matrix to map the h-dimensional hidden state back to 1-dimensional output y t,j ∈ R. η j ∈ R h is the j-th row of η. The output Y from ( 5) is denoted as Y ∆ = EMA(X). Because we do not need to explicitly compute h (j) t to get the output y t,j , and the time and space complexity is similar to the standard EMA in (2) (see Appendix A for the details). < l a t e x i t s h a 1 _ b a s e 6 4 = " e l 2 k j u N X N N 2 b Q y u J n h 8 m m S f 2 t m s = " > A A A y K 3 i c l V v J k t v I E e W M t 7 G 8 z d h H X x B u K W b s a H W I G o X t 4 / S + s b v Z + 6 L W K E A w C U K N T a g i 2 C 0 G / R e + 2 l / g r / H J D l / 9 H 8 6 q Q i G z 0 K A U V s R I q P e y E r W 8 r E y A m E E e R 0 K + e P G v z z 7 / w Q 9 / 9 O O f f P H T J z / 7 + S 9 + + a s v v / r 1 h c g m R Q D n Q R Z n x d X A F x B H K Z z L S M Z w l R f g J 4 M Y L g d 3 6 4 q / L K E Q U Z a e y Y c c 3 i R + m E a j K P A l Q q 9 v E 1 + O B 6 P Z / d f z t 1 8 u v V h 5 o f 9 4 j y + 6 1 c V S p / r T f / t V 9 8 n t M A s m C a Q y i H 0 h X n d f 5 P L N z C 9 k F M Q w f 3 I 7 E Z D 7 w Z 0 f w m u 8 T P 0 E x J u Z H v P c e 4 b I 0 B t l B f 6 X S k + j v M f M T 4 R 4 S A Z o q c Y o m p w C F 3 H K o 1 g e J I 0 h y N G f 3 8 y i N J 9 I S A M z g t E k 9 m T m q Z X x h l E B g Y w f 8 M I P i g g n 4 Q V j v / A D i e v n 3 O L e z O H J k 2 f e g V / c e Q L 7 4 W o K L x t 5 g Z + b a z W 1 A k Z Q F F E a q p s M o z I S 1 m w U h Z M C 0 G 0 K 0 y B L E j 8 d z m 4 R j G E k 5 7 P Z L S T e N z 2 8 / v 0 c b 9 M 0 C n D F o b B m 6 7 q l D J t 2 R R S O a 2 8 n q t F m J b P c 2 p x l e Z v F I J M y S 6 z R m m 4 9 s q s m 7 l s z / / H Q K 5 O B N R k s c h J Y i 2 C R x d B a D P V t n n k 7 O L 1 Y T d H z P b R X 2 w g j D I q h h 4 u T u D 7 w W o H z 1 9 0 3 6 G U w 8 p a 6 y g l 6 2 d L b Y v Y N R Q L L X p x N o X g e Y I i t o N x h p N c V R k v d m d n C v 9 x i a 2 Y c t P X H 8 U b S j 1 e 8 L d S D k B g c a v u F 2 j P k j c s t 6 3 K r 6 V L T c p r Z m y 6 9 r G 4 r P G v k 4 Z S q x k v b 4 / 3 E H 1 K X p W + X X j 3 q t l z 3 s V f f c l e v 0 N U z 7 9 Q I e 9 F 6 m J u h + s 3 o q z B g w 2 / 1 Y F f E 9 D 6 1 v U 9 b e p / Y X j p I p 1 k d a S v 1 y p i 7 C 7 0 0 d R w u W J u m w 3 E B w F z y 6 Q i z c I 9 d 0 r o x 5 9 8 + d u 6 n H u A 2 q M 4 t I o L 3 Z t L W Z P G s H T + T P I f C U 3 6 M m 8 3 K z W a b m 1 W v 8 K e 0 8 A 1 n z 5 8 / 9 8 s s G n o T o c 6 n a O T l m R A R 5 o 1 q H f L Y x / C p b r B 4 e O q M z D G a W i a p G N O 9 s v m / Z 1 k 5 W q 8 d r X / S E U 4 6 D U G f x M Z W V P P R e D 0 k V I v l r a / n z x c q B Y f n x 2 G G S W G c t E w U O T O 8 2 u i j M 2 W u H k 1 1 1 b p a b X F l N W / v h 5 O o f S 0 U v e l 1 5 v R a / W S v R 8 u a 4 8 l V T Z 0 J U K F m v O r q Y 9 t i + j c F 3 K / 7 9 9 3 + d q r 1 D X D U 6 n r x N C v N Q R Q r v c b q A s 9 1 t F B X l c N R n G W F p v W V 4 f V l Z Y D U I J l 1 m 0 l H F h g L 8 5 m u l g I / n m 0 0 D U o / j o b c 4 K 2 5 L p K Z o e a P X I K Q 7 R 0 0 M 6 c p Q S 5 U r s t F F G e p h t X q o p M s 8 U q / i H y M W V F p H K Q / U 7 7 v Z Z o V C f p 9 e o v Q 0 7 l d 0 a J B + 8 Q M X G Z A T O A y A T F D l x n O a 7 0 V 4 F J A n U Y u M y I m d J m Q m L H L j I m J X C Y i 5 p 3 L v G O D u 3 O p O + o U u 0 w 8 1 1 o u E i 8 S G L d Y W Q 8 f 1 J l n d n H Z e z c R 0 h t m 6 d f S U + U t a v J B H U D O 1 n h J 5 T t 1 f a d 0 1 8 x l M m J y l 8 n Z J N 6 7 1 H v q V L h M Q Y x w G c H c S Z e S 1 G n i M h N i S p c p i Z m 6 z J S Y e 5 e 5 J + b B Z R 7 Y 4 D 6 4 1 I e 5 K b d s F G C S z g o b A 2 U V K r O Z f b q h 4 K k H L s c m V K y F b r M b l o x k Q V I O C G Y R U g Y E s / A o h w Q P G Q w E s 7 g o R w S P + F B C w l l I l G O C W T y U E 4 J Z M J T v C H 7 H 4 D u C W R S U M c E x g x O C E z 5 C t t Z 8 k T O C m a T L n O C c w e 8 J Z l o u C 4 K Z k E t B s O D 7 S r D k I 2 S r w v V b E s z E W 0 4 J Z s o t 7 w l m s i 0 f C H 5 g 8 A e C r V 4 3 Y 1 B P y / o J s H C 1 W 5 3 1 R n i t J z Q Y 9 b W e 0 W A k 2 H p K g 9 F h + z k N R o 2 t J z U Y S b a e 1 W B k 2 X p a g 9 F m 6 3 m N 3 M I T G 4 x K m 2 e 2 5 R Y e 2 m D 0 2 j y 2 L Z e 4 X O J M v u 1 M r s m F x z I Y E T c P Z s u 1 n c w 1 u f B w B i P q 5 v F s u Y X n M x h 1 t 5 7 Q Y C T e e k a D 0 X n r K Q 1 G 7 K 3 n N B j F t 5 / U Y H S / + K z G i C i i o C 5 Y k l W K k l U W s M k a 4 W s U V M k 6 w e s M 3 i B 4 g 8 G b B G 8 y e I v g L Q Z v E 7 z N 4 B 2 C d x i 8 S / A u H / g e 4 X v M f J / g f Q b 3 C O 4 x + I D g A w Y f E n z I 4 C O C j / h Q + o T 3 m f k x w c c M P i H 4 h M G n B J 8 y + I z g M w a f E 3 z O 4 A u C L / g I L w m / Z O Z X B F 8 x + J r g a w b f E H x j 6 u b W k 9 Z V H h j p M a G u M o V r + T F u j X P r L r f O u Q 2 X 2 + D c p s t t c m 7 L 5 b Y 4 t + 1 y 2 z z a t D o Z u c M 7 7 r r c L u f 2 X G 6 P c / s u t 8 + 5 n s v 1 O H f g c g e c O 3 S 5 Q 2 c S R y 5 5 x D v 2 X a 7 P u W O X O + b c i c u d c O 7 U 5 U 6 d w Z y 5 5 B n v e O 5 y 5 5 y 7 c L k L z l 2 6 3 C X n r l z u i n P X L n f N u R u X q 6 V / w a v h 8 g P o p w t 8 q n 1 R d y 6 z F G b 2 S d d i y c R A t w m l j 7 p M V n h d I h u q g h k y M M i A V l M X J Q h R M a J L E U S o H C 6 r o V D d o a s O R K j a 0 L U G I l R j 6 A o D E a o s d F 2 B S M R u / 8 5 A V E b o I g I R K h 5 0 6 Y B I z F b C I A k h q U F S t o I G o Z J A F w S I U C G g y w B E q J j V u R 8 R y v k 6 4 y P C H s R 0 o k e I E n x Z 7 Q z b l 9 I g l M x 1 K k e E U r h O 4 I h Q 4 t Z p G x E q U n W u R u R D 2 7 H p P F V B 6 c f 5 W O 2 5 / r e W Y T m o B K K 1 Y U F 6 A q P X F h U V + 8 l g q H q Y C y K y B E K F 6 3 8 J 1 r J U k r Q B W q J H h P B v h o k o T F R n / S 9 1 t v K t p F t P Z T b j M 5 g p y d o W K j a g F q p 1 y K Y 1 U y q 1 L V T p i F q o 0 H B O 7 2 y U P s f E 4 o g j a q E s 3 7 H B o y b v 2 A L N l B b r 6 c + U D m 0 L V 5 Q t J W o w o x b q L 6 c W a u 8 9 t V B 3 B V + t m R J c v U g z p T X b w u W e U A t 1 V l I L N T a l F u r r n l q o r Q d q o a 4 + z K s f y D D r 3 l e 3 1 i k X 9 U a p V i d a R N Y o A H R + R Y j y q s 6 q i F A 2 1 b k U E c q h O o M i Q p l T 5 0 1 E q J b T y R I R S p I 6 R S J C q V E n R k Q o I e p 0 i A i l Q Z 0 E E a H k p 1 M f I p T y d M J D h O o 0 n e U Q Y S W a T m 4 I U V L T K Q 0 R S m U 6 k S F C C U y n L 0 S o H N M 5 C 5 E z 5 v r c Q J S i d I J C h B K T T k u I U D r S y Q g R S k I 6 B S F y z V z f G O i G 7 S K l i Q H P E k l / X B 3 L t 3 j F F t C e A o r p V S d B P b 0 q m B V 3 a g L a n E 5 n k A r 1 K / E G B L F f A E p r v K p O I 7 y l q Q D F K F I v V S E N s m G U h u j N n 8 Q K E a P 6 O p n P h H o f f A p y k Y N B F g 8 / 5 W Z w P 8 d Y f N J 8 q Z s K / a u i S a S V Q / 1 G u 5 q c N E V n K l g M y D W L U Z k p 1 y 2 2 T g s v N y x I k S A 3 L U a x I L c s R t E g t y 1 G 8 S B 3 L E Y R I X c t R j E h 9 y x G U S H 3 L U Z x I X s W 6 7 F B H 1 i Q g k M e W o z C Q x 5 Z j M o / 2 b d Y n z k 8 t i A F i T y x G I W J P L U Y B Y o 8 s x i V d f L c Y h Q r 8 s J i F C 3 y 0 m I U L / L K Y h Q x 8 t p i L G b k j Q W r O g 3 l v F 3 4 + d j Q o X 0 M D p w H k X C N w S S O c J 3 B d E y G G w w m h Y S b D C a R h F s M J p 2 E 2 w w m q Y Q 7 D C a 1 h L s M J s G E e w w m z Y T 7 D N 6 n x Q l 7 D K d D N T x g M I k n P G Q w 6 S c 8 Y j B J K O w z m M 7 Z 8 J j B x 2 w o J w w n L Y W n D C Y 5 h W c M J k W F 5 w w m U Y U X D G a P w e E l w 0 l b 4 R W D S V 7 h N Y P p g S C 8 Y b D R m P r 1 X V b 1 m 6 j f s w z 4 e x a x R j D p S 6 w T S v I S G 4 R q d T 3 z N v Q v H x M B n u 8 J k B 7 e O 4 a h t 7 n s D S D w F S 7 H k f C m 2 S Q e I o Q t 8 I T + n Q Q r z E n h q Q 9 8 s h g d q Y 9 l 4 D 7 H i l P / B l z 9 G C q 2 6 I 6 k U L F N K A l U 7 B B K + h S 7 h L K 3 N G K P Y J K n 2 C e U D j X R I 5 S 0 K Q 4 I P W C O D w k m a Y o j Q k m Z o k 8 o C V M c E 0 r n m z g h l F Q p T g l l z 7 D i j G A S p T g n l D Q p L g i l o 0 5 c E n r J H F 8 R T I I U 1 4 S S H s U N o V a O m y n W g q A f L n z z O u a + K h i p I O i 5 j w W q Y l y l F s p 1 j V o o 0 3 U q a 1 W V u E E k n n q b 1 E I l b V E L F b R N L V T O D r V Q M b v U Q q H s U Q s F s k 8 t F E a P W i i I A z Y Y F M I h k S i A I 2 r h x v e Z K e 7 4 M Z G 4 0 y f U w h 0 + p R Z u 7 B m 1 c E P P q Y U b e U E t 3 M B L a u G + X V E L 9 + u a 3 R 4 3 6 o b d s C q + q s J L b R v w b Z O m B l O H i w p k / f E e R r O B l 7 1 p J M f Z R H p Y A H l T z G 8 5 F G 6 J B F Q j O f V R d X 9 Z C 0 E b P i o O Q R d Q 0 K i g Q J d Q 0 K i h Q B d R Q F V U h W 7 U K H s 9 p + s o a B R S o C s p a J R S o G s p a B R T o K s p o H L K w r s 1 z F 7 A 6 Z I K G j U V 6 K I K G l U V 6 L I K q K 6 q 0 I M a Z S / a d G U F j d I K d G 0 F V F x Z u F / D f Q 4 f 1 z B 7 o 6 Z L L G j U W K C L L G h U W a D L L G j U W a A L L W h U W q B L L W j U W q C L L W h U W 6 D L L a B 6 y 8 L X N c z e k O m S C 3 j N h U 8 Q m I B k M Q F v k g 6 h i B / U p 0 9 D X / p e C C k U m H t U O x K o 9 8 F E J S J X u 7 k y n c / y t 7 P b I p n p h k 6 D y i s k e V R E m A C d / v V H i I M H n f z 0 h y T q J p g t G 7 7 t N y Z j X + K D v H s L x 7 L P L f v z t s E k 2 R D i j 0 1 E G 9 Q z M a 1 5 8 2 m m X 1 n 1 G 1 a N c U f x E C r L W 9 2 o h 1 / 3 w M N C Z s H Y F + q j W n 8 i M / 1 o B Y U z x M b H s L m x q Q d Z d X k 8 g C E 4 d q b Z Y l c g g U e P t T N N F E N g f i h 2 r W M / j / 0 A 5 v V X O b 0 K m H v P v O r a X W C 3 / + a 8 z n 6 b z Z H 0 B P v w p 9 d k T + Y 8 0 z f c o s r Y K j d 6 x s X c v p 5 z i Q L C e f 3 C r U k F k i a p W t E o U q v n m o l s J B P / n i w t 0 L T D r J H p L 6 H M m 7 j H X v J 4 o q b / Q b 0 l a E x u v z f n 3 0 H t 9 x 7 t 4 Y V f 0 B B U o 3 k D i f / 4 B W 5 / k T H L 0 0 d b s J 6 V R K u G T n S X W T w q / E S 9 s x p P s w J r V u E / C O 9 p 7 / u X T 9 U H Q P p r 9 U l q v m g V O U p A 6 K / Q n t 5 C H D M b + + b 0 m b e G q R D j P l V / P W D Q Q 6 K + h l O F s X H K r N X n q N k k 1 N l T 1 8 m R h G X t X m T e M A P l b h r d R T k M I 3 + l 8 U F z V i S x e t M / n / W + f z F v I b M U F N d t 4 + R U 9 3 v Z 3 A 9 N 5 o r K W 7 p p O f S + v 4 3 S k X x o d s 3 9 Q r 1 G x r P D V w F z C n j i C j 8 E L 0 q 9 N K u K f A n 3 K 9 7 6 O B N q f T J V E A Z j b w O f i V P 4 W n i D L L t b U W H J 3 v Y c 5 e q Q z o o / o M 6 L U I 8 A / 7 1 d V l c f M 1 T H p T G M 9 D u b F k u t W D T T f y + w O E N N n a l P B W O Q t / 4 A Y y 3 O p o M C / L s n b 7 9 c 6 j b / X 4 n H F x c v V 7 p / X H l 1 / G r p u 7 X q / 6 P 4 o v P b z u 8 6 3 3 S 6 n T 9 1 v u v s d P q d 8 0 7 Q y T p / 7 f y t 8 / f u P 7 r / 7 P 6 7 + x 9 j + v l n V Z / f d J w / 3 f / + D 3 y c U 5 U = < / l a t e x i t > x 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 g + g L f V o a w Q t 1 9 v U f Z I 7 Y m K 0 5 f I = " > A A A y K n i c l V t Z k 9 v G E a a d y 1 E u O 3 n M C y o r l Z 3 U a m s p q 5 I 8 e u + L u 8 u 9 D 6 + s A s E m C C 0 u Y Y b g r l j M r 8 h r 8 g v y a / L m y m t + S H p m M O g e L C h X V G U J 8 3 0 9 j T m + n m 6 A 8 C C P I y F X V 7 / / 5 N M f / f g n P / 3 Z Z z 9 / 9 o t f / u r X v / n 8 i 9 9 e i m x S B H  A R Z H F W X A 9 8 A X G U w o W M Z A z X e Q F + M o j h a n C / o f i r E g o R Z e m 5 f M z h T e K H a T S K A l 8 i d H u X + H I 8 G M 0 e 5 m 8 / X 1 p d W d V / v K c X 3 e p i q V P 9 6 b / 9 o v v s b p g F k w R S G c S + E N 9 2 V 3 P 5 Z u Y X M g p i m D + 7 m w j I / e D e D + F b v E z 9 B M S b m R 7 y 3 H u B y N A b Z Q X + l 0 p P o 7 z H z E + E e E w G a K m G K J q c A h d x y q N Y H i S N I c j R X 9 / M o j S f S E g D M 4 L R J P Z k 5 q m F 8 Y Z R A Y G M H / H C D 4 o I J + E F Y 7 / w A 4 n L 5 9 z i w c z h 2 b M X 3 q F f 3 H s C + + F i C i 8 b e Y G f m 2 s 1 t Q J G U B R R G q q b D K M y E t Z s F I W T A t B t C t M g S x I / H c 7 u E I x h J O e z 2 R 0 k 3 l c 9 v P 7 j H G / T N A p w x a G w Z h u 6 p Q y b d k U U j m t v p 6 r R Z i W z 3 N q c Z 3 m b x S C T M k u s 0 b p u P b G r J u 5 b M / / p 0 C u T g T U Z L H I S W I t g k c X Q W g z 1 b V 5 4 u z i 9 W E 3 R 8 z 2 0 V 9 s I I 4 y J o Y e L k 7 g + 8 F q B 8 2 + 7 b 9 D L Y O Q t d Z U T 9 L K t t 8 X s G 4 o E l r 0 4 m 0 L x M s A I W 0 G 5 w 0 i v K 4 y W u j O z h X + 7 w 9 b M O G j r j + O N p B + v e N u o B y E x O N T 2 C 7 V n y B u X 2 9 b l d t O l p u U 0 s z d d e l X d V n j W y M M p V Y 1 X t s f 7 i T + k L k t f L 7 1 + 0 m 2 5 7 m O v v u a u X q O r F 9 6 Z E f a i 9 T A 3 Q / W b 0 V d h w I b f 6 s G u i O l 9 Z n u f t f Q + t b 1 0 k E 6 z O t J W 6 p U x d x d 6 a e o 4 X L A 2 T Y f j A o C 5 5 N M R Z u G e u q R 1 Y 8 6 / f u r c T z 3 A b V C d W 0 Q E 7 8 2 k r c n i W T t + J n k O h a f 8 G D d b l Z u t N j d r X u F P a e E b z l 6 + f O m X W T T 0 J k K d T 9 H I y z M h I k w b 1 T r k s Y / h U 9 1 g 8 f D U G Z l j N L V M U j G m e 2 X z f 8 + y c r R R O 9 r 4 Q U c 4 6 T Q E f R I b W 1 H N R + P 1 k F A t l r e + X r 5 c q B Q c n h + H G S a F c d I y U e T M 8 G q j j 8 6 U u X o y 1 T X r a q 3 F l d W 8 v R 9 O o v a 1 U P S m 1 7 n T a + 0 H e z 1 Z 1 h x P r m r q T I A K N e N V V x / b F t O / K e B + 3 b / v 9 r d T r W + A o 1 b X i 6 d Z a Q 6 i W O k 1 V h d 4 r q O F u q o c j u I s K z S t r w y v L y s D p A b J r N t M O r L A W J j P d L E U + P F s s 2 l Q + n E 0 5 A Z v z X W R z A w 1 f + I S h G z v o J k 5 T Q l y o X J d L q I 4 S z W s V h e d Z I l X + k X k Y 8 y K S u M g / Z n y / S D T r E j Q 7 / M 7 h J 7 P 7 Y o W D d o n Z u A y A 2 I C l w m I G b r M c F 7 r r Q C X A u o 0 c p k R M a H L h M S M X W Z M T O Q y E T H v X O Y d G 9 y 9 S 9 1 T p 9 h l 4 r n W c p F 4 k c C 4 x c J 6 + K j O P L O L y 9 6 7 i Z D e M E u / l J 4 q b 1 G T j + o A c r b G S y r f q e s 7 p b t m L p M R k 7 t M z i b x 3 q X e U 6 f C Z Q p i h M s I 5 k 6 6 l K R O E 5 e Z E F O 6 T E n M 1 G W m x D y 4 z A M x j y 7 z y A b 3 w a U + z E 2 5 Z a M A k 3 R W 2 B g o q 1 C Z z e z D D Q V P P X A 5 N q F i L X S b 3 b B k J A u S c k A w i 5 A y I J i F R z k k e M h g I J j F R T k i e M S H E h L O Q q I c E 8 z i o Z w Q z I K h f E f w O w b f E 8 y i o I w J j h m c E J z w E b K 1 5 o u c E c w k X e Y E 5 w x + T z D T c l k Q z I R c C o I F 3 1 e C J R 8 h W x W u 3 5 J g J t 5 y S j B T b v l A M J N t + U j w I 4 M / E G z 1 u h W D e l r W T 4 C F q 9 3 q r D f C a z 2 h w a i v 9 Y w G I 8 H W U x q M D t v P a T B q b D 2 p w U i y 9 a w G I 8 v W 0 x q M N l v P a + Q W n t h g V N o 8 s y 2 3 8 N A G o 9 f m s W 2 5 x O U S Z / J t Z 3 J N L j y W w Y i 4 e T B b r u 1 k r s m F h z M Y U T e P Z 8 s t P J / B q L v 1 h A Y j 8 d Y z G o z O W 0 9 p M G J v P a f B K L 7 9 p A a j + 8 V n N U Z E E Q V 1 w Z K s U Z S M l n h d Y l s q A p m y M A g A 1 p N X Z Q g R M W I L k U Q o X K 4 r I Z C d Y e u O h C h a k P X G o h Q j a E r D E S o s t B 1 B S I R u / 0 7 A 1 E Z o Y s I R K h 4 0 K U D I j F b C Y M k h K Q G S d k K G o R K A l 0 Q I E K F g C 4 D E K F i V u d + R C j n 6 4 y P C H s Q 0 4 k e I U r w Z b U z b F 9 K g 1 A y 1 6 k c E U r h O o E j Q o l b p 2 1 E q E j V u R q R D 2 3 H p v N U B a U f 5 2 O 1 5 / r f W o b l o B K I 1 o Y F 6 Q m M X l t U V O w n g 6 H q Y S 6 I y B I I F a 7 / J V j L U k n S B m i J H h H C v x k m o j B R n f W / 1 N n K t 5 J u P Z X Z j M 9 g p i R r W 6 j Y g F q o 1 i G b 1 k y p 1 L Z Q p S N q o U L D O b 2 z U f o c E 4 s j j q i F s n z H B o + a v G c L N F N a r K c / U z q 0 L V x R t p S o w Y x a q L + c W q i 9 9 9 R C 3 R V 8 t W Z K c P U i z Z T W b A u X e 0 I t 1 F l J L d T Y l F q o r w d q o b Y e q Y W 6 + j C v f i D D r P t Q 3 V q n X N Q b p V q d a B F Z p w D Q + R U h y q s 6 q y J C 2 V T n U k Q o h + o M i g h l T p 0 3 E a F a T i d L R C h J 6 h S J C K V G n R g R o Y S o 0 y E i l A Z 1 E k S E k p 9 O f Y h Q y t M J D x G q 0 3 S W Q 4 S V a D q 5 I U R J T a c 0 R C i V 6 U S G C C U w n b 4 Q o X J M 5 y x E z p n r C w N R i t I J C h F K T D o t I U L p S C c j R C g J 6 R S E y A 1 z f W u g W 7 a L l C Y G P E s k / X F 1 L N / h F V t A e w o o p l e d B P X 0 q m B W 3 J k J a H M 6 n U M q 1 K / E m x D E f g E o r f G a O o 3 w l q Y C F K N I v V S F N M i G U R q i N 3 8 S K 0 S M 6 u t k P h P q f f A Z y E U O B l k 8 / C E 3 g 4 c 5 x u K z 5 k v d V O h f F U 0 i r R z q N 9 r V 5 K Q p O l P B Y k C u W 4 z K T L l h s Q 1 a e L l p Q Y o E u W U x i g W 5 b T G K B r l j M Y o H u W s x i g i 5 Z z G K C b l v M Y o K e W A x i g v Z s 1 i P D f r Q g h Q c 8 s h i F B 7 y 2 G J U / s m + x f r M 4 Y k F K U j k q c U o T O S Z x S h Q 5 L n F q K y T F x a j W J G X F q N o k V c W o 3 i R 1 x a j i J E 3 F m M x I 2 8 t W N V p K O e d w s / H h g 7 t Y 3 D g P I i E 6 w w m c Y Q b D K Z j M t x k M C k k 3 G I w i S T c Z j D p J N x h M E k l 3 G U w q S X c Y z A J J t x n M G k m P G D w A S 1 O 2 G M 4 H a r h I Y N J P O E R g 0 k / 4 T G D S U J h n 8 F 0 z o Y n D D 5 h Q z l l O G k p P G M w y S k 8 Z z A p K r x g M I k q v G Q w e w w O r x h O 2 g q v G U z y C m 8 Y T A 8 E 4 S 2 D j c b U r + + y q t 9 E / Z 5 l w N + z i H W C S V 9 i g 1 C S l 9 g k V K v r h b e p f / m Y C P B 8 T 4 D 0 8 N 4 x D L 2 t Z W 8 A g a 9 w O Y 6 E N 8 0 m 8 R A h b I E n 9 O 8 k W G F O C k 9 9 4 J P F 6 E h 9 L A M P O V a c + j f g 6 s d Q s U 1 3 J I W K H U J J o G K X U N K n 2 C O U v a U R + w S T P M U B o X S o i R 6 h p E 1 x S O g h c 3 x E M E l T H B N K y h R 9 Q k m Y 4 o R Q O t / E K a G k S n F G K H u G F e c E k y j F B a G k S X F J K B 1 1 4 o r Q K + b 4 m m A S p L g h l P Q o b g m 1 c t x K s R Y E / X D h m 9 c x D 1 X B S A V B z 3 0 s U B X j G r V Q r u v U Q p l u U F m r q s R N I v H U 2 6 I W K m m b W q i g H W q h c n a p h Y r Z o x Y K Z Z 9 a K J A D a q E w e t R C Q R y y w a A Q j o h E A R x T C z e + z 0 x x x 0 + I x J 0 + p R b u 8 B m 1 c G P P q Y U b e k E t 3 M h L a u E G X l E L 9 + 2 a W r h f N + z 2 u F G 3 7 I Z V 8 V U V X m o I 2 v B J c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 s 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e q 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 i h 7 0 a Y r K 2 i U V q B r K 6 D i y s L 9 G u 5 z + K S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q l h 9 o Z M l 1 z A a y 5 8 g s A E J I s J e J N 0 C E X 8 q D 5 9 G v r S 9 0 J I o c D c o 9 q R Q L 0 P J i o R u d r N l e l 8 l r + d 3 R X J T D d 0 G l R e I c m j I s I E 6 P S v P 0 I c P O r k p z 8 k U T f B b N n w b b 8 x G f s S H + T d W z i W f W 7 Z n 7 c N J s m G E H 9 s I t q g n o l p z Z t P M / 3 K q t + w a o w 7 i o d Q W d 7 p R j 3 8 u g c e F j I L x r 5 Q H 9 X 6 E 5 n p R y s o n C E 2 P o b N j U 0 9 y K r L 0 w E M w b E z z R a 7 A g k 8 e q y d a a I Y A v N D s W s d + 3 n s B z C v v 8 r p V c D c e + F V 1 + 4 C u / 2 3 5 n X 2 2 2 q O p C f Y h z + 9 J n s 6 5 5 m + 4 R Z V x l a 5 0 T M u 5 v b 1 n E s U E M 7 r F 2 5 N K p A 0 S d W K R p F a P d d M Z C O Z + A 9 k a Y G m H W a N T H 8 J Z d 7 E P f W S x x M 1 / Q / q L U F j c g e 9 O f 8 O 6 q D 3 Z A 8 v / Y K G o B r N G 0 j 8 x y 9 w + 4 u M W Z 4 9 2 Y K N r C R a N X S i u 8 r i U e E n 6 p 3 V e J o V W L M K / 1 F 4 z 3 v f v X q u P g D S X 6 t P U v N F q 8 h R A k J / h f b 8 D u K Y 2 d g 3 p y + 8 d U y F G P e p + u s R g x 4 S 9 T W c K o y N U 2 a t P k f N J q H O n r p O j i Q s a / c i 8 4 Y Z K H f T 6 D 7 K Y R j 5 K 4 0 P m r M i i d W b / v m s 9 9 3 q v I X M U l B c t 4 2 T U 9 3 v V X M / N J k r K m / p p u X Q + + 4 u S k f y s d k 1 9 w v 1 G h n P D l 8 F z B n g i S v 8 E L w o 9 d K s K v I l P K x 4 G + N M q P X J V E E Y j L 1 N f C Z O 4 U v h D b L s f k W F J X v b c 5 y r Q z o r / o Q 6 L 0 I 9 A v z 3 b l l d f c x Q H Z f G M N L v b F o s t W L R T P + 9 w O I c N X W u P h W M Q d 7 5 A 4 y 1 O J s O C v D v n 7 3 9 f K n b / H 8 l n l 5 c v l r p / n n l 9 c n r p W / W q / + P 4 r P O 7 z t / 6 H z V 6 X b + 0 v m m s 9 v p d y 4 6 Q S f t / L 3 z j 8 4 / u / / q / r v 7 f f c / x v T T T 6 o + v + s 4 f 7 r / / R + P Q l N k < / l a t e x i t > x Dense SiLU Dense SiLU < l a t e x i t s h a 1 _ b a s e 6 4 = " J p Q n j I h 7 n 8 R p J C W 2 Y B x I F M 4 w 5 4 g = " > A A A y K n i c l V t Z k 9 v G E a a d y 1 E u O 3 n M C y o r l Z 3 U a m s p q 5 I 8 e u + L u 8 u 9 D 6 + s A s E m C C 0 u Y Y b g r l j M r 8 h r 8 g v y a / L m y m t + S H p m M O g e L C h X V G U J 8 3 0 9 j T m + n m 6 A 8 C C P I y F X V 7 / / 5 N M f / f g n P / 3 Z Z z 9 / 9 o t f / u r X v / n 8 i 9 9 e i m x S B H  A R Z H F W X A 9 8 A X G U w o W M Z A z X e Q F + M o j h a n C / o f i r E g o R Z e m 5 f M z h T e K H a T S K A l 8 i d H u X + H I 8 G M 2 O 5 2 8 / X 1 p d W d V / v K c X 3 e p i q V P 9 6 b / 9 o v v s b p g F k w R S G c S + E N 9 2 V 3 P 5 Z u Y X M g p i m D + 7 m w j I / e D e D + F b v E z 9 B M S b m R 7 y 3 H u B y N A b Z Q X + l 0 p P o 7 z H z E + E e E w G a K m G K J q c A h d x y q N Y H i S N I c j R X 9 / M o j S f S E g D M 4 L R J P Z k 5 q m F 8 Y Z R A Y G M H / H C D 4 o I J + E F Y 7 / w A 4 n L 5 9 z i w c z h 2 b M X 3 q F f 3 H s C + + F i C i 8 b e Y G f m 2 s 1 t Q J G U B R R G q q b D K M y E t Z s F I W T A t B t C t M g S x I / H c 7 u E I x h J O e z 2 R 0 k 3 l c 9 v P 7 j H G / T N A p w x a G w Z h u 6 p Q y b d k U U j m t v p 6 r R Z i W z 3 N q c Z 3 m b x S C T M k u s 0 b p u P b G r J u 5 b M / / p 0 C u T g T U Z L H I S W I t g k c X Q W g z 1 b V 5 4 u z i 9 W E 3 R 8 z 2 0 V 9 s I I 4 y J o Y e L k 7 g + 8 F q B 8 2 + 7 b 9 D L Y O Q t d Z U T 9 L K t t 8 X s G 4 o E l r 0 4 m 0 L x M s A I W 0 G 5 w 0 i v K 4 y W u j O z h X + 7 w 9 b M O G j r j + O N p B + v e N u o B y E x O N T 2 C 7 V n y B u X 2 9 b l d t O l p u U 0 s z d d e l X d V n j W y M M p V Y 1 X t s f 7 i T + k L k t f L 7 1 + 0 m 2 5 7 m O v v u a u X q O r F 9 6 Z E f a i 9 T A 3 Q / W b 0 V d h w I b f 6 s G u i O l 9 Z n u f t f Q + t b 1 0 k E 6 z O t J W 6 p U x d x d 6 a e o 4 X L A 2 T Y f j A o C 5 5 N M R Z u G e u q R 1 Y 8 6 / f u r c T z 3 A b V C d W 0 Q E 7 8 2 k r c n i W T t + J n k O h a f 8 G D d b l Z u t N j d r X u F P a e E b z l 6 + f O m X W T T 0 J k K d T 9 H I y z M h I k w b 1 T r k s Y / h U 9 1 g 8 f D U G Z l j N L V M U j G m e 2 X z f 8 + y c r R R O 9 r 4 Q U c 4 6 T Q E f R I b W 1 H N R + P 1 k F A t l r e + X r 5 c q B Q c n h + H G S a F c d I y U e T M 8 G q j j 8 6 U u X o y 1 T X r a q 3 F l d W 8 v R 9 O o v a 1 U P S m 1 7 n T a + 0 H e z 1 Z 1 h x P r m r q T I A K N e N V V x / b F t O / K e B + 3 b / v 9 r d T r W + A o 1 b X i 6 d Z a Q 6 i W O k 1 V h d 4 r q O F u q o c j u I s K z S t r w y v L y s D p A b J r N t M O r L A W J j P d L E U + P F s s 2 l Q + n E 0 5 A Z v z X W R z A w 1 f + I S h G z v o J k 5 T Q l y o X J d L q I 4 S z W s V h e d Z I l X + k X k Y 8 y K S u M g / Z n y / S D T r E j Q 7 / M 7 h J 7 P 7 Y o W D d o n Z u A y A 2 I C l w m I G b r M c F 7 r r Q C X A u o 0 c p k R M a H L h M S M X W Z M T O Q y E T H v X O Y d G 9 y 9 S 9 1 T p 9 h l 4 r n W c p F 4 k c C 4 x c J 6 + K j O P L O L y 9 6 7 i Z D e M E u / l J 4 q b 1 G T j + o A c r b G S y r f q e s 7 p b t m L p M R k 7 t M z i b x 3 q X e U 6 f C Z Q p i h M s I 5 k 6 6 l K R O E 5 e Z E F O 6 T E n M 1 G W m x D y 4 z A M x j y 7 z y A b 3 w a U + z E 2 5 Z a M A k 3 R W 2 B g o q 1 C Z z e z D D Q V P P X A 5 N q F i L X S b 3 b B k J A u S c k A w i 5 A y I J i F R z k k e M h g I J j F R T k i e M S H E h L O Q q I c E 8 z i o Z w Q z I K h f E f w O w b f E 8 y i o I w J j h m c E J z w E b K 1 5 o u c E c w k X e Y E 5 w x + T z D T c l k Q z I R c C o I F 3 1 e C J R 8 h W x W u 3 5 J g J t 5 y S j B T b v l A M J N t + U j w I 4 M / E G z 1 u h W D e l r W T 4 C F q 9 3 q r D f C a z 2 h w a i v 9 Y w G I 8 H W U x q M D t v P a T B q b D 2 p w U i y 9 a w G I 8 v W 0 x q M N l v P a + Q W n t h g V N o 8 s y 2 3 8 N A G o 9 f m s W 2 5 x O U S Z / J t Z 3 J N L j y W w Y i 4 e T B b r u 1 k r s m F h z M Y U T e P Z 8 s t P J / B q L v 1 h A Y j 8 d Y z G o z O W 0 9 p M G J v P a f B K L 7 9 p A a j + 8 V n N U Z E E Q V 1 w Z K s U Z S s s Y B N 1 g l f p 6 B K N g j e Y P A m w Z s M 3 i J 4 i 8 H b B G 8 z e I f g H Q b v E r z L 4 D 2 C 9 / j A 9 w n f Z + Y H B B 8 w u E d w j 8 G H B B 8 y + I j g I w Y f E 3 z M h 9 I n v M / M T w g + Y f A p w a c M P i P 4 j M H n B J 8 z + I L g C w Z f E n z J R 3 h F + B U z v y b 4 m s E 3 B N 8 w + J b g W 1 M 3 t 5 6 0 r v L A S I 8 J d Y 0 p X M u P c v g 3 K X L X X L u y u W u O H f t c t e c u 3 G 5 G 8 7 d u l w t / U t e D Z c f Q D 9 d 4 F P t a t 2 5 z F K Y 2 S d d i y U T A 9 0 l l D 7 q M l n h d Y l s q A p m y M A g A 1 p N X Z Q g R M W I L k U Q o X K 4 r I Z C d Y e u O h C h a k P X G o h Q j a E r D E S o s t B 1 B S I R u / 0 7 A 1 E Z o Y s I R K h 4 0 K U D I j F b C Y M k h K Q G S d k K G o R K A l 0 Q I E K F g C 4 D E K F i V u d + R C j n 6 4 y P C H s Q 0 4 k e I U r w Z b U z b F 9 K g 1 A y 1 6 k c E U r h O o E j Q o l b p 2 1 E q E j V u R q R D 2 3 H p v N U B a U f 5 2 O 1 5 / r f W o b l o B K I 1 o Y F 6 Q m M X l t U V O w n g 6 H q Y S 6 I y B I I F a 7 / J V j L U k n S B m i J H h H C v x k m o j B R n f W / 1 N n K t 5 J u P Z X Z j M 9 g p i R r W 6 j Y g F q o 1 i G b 1 k y p 1 L Z Q p S N q o U L D O b 2 z U f o c E 4 s j j q i F s n z H B o + a v G c L N F N a r K c / U z q 0 L V x R t p S o w Y x a q L + c W q i 9 9 9 R C 3 R V 8 t W Z K c P U i z Z T W b A u X e 0 I t 1 F l J L d T Y l F q o r w d q o b Y e q Y W 6 + j C v f i D D r P t Q 3 V q n X N Q b p V q d a B F Z p w D Q + R U h y q s 6 q y J C 2 V T n U k Q o h + o M i g h l T p 0 3 E a F a T i d L R C h J 6 h S J C K V G n R g R o Y S o 0 y E i l A Z 1 E k S E k p 9 O f Y h Q y t M J D x G q 0 3 S W Q 4 S V a D q 5 I U R J T a c 0 R C i V 6 U S G C C U w n b 4 Q o X J M 5 y x E z p n r C w N R i t I J C h F K T D o t I U L p S C c j R C g J 6 R S E y A 1 z f W u g W 7 a L l C Y G P E s k / X F 1 L N / h F V t A e w o o p l e d B P X 0 q m B W 3 J k J a H M 6 n U M q 1 K / E m x D E f g E o r f G a O o 3 w l q Y C F K N I v V S F N M i G U R q i N 3 8 S K 0 S M 6 u t k P h P q f f A Z y E U O B l k 8 / C E 3 g 4 c 5 x u K z 5 k v d V O h f F U 0 i r R z q N 9 r V 5 K Q p O l P B Y k C u W 4 z K T L l h s Q 1 a e L l p Q Y o E u W U x i g W 5 b T G K B r l j M Y o H u W s x i g i 5 Z z G K C b l v M Y o K e W A x i g v Z s 1 i P D f r Q g h Q c 8 s h i F B 7 y 2 G J U / s m + x f r M 4 Y k F K U j k q c U o T O S Z x S h Q 5 L n F q K y T F x a j W J G X F q N o k V c W o 3 i R 1 x a j i J E 3 F m M x I 2 8 t W N V p K O e d w s / H h g 7 t Y 3 D g P I i E 6 w w m c Y Q b D K Z j M t x k M C k k 3 G I w i S T c Z j D p J N x h M E k l 3 G U w q S X c Y z A J J t x n M G k m P G D w A S 1 O 2 G M 4 H a r h I Y N J P O E R g 0 k / 4 T G D S U J h n 8 F 0 z o Y n D D 5 h Q z l l O G k p P G M w y S k 8 Z z A p K r x g M I k q v G Q w e w w O r x h O 2 g q v G U z y C m 8 Y T A 8 E 4 S 2 D j c b U r + + y q t 9 E / Z 5 l w N + z i H W C S V 9 i g 1 C S l 9 g k V K v r h b e p f / m Y C P B 8 T 4 D 0 8 N 4 x D L 2 t Z W 8 A g a 9 w O Y 6 E N 8 0 m 8 R A h b I E n 9 O 8 k W G F O C k 9 9 4 J P F 6 E h 9 L A M P O V a c + j f g 6 s d Q s U 1 3 J I W K H U J J o G K X U N K n 2 C O U v a U R + w S T P M U B o X S o i R 6 h p E 1 x S O g h c 3 x E M E l T H B N K y h R 9 Q k m Y 4 o R Q O t / E K a G k S n F G K H u G F e c E k y j F B a G k S X F J K B 1 1 4 o r Q K + b 4 m m A S p L g h l P Q o b g m 1 c t x K s R Y E / X D h m 9 c x D 1 X B S A V B z 3 0 s U B X j G r V Q r u v U Q p l u U F m r q s R N I v H U 2 6 I W K m m b W q i g H W q h c n a p h Y r Z o x Y K Z Z 9 a K J A D a q E w e t R C Q R y y w a A Q j o h E A R x T C z e + z 0 x x x 0 + I x J 0 + p R b u 8 B m 1 c G P P q Y U b e k E t 3 M h L a u E G X l E L 9 + 2 a W r h f N + z 2 u F G 3 7 I Z V 8 V U V X m r b g G + b N D W Y O l x U I O u P 9 z C a D b z s T S M 5 z i b S w w L I m 2 J + y 6 F w S y S g G s m p j 6 r 7 y 1 o I 2 v B J c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 s 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e q 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 i h 7 0 a Y r K 2 i U V q B r K 6 D i y s L 9 G u 5 z + K S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q l h 9 o Z M l 1 z A a y 5 8 g s A E J I s J e J N 0 C E X 8 q D 5 9 G v r S 9 0 J I o c D c o 9 q R Q L 0 P J i o R u d r N l e l 8 l r + d 3 R X J T D d 0 G l R e I c m j I s I E 6 P S v P 0 I c P O r k p z 8 k U T f B b N n w b b 8 x G f s S H + T d W z i W f W 7 Z n 7 c N J s m G E H 9 s I t q g n o l p z Z t P M / 3 K q t + w a o w 7 i o d Q W d 7 p R j 3 8 u g c e F j I L x r 5 Q H 9 X 6 E 5 n p R y s o n C E 2 P o b N j U 0 9 y K r L 0 w E M w b E z z R a 7 A g k 8 e q y d a a I Y A v N D s W s d + 3 n s B z C v v 8 r p V c D c e + F V 1 + 4 C u / 2 3 5 n X 2 2 2 q O p C f Y h z + 9 J n s 6 5 5 m + 4 R Z V x l a 5 0 T M u 5 v b 1 n E s U E M 7 r F 2 5 N K p A 0 S d W K R p F a P d d M Z C O Z + A 9 k a Y G m H W a N T H 8 J Z d 7 E P f W S x x M 1 / Q / q L U F j c g e 9 O f 8 O 6 q D 3 Z A 8 v / Y K G o B r N G 0 j 8 x y 9 w + 4 u M W Z 4 9 2 Y K N r C R a N X S i u 8 r i U e E n 6 p 3 V e J o V W L M K / 1 F 4 z 3 v f v X q u P g D S X 6 t P U v N F q 8 h R A k J / h f b 8 D u K Y 2 d g 3 p y + 8 d U y F G P e p + u s R g x 4 S 9 T W c K o y N U 2 a t P k f N J q H O n r p O j i Q s a / c i 8 4 Y Z K H f T 6 D 7 K Y R j 5 K 4 0 P m r M i i d W b / v m s 9 9 3 q v I X M U l B c t 4 2 T U 9 3 v V X M / N J k r K m / p p u X Q + + 4 u S k f y s d k 1 9 w v 1 G h n P D l 8 F z B n g i S v 8 E L w o 9 d K s K v I l P K x 4 G + N M q P X J V E E Y j L 1 N f C Z O 4 U v h D b L s f k W F J X v b c 5 y r Q z o r / o Q 6 L 0 I 9 A v z 3 b l l d f c x Q H Z f G M N L v b F o s t W L R T P + 9 w O I c N X W u P h W M Q d 7 5 A 4 y 1 O J s O C v D v n 7 3 9 f K n b / H 8 l n l 5 c v l r p / n n l 9 c n r p W / W q / + P 4 r P O 7 z t / 6 H z V 6 X b + 0 v m m s 9 v p d y 4 6 Q S f t / L 3 z j 8 4 / u / / q / r v 7 f f c / x v T T T 6 o + v + s 4 f 7 r / / R + X X F M 7 < / l a t e x i t > O < l a t e x i t s h a 1 _ b a s e 6 4 = " M r l 2 v q 5 b V F J E L B E Y L i e O I / 8 A 3 H w = " > A A A y J H i c l V v J c t z I E e W M t z G 9 a e w I X 3 x B m J J n 7 K A Y p G b C 9 n G 4 b 0 2 y u S / T G g U a n Y 2 G i E 2 o a p B U u / 0 x P v h i f 4 p v D h 9 8 8 W f 4 7 K w q F D I L j d a E F S E R 9 V 5 W o p a X l Q k Q 6 u d x J O T q 6 r 8 / + v g 7 3 / 3 e 9 3 / w y Q 8 X f / T j n / z 0 Z 8 8 + / f m V y M Z F A J d B F m f F T d 8 X E E c p X M p I x n C T F + A n / R i u + / e b i r 8 u o R B R l l 7 I p x x e J 3 6 Y R s M o 8 C V C b 5 7 9 s p f 4 c t Q f T k 6 n v d / Y 6 8 P p m 2 d L q y u r + o 8 3 e 7 F W X S w t V H + 6 b z 5 d / W 9 v k A X j B F I Z x L 4 Q X 6 + t 5 v L 1 x C 9 k F M Q w X e y N B e R + c O + H 8 D V e p n 4 C 4 v V E T 2 D q v U B k 4 A 2 z A v + m 0 t M o 7 z H x E y G e k j 5 a q i G K J q f A e Z z y K J b 7 S W M I c v j H 1 5 M o z c c S 0 s C M Y D i O P Z l 5 a p m 8 Q V R A I O M n v P C D I s J J e M H I L / x A 4 m I u L r 7 w j v z i 3 h N o g 8 s o v G z o B X 5 u r t U 0 C h h C U U R p q B w O o j I S 1 m w Y h e M C c J Q p P A R Z k v j p Y N J D M I a h n E 4 m P U i 8 z z t 4 / d v p d H H G K M D V h c K a b e q W M m z a F V E 4 q r 2 d q U a b l c x y a 3 O R 5 W 0 W / U z K L L F G G 7 o 1 Y 1 d N 3 L d m / u z Q K 5 O + N e n P c x J Y i 2 C e x c B a D P R t X n h 7 O L 1 Y T d H z P b R X W w Z D j I a B h 4 u T u D 7 w W o H T r 9 d e o 5 f + 0 F t a U 0 7 Q y 4 7 e F r N v K A h Y 9 u L s A Y q X A c b W y m I P X e p 1 h e H S 2 s R s 4 Z 9 7 2 J o Y B 2 3 9 c b y R 9 O M V b w f 1 I C Q G g t p + o f Y M e e N y x 7 r c a b r U t H z I 7 E 2 X X l W 3 F Z 4 1 8 n B K V e O V 7 f F u 7 A + o y 9 I X S 1 / O d F u u + 9 i r L 7 i r L 9 H V C + / c C H v e e p i b o f r N 6 K s w Y M N v 9 W B X x P Q + t 7 3 P W 3 q f 2 V 4 6 I B + y O t J W 6 p U x d x d 6 a e o 4 n L M 2 T Y e j A o C 5 5 N M R Z u F m X d K 6 M e d f z D r 3 U w 9 w G 1 T n F h H B O z N p a z J / 1 o 6 f c Z 5 D 4 S k / x s 1 2 5 W a 7 z c 2 6 V / g P t P A N Z y 9 f v v T L L B p 4 Y 6 H O p 2 j o 5 Z k Q E S a M a h 3 y 2 M f w q W 4 w f 3 j q P M w x m l o m q R j T v b L 5 v 2 d Z O d q s H W 1 + q y O c d B q C P o m N r a j m o / F 6 S K g W y 1 t f L 1 / O V Q o O z 4 / D D B P A K G m Z K H J m e L X R B 2 f K X M 1 M d d 2 6 W m 9 x Z T V v 7 4 e T q H 3 N F b 3 p d e H 0 W v / W X j P L m u P J V U 2 d C V C h Z r z q 6 k P b Y v o 3 B d y t + 3 f d / n a q 9 Q 1 w 1 O p 6 / j Q r z U E U K 7 3 G 6 g L P d b R Q V 5 X D Y Z x l h a b 1 l e H 1 Z W W A V D + Z r D W T j i w w F q Y T X R g F f j z Z a h q U f h w N u M E b c 1 0 k E 0 N N Z 1 y C k O 0 d N D O l K U E u V K 7 L R R R n q Y b V 6 q K T L P F K v 4 h 8 j F l R a R y k P 1 G + H 2 W a F Q n 6 f d 5 D 6 P n U r m j R o H 1 i + i 7 T J y Z w m Y C Y g c s M p r X e C n A p o E 5 D l x k S E 7 p M S M z I Z U b E R C 4 T E f P W Z d 6 y w d 2 7 1 D 1 1 i l 0 m n m o t F 4 k X C Y x b L K k H T + r M M 7 u 4 7 L 0 d C + k N s v Q z 6 a l S F j X 5 p A 4 g Z 2 u 8 p P K d u r 5 T u m v m M h k x u c v k b B L v X O o d d S p c p i B G u I x g 7 q R L S e o 0 d p k x M a X L l M Q 8 u M w D M Y 8 u 8 0 j M k 8 s 8 s c G 9 d 6 n 3 U 1 N u 2 S j A J J 0 V N g b K K l Q m E / s g Q 8 F T D 1 y O T K h Y C 9 1 m N y w Z y Y K k 7 B P M I q Q M C G b h U Q 4 I H j A Y C G Z x U Q 4 J H v K h h I S z k C h H B L N 4 K M c E s 2 A o 3 x L 8 l s H 3 B L M o K G O C Y w Y n B C d 8 h G y t + S J n B D N J l z n B O Y P f E c y 0 X B Y E M y G X g m D B 9 5 V g y U f I V o X r t y S Y i b d 8 I J g p t 3 w k m M m 2 f C L 4 i c H v C b Z 6 3 Y 5 B P R n r J 8 D C 1 W 5 1 1 h v h t Z 7 Q Y N T X e k a D k W D r K Q 1 G h + 3 n N B g 1 t p 7 U Y C T Z e l a D k W X r a Q 1 G m 6 3 n N X J z T 2 w w K m 2 e 2 Z a b e 2 i D 0 W v z 2 L Z c 4 n K J M / m 2 M 7 k m 5 x 7 L Y E T c P J g t 1 3 Y y 1 + T c w x m M q J v H s + X m n s 9 g 1 N 1 6 Q o O R e O s Z D U b n r a c 0 G L G 3 n t N g F N 9 + U o P R / f y z G i O i i I K 6 Y E n W K U r W W c A m G 4 R v U F A l m w R v M n i L 4 C 0 G b x O 8 z e A d g n c Y v E v w L o P 3 C N 5 j 8 D 7 B + 3 z g B 4 Q f M P N D g g 8 Z 3 C G 4 w + A j g o 8 Y f E z w M Y N P C D 7 h Q + k S 3 m X m p w S f M v i M 4 D M G n x N 8 z u A L g i 8 Y f E n w J Y O v C L 7 i I 7 w m / J q Z 3 x B 8 w + B b g m 8 Z f E f w n a m b W 0 9 a V 3 l g p M e E u s 4 U r u X H u A 3 O b b r c J u e 2 X G 6 L c 9 s u t 8 2 5 H Z f b 4 d y u y + 3 y a N P q Z O Q e 7 7 j v c v u c O 3 C 5 A 8 4 d u t w h 5 z o u 1 + H c k c s d c e 7 Y 5 Y 6 d S Z y 4 5 A n v 2 H W 5 L u d O X e 6 U c 2 c u d 8 a 5 c 5 c 7 d w Z z 4 Z I X v O O l y 1 1 y 7 s r l r j h 3 7 X L X n L t x u R v O 3 b r c L e f u X K 6 W / h W v h s v 3 o J 8 u 8 K l 2 t e 5 c Z i l M 7 J O u x Z K x g X o J p Y + 6 T F Z 4 X S I b q o I Z 0 j d I n 1 Z T F y U I U T G i S x F E q B w u q 6 F Q 3 a G r D k S o 2 t C 1 B i J U Y + g K A x G q L H R d g U j E b v / W Q F R G 6 C I C E S o e d O m A S M x W w i A J I a l B U r a C B q G S Q B c E i F A h o M s A R K i Y 1 b k f E c r 5 O u M j w h 7 E d K J H i B J 8 W e 0 M 2 5 f S I J T M d S p H h F K 4 T u C I U O L W a R s R K l J 1 r k b k f d u x 6 T x V Q e n H + U j t u f 5 Z y 7 D s V w L R 2 r A g P Y H R a 4 u K i v 2 k P 1 A 9 z A U R W Q K h w v V P g r U s l S R t g J b o E S H 8 l 2 E i C h P V W f + k z l a + l X T r q U w m f A Y T J V n b Q s U G 1 E K 1 D t i 0 J k q l t o U q H V I L F R p O 6 Z 2 N 0 u e I W B x x R C 2 U 5 V s 2 e N T k P V u g i d J i P f 2 J 0 q F t 4 Y q y p U Q N Z t R C / e X U Q u 2 9 o x b q r u C r N V G C q x d p o r R m W 7 j c Y 2 q h z k p q o c Y e q I X 6 e q Q W a u u J W q i r 9 6 b s O 1 J Z 9 7 G 6 t U 6 5 q D d K t T r R I r J B A a D z K 0 K U V 3 V W R Y S y q c 6 l i F A O 1 R k U E c q c O m 8 i Q r W c T p a I U J L U K R I R S o 0 6 M S J C C V G n Q 0 Q o D e o k i A g l P 5 3 6 E K G U p x M e I l S n 6 S y H C C v R d H J D i J K a T m m I U C r T i Q w R S m A 6 f S F C 5 Z j O W Y h c M N e X B q I U p R M U I p S Y d F p C h N K R T k a I U B L S K Q i R W + b 6 z k B 3 b B c p T f R 5 l k i 6 o + p Y 7 u E V W 0 B 7 C i i m U 5 0 E 9 f S q Y F b c u Q l o c z p d Q C r U b 4 S 3 I I j 9 A l B a o 3 V 1 G u E t T Q U o h p F 6 q Q p p k A 2 i N E R v / j h W i B j W 1 8 l 0 I t T 7 4 H O Q 8 x z 0 s 3 j w b W 7 6 j 1 O M x c X m S 9 1 U 6 N 8 q m k R a O d R v t K v J S V N 0 p o L F g N y w G J W Z c t N i m 7 T w c s u C F A l y 2 2 I U C 3 L H Y h Q N c t d i F A 9 y z 2 I U E X L f Y h Q T 8 s B i F B X y 0 G I U F 7 J j s Q 4 b 9 J E F K T j k s c U o P O S J x a j 8 k 1 2 L d Z n D U w t S k M g z i 1 G Y y H O L U a D I C 4 t R W S c v L U a x I q 8 s R t E i r y 1 G 8 S J v L E Y R I 2 8 t x m J G 3 l m w q t N Q z r u F n 4 8 M H d r H 4 M B 5 E A k 3 G E z i C D c Z T M d k u M V g U k i 4 z W A S S b j D Y N J J u M t g k k q 4 x 2 B S S 7 j P Y B J M e M B g 0 k x 4 y O B D W p y w w 3 A 6 V M M j B p N 4 w m M G k 3 7 C E w a T h M I u g + m c D U 8 Z f M q G c s Z w 0 l J 4 z m C S U 3 j B Y F J U e M l g E l V 4 x W D 2 G B x e M 5 y 0 F d 4 w m O Q V 3 j K Y H g j C O w Y b j a n f v s u q f h P 1 e 5 Y + f 8 8 i N g g m f Y l N Q k l e Y o t Q r a 4 X 3 p b + z c d Y g O d 7 A q S H 9 4 5 h 4 G 0 v e 3 0 I f I X L U S S 8 h 2 w c D x D C F n h C / 5 4 E K 8 x x 4 a m P e b I Y H a m P Z e A x x 4 p T / w 6 4 + m W o 2 K E 7 k k L F L q E k U L F H K O l T 7 B P K 3 t K I A 4 J J n u K Q U D r U R I d Q 0 q Y 4 I v S I O T 4 m m K Q p T g g l Z Y o u o S R M c U o o n W / i j F B S p T g n l D 3 D i g u C S Z T i k l D S p L g i l I 4 6 c U 3 o N X N 8 Q z A J U t w S S n o U d 4 R a O W 6 n W A u C f r j w z e u Y x 6 p g p I K g 4 z 4 W q I p x n V o o 1 w 1 q o U w 3 q a x V V e I W k X j q b V M L l b R D L V T Q L r V Q O X v U Q s X s U w u F c k A t F M g h t V A Y H W q h I I 7 Y Y F A I x 0 S i A E 6 o h R v f Z a a 4 4 6 d E 4 k 6 f U Q t 3 + J x a u L E X 1 M I N v a Q W b u Q V t X A D r 6 m F + 3 Z D L d y v W 3 Z 7 3 K g 7 d s O q + K o K L 7 V t w L d N m h p M H S 4 q k P W H e h j N B l 7 2 H i I 5 y s b S w w L I e 8 D 8 l k P h l k h A N Z J T H 1 X 3 l 7 U Q t O F M c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 q 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e r 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 S h 7 0 a Y r K 2 i U V q B r K 6 D i y s L d G u 5 y + L S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q 1 h 9 o Z M l 1 z A a y 5 8 g s A E J I s x e O N 0 A E X 8 p D 5 9 G v j S 9 0 J I o c D c o 9 q R Q L 3 3 x y o R u d r N l e l 0 k r + Z 9 I p k o h s 6 D S q v k O R R E W E C d P r X H y H 2 n 3 T y 0 x + S q J t g t m z 4 t t + Y j H y J D / L u L R z L L r f s T t s G k 2 Q D i D 8 0 E W 1 Q z 8 S 0 p s 2 n m W 5 l 1 W 1 Y N c Y d x Q O o L H u 6 U Q + / 7 o G H h c y C k S / U B 7 T + W G b 6 0 Q o K Z 4 i N j 2 F z Y 1 M P s u o y O 4 A B O H a m 2 W J X I I F H j 7 U z T R R D Y H 5 R 7 F r H f h 7 7 A U z r r 3 I 6 F T D 1 X n j V t b v A b v / t a Z 3 9 t p s j 6 Q j 2 4 U + n y Z 5 N e a Z v u E W V s V V u 9 I y L q X 0 9 5 x I F h N P 6 h V u T C i R N U r W i Y a R W z z U T 2 V A m / i N Z W q B p h 1 k j 0 1 9 C m T d x s 1 7 y e K y m / 1 6 9 J W h M 7 r A z 5 d 9 B H X Z m 9 v D K L 2 g I q t G 8 g c Q f f o H b X 2 T M 8 n x m C z a z k m j V 0 I n u O o u H h Z + o d 1 a j h 6 z A m l X 4 T 8 J 7 3 v n m 1 X P 1 A Z D + M n 2 c m i 9 a R Y 4 S E P o r t O c 9 i G N m Y 9 + c v v A 2 M B V i 3 K f q n y c M e k j U 1 3 C q M D Z O m b X 6 H D U b h z p 7 6 j o 5 k r C s 3 Y v M G 2 S g 3 D 1 E 9 1 E O g 8 h f a X z Q n B V J r N 7 0 T y e d b 1 a n L W S W g u L W 2 j j 5 o P u 9 a u 6 H J n N F 5 S 3 d t B w 6 3 / S i d C i f m l 1 z v 1 C v k f H s 8 F X A n A O e u M I P w Y t S L 8 2 q I l / C 4 4 q 3 O c q E W p 9 M F Y T B y N v C Z + I U P h N e P 8 v u V 1 R Y s r c 9 J 7 k 6 p L P i d 6 j z I t Q j w J + 9 Z X X 1 I U N 1 X B r D S L + z a b H U i k U z / e 8 c i w v U 1 I X 6 V D A G 2 f P 7 G G t x 9 t A v w L 9 f f P N s a a 3 5 / y J m L 6 5 e r a z 9 f u X L 0 y + X v t q o / s / E J w u / W v j 1 w u c L a w t / W P h q Y W + h u 3 C 5 E C z 8 a e E v C 3 9 b + P v a X 9 f + s f b P t X 8 Z 0 4 8 / q v r 8 Y s H 5 s / a f / w E Q 1 1 D y < / l a t e x i t > Q&K < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 H a I A i 1 J f / A G Y M f C q P H w v O d c j x o = " > A A A y F n i c l V v J c t z I E e V 4 H d P b j H 3 0 B W F K M W O H x C A 1 C t v H 4 b 4 1 y e a + T G s U a H Q 2 G i I 2 o a r R p D r a X + G D L / a n + O b w 1 V f / i M / O q k I h s 9 B o T V g R E l H v Z S V q e V m Z A K F + H k d C r q 3 9 5 5 P v f f 8 H P / z R j z / 9 y f J P f / b z X / z y s 8 9 / d S 2 y c R H A V Z D F W X H b 9 w X E U Q p X M p I x 3 O Y F + E k / h p v + w 5 b i b 0 o o R J S l l / I p h z e J H 6 b R M A p 8 i d B 9 L / H l q D + c X s / e f r a y t r q m / 3 j z F + v V x c p S 9 a f 7 9 v O 1 / / Y G W T B O I J V B 7 A v x z f p a L t 9 M / U J G Q Q y z 5 d 5 Y Q O 4 H D 3 4 I 3 + B l 6 i c g 3 k z 1 k G f e c 0 Q G 3 j A r 8 G 8 q P Y 3 y H l M / E e I p 6 a O l G q J o c g p c x C m P 4 k U / a Q x B D v / 0 Z h q l + V h C G p g R D M e x J z N P L Y w 3 i A o I Z P y E F 3 5 Q R D g J L x j 5 h R 9 I X L 7 l 5 e f e s V 8 8 e A J t c O G E l w 2 9 w M / N t Z p G A U M o i i g N l c N B V E b C m g 2 j c F w A j j K F S Z A l i Z 8 O p j 0 E Y x j K 2 X T a g 8 T 7 s o P X v 5 v N l u e M A l x d K K z Z l m 4 p w 6 Z d E Y W j 2 t u 5 a r R Z y S y 3 N p d Z 3 m b R z 6 T M E m u 0 q V t z d t X E f W v m z w + 9 M u l b k / 4 i J 4 G 1 C B Z Z D K z F Q N / m u b e P 0 4 v V F D 3 f Q 3 u 1 Z T B E / Q 8 8 X J z E 9 Y H X C p x 9 s / 4 G v f S H 3 s q 6 c o J e d v W 2 m H 1 D Q c A L L 8 4 m U L w M M J p W l 3 v o U q 8 r D F f W p 2 Y L / 9 z D 1 t Q 4 a O u P 4 4 2 k H 6 9 6 u 6 g H I T E Q 1 P Y L t W f I G 5 e 7 1 u V u 0 6 W m 5 S S z N 1 1 5 V d 1 W e N b I w y l V j V e 2 x / u x P 6 A u K 1 + t v J 7 r 9 q L u Y 6 + + 4 q 5 e o 6 v n 3 o U R 9 q L 1 M D d D 9 Z v R V 2 H A h t / q w a 6 I 6 X 1 h e 1 + 0 9 D 6 3 v X R A T r I 6 0 l b r l T F 3 F 3 p p 6 j h c s D Z N h 6 M C g L n k 0 x F m 4 e Z d 0 r o x 5 1 / N O / d T D 3 A b V O c W E c F 7 M 2 l r s n j W j p 9 x n k P h K T / G z U 7 l Z q f N z Y Z X + B N a + I a z l y 9 f + m U W D b y x U O d T N P T y T I g I U 0 S 1 D n n s Y / h U N 1 g 8 P H U e 5 h h N L Z N U j O l e 2 f z f s 6 w c b d W O t r 7 T E U 4 6 D U G f x M Z W V P P R e D 0 k V I v l r a + X L x c q B Y f n x 2 G G C W C U t E w U O T O 8 2 u i j M 2 W u 5 q a 6 Y V 1 t t L i y m r f 3 w 0 n U v h a K 3 v S 6 d H p t f G e v u W X N 8 e S q p s 4 E q F A z X n X 1 s W 0 x / Z s C 7 t b 9 u 2 5 / O 9 X 6 B j h q d b 1 4 m p X m I I q V X m N 1 g e c 6 W q i r y u E w z r J C 0 / r K 8 P q y M k C q n 0 z X m 0 l H F h g L s 6 k u j A I / n m 4 3 D U o / j g b c 4 K 2 5 L p K p o W Z z L k H I 9 g 6 a m d G U I B c q 1 + U i i r N U w 2 p 1 0 U m W e K V f R D 7 G r K g 0 D t K f K t + P M s 2 K B P 0 + 6 y H 0 b G Z X t G j Q P j F 9 l + k T E 7 h M Q M z A Z Q a z W m 8 F u B R Q p 6 H L D I k J X S Y k Z u Q y I 2 I i l 4 m I e e c y 7 9 j g H l z q g T r F L h P P t J a L x I s E x i 0 W 0 Y M n d e a Z X X z h v R s L 6 Q 2 y 9 A v p q V I W N f m k D i B n a 7 y k 8 p 2 6 v l O 6 a + Y y G T G 5 y + R s E u 9 d 6 j 1 1 K l y m I E a 4 j G D u p E t J 6 j R 2 m T E x p c u U x E x c Z k L M o 8 s 8 E v P k M k 9 s c B 9 c 6 s P M l F s 2 C j B J Z 4 W N g b I K l e n U P s h Q 8 N Q D l y M T K t Z C t 9 k N S 0 a y I C n 7 B L M I K Q O C W X i U A 4 I H D A a C W V y U Q 4 K H f C g h 4 S w k y h H B L B 7 K M c E s G M p 3 B L 9 j 8 A P B L A r K m O C Y w Q n B C R 8 h W 2 u + y B n B T N J l T n D O 4 P c E M y 2 X B c F M y K U g W P B 9 J V j y E b J V 4 f o t C W b i L S c E M + W W j w Q z 2 Z Z P B D 8 x + A P B V q 8 7 M a g n Y / 0 E W L j a r c 5 6 I 7 z W E x q M + l r P a D A S b D 2 l w e i w / Z w G o 8 b W k x q M J F v P a j C y b D 2 t w W i z 9 b x G b u G J D U a l z T P b c g s P b T B 6 b R 7 b l k t c L n E m 3 3 Y m 1 + T C Y x m M i J s H s + X a T u a a X H g 4 g x F 1 8 3 i 2 3 M L z G Y y 6 W 0 9 o M B J v P a P B 6 L z 1 l A Y j 9 t Z z G o z i 2 0 9 q M L p f f F Z j R B R R U B c s y Q Z F y Q Y L 2 G S T 8 E 0 K q m S L 4 C 0 G b x O 8 z e A d g n c Y v E v w L o P 3 C N 5 j 8 D 7 B + w w + I P i A D / y Q 8 E N m f k T w E Y M 7 B H c Y f E z w M Y N P C D 5 h 8 C n B p 3 w o X c K 7 z P y M 4 D M G n x N 8 z u A L g i 8 Y f E n w J Y O v C L 5 i 8 D X B 1 3 y E N 4 T f M P N b g m 8 Z f E f w H Y P v C b 4 3 d X P r S e s q D 4 z 0 m F A 3 m M K 1 / B i 3 y b k t l 9 v i 3 L b L b X N u x + V 2 O L f r c r u c 2 3 O 5 P R 5 t W p 2 M 3 O c d D 1 z u g H O H L n f I u S O X O + J c x + U 6 n D t 2 u W P O n b j c i T O J U 5 c 8 5 R 2 7 L t f l 3 J n L n X H u 3 O X O O X f h c h f O Y C 5 d 8 p J 3 v H K 5 K 8 5 d u 9 w 1 5 2 5 c 7 o Z z t y 5 3 y 7 k 7 l 7 v j 3 L 3 L 1 d K / 5 t V w + Q H 0 0 w U + 1 a 7 V n c s s h a l 9 0 r V Y M j Z Q L 6 H 0 U Z f J C q 9 L Z E N V M E P 6 B u n T a u q i B C E q R n Q p g g i V w 2 U 1 F K o 7 d N W B C F U b u t Z A h G o M X W E g Q p W F r i s Q i d j t 3 x m I y g h d R C B C x Y M u H R C J 2 U o Y J C E k N U j K V t A g V B L o g g A R K g R 0 G Y A I F b M 6 9 y N C O V 9 n f E T Y g 5 h O 9 A h R g i + r n W H 7 U h q E k r l O 5 Y h Q C t c J H B F K 3 D p t I 0 J F q s 7 V i H x o O z a d p y o o / T g f q T 3 X P 2 s Z l v 1 K I F o b F q Q n M H p t U V G x n / Q H q o e 5 I C J L I F S 4 / k m w l q W S p A 3 Q E j 0 i h P 8 y T E R h o j r r n 9 T Z y r e S b j 2 V 6 Z T P Y K o k a 1 u o 2 I B a q N Y B m 9 Z U q d S 2 U K V D a q F C w x m 9 s 1 H 6 H B G L I 4 6 o h b J 8 x w a P m n x g C z R V W q y n P 1 U 6 t C 1 c U b a U q M G M W q i / n F q o v f f U Q t 0 V f L W m S n D 1 I k 2 V 1 m w L l 3 t M L d R Z S S 3 U 2 I R a q K 9 H a q G 2 n q i F u v p g y r 5 j l X U f q 1 v r l I t 6 o 1 S r E y 0 i m x Q A O r 8 i R H l V Z 1 V E K J v q X I o I 5 V C d Q R G h z K n z J i J U y + l k i Q g l S Z 0 i E a H U q B M j I p Q Q d T p E h N K g T o K I U P L T q Q 8 R S n k 6 4 S F C d Z r O c o i w E k 0 n N 4 Q o q e m U h g i l M p 3 I E K E E p t M X I l S O 6 Z y F y C V z f W U g S l E 6 Q S F C i U m n J U Q o H e l k h A g l I Z 2 C E L l j r u 8 N d M 9 2 k d J E n 2 e J p D u q j u U e X r E F t K e A Y j r V S V B P r w p m x V 2 Y g D a n 0 y W k Q v 1 G e B u C 2 C 8 A p T X a U K c R 3 t J U g G I Y q Z e q k A b Z I E p D 9 O a P Y 4 W I Y X 2 d z K Z C v Q + + A L n I Q T + L B 9 / l p v 8 4 w 1 h c b r 7 U T Y X + r a J J p J V D / U a 7 m p w 0 R W c q W A z I T Y t R m S m 3 L L Z F C y + 3 L U i R I H c s R r E g d y 1 G 0 S D 3 L E b x I P c t R h E h D y x G M S E P L U Z R I Y 8 s R n E h O x b r s E E f W 5 C C Q 5 5 Y j M J D n l q M y j / Z t V i X O T y z I A W J P L c Y h Y m 8 s B g F i r y 0 G J V 1 8 s p i F C v y 2 m I U L f L G Y h Q v 8 t Z i F D H y z m I s Z u S 9 B a s 6 D e W 8 V / j 5 y N C h f Q w O n A e R c J P B J I 5 w i 8 F 0 T I b b D C a F h D s M J p G E u w w m n Y R 7 D C a p h P s M J r W E B w w m w Y S H D C b N h E c M P q L F C T s M p 0 M 1 P G Y w i S c 8 Y T D p J z x l M E k o 7 D K Y z t n w j M F n b C j n D C c t h R c M J j m F l w w m R Y V X D C Z R h d c M Z o / B 4 Q 3 D S V v h L Y N J X u E d g + m B I L x n s N G Y + u 2 7 r O o 3 U b 9 n 6 f P 3 L G K T Y N K X 2 C K U 5 C W 2 C d X q e u 5 t 6 9 9 8 j A V 4 v i d A e n j v G A b e z g u v D 4 G v c D m K h D f J x v E A I W y B J / T v S b D C H B e e + p g n i 9 G R + l g G H n O s O P X v g K t f h o p d u i M p V O w R S g I V + 4 S S P s U B o e w t j T g k m O Q p j g i l Q 0 1 0 C C V t i m N C j 5 n j E 4 J J m u K U U F K m 6 B J K w h R n h N L 5 J s 4 J J V W K C 0 L Z M 6 y 4 J J h E K a 4 I J U 2 K a 0 L p q B M 3 h N 4 w x 7 c E k y D F H a G k R 3 F P q J X j T o q 1 I O i H C 9 + 8 j n m s C k Y q C D r u Y 4 G q G D e o h X L d p B b K d I v K W l U l b h O J p 9 4 O t V B J u 9 R C B e 1 R C 5 W z T y 1 U z A G 1 U C i H 1 E K B H F E L h d G h F g r i m A 0 G h X B C J A r g l F q 4 8 V 1 m i j t + R i T u 9 D m 1 c I c v q I U b e 0 k t 3 N A r a u F G X l M L N / C G W r h v t 9 T C / b p j t 8 e N u m c 3 r I q v q v B S 2 w Z 8 2 6 S p w d T h o g J Z f 6 i H 0 W z g F 9 4 k k q N s L D 0 s g L w J 5 r c c C r d E A q q R n P q o u r + s h a A N 5 4 p D 0 A U U N C o o 0 C U U N G o o 0 E U U U B V V o d s 1 y l 7 P 6 T o K G o U U 6 E o K G q U U 6 F o K G s U U 6 G o K q J y y 8 E E N s x d w u q S C R k 0 F u q i C R l U F u q w C q q s q 9 L h G 2 Y s 2 X V l B o 7 Q C X V s B F V c W 7 t Z w l 8 N n N c z e q O k S C x o 1 F u g i C x p V F u g y C x p 1 F u h C C x q V F u h S C x q 1 F u h i C x r V F u h y C 6 j e s v B d D b M 3 Z L r k A l 5 z 4 R M E J i B Z j M E b p w M o 4 i f 1 6 d P A l 7 4 X Q g o F 5 h 7 V j g T q v T 9 W i c j V b q 5 M Z 9 P 8 7 b R X J F P d 0 G l Q e Y U k j 4 o I E 6 D T v / 4 I s f + k k 5 / + k E T d B L N l w 7 f 9 x m T k S 3 y Q d 2 / h W H a 5 Z X f W N p g k G 0 D 8 s Y l o g 3 o m p j V r P s 1 0 K 6 t u w 6 o x 7 i g e Q G X Z 0 4 1 6 + H U P P C x k F o x 8 o T 6 g 9 c c y 0 4 9 W U D h D b H w M m x u b e p B V l / k B D M C x M 8 0 W u w I J P H q s n W m i G A L z i 2 L X O v b z 2 A 9 g V n + V 0 6 m A m f f c q 6 7 d B X b 7 7 8 z q 7 L f T H E l H s A 9 / O k 3 2 f M Y z f c M t q o y t c q N n X M z s 6 z m X K C C c 1 S / c m l Q g a Z K q F Q 0 j t X q u m c i G M v E f y d I C T T v M G p n + E s q 8 i Z v 3 k s d j N f 0 P 6 i 1 B Y 3 J H n R n / D u q o M 7 e H 1 3 5 B Q 1 C N 5 g 0 k / v A L 3 P 4 i Y 5 Y X c 1 u w l Z V E q 4 Z O d D d Z P C z 8 R L 2 z G k 2 y A m t W 4 T 8 J 7 1 n n 2 1 f P 1 A d A + s v 0 c W q + a B U 5 S k D o r 9 C e 9 S C O m Y 1 9 c / r c 2 8 R U i H G f q n + e M O g h U V / D q c L Y O G X W 6 n P U b B z q 7 K n r 5 E j C C + 1 e Z N 4 g A + V u E j 1 E O Q w i f 7 X x Q X N W J L F 6 0 z + b d r 5 d m 7 W Q W Q q K W 2 / j 5 E T 3 e 9 X c D 0 3 m i s p b u m k 5 d L 7 t R e l Q P j W 7 5 n 6 h X i P j 2 e G r g L k A P H G F H 4 I X p V 6 a V U W + h M d V b 2 u U C b U + m S o I g 5 G 3 j c / E K X w h v H 6 W P a y q s G R v e 0 5 z d U h n x e 9 R 5 0 W o R 4 A / e y / U 1 c c M 1 X F p D C P 9 z q b F U i s W z f S / C y w u U V O X 6 l P B G G T P 7 2 O s x d m k X 4 D / s P z 2 s 5 X 1 5 v + L m L + 4 f r W 6 / o f V 1 2 e v V 7 7 e r P 7 P x K d L v 1 n 6 7 d K X S + t L f 1 z 6 e m l / q b t 0 t R Q s p U t / W f r b 0 t / X / 7 r + j / V / r v / L m H 7 v k 6 r P r 5 e c P + v / / h 8 / 4 E u 7 < / l a t e x i t > V < l a t e x i t s h a 1 _ b a s e 6 4 = " F a  V M 9 E 5 K q Z 5 R 9 2 Y o A v w T e z 5 O g p g = " > A A A C Q n i c b V D L S g M x F M 3 4 t r 6 q L t 0 E i 1 A R y o y I u h T d C G 4 U b K 1 0 a s m k d 9 p g 5 k F y R y h D v s 2 N X + D O D 3 D j Q h G 3 L k x f 4 O t A 4 O S c c 8 n N C V I p N L r u k z M x O T U 9 M z s 3 X 1 h Y X F p e K a 6 u 1 X S S K Q 5 V n s h E 1 Q O m Q Y o Y q i h Q Q j 1 V w K J A w l V w e 9 L 3 r + 5 A a Z H E l 9 h L o R m x T i x C w R l a q V W 8 D n 0 J I d K y H y r G c z 9 i 2 A 3 C / O L s 5 t K Y 3 E e W l c d a 3 W y b n f E l M K 1 h W E W 5 A m m M r 0 S n i 9 t j v 2 Z a x Z J b c Q e g f 4 k 3 I i U y w n m r + O i 3 E 5 5 F E C O X T O u G 5 6 b Y z J l C w S W Y g p 9 p S B m / Z R 1 o W B q z C H Q z H 1 R g 6 J Z V 2 j R M l D 0 x 0 o H 6 f S J n k d a 9 K L D J / o b 6 t 9 c X / / M a G Y a H z V z E a Y Y Q 8 + F D Y S Y p J r T f J 2 0 L B R x l z x L G l b C 7 U t 5 l t k u 0 r R d s C d 7 v L / 8 l t d 2 K t 1 / Z u 9 g r H R 2 P 6 p g j G 2 S T l I l H D s g R O S X n p E o 4 u S f P 5 J W 8 O Q / O i / P u f A y j E 8 5 o Z p 3 8 g P P 5 B f u 1 s 3 0 = < / l a t e x i t > f ✓ QK T ⌧ (X) + b rel ◆ V Mega Layer: EMA & Gated Attention Input Embedding Feedforward Norm Add & Norm < l a t e x i t s h a 1 _ b a s e 6 4 = " Q V M V k t g n 5 U I r O K Z V 6 u P v l z 8 g M t E = " > A A A B 7 3 i c b V A 9 S w N B E J 2 L X z F + R S 1 t F o N g F e 5 E 1 D J o Y 2 E R w X x A c o S 9 z V 6 y Z G / v 3 J 0 T w p E / Y W O h i K 1 / x 8 5 / 4 y a 5 Q h M f D D z e m 2 F m X p B I Y d B 1 v 5 3 C y u r a + k Z x s 7 S 1 v b O 7 V 9 4 / a J o 4 1 Y w 3 W C x j 3 Q 6 o 4 V I o 3 k C B k r c T z W k U S N 4 K R j d T v / X E t R G x e s B x w v 2 I D p Q I B a N o p X Y X R c Q N u e u V K 2 7 V n Y E s E y 8 n F c h R 7 5 W / u v 2 Y p R F X y C Q 1 p u O 5 C f o Z 1 S i Y 5 J N S N z U 8 o W x E B 7 x j q a J 2 j Z / N 7 p 2 Q E 6 v 0 S R h r W w r J T P 0 9 k d H I m H E U 2 M 6 I 4 t A s e l P x P 6 + T Y n j l Z 0 I l K X L F 5 o v C V B K M y f R 5 0 h e a M 5 R j S y j T w t 5 K 2 J B q y t B G V L I h e I s v L 5 P m W d W 7 q J 7 f n 1 d q 1 3 k c R T i C Y z g F D y 6 h B r d Q h w Y w k P A M r / D m P D o v z r v z M W 8 t O P n M I f y B 8 / k D q 9 K P v Q = = < / l a t e x i t > ⇥L < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 g + g L f V o a w Q t 1 9 v U f Z I 7 Y m K 0 5 f I = " > A A A y K n i c l V t Z k 9 v G E a a d y 1 E u O 3 n M C y o r l Z 3 U a m s p q 5 I 8 e u + L u 8 u 9 D 6 + s A s E m C C 0 u Y Y b g r l j M r 8 h r 8 g v y a / L m y m t + S H p m M O g e L C h X V G U J 8 3 0 9 j T m + n m 6 A 8 C C P I y F X V 7 / / 5 N M f / f g n P / 3 Z Z z 9 / 9 o t f / u r X v / n 8 i 9 9 e i m x S B H A R Z H F W X A 9 8 A X G U w o W M Z A z X e Q F + M o j h a n C / o f i r E g o R Z e m 5 f M z h T e K H a T S K A l 8 i d H u X + H I 8 G M 0 e 5 m 8 / X 1 p d W d V / v K c X 3 e p i q V P 9 6 b / 9 o v v s b p g F k w R S G c S + E N 9 2 V 3 P 5 Z u Y X M g p i m D + 7 m w j I / e D e D + F b v E z 9 B M S b m R 7 y 3 H u B y N A b Z Q X + l 0 p P o 7 z H z E + E e E w G a K m G K J q c A h d x y q N Y H i S N I c j R X 9 / M o j S f S E g D M 4 L R J P Z k 5 q m F 8 Y Z R A Y G M H / H C D 4 o I J + E F Y 7 / w A 4 n L 5 9 z i w c z h 2 b M X 3 q F f 3 H s C + + F i C i 8 b e Y G f m 2 s 1 t Q J G U B R R G q q b D K M y E t Z s F I W T A t B t C t M g S x I / H c 7 u E I x h J O e z 2 R 0 k 3 l c 9 v P 7 j H G / T N A p w x a G w Z h u 6 p Q y b d k U U j m t v p 6 r R Z i W z 3 N q c Z 3 m b x S C T M k u s 0 b p u P b G r J u 5 b M / / p 0 C u T g T U Z L H I S W I t g k c X Q W g z 1 b V 5 4 u z i 9 W E 3 R 8 z 2 0 V 9 s I I 4 y J o Y e L k 7 g + 8 F q B 8 2 + 7 b 9 D L Y O Q t d Z U T 9 L K t t 8 X s G 4 o E l r 0 4 m 0 L x M s A I W 0 G 5 w 0 i v K 4 y W u j O z h X + 7 w 9 b M O G j r j + O N p B + v e N u o B y E x O N T 2 C 7 V n y B u X 2 9 b l d t O l p u U 0 s z d d e l X d V n j W y M M p V Y 1 X t s f 7 i T + k L k t f L 7 1 + 0 m 2 5 7 m O v v u a u X q O r F 9 6 Z E f a i 9 T A 3 Q / W b 0 V d h w I b f 6 s G u i O l 9 Z n u f t f Q + t b 1 0 k E 6 z O t J W 6 p U x d x d 6 a e o 4 X L A 2 T Y f j A o C 5 5 N M R Z u G e u q R 1 Y 8 6 / f u r c T z 3 A b V C d W 0 Q E 7 8 2 k r c n i W T t + J n k O h a f 8 G D d b l Z u t N j d r X u F P a e E b z l 6 + f O m X W T T 0 J k K d T 9 H I y z M h I k w b 1 T r k s Y / h U 9 1 g 8 f D U G Z l j N L V M U j G m e 2 X z f 8 + y c r R R O 9 r 4 Q U c 4 6 T Q E f R I b W 1 H N R + P 1 k F A t l r e + X r 5 c q B Q c n h + H G S a F c d I y U e T M 8 G q j j 8 6 U u X o y 1 T X r a q 3 F l d W 8 v R 9 O o v a 1 U P S m 1 7 n T a + 0 H e z 1 Z 1 h x P r m r q T I A K N e N V V x / b F t O / K e B + 3 b / v 9 r d T r W + A o 1 b X i 6 d Z a Q 6 i W O k 1 V h d 4 r q O F u q o c j u I s K z S t r w y v L y s D p A b J r N t M O r L A W J j P d L E U + P F s s 2 l Q + n E 0 5 A Z v z X W R z A w 1 f + I S h G z v o J k 5 T Q l y o X J d L q I 4 S z W s V h e d Z I l X + k X k Y 8 y K S u M g / Z n y / S D T r E j Q 7 / M 7 h J 7 P 7 Y o W D d o n Z u A y A 2 I C l w m I G b r M c F 7 r r Q C X A u o 0 c p k R M a H L h M S M X W Z M T O Q y E T H v X O Y d G 9 y 9 S 9 1 T p 9 h l 4 r n W c p F 4 k c C 4 x c J 6 + K j O P L O L y 9 6 7 i Z D e M E u / l J 4 q b 1 G T j + o A c r b G S y r f q e s 7 p b t m L p M R k 7 t M z i b x 3 q X e U 6 f C Z Q p i h M s I 5 k 6 6 l K R O E 5 e Z E F O 6 T E n M 1 G W m x D y 4 z A M x j y 7 z y A b 3 w a U + z E 2 5 Z a M A k 3 R W 2 B g o q 1 C Z z e z D D Q V P P X A 5 N q F i L X S b 3 b B k J A u S c k A w i 5 A y I J i F R z k k e M h g I J j F R T k i e M S H E h L O Q q I c E 8 z i o Z w Q z I K h f E f w O w b f E 8 y i o I w J j h m c E J z w E b K 1 5 o u c E c w k X e Y E 5 w x + T z D T c l k Q z I R c C o I F 3 1 e C J R 8 h W x W u 3 5 J g J t 5 y S j B T b v l A M J N t + U j w I 4 M / E G z 1 u h W D e l r W T 4 C F q 9 3 q r D f C a z 2 h w a i v 9 Y w G I 8 H W U x q M D t v P a T B q b D 2 p w U i y 9 a w G I 8 v W 0 x q M N l v P a + Q W n t h g V N o 8 s y 2 3 8 N A G o 9 f m s W 2 5 x O U S Z / J t Z 3 J N L j y W w Y i 4 e T B b r u 1 k r s m F h z M Y U T e P Z 8 s t P J / B q L v 1 h A Y j 8 d Y z G o z O W 0 9 p M G J v P a f B K L 7 9 p A a j + 8 V n N U Z E E Q V 1 w Z K s U Z S s s Y B N 1 g l f p 6 B K N g j e Y P A m w Z s M 3 i J 4 i 8 H b B G 8 z e I f g H Q b v E r z L 4 D 2 C 9 / j A 9 w n f Z + Y H B B 8 w u E d w j 8 G H B B 8 y + I j g I w Y f E 3 z M h 9 I n v M / M T w g + Y f A p w a c M P i P 4 j M H n B J 8 z + I L g C w Z f E n z J R 3 h F + B U z v v g 3 K X L X X L u y u W u O H f t c t e c u 3 G 5 G 8 7 d u l w t / U t e D Z c f Q D 9 d 4 F P t a t 2 5 z F K Y 2 S d d i y U T A 9 0 l l D 7 q M l n h d Y l s q A p m y M A g A 1 p N X Z Q g R M W I L k U Q o X K 4 r I Z C d Y e u O h C h a k P X G o h Q j a E r D E S o s t B 1 B S I R u / 0 7 A 1 E Z o Y s I R K h 4 0 K U D I j F b C Y M k h K Q G S d k K G o R K A l 0 Q I E K F g C 4 D E K F i V u d + R C j n 6 4 y P C H s Q 0 4 k e I U r w Z b U z b F 9 K g 1 A y 1 6 k c E U r h O o E j Q o l b p 2 1 E q E j V u R q R D 2 3 H p v N U B a U f 5 2 O 1 5 / r f W o b l o B K I 1 o Y F 6 Q m M X l t U V O w n g 6 H q Y S 6 I y B I I F a 7 / J V j L U k n S B m i J H h H C v x k m o j B R n f W / 1 N n K t 5 J u P Z X Z j M 9 g p i R r W 6 j Y g F q o 1 i G b 1 k y p 1 L Z Q p S N q o U L D O b 2 z U f o c E 4 s j j q i F s n z H B o + a v G c L N F N a r K c / U z q 0 L V x R t p S o w Y x a q L + c W q i 9 9 9 R C 3 R V 8 t W Z K c P U i z Z T W b A u X e 0 I t 1 F l J L d T Y l F q o r w d q o b Y e q Y W 6 + j C v f i D D r P t Q 3 V q n X N Q b p V q d a B F Z p w D Q + R U h y q s 6 q y J C 2 V T n U k Q o h + o M i g h l T p 0 3 E a F a T i d L R C h J 6 h S J C K V G n R g R o Y S o 0 y E i l A Z 1 E k S E k p 9 O f Y h Q y t M J D x G q 0 3 S W Q 4 S V a D q 5 I U R J T a c 0 R C i V 6 U S G C C U w n b 4 Q o X J M 5 y x E z p n r C w N R i t I J C h F K T D o t I U L p S C c j R C g J 6 R S E y A 1 z f W u g W 7 a L l C Y G P E s k / X F 1 L N / h F V t A e w o o p l e d B P X 0 q m B W 3 J k J a H M 6 n U M q 1 K / E m x D E f g E o r f G a O o 3 w l q Y C F K N I v V S F N M i G U R q i N 3 8 S K 0 S M 6 u t k P h P q f f A Z y E U O B l k 8 / C E 3 g 4 c 5 x u K z 5 k v d V O h f F U 0 i r R z q N 9 r V 5 K Q p O l P B Y k C u W 4 z K T L l h s Q 1 a e L l p Q Y o E u W U x i g W 5 b T G K B r l j M Y o H u W s x i g i 5 Z z G K C b l v M Y o K e W A x i g v Z s 1 i P D f r Q g h Q c 8 s h i F B 7 y 2 G J U / s m + x f r M 4 Y k F K U j k q c U o T O S Z x S h Q 5 L n F q K y T F x a j W J G X F q N o k V c W o 3 i R 1 x a j i J E 3 F m M x I 2 8 t W N V p K O e d w s / H h g 7 t Y 3 D g P I i E 6 w w m c Y Q b D K Z j M t x k M C k k 3 G I w i S T c Z j D p J N x h M E k l 3 G U w q S X c Y z A J J t x n M G k m P G D w A S 1 O 2 G M 4 H a r h I Y N J P O E R g 0 k / 4 T G D S U J h n 8 F 0 z o Y n D D 5 h Q z l l O G k p P G M w y S k 8 Z z A p K r x g M I k q v G Q w e w w O r x h O 2 g q v G U z y C m 8 Y T A 8 E 4 S 2 D j c b U r + + y q t 9 E / Z 5 l w N + z i H W C S V 9 i g 1 C S l 9 g k V K v r h b e p f / m Y C P B 8 T 4 D 0 8 N 4 x D L 2 t Z W 8 A g a 9 w O Y 6 E N 8 0 m 8 R A h b I E n 9 O 8 k W G F O C k 9 9 4 J P F 6 E h 9 L A M P O V a c + j f g 6 s d Q s U 1 3 J I W K H U J J o G K X U N K n 2 C O U v a U R + w S T P M U B o X S o i R 6 h p E 1 x S O g h c 3 x E M E l T H B N K y h R 9 Q k m Y 4 o R Q O t / E K a G k S n F G K H u G F e c E k y j F B a G k S X F J K B 1 1 4 o r Q K + b 4 m m A S p L g h l P Q o b g m 1 c t x K s R Y E / X D h m 9 c x D 1 X B S A V B z 3 0 s U B X j G r V Q r u v U Q p l u U F m r q s R N I v H U 2 6 I W K m m b W q i g H W q h c n a p h Y r Z o x Y K Z Z 9 a K J A D a q E w e t R C Q R y y w a A Q j o h E A R x T C z e + z 0 x x x 0 + I x J 0 + p R b u 8 B m 1 c G P P q Y U b e k E t 3 M h L a u E G X l E L 9 + 2 a W r h f N + z 2 u F G 3 7 I Z V 8 V U V X m o I 2 v B J c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 s 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e q 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 i h 7 0 a Y r K 2 i U V q B r K 6 D i y s L 9 G u 5 z + K S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q l h 9 o Z M l 1 z A a y 5 8 g s A E J I s J e J N 0 C E X 8 q D 5 9 G v r S 9 0 J I o c D c o 9 q R Q L 0 P J i o R u d + F V 1 + 4 C u / 2 3 5 n X 2 2 2 q O p C f Y h z + 9 J n s 6 5 5 m + 4 R Z V x l a 5 0 T M u 5 v b 1 n E s U E M 7 r F 2 5 N K p A 0 S d W K R p F a P d d M Z C O Z + A 9 k a Y G m H W a N T H 8 J Z d 7 E P f W S x x M 1 / Q / q L U F j c g e 9 O f 8 O 6 q D 3 Z A 8 v / Y K G o B r N G 0 j 8 x y 9 w + 4 u M W Z 4 9 2 Y K N r C R a N X S i u 8 r i U e E n 6 p 3 V e J o V W L M K / 1 F 4 z 3 v f v X q u P g D S X 6 t P U v N F q 8 h R A k J / h f b 8 D u K Y 2 d g 3 p y + 8 d U y F G P e p + u s R g x 4 S 9 T W c K o y N U 2 a t P k f N J q H O n r p O j i Q s a / c i 8 4 Y Z K H f T 6 D 7 K Y R j 5 K 4 0 P m r M i i d W b / v m s 9 9 3 q v I X M U l B c t 4 2 T U 9 3 v V X M / N J k r K m / p p u X Q + + 4 u S k f y s d k 1 9 w v 1 G h n P D l 8 F z B n g i S v 8 E L w o 9 d K s K v I l P K x 4 G + N M q P X J V E E Y j L 1 N f C Z O 4 U v h D b L s f k W F J X v b c 5 y r Q z o r / o Q 6 L 0 I 9 A v z 3 b l l d f c x Q H Z f G M N L v b F o s t W L R T P + 9 w O I c N X W u P h W M Q d 7 5 A 4 y 1 O J s O C v D v n 7 3 9 f K n b / H 8 l n l 5 c v l r p / n n l 9 c n r p W / W q / + P 4 r P O 7 z t / 6 H z V 6 X b + 0 v m m s 9 v p d y 4 6 Q S f t / L 3 z j 8 4 / u / / q / r v 7 f f c / x v T T T 6 o + v + s 4 f 7 r / / R + P Q l N k < / l a t e x i t > x Layer input

EMA output

< l a t e x i t s h a 1 _ b a s e 6 4 = " e l 2 k j u N  X N N 2 b Q y u J n h 8 m m S f 2 t m s = " > A A A y K 3 i c l V v J k t v I E e W M t 7 G 8 z d h H X x B u K W b s a H W I G o X t 4 / S + s b v Z + 6 L W K E A w C U K N T a g i 2 C 0 G / R e + 2 l / g r / H J D l / 9 H 8 6 q Q i G z 0 K A U V s R I q P e y E r W 8 r E y A m E E e R 0 K + e P G v z z 7 / w Q 9 / 9 O O f f P H T J z / 7 + S 9 + + a s v v / r 1 h c g m R Q D n Q R Z n x d X A F x B H K Z z L S M Z w l R f g J 4 M Y L g d 3 6 4 q / L K E Q U Z a e y Y c c 3 i R + m E a j K P A l Q q 9 v E 1 + O B 6 P Z / d f z t 1 8 u v V h 5 o f 9 4 j y + 6 1 c V S p / r T f / t V 9 8 n t M A s m C a Q y i H 0 h X n d f 5 P L N z C 9 k F M Q w f 3 I 7 E Z D 7 w Z 0 f w m u 8 T P 0 E x J u Z H v P c e 4 b I 0 B t l B f 6 X S k + j v M f M T 4 R 4 S A Z o q c Y o m p w C F 3 H K o 1 g e J I 0 h y N G f 3 8 y i N J 9 I S A M z g t E k 9 m T m q Z X x h l E B g Y w f 8 M I P i g g n 4 Q V j v / A D i e v n 3 O L e z O H J k 2 f e g V / c e Q L 7 4 W o K L x t 5 g Z + b a z W 1 A k Z Q F F E a q p s M o z I S 1 m w U h Z M C 0 G 0 K 0 y B L E j 8 d z m 4 R j G E k 5 7 P Z L S T e N z 2 8 / v 0 c b 9 M 0 C n D F o b B m 6 7 q l D J t 2 R R S O a 2 8 n q t F m J b P c 2 p x l e Z v F I J M y S 6 z R m m 4 9 s q s m 7 l s z / / H Q K 5 O B N R k s c h J Y i 2 C R x d B a D P V t n n k 7 O L 1 Y T d H z P b R X 2 w g j D I q h h 4 u T u D 7 w W o H z 1 9 0 3 6 G U w 8 p a 6 y g l 6 2 d L b Y v Y N R Q L L X p x N o X g e Y I i t o N x h p N c V R k v d m d n C v 9 x i a 2 Y c t P X H 8 U b S j 1 e 8 L d S D k B g c a v u F 2 j P k j c s t 6 3 K r 6 V L T c p r Z m y 6 9 r G 4 r P G v k 4 Z S q x k v b 4 / 3 E H 1 K X p W + X X j 3 q t l z 3 s V f f c l e v 0 N U z 7 9 Q I e 9 F 6 m J u h + s 3 o q z B g w 2 / 1 Y F f E 9 D 6 1 v U 9 b e p / Y X j p I p 1 k d a S v 1 y p i 7 C 7 0 0 d R w u W J u m w 3 E B w F z y 6 Q i z c I 9 d 0 r o x 5 9 8 + d u 6 n H u A 2 q M 4 t I o L 3 Z t L W Z P G s H T + T P I f C U 3 6 M m 8 3 K z W a b m 1 W v 8 K e 0 8 A 1 n z 5 8 / 9 8 s s G n o T o c 6 n a O T l m R A R 5 o 1 q H f L Y x / C p b r B 4 e O q M z D G a W i a p G N O 9 s v m / Z 1 k 5 W q 8 d r X / S E U 4 6 D U G f x M Z W V P P R e D 0 k V I v l r a / n z x c q B Y f n x 2 G G S W G c t E w U O T O 8 2 u i j M 2 W u H k 1 1 1 b p a b X F l N W / v h 5 O o f S 0 U v e l 1 5 v R a / W S v R 8 u a 4 8 l V T Z 0 J U K F m v O r q Y 9 t i + j c F 3 K / 7 9 9 3 + d q r 1 D X D U 6 n r x N C v N Q R Q r v c b q A s 9 1 t F B X l c N R n G W F p v W V 4 f V l Z Y D U I J l 1 m 0 l H F h g L 8 5 m u l g I / n m 0 0 D U o / j o b c 4 K 2 5 L p K Z o e a P X I K Q 7 R 0 0 M 6 c p Q S 5 U r s t F F G e p h t X q o p M s 8 U q / i H y M W V F p H K Q / U 7 7 v Z Z o V C f p 9 e o v Q 0 7 l d 0 a J B + 8 Q M X G Z A T O A y A T F D l x n O a 7 0 V 4 F J A n U Y u M y I m d J m Q m L H L j I m J X C Y i 5 p 3 L v G O D u 3 O p O + o U u 0 w 8 1 1 o u E i 8 S G L d Y W Q 8 f 1 J l n d n H Z e z c R 0 h t m 6 d f S U + U t a v J B H U D O 1 n h J 5 T t 1 f a d 0 1 8 x l M m J y l 8 n Z J N 6 7 1 H v q V L h M Q Y x w G c H c S Z e S 1 G n i M h N i S p c p i Z m 6 z J S Y e 5 e 5 J + b B Z R 7 Y 4 D 6 4 1 I e 5 K b d s F G C S z g o b A 2 U V K r O Z f b q h 4 K k H L s c m V K y F b r M b l o x k Q V I O C G Y R U g Y E s / A o h w Q P G Q w E s 7 g o R w S P + F B C w l l I l G O C W T y U E 4 J Z M J T v C H 7 H 4 D u C W R S U M c E x g x O C E z 5 C t t Z 8 k T O C m a T L n O C c w e 8 J Z l o u C 4 K Z k E t B s O D 7 S r D k I 2 S r w v V b E s z E W 0 4 J Z s o t 7 w l m s i 0 f C H 5 g 8 A e C r V 4 3 Y 1 B P y / o J s H C 1 W 5 3 1 R n i t J z Q Y 9 b W e 0 W A k 2 H p K g 9 F h + z k N R o 2 t J z U Y S b a e 1 W B k 2 X p a g 9 F m 6 3 m N 3 M I T G 4 x K m 2 e 2 5 R Y e 2 m D 0 2 j y 2 L Z e 4 X O J M v u 1 M r s m F x z I Y E T c P Z s u 1 n c w 1 u f B w B i P q 5 v F s u Y X n M x h 1 t 5 7 Q Y C T e e k a D 0 X n r K Q 1 G 7 K 3 n N B j F t 5 / U Y H S / + K z G i C i i o C 5 Y k l W K k l U W s M k a 4 W s U V M k 6 w e s M 3 i B 4 g 8 G b B G 8 y e I v g L Q Z v E 7 z N 4 B 2 C d x i 8 S / A u H / g e 4 X v M f J / g f Q b 3 C O 4 x + I D g A w Y f E n z I 4 C O C j / h Q + o T 3 m f k x w c c M P i H 4 h M G n B J 8 y + I z g M w a f E 3 z O 4 A u C L / g I L w m / Z O Z X B F 8 x + J r g a w b f E H x j 6 u b W k 9 Z V H h j p M a G u M o V r + T F u j X P r L r f O u Q 2 X 2 + D c c L k L z l 2 6 3 C X n r l z u i n P X L n f N u R u X q 6 V / w a v h 8 g P o p w t 8 q n 1 R d y 6 z F G b 2 S d d i y c R A t w m l j 7 p M V n h d I h u q g h k y M M i A V l M X J Q h R M a J L E U S o H C 6 r o V D d o a s O R K j a 0 L U G I l R j 6 A o D E a o s d F 2 B S M R u / 8 5 A V E b o I g I R K h 5 0 6 Y B I z F b C I A k h q U F S t o I G o Z J A F w S I U C G g y w B E q J j V u R 8 R y v k 6 4 y P C H s R 0 o k e I E n x Z 7 Q z b l 9 I g l M x 1 K k e E U r h O 4 I h Q 4 t Z p G x E q U n W u R u R D 2 7 H p P F V B 6 c f 5 W O 2 5 / r e W Y T m o B K K 1 Y U F 6 A q P X F h U V + 8 l g q H q Y C y K y B E K F 6 3 8 J 1 r J U k r Q B W q J H h P B v h o k o T F R n / S 9 1 t v K t p F t P Z T b j M 5 g p y d o W K j a g F q p 1 y K Y 1 U y q 1 L V T p i F q o 0 H B O 7 2 y U P s f E 4 o g j a q E s 3 7 H B o y b v 2 A L N l B b r 6 c + U D m 0 L V 5 Q t J W o w o x b q L 6 c W a u 8 9 t V B 3 B V + t m R J c v U g z p T X b w u W e U A t 1 V l I L N T a l F u r r n l q o r Q d q o a 4 + z K s f y D D r 3 l e 3 1 i k X 9 U a p V i d a R N Y o A H R + R Y j y q s 6 q i F A 2 1 b k U E c q h O o M i Q p l T 5 0 1 E q J b T y R I R S p I 6 R S J C q V E n R k Q o I e p 0 i A i l Q Z 0 E E a H k p 1 M f I p T y d M J D h O o 0 n e U Q Y S W a T m 4 I U V L T K Q 0 R S m U 6 k S F C C U y n L 0 S o H N M 5 C 5 E z 5 v r c Q J S i d I J C h B K T T k u I U D r S y Q g R S k I 6 B S F y z V z f G O i G 7 S K l i Q H P E k l / X B 3 L t 3 j F F t C e A o r p V S d B P b 0 q m B V 3 a g L a n E 5 n k A r 1 K / E G B L F f A E p r v K p O I 7 y l q Q D F K F I v V S E N s m G U h u j N n 8 Q K E a P 6 O p n P h H o f f A p y k Y N B F g 8 / 5 W Z w P 8 d Y f N J 8 q Z s K / a u i S a S V Q / 1 G u 5 q c N E V n K l g M y D W L U Z k p 1 y 2 2 T g s v N y x I k S A 3 L U a x I L c s R t E g t y 1 G 8 S B 3 L E Y R I X c t R j E h 9 y x G U S H 3 L U Z x I X s W 6 7 F B H 1 i Q g k M e W o z C Q x 5 Z j M o / 2 b d Y n z k 8 t i A F i T y x G I W J P L U Y B Y o 8 s x i V d f L c Y h Q r 8 s J i F C 3 y 0 m I U L / L K Y h Q x 8 t p i L G b k j Q W r O g 3 l v F 3 4 + d j Q o X 0 M D p w H k X C N w S S O c J 3 B d E y G G w w m h Y S b D C a R h F s M J p 2 E 2 w w m q Y Q 7 D C a 1 h L s M J s G E e w w m z Y T 7 D N 6 n x Q l 7 D K d D N T x g M I k n P G Q w 6 S c 8 Y j B J K O w z m M 7 Z 8 J j B x 2 w o J w w n L Y W n D C Y 5 h W c M J k W F 5 w w m U Y U X D G a P w e E l w 0 l b 4 R W D S V 7 h N Y P p g S C 8 Y b D R m P r 1 X V b 1 m 6 j f s w z 4 e x a x R j D p S 6 w T S v I S G 4 R q d T 3 z N v Q v H x M B n u 8 J k B 7 e O 4 a h t 7 n s D S D w F S 7 H k f C m 2 S Q e I o Q t 8 I T + n Q Q r z E n h q Q 9 8 s h g d q Y 9 l 4 D 7 H i l P / B l z 9 G C q 2 6 I 6 k U L F N K A l U 7 B B K + h S 7 h L K 3 N G K P Y J K n 2 C e U D j X R I 5 S 0 K Q 4 I P W C O D w k m a Y o j Q k m Z o k 8 o C V M c E 0 r n m z g h l F Q p T g l l z 7 D i j G A S p T g n l D Q p L g i l o 0 5 c E n r J H F 8 R T I I U 1 4 S S H s U N o V a O m y n W g q A f L n z z O u a + K h i p I O i 5 j w W q Y l y l F s p 1 j V o o 0 3 U q a 1 W V u E E k n n q b 1 E I l b V E L F b R N L V T O D r V Q M b v U Q q H s U Q s F s k 8 t F E a P W i i I A z Y Y F M I h k S i A I 2 r h x v e Z K e 7 4 M Z G 4 0 y f U w h 0 + p R Z u 7 B m 1 c E P P q Y U b e U E t 3 M B L a u G + X V E L 9 + u a 3 R 4 3 6 o b d s C q + q s J L b R v w b Z O m B l O H i w p k / f E e R r O B l 7 1 p J M f Z R H p Y A H l T z G 8 5 F G 6 J B F Q j O f V R d X 9 Z C 0 E b P i o O Q R d Q 0 K i g Q J d Q 0 K i h Q B d R Q F V U h W 7 U K H s 9 p + s o a B R S o C s p a J R S o G s p a B R T o K s p o H L K w r s 1 z F 7 A 6 Z I K G j U V 6 K I K G l U V 6 L I K q K 6 q 0 I M a Z S / a d G U F j d I K d G 0 F V F x Z u F / D f Q 4 f 1 z B 7 o 6 Z L L G j U W K C L L G h U W a D L L G j U W a A L L W h U W q B L L W j U W q C L L W h U W 6 D L L a B 6 y 8 L X N c z e k O m S C 3 j N h U 8 Q m I B k M Q F v k g 6 h i B / U p 0 9 D X / p e C C k U m H t U O x K o 9 8 F E J S J X u 7 k y n c / y t 7 P b I p n p h k 6 D y i s k e V R E m A C d / v V H i I M H n f z 0 h y T q J p g t G 7 7 t N y Z j X + K D v H s L x 7 L P L f v z t s E k 2 R D i j 0 1 E G 9 Q z M a 1 5 8 2 m m X 1 n 1 G 1 a N c U f x E C r L W 9 2 o h 1 / 3 w M N C Z s H Y F + q j W n 8 i M / 1 o B Y U z x M b H s L m x q Q d Z d X k 8 g C E 4 d q b Z Y l c g g U e P t T N N F E N g f i h 2 r W M / j / 0 A 5 v V X O b 0 K m H v P v O r a X W C 3 / + a 8 z n 6 b z Z H 0 B P v w p 9 d k T + Y 8 0 z f c o s r Y K j d 6 x s X c v p 5 z i Q L C e f 3 C r U k F k i a p W t E o U q v n m o l s J B P / n i w t 0 L T D r J H p L 6 H M m 7 j H X v J 4 o q b / Q b 0 l a E x u v z f n 3 0 H t 9 x 7 t 4 Y V f 0 B B U o 3 k D i f / 4 B W 5 / k T H L 0 0 d b s J 6 V R K u G T n S X W T w q / E S 9 s x p P s w J r V u E / C O 9 p 7 / u X T 9 U H Q P p r 9 U l q v m g V O U p A 6 K / Q n t 5 C H D M b + + b 0 m b e G q R D j P l V / P W D Q Q 6 K + h l O F s X H K r N X n q N k k 1 N l T 1 8 m R h G X t X m T e M A P l b h r d R T k M I 3 + l 8 U F z V i S x e t M / n / W + f z F v I b M U F N d t 4 + R U 9 3 v Z 3 A 9 N 5 o r K W 7 p p O f S + v 4 3 S k X x o d s 3 9 Q r 1 G x r P D V w F z C n j i C j 8 E L 0 q 9 N K u K f A n 3 K 9 7 6 O B N q f T J V E A Z j b w O f i V P 4 W n i D L L t b U W H J 3 v Y c 5 e q Q z o o / o M 6 L U I 8 A / 7 1 d V l c f M 1 T H p T G M 9 D u b F k u t W D T T f y + w O E N N n a l P B W O Q t / 4 A Y y 3 O p o M C / L s n b 7 9 c 6 j b / X 4 n H F x c v V 7 p / X H l 1 / G r p u 7 X q / 6 P 4 o v P b z u 8 6 3 3 S 6 n T 9 1 v u v s d P q d 8 0 7 Q y T p / 7 f y t 8 / f u P 7 r / 7 P 6 7 + x 9 j + v l n V Z / f d J w / 3 f / + D 3 y c U 5 U = < / l a t e x i t > x 0 Gate < l a t e x i t s h a 1 _ b a s e 6 4 = " h I x 9 X T h z v l 7 x 2 x j m g V y u O d 5 J 6 w 8 = " > A A A y J n i c l V t Z k 9 v G E V 4 7 l 6 N c d v K Y F 1 R W K j u p 1 Z Y o q 5 I 8 e u + L u 8 u 9 D 1 N W g W A T x C 4 u Y Y b g r l j M b 8 h r 8 g v y a / K W S u U t P y U 9 M x h 0 D x a U K 6 q y h P m + n s Y c X 0 8 3 Q H i Q x 5 G Q r 1 7 9 5 5 N P f / D D H / 3 4 J 5 / 9 9 N n P f v 6 L X / 7 q 8 y 9 + f S m y S R H A R Z D F W X E 9 8 A X E U Q o X M p I x X O c F + M k g h q v B / Y b i r 0 o o R J S l 5 / I x h 7 e J H 6 b R K A p 8 i d B l P / S T x H / 3 + f K r 1 V f 6 j / f 0 o l N d L C 9 V f 3 r v v u g 8 6 w + z Y J J A K o P Y F + L b z q t c v p 3 5 h Y y C G O b P + h M B u R / c + y F 8 i 5 e p n 4 B 4 O 9 P D n X s v E B l 6 o 6 z A / 1 L p a Z T 3 m P m J E I / J A C 0 T X 4 5 F k 1 P g I k 5 5 F C u D p D E E O f r z 2 1 m U 5 h M J a W B G M J r E n s w 8 t S j e M C o g k P E j X v h B E e E k v G D s F 3 4 g c e m c W z y Y O T x 7 9 s I 7 9 I t 7 T 2 A / X E j h Z S M v 8 H N z r a Z W w A i K I k p D d Z N h V E b C m o 2 i c F I A u k 1 h G m S 4 / O l w 1 k c w h p G c z 2 Z 9 S L y v u n j 9 + z n e p m k U 4 I p D Y c 0 2 d E s Z N u 2 K K B z X 3 k 5 V o 8 1 K Z r m 1 O c / y N o t B J m W W W K N 1 3 X p i V 0 3 c t 2 b + 0 6 F X J g N r M l j k J L A W w S K L o b U Y 6 t u 8 8 H Z x e r G a o u d 7 a K + 2 E U Y Y D 0 M P F y d x f e C 1 A u f f d t 6 i l 8 H I W + 4 o J + h l W 2 + L 2 T c U C a x 4 c T a F 4 m W A 0 b W K c o e R X l c Y L X d m Z g v / 0 s f W z D h o 6 4 / j j a Q f r 3 r b q A c h M T j U 9 g u 1 Z 8 g b l 9 v W 5 X b T p a b l N L M 3 X X 5 d 3 V Z 4 1 s j D K V W N 1 7 b H + 4 k / p C 7 L X y + / e d J t p e 5 j r 7 7 m r t 6 g q x f e m R H 2 o v U w N 0 P 1 m 9 F X Y c C G 3 + r B r o j p f W Z 7 n 7 X 0 P r W 9 d J B O s z r S V u u V M X c X e m n q O F y w N k 2 H 4 w K A u e T T E W b h n r q k d W P O v 3 7 q 3 E 8 9 w G 1 Q n V t E B O / N p K 3 J 4 l k 7 f i Z 5 D o W n / B g 3 W 5 W b r T Y 3 a 1 7 h T 2 n h G 8 5 e v n z p l 1 k 0 9 C Z C n U / R y M s z I S J M G d U 6 5 L G P 4 V P d Y P H w 1 B m Z Y z S 1 T F I x p n t l 8 3 / P s n K 0 U T v a + F 5 H O O k 0 B H 0 S G 1 t R z U f j 9 Z B Q L Z a 3 v l 6 + X K g U H J 4 f h x k m h X H S M l H k z P B q o 4 / O l L l 6 M t U 1 6 2 q t x Z X V v L 0 f T q L 2 t V D 0 p t e 5 0 2 v t e 3 s 9 W d Y c T 6 5 q 6 k y A C j X j V V c f 2 x b T v y n g X t 2 / 5 / a 3 U 6 1 v g K N W 1 4 u n W W k O o l j p N V Y X e K 6 j h b q q H I 7 i L C s 0 r a 8 M r y 8 r A 6 Q G y a z T T D q y w F i Y z / q q 2 g j 8 e L b Z N C j 9 O B p y g 3 f m u k h m h p o / c Q l C t n f Q z J y m B L l Q u S 4 X U Z y l G l a r i 0 6 y x C v 9 I v I x Z k W l c Z D + T P l + k G l W J O j 3 e R + h 5 3 O 7 o k W D 9 o k Z u M y A m M B l A m K G L j O c 1 3 o r w K W A O o 1 c Z k R M 6 D I h M W O X G R M T u U x E z J 3 L 3 L H B 3 b v U P X W K X S a e a y 0 X i R c J j F s s q o e P 6 s w z u 7 j i 3 U 2 E 9 I  Z Z + q X 0 V H m L m n x U B 5 C z N V 5 S + U 5 d 3 y n d N X O Z j J j c Z X I 2 i f c u 9 Z 4 6 F S 5 T E C N c R j B 3 0 q U k d Z q 4 z I S Y 0 m V K Y q Y u M y X m w W U e i H l 0 m U c 2 u A 8 u 9 W F u y i 0 b B Z i k s 8 L G Q F m F y s x E 1 G D E g q c e u B y b U L E W u s 1 u W D K S B U k 5 I J h F S B k Q z M K j H B I 8 Z D A Q z O K i H B E 8 4 k M J C W c h U Y 4 J Z v F Q T g h m w V D e E X z H 4 H u C W R S U M c E x g x O C E z 5 C t t Z 8 k T O C m a T L n O C c w e 8 J Z l o u C 4 K Z k E t B s O D 7 S r D k I 2 S r w v V b E s z E W 0 4 J Z s o t H w h m s i 0 f C X 5 k 8 A e C r V 6 3 Y l B P y / o J s H C 1 W 5 3 1 R n i t J z Q Y 9 b W e 0 W A k 2 H p K g 9 F h + z k N R o 2 t J z U Y S b a e 1 W B k 2 X p a g 9 F m 6 3 m N 3 M I T G 4 x K m 2 e 2 5 R Y e 2 m D 0 2 j y 2 L Z e 4 X O J M v u 1 M r s m F x z I Y E T c P Z s u 1 n c w 1 u f B w B i P q 5 v F s u Y X n M x h 1 t 5 7 Q Y C T e e k a D 0 X n r K Q 1 G 7 K 3 n N B j F t 5 / U Y H S / + K z G i C i i o C 5 Y k j W K k j U W s M k 6 4 e s U V M k G w R s M 3 i R 4 k 8 F b B G 8 x e J v g b Q b v E L z D 4 F 2 C d x m 8 R / A e H / g + 4 f v M / I D g A w Z 3 C e 4 y + J D g Q w Y f E X z E 4 G O C j / l Q e o T 3 m P k J w S c M P i X 4 l M F n B J 8 x + J z g c w Z f E H z B 4 E u C L / k I r w i / Y u b X B F 8 z + I b g G w b f E n x r 6 u b W k 9 Z V H h j p M a G u M Y V r + T F u n X M b L r f B u U 2 X 2 + T c l s t t c W n M 3 L n f D u V u X q 6 V / y a v h 8 g P o p w t 8 q n 1 V d y 6 z F G b 2 S d d i y c R A / Y T S R 1 0 m K 7 w u k Q 1 V w Q w Z G G R A q 6 m L E o S o G N G l C C J U D p f V U K j u 0 F U H I l R t 6 F o D E a o x d I W B C F U W u q 5 A J G K 3 v z M Q l R G 6 i E C E i g d d O i A S s 5 U w S E J I a p C U r a B B q C T Q B Q E i V A j o M g A R K m Z 1 7 k e E c r 7 O + I i w B z G d 6 B G i B F 9 W O 8 P 2 p T Q I J X O d y h G h F K 4 T O C K U u H X a R o S K V J 2 r E f n Q d m w 6 T 1 V Q + n E + V n u u / 6 1 l W A 4 q g W h t W J C e w O i 1 R U X F f j I Y q h 7 m g o g s g V D h + l + C t S y V J G 2 A l u g R I f y b Y S I K E 9 V Z / 0 u d r X w r 6 d Z T m c 3 4 D G Z K s r a F i g 2 o h W o d s m n N l E p t C 1 U 6 o h Y q N J z T O x u l z z G x O O K I W i j L O z Z 4 1 O Q 9 W 6 C Z 0 m I 9 / Z n S o W 3 h i r K l R A 1 m 1 E L 9 5 d R C 7 b 2 n F u q u 4 K s 1 U 4 K r F 2 m m t G Z b u N w T a q H O S m q h x q b U Q n 0 9 U A u 1 9 U g t 1 N W H e f U D G W b d h + r W O u W i 3 i j V 6 k S L y D o F g M 6 v C F F e 1 V k V E c q m O p c i Q j l U Z 1 B E K H P q v I k I 1 X I 6 W S J C S V K n S E Q o N e r E i A g l R J 0 O E a E 0 q J M g I p T 8 d O p D h F K e T n i I U J 2 m s x w i r E T T y Q 0 h S m o 6 p S F C q U w n M k Q o g e n 0 h Q i V Y z p n I X L O X F 8 Y i F K U T l C I U G L S a Q k R S k c 6 G S F C S U i n I E R u m O t b A 9 2 y X a Q 0 M e B Z I u m N q 2 O 5 j 1 d s A e 0 p o J h u d R L U 0 6 u C W X F n J q D N 6 X Q O q V C / E m 9 C E P s F o L T G a + o 0 w l u a C l C M I v V S F d I g G 0 Z p i N 7 8 S a w Q M a q v k / l M q P f B Z y A X O R h k 8 f D 7 3 A w e 5 h i L z 5 o v d V O h f 1 U 0 i b R y q N 9 o V 5 O T p u h M B Y s B u W 4 x K j P l h s U 2 a O H l p g U p E u S W x S g W 5 L b F K B r k j s U o H u S u x S g i 5 J 7 F K C b k v s U o K u S B x S g u Z N d i X T b o Q w t S c M g j i 1 F 4 y G O L U f k n e x b r M Y c n F q Q g k a c W o z C R Z x a j Q J H n F q O y T l 5 Y j G J F X l q M o k V e W Y z i R V 5 b j C J G 3 l i M x Y y 8 t W B V p 6 G c d w o / H x s 6 t I / B g f M g E q 4 z m M Q R b j C Y j s l w k 8 G k k H C L w S S S c J v B p J N w h 8 E k l X C X w a S W c I / B J J h w n 8 G k m f C A w Q e 0 O G G X 4 X S o h o c M J v G E R w w m / Y T H D C Y J h T 0 G 0 z k b n j D 4 h A 3 l l O G k p f C M w S S n 8 J z B p K j w g s E k q v C S w e w x O L x i O G k r v G Y w y S u 8 Y T A 9 E I S 3 D D Y a U 7 + + y 6 p + E / V 7 l g F / z y L W C S Z 9 i Q 1 C S V 5 i k 1 C t r h f e p v 7 l Y y L A 8 z 0 B 0 s N 7 x z D 0 t l a 8 A Q S + w u U 4 E t 4 0 m 8 R D h L A F n t C / k 2 C F O S k 8 9 Y F P F q M j 9 b E M P O R Y c e r f g K s f Q 8 U 2 3 Z E U K n Y I J Y G K X U J J n 2 K P U P a W R u w T T P I U B 4 T S o S a 6 h J I 2 x S G h h 8 z x E c E k T X F M K C l T 9 A g l Y Y o T Q u l 8 E 6 e E k i r F G a H s G V a c E 0 y i F B e E k i b F J a F 0 1 I k r Q q + Y 4 2 u C S Z D i h l D S o 7 g l 1 M p x K 8 V a E P T D h W 9 e x z x U B S M V B F 3 3 s U B V j G v U Q r m u U w t l u k F l r a o S N 4 n E U 2 + L W q i k b W q h g n a o h c r Z p R Y q Z o 9 a K J R 9 a q F A D q i F w u h S C w V x y A a D Q j g i E g V w T C 3 c + B 4 z x R 0 / I R J 3 + p R a u M N n 1 M K N P a c W b u g F t X A j L 6 m F G 3 h F L d y 3 a 2 r h f t 2 w 2 + N G 3 b I b V s V X V X i p b Q O + b d L U Y O p w U Y G s P 9 7 D a D b w i j e N 5 D i b S A 8 L I G + K + S 2 H w i 2 R g G o k p z 6 q 7 i 9 r I W j D J 8 U h 6 A I K G h U U 6 B I K G j U U 6 C I K q I q q 0 M 0 a Z a / n d B 0 F j U I K d C U F j V I K d C 0 F j W I K d D U F V E 5 Z e K + G 2 Q s 4 X V J B o 6 Y C X V R B o 6 o C X V Y B 1 V U V e l i j 7 E W b r q y g U V q B r q 2 A i i s L 9 2 q 4 x + G T G m Z v 1 H S J B Y 0 a C 3 S R B Y 0 q C 3 S Z B Y 0 6 C 3 S h B Y 1 K C 3 S p B Y 1 a C 3 S x B Y 1 q C 3 S 5 B V R v W f i m h t k b M l 1 y A a + 5 8 A k C E 5 A s J u B N 0 i E U 8 a P 6 9 G n o S 9 8 L I Y U C c 4 9 q R w L 1 P p i o R O R q N 1 e m 8 1 n + b t Y v k p l u 6 D S o v E K S R 0 W E C d D p X 3 + E O H j U y U 9 / S K J u g t m y 4 d t + Y z L 2 J T 7 I u 7 d w L H v c s j d v G 0 y S D S H + 2 E S 0 Q T 0 T 0 5 o 3 n 2 Z 6 l V W v Y d U Y d x Q P o b L s 6 0 Y 9 / L o H H h Y y C 8 a + U B / V + h O Z 6 U c r K J w h N j 6 G z Y 1 N P c i q y 9 M B D M G x M 8 0 W u w I J P H q s n W m i G A L z Q 7 F r H f t 5 7 A c w r 7 / K 6 V b A 3 H v h V d f u A r v 9 t + Z 1 9 t t q j q Q r 2 I c / 3 S Z 7 O u e Z v u E W V c Z W u d E z L u b 2 9 Z x L F B D O 6 x d u T S q Q N E n V i k a R W j 3 X T G Q j m f g P Z G m B p h 1 m j U x / C W X e x D 3 1 k s c T N f 0 P 6 i 1 B Y 3 I H 3 T n / D u q g + 2 Q P L / 2 C h q A a z R t I / M c v c P u L j F m e P d m C j a w k W j V 0 o r v K 4 l H h J + q d 1 X i a F V i z C v 9 R e M + 7 3 7 1 + r j 4 A 0 l + r T 1 L z R a v I U Q J C f 4 X 2 v A 9 x z G z s m 9 M X 3 j q m Q o z 7 V P 3 1 i E E P i f o a T h X G x i m z V p + j Z p N Q Z 0 9 d J 0 c S V r R 7 k X n D D J S 7 a X Q f 5 T C M / N X G B 8 1 Z k c T q T f 9 8 1 v 3 u 1 b y F z F J Q X K e N k 1 P d 7 3 V z P z S Z K y p v 6 a b l 0 P 2 u H 6 U j + d j s m v u F e o 2 M Z 4 e v A u Y M 8 M Q V f g h e l H p p V h X 5 E h 5 W v Y 1 x J t T 6 Z K o g D M b e J j 4 T p / C l 8 A Z Z d r + q w p K 9 7 T n O 1 S G d F X 9 A n R e h H g H + 2 1 9 R V x 8 z V M e l M Y z 0 O 5 s W S 6 1 Y N N N / L 7 A 4 R 0 2 d q 0 8 F Y 5 B 9 f 4 C x F m f T Q Q H + / b N 3 n y 9 3 m v + v x N O L y 9 e r n T + u v j l 5 s / z N e v X / U X y 2 9 N u l 3 y 1 9 t d R Z + t P S N 0 u 7 S 7 2 l i 6 V g 6 W 7 p r 0 t / W / p 7 5 x + d f 3 b + 1 f m 3 M f 3 0 k 6 r P b 5 a c P 5 3 / / g 8 Y F F F d < / l a t e x i t >

Gate

< l a t e x i t s h a 1 _ b a s e 6 4 = " M r l 2 v q 5 b V  F J E L B E Y L i e O I / 8 A 3 H w = " > A A A y J H i c l V v J c t z I E e W M t z G 9 a e w I X 3 x B m J J n 7 K A Y p G b C 9 n G 4 b 0 2 y u S / T G g U a n Y 2 G i E 2 o a p B U u / 0 x P v h i f 4 p v D h 9 8 8 W f 4 7 K w q F D I L j d a E F S E R 9 V 5 W o p a X l Q k Q 6 u d x J O T q 6 r 8 / + v g 7 3 / 3 e 9 3 / w y Q 8 X f / T j n / z 0 Z 8 8 + / f m V y M Z F A J d B F m f F T d 8 X E E c p X M p I x n C T F + A n / R i u + / e b i r 8 u o R B R l l 7 I p x x e J 3 6 Y R s M o 8 C V C b 5 7 9 s p f 4 c t Q f T k 6 n v d / Y 6 8 P p m 2 d L q y u r + o 8 3 e 7 F W X S w t V H + 6 b z 5 d / W 9 v k A X j B F I Z x L 4 Q X 6 + t 5 v L 1 x C 9 k F M Q w X e y N B e R + c O + H 8 D V e p n 4 C 4 v V E T 2 D q v U B k 4 A 2 z A v + m 0 t M o 7 z H x E y G e k j 5 a q i G K J q f A e Z z y K J b 7 S W M I c v j H 1 5 M o z c c S 0 s C M Y D i O P Z l 5 a p m 8 Q V R A I O M n v P C D I s J J e M H I L / x A 4 m I u L r 7 w j v z i 3 h N o g 8 s o v G z o B X 5 u r t U 0 C h h C U U R p q B w O o j I S 1 m w Y h e M C c J Q p P A R Z k v j p Y N J D M I a h n E 4 m P U i 8 z z t 4 / d v p d H H G K M D V h c K a b e q W M m z a F V E 4 q r 2 d q U a b l c x y a 3 O R 5 W 0 W / U z K L L F G G 7 o 1 Y 1 d N 3 L d m / u z Q K 5 O + N f v v T L L B p 4 Y 6 H O p 2 j o 5 Z k Q E S a M a h 3 y 2 M f w q W 4 w f 3 j q P M w x m l o m q R j T v b L 5 v 2 d Z O d q s H W 1 + q y O c d B q C P o m N r a j m o / F 6 S K g W y 1 t f L 1 / O V Q o O z 4 / D D B P A K G m Z K H J m e L X R B 2 f K X M 1 M d d 2 6 W m 9 x Z T V v 7 4 e T q H 3 N F b 3 p d e H 0 W v / W X j P L m u P J V U 2 d C V C h Z r z q 6 k P b Y v o 3 B d y t + 3 f d / n a q 9 Q 1 w 1 O p 6 / j Q r z U E U K 7 3 G 6 g L P d b R Q V 5 X D Y Z x l h a b 1 l e H 1 Z W W A V D + Z r D W T j i w w F q Y T X R g F f j z Z a h q U f h w N u M E b c 1 0 k E 0 N N Z 1 y C k O 0 d N D O l K U E u V K 7 L R R R n q Y b V 6 q K T L P F K v 4 h 8 j F l R a R y k P 1 G + H 2 W a F Q n 6 f d 5 D 6 P n U r m j R o H 1 i + i 7 T J y Z w m Y C Y g c s M p r X e C n A p o E 5 D l x k S E 7 p M S M z I Z U b E R C 4 T E f P W Z d 6 y w d 2 7 1 D 1 1 i l 0 m n m o t F 4 k X C Y x b L K k H T + r M M 7 u 4 7 L 0 d C + k N s v Q z 6 a l S F j X 5 p A 4 g Z 2 u 8 p P K d u r 5 T u m v m M h k x u c v k b B L v X O o d d S p c p i B G u I x g 7 q R L S e o 0 d p k x M a X L l M Q 8 u M w D M Y 8 u 8 0 j M k 8 s 8 s c G 9 d 6 n 3 U 1 N u 2 S j A J J 0 V N g b K K l Q m E / s g Q 8 F T D 1 y O T K h Y C 9 1 m N y w Z y Y K k 7 B P M I q Q M C G b h U Q 4 I H j A Y C G Z x U Q 4 J H v K h h I S z k C h H B L N 4 K M c E s 2 A o 3 x L 8 l s H 3 B L M o K G O C Y w Y n B C d 8 h G y t + S J n B D N J l z n B O Y P f E c y 0 X B Y E M y G X g m D B 9 5 V g y U f I V o X r t y S Y i b d 8 I J g p t 3 w k m M m 2 f C L 4 i c H v C b Z 6 3 Y 5 B P R n r J 8 D C 1 W 5 1 1 h v h t Z 7 Q Y N T X e k a D k W D r K Q 1 G h + 3 n N B g 1 t p 7 U Y C T Z e l a D k W X r a Q 1 G m 6 3 n N X J z T 2 w w K m 2 e 2 Z a b e 2 i D 0 W v z 2 L Z c 4 n K J M / m 2 M 7 k m 5 x 7 L Y E T c P J g t 1 3 Y y 1 + T c w x m M q J v H s + X m n s 9 g 1 N 1 6 Q o O R e O s Z D U b n r a c 0 G L G 3 n t N g F N 9 + U o P R / f y z G i O i i I K 6 Y E n W K U r W W c A m G 4 R v U F A l m w R v M n i L 4 C 0 G b x O 8 z e A d g n c Y v E v w L o P 3 C N 5 j 8 D 7 B + 3 z g B 4 Q f M P N D g g 8 Z 3 C G 4 w + A j g o 8 Y f E z w M Y N P C D 7 h Q + k S 3 m X m p w S f M v i M 4 D M G n x N 8 z u A L g i 8 Y f E n w J Y O v s r l r j h 3 7 X L X n L t x u R v O 3 b r c L e f u X K 6 W / h W v h s v 3 o J 8 u 8 K l 2 t e 5 c Z i l M 7 J O u x Z K x g X o J p Y + 6 T F Z 4 X S I b q o I Z 0 j d I n 1 Z T F y U I U T G i S x F E q B w u q 6 F Q 3 a G r D k S o 2 t C 1 B i J U Y + g K A x G q L H R d g U j E b v / W Q F R G 6 C I C E S o e d O m A S M x W w i A J I a l B U r a C B q G S Q B c E i F A h o M s A R K i Y 1 b k f E c r 5 O u M j w h 7 E d K J H i B J 8 W e 0 M 2 5 f S I J T M d S p H h F K 4 T u C I U O L W a R s R K l J 1 r k b k f d u x 6 T x V Q e n H + U j t u f 5 Z y 7 D s V w L R 2 r A g P Y H R a 4 u K i v 2 k P 1 A 9 z A U R W Q K h w v V P g r U s l S R t g J b o E S H 8 l 2 E i C h P V W f + k z l a + l X T r q U w m f A Y T J V n b Q s U G 1 E K 1 D t i 0 J k q l t o U q H V I L F R p O 6 Z 2 N 0 u e I W B x x R C 2 U 5 V s 2 e N T k P V u g i d J i P f 2 J 0 q F t 4 Y q y p U Q N Z t R C / e X U Q u 2 9 o x b q r u C r N V G C q x d p o r R m W 7 j c Y 2 q h z k p q o c Y e q I X 6 e q Q W a u u J W q i r 9 6 b s O 1 J Z 9 7 G 6 t U 6 5 q D d K t T r R I r J B A a D z K 0 K U V 3 V W R Y S y q c 6 l i F A O 1 R k U E c q c O m 8 i Q r W c T p a I U J L U K R I R S o 0 6 M S J C C V G n Q 0 Q o D e o k i A g l P 5 3 6 E K G U p x M e I l S n 6 S y H C C v R d H J D i J K a T m m I U C r T i Q w R S m A 6 f S F C 5 Z j O W Y h c M N e X B q I U p R M U I p S Y d F p C h N K R T k a I U B L S K Q i R W + b 6 z k B 3 b B c p T f R 5 l k i 6 o + p Y 7 u E V W 0 B 7 C i i m U 5 0 E 9 f S q Y F b c u Q l o c z p d Q C r U b 4 S 3 I I j 9 A l B a o 3 V 1 G u E t T Q U o h p F 6 q Q p p k A 2 i N E R v / j h W i B j W 1 8 l 0 I t T 7 4 H O Q 8 x z 0 s 3 j w b W 7 6 j 1 O M x c X m S 9 1 U 6 N 8 q m k R a O d R v t K v J S V N 0 p o L F g N y w G J W Z c t N i m 7 T w c s u C F A l y 2 2 I U C 3 L H Y h Q N c t d i F A 9 y z 2 I U E X L f Y h Q T 8 s B i F B X y 0 G I U F 7 J j s Q 4 b 9 J E F K T j k s c U o P O S J x a j 8 k 1 2 L d Z n D U w t S k M g z i 1 G Y y H O L U a D I C 4 t R W S c v L U a x I q 8 s R t E i r y 1 G 8 S J v L E Y R I 2 8 t x m J G 3 l m w q t N Q z r u F n 4 8 M H d r H 4 M B 5 E A k 3 G E z i C D c Z T M d k u M V g U k i 4 z W A S S b j D Y N J J u M t g k k q 4 x 2 B S S + m W o 2 K E 7 k k L F L q E k U L F H K O l T 7 B P K 3 t K I A 4 J J n u K Q U D r U R I d Q 0 q Y 4 I v S I O T 4 m m K Q p T g g l Z Y o u o S R M c U o o n W / i j F B S p T g n l D 3 D i g u C S Z T i k l D S p L g i l I 4 6 c U 3 o N X N 8 Q z A J U t w S S n o U d 4 R a O W 6 n W A u C f r j w z e u Y x 6 p g p I K g 4 z 4 W q I p x n V o o 1 w 1 q o U w 3 q a x V V e I W k X j q b V M L l b R D L V T Q L r V Q O X v U Q s X s U w u F c k A t F M g h t V A Y H W q h I I 7 Y Y F A I x 0 S i A E 6 o h R v f Z a a 4 4 6 d E 4 k 6 f U Q t 3 + J x a u L E X 1 M I N v a Q W b u Q V t X A D r 6 m F + 3 Z D L d y v W 3 Z 7 3 K g 7 d s O q + K o K L 7 V t w L d N m h p M H S 4 q k P W H e h j N B l 7 2 H i I 5 y s b S w w L I e 8 D 8 l k P h l k h A N Z J T H 1 X 3 l 7 U Q t O F M c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 q 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e r 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 S h 7 0 a Y r K 2 i U V q B r K 6 D i y s L d G u 5 y + L S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q 1 h 9 o Z M l 1 z A a y 5 8 g s A E J I s x e O N 0 A E X 8 p D 5 9 G v j S 9 0 J I o c D c o 9 q R Q L 3 3 x y o R u d r N l e l 0 k r + Z 9 I p k o h s 6 D S q v k O R R E W E C d P r X H y H 2 n 3 T y 0 x + S q J t g t m z 4 t t + Y j H y J D / L u L R z L L r f s T t s G k 2 Q D i D 8 0 E W 1 Q z 8 S 0 p s 2 n m W 5 l 1 W 1 Y N c Y d x Q O o L H u 6 U Q + / 7 o G H h c y C k S / U B 7 T + W G b 6 0 Q o K Z 4 i N j 2 F z Y 1 M P s u o y O 4 A B O H a m 2 W J X I I F H j 7 U z T R R D Y H 5 R 7 F r H f h 7 7 A U z r r 3 I 6 F T D 1 X n j V t b v A b v / t a Z 3 9 t p s j 6 Q j 2 4 U + n y Z 5 N e a Z v u E W V s V V u 9 I y L q X 0 9 5 x I F h N P 6 h V u T C i R N U r W i Y a R W z z U T 2 V A m / i N Z W q B p h 1 k j 0 1 9 C m T d x s 1 7 y e K y m / 1 6 9 J W h M 7 r A z 5 d 9 B H X Z m 9 v D K L 2 g I q t G 8 g c Q f f o H b X 2 T M 8 n x m C z a z k m j V 0 I n u O o u H h Z + o d 1 a j h 6 z A m l X 4 T 8 J 7 3 v n m 1 X P 1 A Z D + M n 2 c m i 9 a R Y 4 S E P o r t O c 9 i G N m Y 9 + c v v A 2 M B V i 3 K f q n y c M e k j U 1 3 C q M D Z O m b X 6 H D U b h z p 7 6 j o 5 k r C s 3 Y v M G 2 S g 3 D 1 E 9 1 E O g 8 h f a X z Q n B V J r N 7 0 T y e d b 1 a n L W S W g u L W 2 j j 5 o P u 9 a u 6 H J n N F 5 S 3 d t B w 6 3 / S i d C i f m l 1 z v 1 C v k f H s 8 F X A n A O e u M I P w Y t S L 8 2 q I l / C 4 4 q 3 O c q E W p 9 M F Y T B y N v C Z + I U P h N e P 8 v u V 1 R Y s r c 9 J 7 k 6 p L P i d 6 j z I t Q j w J + 9 Z X X 1 I U N 1 X B r D S L + z a b H U i k U z / e 8 c i w v U 1 I X 6 V D A G 2 f P 7 G G t x 9 t A v w L 9 f f P N s a a 3 5 / y J m L 6 5 e r a z 9 f u X L 0 y + X v t q o / s / E J w u / W v j 1 w u c L a w t / W P h q Y W + h u 3 C 5 E C z 8 a e E v C 3 9 b + P v a X 9 f + s f b P t X 8 Z 0 4 8 / q v r 8 Y s H 5 s / a f / w E Q 1 1 D y < / l a t e x i t > Q&K < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 H a I A i 1 J f / A G Y M f C q P H w v O d c j x o = " > A A A y F n i c l V v J c t z I E e V 4 H d P b j H 3 0 B W F K M W O H x C A 1 C t v H 4 b 4 1 y e a + T G s U a H Q 2 G i I 2 o a r R p D r a X + G D L / a n + O b w 1 V f / i M / O q k I h s 9 B o T V g R E l H v Z S V q e V m Z A K F + H k d C r q 3 9 5 5 P v f f 8 H P / z R j z / 9 y f J P f / b z X / z y s 8 9 / d S 2 y c R H A V Z D F W X H b 9 w X E U Q p X M p I x 3 O Y F + E k / h p v + w 5 b i b 0 o o R J S l l / I p h z e J H 6 b R M A p 8 i d B 9 L / H l q D + c X s / e f r a y t r q m / 3 j z F + v V x c p S 9 a f 7 9 v O 1 / / Y G W T B O I J V B 7 A v x z f p a L t 9 M / U J G Q Q y z 5 d 5 Y Q O 4 H D 3 4 I 3 + B l 6 i c g 3 k z 1 k G f e c 0 Q G 3 j A r 8 G 8 q P Y 3 y H l M / E e I p 6 a O l G q J o c g p c x C m P 4 k U / a Q x B D v / 0 Z h q l + V h C G p g R D M e x J z N P L Y w 3 i A o I Z P y E F 3 5 Q R D g J L x j 5 h R 9 I X L 7 l 5 e f e s V 8 8 e A J t c O G E l w 2 9 w M / N t Z p G A U M o i i g N l c N B V E b C m g 2 j c F w A j j K F S Z A l i Z 8 O p j 0 E Y x j K 2 X T a g 8 T 7 s o P X v 5 v N l u e M A l x d K K z Z l m 4 p w 6 Z d E Y W j 2 t u 5 a r R Z y S y 3 N p d Z 3 m b R z 6 T M E m u 0 q V t z d t X E f W v m z w + 9 M u l b k / 4 i J 4 G 1 C B Z Z D K z F Q N / m u b e P 0 4 v V F D 3 f Q 3 u 1 Z T B E / Q 8 8 X J z E 9 Y H X C p x 9 s / 4 G v f S H 3 s q 6 c o J e d v W 2 m H 1 D Q c A L L 8 4 m U L w M M J p W l 3 v o U q 8 r D F f W p 2 Y L / 9 z D 1 t Q 4 a O u P 4 4 2 k H 6 9 6 u 6 g H I T E Q 1 P Y L t W f I G 5 e 7 1 u V u 0 6 W m 5 S S z N 1 1 5 V d 1 W e N b I w y l V j V e 2 x / u x P 6 A u K 1 + t v J 7 r 9 q L u Y 6 + + 4 q 5 e o 6 v n 3 o U R 9 q L 1 M D d D 9 Z v R V 2 H A h t / q w a 6 I 6 X 1 h e 1 + 0 9 D 6 3 v X R A T r I 6 0 l b r l T F 3 F 3 p p 6 j h c s D Z N h 6 M C g L n k 0 x F m 4 e Z d 0 r o x 5 1 / N O / d T D 3 A b V O c W E c F 7 M 2 l r s n j W j p 9 x n k P h K T / G z U 7 l Z q f N z Y Z X + B N a + I a z l y 9 f + m U W D b y x U O d T N P T y T I g I U 0 S 1 D n n s Y / h U N 1 g 8 P H U e 5 h h N L Z N U j O l e 2 f z f s 6 w c b d W O t r 7 T E U 4 6 D U G f x M Z W V P P R e D 0 k V I v l r a + X L x c q B Y f n x 2 G G C W C U t E w U O T O 8 2 u i j M 2 W u 5 q a 6 Y V 1 t t L i y m r f 3 w 0 n U v h a K 3 v S 6 d H p t f G e v u W X N 8 e S q p s 4 E q F A z X n X 1 s W 0 x / Z s C 7 t b 9 u 2 5 / O 9 X 6 B j h q d b 1 4 m p X m I I q V X m N 1 g e c 6 W q i r y u E w z r J C 0 / r K 8 P q y M k C q n 0 z X m 0 l H F h g L s 6 k u j A I / n m 4 3 D U o / j g b c 4 K 2 5 L p K p o W Z z L k H I 9 g 6 a m d G U I B c q 1 + U i i r N U w 2 p 1 0 U m W e K V f R D 7 G r K g 0 D t K f K t + P M s 2 K B P 0 + 6 y H 0 b G Z X t G j Q P j F 9 l + k T E 7 h M Q M z A Z Q a z W m 8 F u B R Q p 6 H L D I k J X S Y k Z u Q y I 2 I i l 4 m I e e c y 7 9 j g H l z q g T r F L h P P t J a L x I s E x i 0 W 0 Y M n d e a Z X X z h v R s L 6 Q 2 y 9 A v p q V I W N f m k D i B n a 7 y k 8 p 2 6 v l O 6 a + Y y G T G 5 y + R s E u 9 d 6 j 1 1 K l y m I E a 4 j G D u p E t J 6 j R 2 m T E x p c u U x E x c Z k L M o 8 s 8 E v P k M k 9 s c B 9 c 6 s P M l F s 2 C j B J Z 4 W N g b I K l e n U P s h Q 8 N Q D l y M T K t Z C t 9 k N S 0 a y I C n 7 B L M I K Q O C W X i U A 4 I H D A a C W V y U Q 4 K H f C g h 4 S w k y h H B L B 7 K M c E s G M p 3 B L 9 j 8 A P B L A r K m O C Y w Q n B C R 8 h W 2 u + y B n B T N J l T n D O 4 P c E M y 2 X B c F M y K U g W P B 9 J V j y E b J V 4 f o t C W b i L S c E M + W W j w Q z 2 Z Z P B D 8 x + A P B V q 8 7 M a g n Y / 0 E W L j a r c 5 6 I 7 z W E x q M + l r P a D A S b D 2 l w e i w / Z w G o 8 b W k x q M J F v P a j C y b D 2 t w W i z 9 b x G b u G J D U a l z T P b c g s P b T B 6 b R 7 b l k t c L n E m 3 3 Y m 1 + T C Y x m M i J s H s + X a T u a a X H g 4 g x F 1 8 3 i 2 3 M L z G Y y 6 W 0 9 o M B J v P a P B 6 L z 1 l A Y j 9 t Z z G o z i 2 0 9 q M L p f f F Z j R B R R U B c s y Q Z F y Q Y L 2 G S T 8 E 0 K q m S L 4 C 0 G b x O 8 z e A d g n c Y v E v w L o P 3 C N 5 j 8 D 7 B + w w + I P i A D / y Q 8 E N m f k T w E Y M 7 B H c Y f E z w M Y N P C D 5 h 8 C n B p 3 w o X c K 7 z P y M 4 D M G n x N 8 z u A L g i 8 Y f E n w J Y O v C L 5 i 8 D X B 1 3 y E N 4 T f M P N b g m 8 Z f E f w H Y P v C b 4 3 d X P r S e s q D 4 z 0 m F A 3 m M K 1 / B i 3 y b k t l 9 v i 3 L b L b X N u x + V 2 O L f r c r u c 2 3 O 5 P R 5 t W p 2 M 3 O c d D 1 z u g H O H L n f I u S O X O + J c x + U 6 n D t 2 u W P O n b j c i T O J U 5 c 8 5 R 2 7 L t f l 3 J n L n X H u 3 O X O O X f h c h f O Y C 5 d 8 p J 3 v H K 5 K 8 5 d u 9 w 1 5 2 5 c 7 o Z z t y 5 3 y 7 k 7 l 7 v j 3 L 3 L 1 d K / 5 t V w + Q H 0 0 w U + 1 a 7 V n c s s h a l 9 0 r V Y M j Z Q L 6 H 0 U Z f J C q 9 L Z E N V M E P 6 B u n T a u q i B C E q R n Q p g g i V w 2 U 1 F K o 7 d N W B C F U b u t Z A h G o M X W E g Q p W F r i s Q i d j t 3 x m I y g h d R C B C x Y M u H R C J 2 U o Y J C E k N U j K V t A g V B L o g g A R K g R 0 G Y A I F b M 6 9 y N C O V 9 n f E T Y g 5 h O 9 A h R g i + r n W H 7 U h q E k r l O 5 Y h Q C t c J H B F K 3 D p t I 0 J F q s 7 V i H x o O z a d p y o o / T g f q T 3 X P 2 s Z l v 1 K I F o b F q Q n M H p t U V G x n / Q H q o e 5 I C J L I F S 4 / k m w l q W S p A 3 Q E j 0 i h P 8 y T E R h o j r r n 9 T Z y r e S b j 2 V 6 Z T P Y K o k a 1 u o 2 I B a q N Y B m 9 Z U q d S 2 U K V D a q F C w x m 9 s 1 H 6 H B G L I 4 6 o h b J 8 x w a P m n x g C z R V W q y n P 1 U 6 t C 1 c U b a U q M G M W q i / n F q o v f f U Q t 0 V f L W m S n D 1 I k 2 V 1 m w L l 3 t M L d R Z S S 3 U 2 I R a q K 9 H a q G 2 n q i F u v p g y r 5 j l X U f q 1 v r l I t 6 o 1 S r E y 0 i m x Q A O r 8 i R H l V Z 1 V E K J v q X I o I 5 V C d Q R G h z K n z J i J U y + l k i Q g l S Z 0 i E a H U q B M j I p Q Q d T p E h N K g T o K I U P L T q Q 8 R S n k 6 4 S F C d Z r O c o i w E k 0 n N 4 Q o q e m U h g i l M p 3 I E K E E p t M X I l S O 6 Z y F y C V z f W U g S l E 6 Q S F C i U m n J U Q o H e l k h A g l I Z 2 C E L l j r u 8 N d M 9 2 k d J E n 2 e J p D u q j u U e X r E F t K e A Y j r V S V B P r w p m x V 2 Y g D a n 0 y W k Q v 1 G e B u C 2 C 8 A p T X a U K c R 3 t J U g G I Y q Z e q k A b Z I E p D 9 O a P Y 4 W I Y X 2 d z K Z C v Q + + A L n I Q T + L B 9 / l p v 8 4 w 1 h c b r 7 U T Y X + r a J J p J V D / U a 7 m p w 0 R W c q W A z I T Y t R m S m 3 L L Z F C y + 3 L U i R I H c s R r E g d y 1 G 0 S D 3 L E b x I P c t R h E h D y x G M S E P L U Z R I Y 8 s R n E h O x b r s E E f W 5 C C Q 5 5 Y j M J D n l q M y j / Z t V i X O T y z I A W J P L c Y h Y m 8 s B g F i r y 0 G J V 1 8 s p i F C v y 2 m I U L f L G Y h Q v 8 t Z i F D H y z m I s Z u S 9 B a s 6 D e W 8 V / j 5 y N C h f Q w O n A e R c J P B J I 5 w i 8 F 0 T I b b D C a F h D s M J p G E u w w m n Y R 7 D C a p h P s M J r W E B w w m w Y S H D C b N h E c M P q L F C T s M p 0 M 1 P G Y w i S c 8 Y T D p J z x l M E k o 7 D K Y z t n w j M F n b C j n D C c t h R c M J j m F l w w m R Y V X D C Z R h d c M Z o / B 4 Q 3 D S V v h L Y N J X u E d g + m B I L x n s N G Y + u 2 7 r O o 3 U b 9 n 6 f P 3 L G K T Y N K X 2 C K U 5 C W 2 C d X q e u 5 t 6 9 9 8 j A V 4 v i d A e n j v G A b e z g u v D 4 G v c D m K h D f J x v E A I W y B J / T v S b D C H B e e + p g n i 9 G R + l g G H n O s O P X v g K t f h o p d u i M p V O w R S g I V + 4 S S P s U B o e w t j T g k m O Q p j g i l Q 0 1 0 C C V t i m N C j 5 n j E 4 J J m u K U U F K m 6 B J K w h R n h N L 5 J s 4 J J V W K C 0 L Z M 6 y 4 J J h E K a 4 I J U 2 K a 0 L p q B M 3 h N 4 w x 7 c E k y D F H a G k R 3 F P q J X j T o q 1 I O i H C 9 + 8 j n m s C k Y q C D r u Y 4 G q G D e o h X L d p B b K d I v K W l U l b h O J p 9 4 O t V B J u 9 R C B e 1 R C 5 W z T y 1 U z A G 1 U C i H 1 E K B H F E L h d G h F g r i m A 0 G h X B C J A r g l F q 4 8 V 1 m i j t + R i T u 9 D m 1 c I c v q I U b e 0 k t 3 N A r a u F G X l M L N / C G W r h v t 9 T C / b p j t 8 e N u m c 3 r I q v q v B S 2 w Z 8 2 6 S p w d T h o g J Z f 6 i H 0 W z g F 9 4 k k q N s L D 0 s g L w J 5 r c c C r d E A q q R n P q o u r + s h a A N 5 4 p D 0 A U U N C o o 0 C U U N G o o 0 E U U U B V V o d s 1 y l 7 P 6 T o K G o U U 6 E o K G q U U 6 F o K G s U U 6 G o K q J y y 8 E E N s x d w u q S C R k 0 F u q i C R l U F u q w C q q s q 9 L h G 2 Y s 2 X V l B o 7 Q C X V s B F V c W 7 t Z w l 8 N n N c z e q O k S C x o 1 F u g i C x p V F u g y C x p 1 F u h C C x q V F u h S C x q 1 F u h i C x r V F u h y C 6 j e s v B d D b M 3 Z L r k A l 5 z 4 R M E J i B Z j M E b p w M o 4 i f 1 6 d P A l 7 4 X Q g o F 5 h 7 V j g T q v T 9 W i c j V b q 5 M Z 9 P 8 7 b R X J F P d 0 G l Q e Y U k j 4 o I E 6 D T v / 4 I s f + k k 5 / + k E T d B L N l w 7 f 9 x m T k S 3 y Q d 2 / h W H a 5 Z X f W N p g k G 0 D 8 s Y l o g 3 o m p j V r P s 1 0 K 6 t u w 6 o x 7 i g e Q G X Z 0 4 1 6 + H U P P C x k F o x 8 o T 6 g 9 c c y 0 4 9 W U D h D b H w M m x u b e p B V l / k B D M C x M 8 0 W u w I J P H q s n W m i G A L z i 2 L X O v b z 2 A 9 g V n + V 0 6 m A m f f c q 6 7 d B X b 7 7 8 z q 7 L f T H E l H s A 9 / O k 3 2 f M Y z f c M t q o y t c q N n X M z s 6 z m X K C C c 1 S / c m l Q g a Z K q F Q 0 j t X q u m c i G M v E f y d I C T T v M G p n + E s q 8 i Z v 3 k s d j N f 0 P 6 i 1 B Y 3 J H n R n / D u q o M 7 e H 1 3 5 B Q 1 C N 5 g 0 k / v A L 3 P 4 i Y 5 Y X c 1 u w l Z V E q 4 Z O d D d Z P C z 8 R L 2 z G k 2 y A m t W 4 T 8 J 7 1 n n 2 1 f P 1 A d A + s v 0 c W q + a B U 5 S k D o r 9 C e 9 S C O m Y 1 9 c / r c 2 8 R U i H G f q n + e M O g h U V / D q c L Y O G X W 6 n P U b B z q 7 K n r 5 E j C C + 1 e Z N 4 g A + V u E j 1 E O Q w i f 7 X x Q X N W J L F 6 0 z + b d r 5 d m 7 W Q W Q q K W 2 / j 5 E T 3 e 9 X c D 0 3 m i s p b u m k 5 d L 7 t R e l Q P j W 7 5 n 6 h X i P j 2 e G r g L k A P H G F H 4 I X p V 6 a V U W + h M d V b 2 u U C b U + m S o I g 5 G 3 j c / E K X w h v H 6 W P a y q s G R v e 0 5 z d U h n x e 9 R 5 0 W o R 4 A / e y / U 1 c c M 1 X F p D C P 9 z q b F U i s W z f S / C y w u U V O X 6 l P B G G T P 7 2 O s x d m k X 4 D / s P z 2 s 5 X 1 5 v + L m L + 4 f r W 6 / o f V 1 2 e v V 7 7 e r P 7 P x K d L v 1 n 6 7 d K X S + t L f 1 z 6 e m l / q b t 0 t R Q s p U t / W f r b 0 t / X / 7 r + j / V / r v / L m H 7 v k 6 r P r 5 e c P + v / / h 8 / 4 E u 7 < / l a t e x i t >

Single-head Attention Unit

< l a t e x i t s h a 1 _ b a s e 6 4 = " J p Q n j I h 7 n 8 R p J C W 2 Y B x I F M 4 w 5 4 g = " > A A A y K n i c l V t Z k 9 v G E a a d y 1 E u O 3 n M C y o r l Z 3 U a m s p q 5 I 8 e u + L u 8 u 9 D 6 + s A s E m C C 0 u Y Y b g r l j M r 8 h r 8 g v y a / L m y m t + S H p m M O g e L C h X V G U J 8 3 0 9 j T m + n m 6 A 8 C C P I y F X V 7 / / 5 N M f / f g n P / 3 Z Z z 9 / 9 o t f / u r X v / n 8 i 9 9 e i m x S B H  A R Z H F W X A 9 8 A X G U w o W M Z A z X e Q F + M o j h a n C / o f i r E g o R Z e m 5 f M z h T e K H a T S K A l 8 i d H u X + H I 8 G M 2 O 5 2 8 / X 1 p d W d V / v K c X 3 e p i q V P 9 6 b / 9 o v v s b p g F k w R S G c S + E N 9 2 V 3 P 5 Z u Y X M g p i m D + 7 m w j I / e D e D + F b v E z 9 B M S b m R 7 y 3 H u B y N A b Z Q X + l 0 p P o 7 z H z E + E e E w G a K m G K J q c A h d x y q N Y H i S N I c j R X 9 / M o j S f S E g D M 4 L R J P Z k 5 q m F 8 Y Z R A Y G M H / H C D 4 o I J + E F Y 7 / w A 4 n L 5 9 z i w c z h 2 b M X 3 q F f 3 H s C + + F i C i 8 b e Y G f m 2 s 1 t Q J G U B R R G q q b D K M y E t Z s F I W T A t B t C t M g S x I / H c 7 u E I x h J O e z 2 R 0 k 3 l c 9 v P 7 j H G / T N A p w x a G w Z h u 6 p Q y b d k U U j m t v p 6 r R Z i W z 3 N q c Z 3 m b x S C T M k u s 0 b p u P b G r J u 5 b M / / p 0 C u T g T U Z L H I S W I t g k c X Q W g z 1 b V 5 4 u z i 9 W E 3 R 8 z 2 0 V 9 s I I 4 y J o Y e L k 7 g + 8 F q B 8 2 + 7 b 9 D L Y O Q t d Z U T 9 L K t t 8 X s G 4 o E l r 0 4 m 0 L x M s A I W 0 G 5 w 0 i v K 4 y W u j O z h X + 7 w 9 b M O G j r j + O N p B + v e N u o B y E x O N T 2 C 7 V n y B u X 2 9 b l d t O l p u U 0 s z d d e l X d V n j W y M M p V Y 1 X t s f 7 i T + k L k t f L 7 1 + 0 m 2 5 7 m O v v u a u X q O r F 9 6 Z E f a i 9 T A 3 Q / W b 0 V d h w I b f 6 s G u i O l 9 Z n u f t f Q + t b 1 0 k E 6 z O t J W 6 p U x d x d 6 a e o 4 X L A 2 T Y f j A o C 5 5 N M R Z u G e u q R 1 Y 8 6 / f u r c T z 3 A b V C d W 0 Q E 7 8 2 k r c n i W T t + J n k O h a f 8 G D d b l Z u t N j d r X u F P a e E b z l 6 + f O m X W T T 0 J k K d T 9 H I y z M h I k w b 1 T r k s Y / h U 9 1 g 8 f D U G Z l j N L V M U j G m e 2 X z f 8 + y c r R R O 9 r 4 Q U c 4 6 T Q E f R I b W 1 H N R + P 1 k F A t l r e + X r 5 c q B Q c n h + H G S a F c d I y U e T M 8 G q j j 8 6 U u X o y 1 T X r a q 3 F l d W 8 v R 9 O o v a 1 U P S m 1 7 n T a + 0 H e z 1 Z 1 h x P r m r q T I A K N e N V V x / b F t O / K e B + 3 b / v 9 r d T r W + A o 1 b X i 6 d Z a Q 6 i W O k 1 V h d 4 r q O F u q o c j u I s K z S t r w y v L y s D p A b J r N t M O r L A W J j P d L E U + P F s s 2 l Q + n E 0 5 A Z v z X W R z A w 1 f + I S h G z v o J k 5 T Q l y o X J d L q I 4 S z W s V h e d Z I l X + k X k Y 8 y K S u M g / Z n y / S D T r E j Q 7 / M 7 h J 7 P 7 Y o W D d o n Z u A y A 2 I C l w m I G b r M c F 7 r r Q C X A u o 0 c p k R M a H L h M S M X W Z M T O Q y E T H v X O Y d G 9 y 9 S 9 1 T p 9 h l 4 r n W c p F 4 k c C 4 x c J 6 + K j O P L O L y 9 6 7 i Z D e M E u / l J 4 q b 1 G T j + o A c r b G S y r f q e s 7 p b t m L p M R k 7 t M z i b x 3 q X e U 6 f C Z Q p i h M s I 5 k 6 6 l K R O E 5 e Z E F O 6 T E n M 1 G W m x D y 4 z A M x j y 7 z y A b 3 w a U + z E 2 5 Z a M A k 3 R W 2 B g o q 1 C Z z e z D D Q V P P X A 5 N q F i L X S b 3 b B k J A u S c k A w i 5 A y I J i F R z k k e M h g I J j F R T k i e M S H E h L O Q q I c E 8 z i o Z w Q z I K h f E f w O w b f E 8 y i o I w J j h m c E J z w E b K 1 5 o u c E c w k X e Y E 5 w x + T z D T c l k Q z I R c C o I F 3 1 e C J R 8 h W x W u 3 5 J g J t 5 y S j B T b v l A M J N t + U j w I 4 M / E G z 1 u h W D e l r W T 4 C F q 9 3 q r D f C a z 2 h w a i v 9 Y w G I 8 H W U x q M D t v P a T B q b D 2 p w U i y 9 a w G I 8 v W 0 x q M N l v P a + Q W n t h g V N o 8 s y 2 3 8 N A G o 9 f m s W 2 5 x O U S Z / J t Z 3 J N L j y W w Y i 4 e T B b r u 1 k r s m F h z M Y U T e P Z 8 s t P J / B q L v 1 h A Y j 8 d Y z G o z O W 0 9 p M G J v P a f B K L 7 9 p A a j + 8 V n N U Z E E Q V 1 w Z K s U Z S s s Y B N 1 g l f p 6 B K N g j e Y P A m w Z s M 3 i J 4 i 8 H b B G 8 z e I f g H Q b v E r z L 4 D 2 C 9 / j A 9 w n f Z + Y H B B 8 w u E d w j 8 G H B B 8 y + I j g I w Y f E 3 z M h 9 I n v M / M T w g + Y f A p w a c M P i P 4 j M H n B J 8 z + I L g C w Z f E n z J R 3 h F + B U z v M l n h d Y l s q A p m y M A g A 1 p N X Z Q g R M W I L k U Q o X K 4 r I Z C d Y e u O h C h a k P X G o h Q j a E r D E S o s t B 1 B S I R u / 0 7 A 1 E Z o Y s I R K h 4 0 K U D I j F b C Y M k h K Q G S d k K G o R K A l 0 Q I E K F g C 4 D E K F i V u d + R C j n 6 4 y P C H s Q 0 4 k e I U r w Z b U z b F 9 K g 1 A y 1 6 k c E U r h O o E j Q o l b p 2 1 E q E j V u R q R D 2 3 H p v N U B a U f 5 2 O 1 5 / r f W o b l o B K I 1 o Y F 6 Q m M X l t U V O w n g 6 H q Y S 6 I y B I I F a 7 / J V j L U k n S B m i J H h H C v x k m o j B R n f W / 1 N n K t 5 J u P Z X Z j M 9 g p i R r W 6 j Y g F q o 1 i G b 1 k y p 1 L Z Q p S N q o U L D O b 2 z U f o c E 4 s j j q i F s n z H B o + a v G c L N F N a r K c / U z q 0 L V x R t p S o w Y x a q L + c W q i 9 9 9 R C 3 R V 8 t W Z K c P U i z Z T W b A u X e 0 I t 1 F l J L d T Y l F q o r w d q o b Y e q Y W 6 + j C v f i D D r P t Q 3 V q n X N Q b p V q d a B F Z p w D Q + R U h y q s 6 q y J C 2 V T n U k Q o h + o M i g h l T p 0 3 E a F a T i d L R C h J 6 h S J C K V G n R g R o Y S o 0 y E i l A Z 1 E k S E k p 9 O f Y h Q y t M J D x G q 0 3 S W Q 4 S V a D q 5 I U R J T a c 0 R C i V 6 U S G C C U w n b 4 Q o X J M 5 y x E z p n r C w N R i t I J C h F K T D o t I U L p S C c j R C g J 6 R S E y A 1 z f W u g W 7 a L l C Y G P E s k / X F 1 L N / h F V t A e w o o p l e d B P X 0 q m B W 3 J k J a H M 6 n U M q 1 K / E m x D E f g E o r f G a O o 3 w l q Y C F K N I v V S F N M i G U R q i N 3 8 S K 0 S M 6 u t k P h P q f f A Z y E U O B l k 8 / C E 3 g 4 c 5 x u K z 5 k v d V O h f F U 0 i r R z q N 9 r V 5 K Q p O l P B Y k C u W 4 z K T L l h s Q 1 a e L l p Q Y o E u W U x i g W 5 b T G K B r l j M Y o H u W s x i g i 5 Z z G K C b l v M Y o K e W A x i g v Z s 1 i P D f r Q g h Q c 8 s h i F B 7 y 2 G J U / s m + x f r M 4 Y k F K U j k q c U o T O S Z x S h Q 5 L n F q K y T F x a j W J G X F q N o k V c W o 3 i R 1 x a j i J E 3 F m M x I 2 8 t W N V p K O e d w s / H h g 7 t Y 3 D g P I i E 6 w w m c Y Q b D K Z j M t x k M C k k 3 G I w i S T c Z j D p J N x h M E k l 3 G U w q S X c Y z A J J t x n M G k m P G D w A S 1 O 2 G M 4 H a r h I Y N J P O E R g 0 k / 4 T G D S U J h n 8 F 0 z o Y n D D 5 h Q z l l O G k p P G M w y S k 8 Z z A p K r x g M I k q v G Q w e w w O r x h O 2 g q v G U z y C m 8 Y T A 8 E 4 S 2 D j c b U r + + y q t 9 E / Z 5 l w N + z i H W C S V 9 i g 1 C S l 9 g k V K v r h b e p f / m Y C P B 8 T 4 D 0 8 N 4 x D L 2 t Z W 8 A g a 9 w O Y 6 E N 8 0 m 8 R A h b I E n 9 O 8 k W G F O C k 9 9 4 J P F 6 E h 9 L A M P O V a c + j f g 6 s d Q s U 1 3 J I W K H U J J o G K X U N K n 2 C O U v a U R + w S T P M U B o X S o i R 6 h p E 1 x S O g h c 3 x E M E l T H B N K y h R 9 Q k m Y 4 o R Q O t / E K a G k S n F G K H u G F e c E k y j F B a G k S X F J K B 1 1 4 o r Q K + b 4 m m A S p L g h l P Q o b g m 1 c t x K s R Y E / X D h m 9 c x D 1 X B S A V B z 3 0 s U B X j G r V Q r u v U Q p l u U F m r q s R N I v H U 2 6 I W K m m b W q i g H W q h c n a p h Y r Z o x Y K Z Z 9 a K J A D a q E w e t R C Q R y y w a A Q j o h E A R x T C z e + z 0 x x x 0 + I x J 0 + p R b u 8 B m 1 c G P P q Y U b e k E t 3 M h L a u E G X l E L 9 + 2 a W r h f N + z 2 u F G 3 7 I Z V 8 V U V X m r b g G + b N D W Y O l x U I O u P 9 z C a D b z s T S M 5 z i b S w w L I m 2 J + y 6 F w S y S g G s m p j 6 r 7 y 1 o I 2 v B J c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 s 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e q 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 i h 7 0 a Y r K 2 i U V q B r K 6 D i y s L 9 G u 5 z + K S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q l h 9 o Z M l 1 z A a y 5 8 g s A E J I s J e J N 0 C E X 8 q D 5 9 G v r S 9 0 J I o c D c o 9 q R Q L 0 P J i o R u d r N l e l 8 l r + d 3 R X J T D d 0 G l R e I c m j I s I E 6 P S v P 0 I c P O r k p z 8 k U T f B b N n w b b 8 x G f s S H + T d W z i W f W 7 Z n 7 c N J s m G E H 9 s I t q g n o l p z Z t P M / 3 K q t + w a o w 7 i o d Q W d 7 p R j 3 8 u g c e F j I L x r 5 Q H 9 X 6 E 5 n p R y s o n C E 2 P o b N j U 0 9 y K r L 0 w E M w b E z z R a 7 A g k 8 e q y d a a I Y A v N D s W s d + 3 n s B z C v v 8 r p V c D c e + F V 1 + 4 C u / 2 3 5 n X 2 2 2 q O p C f Y h z + 9 J n s 6 5 5 m + 4 R Z V x l a 5 0 T M u 5 v b 1 n E s U E M 7 r F 2 5 N K p A 0 S d W K R p F a P d d M Z C O Z + A 9 k a Y G m H W a N T H 8 J Z d 7 E P f W S x x M 1 / Q / q L U F j c g e 9 O f 8 O 6 q D 3 Z A 8 v / Y K G o B r N G 0 j 8 x y 9 w + 4 u M W Z 4 9 2 Y K N r C R a N X S i u 8 r i U e E n 6 p 3 V e J o V W L M K / 1 F 4 z 3 v f v X q u P g D S X 6 t P U v N F q 8 h R A k J / h f b 8 D u K Y 2 d g 3 p y + 8 d U y F G P e p + u s R g x 4 S 9 T W c K o y N U 2 a t P k f N J q H O n r p O j i Q s a / c i 8 4 Y Z K H f T 6 D 7 K Y R j 5 K 4 0 P m r M i i d W b / v m s 9 9 3 q v I X M U l B c t 4 2 T U 9 3 v V X M / N J k r K m / p p u X Q + + 4 u S k f y s d k 1 9 w v 1 G h n P D l 8 F z B n g i S v 8 E L w o 9 d K s K v I l P K x 4 G + N M q P X J V E E Y j L 1 N f C Z O 4 U v h D b L s f k W F J X v b c 5 y r Q z o r / o Q 6 L 0 I 9 A v z 3 b l l d f c x Q H Z f G M N L v b F o s t W L R T P + 9 w O I c N X W u P h W M Q d 7 5 A 4 y 1 O J s O C v D v n 7 3 9 f K n b / H 8 l n l 5 c v l r p / n n l 9 c n r p W / W q / + P 4 r P O 7 z t / 6 H z V 6 X b + 0 v m m s 9 v p d y 4 6 Q S f t / L 3 z j 8 4 / u / / q / r v 7 f f c / x v T T T 6 o + v + s 4 f 7 r / / R + X X F M 7 < / l a t e x i t > O < l a t e x i t s h a 1 _ b a s e 6 4 = " V 3 B e y D s u v e q B p L f r K P b E H 4 B 1 h w Q = " > A A A y H n i c l V v J c t z I E e V 4 H d O b x j 7 6 g j C l m L F D Y p A a h e 3 j c N + a Z H N f p j U K N D o b D R G b U N V o U h 3 t L / H B F / t T f H P 4 a v + I z 8 6 q Q i G z 0 O i Z s C I k o t 7 L S t T y s j I B Q v 0 8 j o R c W / v P J 9 / 7 / g 9 + + K M f f / q T 5 Z / + 7 O e / + O W z z 3 5 1 L b J x E c B V k M V Z c d v 3 B c R R C l c y k j H c 5 g X 4 S T + G m / 7 D l u J v S i h E l K W X 8 i m H t 4 k f p t E w C n y J 0 L t n z 3 o j X 0 5 7 i S 9 H / e F 0 f z Z 7 9 2 x l b X V N / / H m L 9 a r i 5 W l 6 k / 3 3 W d r / + 0 N s m C c Q C q D 2 B f i 6 / W 1 X L 6 d + o W M g h h m y 7 2 x g N w P H v w Q v s b L 1 E 9 A v J 3 q o c + 8 F 4 g M v G F W 4 N 9 U e h r l P a Z + I s R T 0 k d L N U b R 5 B S 4 i F M e x c t + 0 h i C H P 7 p 7 T R K 8 7 G E N D A j G I 5 j T 2 a e W i B v E B U Q y P g J L / y g i H A S X j D y C z + Q u I z L y y + 8 Y 7 9 4 8 A T a 4 A I K L x t 6 g Z + b a z W N A o Z Q F F E a K o e D q I y E N R t G 4 b g A H G U K k y B L E j 8 d T H s I x j C U s + m 0 B 4 n 3 R Q e v f z e b L c 8 Z B b i 6 U F i z L d 1 S h k 2 7 I g p H t b d z 1 W i z k l l u b S 6 z v M 2 i n 0 m Z J d Z o U 7 f m 7 K q J + 9 b M n x 9 6 Z d K 3 J v 1 F T g J r E S y y G F i L g b 7 N C 2 8 f p x e r K X q + h / Z q y 2 C I c T D w c H E S 1 w d e K 3 D 2 9 f p b 9 N I f e i v r y g l 6 2 d X b Y v Y N B Q E v v T i b Q P E q w K h a X e 6 h S 7 2 u M F x Z n 5 o t / H M P W 1 P j o K 0 / j j e S f r z q 7 a I e h M R A U N s v 1 J 4 h b 1 z u W p e 7 T Z e a l p P M 3 n T l d X V b 4 V k j D 6 d U N V 7 b H h / G / o C 6 r H y 5 8 m a u 2 8 u 6 j 7 3 6 k r t 6 g 6 5 e e B d G 2 I v W w 9 w M 1 W 9 G X 4 U B G 3 6 r B 7 s i p v e F 7 X 3 R 0 v v c 9 t I B O c n q S F u t V 8 b c X e i l q e N w w d o 0 H Y 4 K A O a S T 0 e Y h Z t 3 S e v G n H 8 5 7 9 x P P c B t U J 1 b R A Q f z K S t y e J Z O 3 7 G e Q 6 F p / w Y N z u V m 5 0 2 N x t e 4 U 9 o 4 R v O X r 1 6 5 Z d Z N P D G Q p 1 P 0 d D L M y E i T B X V O u S x j + F T 3 W D x 8 N R 5 m G M 0 t U x S M a Z 7 Z f N / z 7 J y t F U 7 2 v p O R z j p N A R 9 E h t b U c 1 H 4 / W Q U C 2 W t 7 5 e v V q o F B y e H 4 c Z J o B R 0 j J R 5 M z w a q N v n S l z N T f V D e t q o 8 W V 1 b y 9 H 0 6 i 9 r V Q 9 K b X p d N r 4 z t 7 z S 1 r j i d X N X U m Q I W a 8 a q r b 9 s W 0 7 8 p 4 G 7 d v + v 2 t 1 O t b 4 C j V t e L p 1 l p D q J Y 6 T V W F 3 i u o 4 W 6 q h w O 4 y w r N K 2 v D K 8 v K w O k + s l 0 v Z l 0 Z I G x M D O V U e D H 0 + 2 m Q e n H 0 Y A b v D P X R T I 1 1 G z O J Q j Z 3 k E z M 5 o S 5 E L l u l x E c Z Z q W K 0 u O s k S r / S L y M e Y F Z X G Q f p T 5 f t R p l m R o N / n P Y S e z + y K F g 3 a J 6 b v M n 1 i A p c J i B m 4 z G B W 6 6 0 A l w L q N H S Z I T G h y 4 T E j F x m R E z k M h E x 7 1 3 m P R v c g 0 s 9 U K f Y Z e K Z 1 n K R e J H A u M V i e v C k z j y z i y + 9 9 2 M h v U G W f i 4 9 V c q i J p / U A e R s j Z d U v l P X d 0 p 3 z V w m I y Z 3 m Z x N 4 o N L f a B O h c s U x A i X E c y d d C l J n c Y u M y a m d J m S m I n L T I h 5 d J l H Y p 5 c 5 o k N 7 q N L f Z y Z c s t G A S b p r L A x U F a h M q 2 f Z C h 4 6 o H L k Q k V a 6 H b 7 I Y l I 1 m Q l H 2 C W Y S U A c E s P M o B w Q M G A 8 E s L s o h w U M + l J B w F h L l i G A W D + W Y Y B Y M 5 X u C 3 z P 4 g W A W B W V M c M z g h O C E j 5 C t N V / k j G A m 6 T I n O G f w B 4 K Z l s u C Y C b k U h A s + L 4 S L P k I 2 a p w / Z Y E M / G W E 4 K Z c s t H g p l s y y e C n x j 8 k W C r 1 5 0 Y 1 J O x f g I s X O 1 W Z 7 0 R X u s J D U Z 9 r W c 0 G A m 2 n t J g d N h + T o N R Y + t J D U a S r W c 1 G F m 2 n t Z g t N l 6 X i O 3 8 M Q G o 9 L m m W 2 5 h Y c 2 G L 0 2 j 2 3 L J S 6 X O J N v O 5 N r c u G x D E b E z Y P Z c m 0 n c 0 0 u P J z B i L p 5 P F t u 4 f k M R t 2 t J z Q Y i b e e 0 W B 0 3 n p K g x F 7 6 z k N R v H t J z U Y 3 S 8 + q z E i i i i o C 5 Z k g 6 J k g w V s s k n 4 J g V V s k X w F o O 3 C d 5 m 8 A 7 B O w z e J X i X w X s E 7 z F 4 n + B 9 B h 8 Q f M A H f k j 4 I T M / I v i I w R 2 C O w w + J v i Y w S c E n z D 4 l O B T P p Q u 4 V 1 m f k b w G Y P P C T 5 n 8 A X B F w y + J P i S w V c E X z H 4 m u B r P s I b w m + Y + S 3 B t w y + I / i O w f c E 3 5 u 6 u f W k d Z U H R n p M q B t M 4 V p + j N v k 3 J b L b X F u 2 + W 2 O b f j c j u c 2 3 W 5 X c 7 t u d w e j z a t T k b u 8 4 4 H L n f A u U O X O + T c k c s d c a 7 j c h 3 O H b v c M e d O X O 7 E m c S p S 5 7 y j l 2 X 6 3 L u z O X O O H f u c u e c u 3 C 5 C 2 c w l y 5 5 y T t e u d w V 5 6 5 d 7 p p z N y 5 3 w 7 l b l 7 v l 3 J 3 L 3 X H u 3 u V q 6 V / z a r j 8 C P r p A p 9 q 1 + r O Z Z b C 1 D 7 p W i w Z G 6 i X U P q o y 2 S F 1 y W y o S q Y I X 2 D 9 G k 1 d V G C E B U j u h R B h M r h s h o K 1 R 2 6 6 k C E q g 1 d a y B C N Y a u M B C h y k L X F Y h E 7 P b v D U R l h C 4 i E K H i Q Z c O i M R s J Q y S E J I a J G U r a B A q C X R B g A g V A r o M Q I S K W Z 3 7 E a G c r z M + I u x B T C d 6 h C j B l 9 X O s H 0 p D U L J X K d y R C i F 6 w S O C C V u n b Y R o S J V 5 2 p E P r Y d m 8 5 T F Z R + n I / U n u u f t Q z L f i U Q r Q 0 L 0 h M Y v b a o q N h P + g P V w 1 w Q k S U Q K l z / J F j L U k n S B m i J H h H C f x k m o j B R n f V P 6 m z l W 0 m 3 n s p 0 y m c w V Z K 1 L V R s Q C 1 U 6 4 B N a 6 p U a l u o 0 i G 1 U K H h j N 7 Z K H 2 O i M U R R 9 R C W b 5 n g 0 d N P r A F m i o t 1 t O f K h 3 a F q 4 o W 0 r U Y E Y t 1 F 9 O L d T e B 2 q h 7 g q + W l M l u H q R p k p r t o X L P a Y W 6 q y k F m p s Q i 3 U 1 y O 1 U F t P 1 E J d f T R l 3 7 H K u o / V r X X K R b 1 R q t W J F p F N C g C d X x G i v K q z K i K U T X U u R Y R y q M 6 g i F D m 1 H k T E a r l d L J E h J K k T p G I U G r U i R E R S o g 6 H S J C a V A n Q U Q o + e n U h w i l P J 3 w E K E 6 T W c 5 R F i J p p M b Q p T U d E p D h F K Z T m S I U A L T 6 Q s R K s d 0 z k L k k r m + M h C l K J 2 g E K H E p N M S I p S O d D J C h J K Q T k G I 3 D H X 9 w a 6 Z 7 t I a a L P s 0 T S H V X H c g + v 2 A L a U 0 A x n e o k q K d X B b P i L k x A m 9 P p E l K h f i O 8 D U H s F 4 D S G m 2 o 0 w h v a S p A M Y z U S 1 V I g 2 w Q p S F 6 8 8 e x Q s S w v k 5 m U 6 H e B 1 + A X O S g n 8 W D 7 3 L T f 5 x h L C 4 3 X + q m Q v 9 W 0 S T S y q F + o 1 1 N T p q i M x U s B u S m x a j M l F s W 2 6 K F l 9 s W p E i Q O x a j W J C 7 F q N o k H s W o 3 i Q + x a j i J A H F q O Y k I c W o 6 i Q R x a j u J A d i 3 X Y o I 8 t S M E h T y x G 4 S F P L U b l n + x a r M s c n l m Q g k S e W 4 z C R F 5 Y j A J F X l q M y j p 5 Z T G K F X l t M Y o W e W M x i h d 5 a z G K G H l n M R Y z 8 t 6 C V Z 2 G c t 4 r / H x k 6 N A + B g f O g 0 i 4 y W A S R 7 j F Y D o m w 2 0 G k 0 L C H Q a T S M J d B p N O w j 0 G k 1 T C f Q a T W s I D B p N g w k M G k 2 b C I w Y f 0 e K E H Y b T o R o e M 5 j E E 5 4 w m P Q T n j K Y J B R 2 G U z n b H j G 4 D M 2 l H O G k 5 b C C w a T n M J L B p O i w i s G k 6 j C a w a z x + D w h u G k r f C W w S S v 8 I 7 B 9 E A Q 3 j P Y a E z 9 9 l 1 W 9 Z u o 3 7 P 0 + X s W s U k w 6 U t s E U r y E t u E a n W 9 8 L b 1 b z 7 G A j z f E y A 9 v H c M A 2 / n p d e H w F e 4 H E X C m 2 T j e I A Q t s A T + v c k W G G O C 0 9 9 z J P F 6 E h 9 L A O P O V a c + n f A 1 S 9 D x S 7 d k R Q q 9 g g l g Y p 9 Q k m f 4 o B Q 9 p Z G H B J M 8 h R H h N K h J j q E k j b F M a H H z P E J w S R N c U o o K V N 0 C S V h i j N C 6 X w T 5 4 S S K s U F o e w Z V l w S T K I U V 4 S S J s U 1 o X T U i R t C b 5 j j W 4 J J k O K O U N K j u C f U y n E n x V o Q 9 M O F b 1 7 H P F Y F I x U E H f e x Q F W M G 9 R C u W 5 S C 2 W 6 R W W t q h K 3 i c R T b 4 d a q K R d a q G C 9 q i F y t m n F i r m g F o o l E N q o U C O q I X C 6 F A L B X H M B o N C O C E S B X B K L d z 4 L j P F H T 8 j E n f 6 n F q 4 w x f U w o 2 9 p B Z u 6 B W 1 c C O v q Y U b e E M t 3 L d b a u F + 3 b H b 4 0 b d s x t W x V d V e K l t A 7 5 t 0 t R g 6 n B R g a w / 1 M N o N v B L b x L J U T a W H h Z A 3 g T z W w 6 F W y I B 1 U h O f V T d X 9 Z C 0 I Z z x S H o A g o a F R T o E g o a N R T o I g q o i q r Q 7 R p l r + d 0 H Q W N Q g p 0 J Q W N U g p 0 L Q W N Y g p 0 N Q V U T l n 4 o I b Z C z h d U k G j p g J d V E G j q g J d V g H V V R V 6 X K P s R Z u u r K B R W o G u r Y C K K w t 3 a 7 j L 4 b M a Z m / U d I k F j R o L d J E F j S o L d J k F j T o L d K E F j U o L d K k F j V o L d L E F j W o L d L k F V G 9 Z + K 6 G 2 R s y X X I B r 7 n w C Q I T k C z G 4 I 3 T A R T x k / r 0 a e B L 3 w s h h Q J z j 2 p H A v X e H 6 t E 5 G o 3 V 6 a z a f 5 u 2 i u S q W 7 o N K i 8 Q p J H R Y Q J 0 O l f f 4 T Y f 9 L J T 3 9 I o m 6 C 2 b L h 2 3 5 j o j 7 B z W f u L R z L L r f s z t o G k 2 Q D i L 9 t I t q g n o l p z Z p P M 9 3 K q t u w a o w 7 i g d Q W f Z 0 o x 5 + 3 Q M P C 5 k F I 1 + o D 2 j 9 s c z 0 o x U U z h A b H 8 P m x q Y e Z N V l f g A D c O x M s 8 W u Q A K P H m t n m i i G w P y i 2 L W O / T z 2 A 5 j V X + V 0 K m D m v f C q a 3 e B 3 f 4 7 s z r 7 7 T R H 0 h H s w 5 9 O k z 2 f 8 U z f c I s q Y 6 v c 6 B k X M / t 6 z i U K C G f 1 C 7 c m F U i a p G p F w 0 i t n m s m s q F M / E e y t E D T D r N G p r + E M m / i 5 r 3 k 8 V h N / 6 N 6 S 9 C Y 3 F F n x r + D O u r M 7 e G 1 X 9 A Q V K N 5 A 4 k / / A K 3 v 8 i Y 5 c X c F m x l J d G q o R P d T R Y P C z 9 R 7 6 x G k 6 z A m l X 4 T 8 J 7 3 v n m 9 X P 1 A Z D + M n 2 c m i 9 a R Y 4 S E P o r t O c 9 i G N m Y 9 + c v v A 2 M R V i 3 K f q n y c M e k j U 1 3 C q M D Z O m b X 6 H D U b h z p 7 6 j o 5 k v B S u x e Z N 8 h A u Z t E D 1 E O g 8 h f b X z Q n B V J r N 7 0 z 6 a d b 9 Z m L W S W g u L W 2 z g 5 0 f 1 e N / d D k 7 m i 8 p Z u W g 6 d b 3 p R O p R P z a 6 5 X 6 j X y H h 2 + C p g L g B P X O G H 4 E W p l 2 Z V k S / h c d X b G m V C r U + m C s J g 5 G 3 j M 3 E K n w u v n 2 U P q y o s 2 d u e 0 1 w d 0 l n x e 9 R 5 E e o R 4 M / e S 3 X 1 b Y b q u D S G k X 5 n 0 2 K p F Y t m + t 8 F F p e o q U v 1 q W A M s u f 3 M d b i b N I v w H 9 Y f v d s Z b 3 5 / y L m L 6 5 f r 6 7 / Y f X N 2 Z u V r z a r / z P x 6 d J v l n 6 7 9 M X S + t I f l 7 5 a 2 l / q L l 0 t B U v l 0 l + W / r b 0 9 / W / r v 9 j / Z / r / z K m 3 / u k 6 v P r J e f P + r / / B 2 z v T q s = < / l a t e x i t > Ĥ output < l a t e x i t s h a 1 _ b a s e 6 4 = "  2 O 3 + 1 Q M u d 3 z y 0 / z U m P N h W e D M D U g = " > A A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s y I q M u i G 5 c V 7 E P b U j J p p g 3 N Z I b k j l C G / o U b F 4 q 4 9 W / c + T d m 2 l l o 6 4 H A 4 Z x 7 y b n H j 6 U w 6 L r f T m F l d W 1 9 o 7 h Z 2 t r e 2 d 0 r 7 x 8 0 T Z R o x h s s k p F u + 9 R w K R R v o E D J 2 7 H m N P Q l b / n j m 8 x v P X F t R K T u c R L z X k i H S g S C U b T S Y z e k O P K D 9 G H a L 1 f c q j s D W S Z e T i q Q o 9 4 v f 3 U H E U t C r p B J a k z H c 2 P s p V S j Y J J P S 9 3 E 8 J i y M R 3 y j q W K h t z 0 0 l n i K T m x y o A E k b Z P I Z m p v z d S G h o z C X 0 7 m S U 0 i 1 4 m / u d 1 E g y u e q l Q c Y J c s f l H Q S I J R i Q 7 n w y E 5 g z l x B L K t L B Z C R t R T R n a k k q 2 B G / x 5 G X S P K t 6 F 9 X z u / N K 7 T q v o w h H c A y n 4 M E l 1 O A W 6 t A A B g q e 4 R X e H O O 8 O O / O x 3 y 0 4 O Q 7 h / A H z u c P 0 Q y R B g = = < / l a t e x i t > Y < l a t e x i t s h a 1 _ b a s e 6 4 = " A P c Q O U h v W 0 4 3 g o G B / 0 d 6 c U R q + a o = " > A A A y G n i c l V v J c t z I E e V 4 H d P b j H 3 0 B W F K M W O H x C A 1 C t v H 4 b 4 1 y e a + T E s K N D o b D R G b U N V o U h 3 t 7 / D B F / t T f H P 4 6 o t / x G d n V a G Q W W j 0 T F g R E l H v Z S V q e V m Z A K F + H k d C r q 3 9 5 5 P v f f 8 H P / z R j z / 9 y f J P f / b z X / z y s 8 9 / d S 2 y c R H A V Z D F W X H b 9 w X E U Q p X M p I x 3 O Y F + E k / h p v + w 5 b i b 0 o o R J S l l / I p h z e J H 6 b R M A p 8 i d D b X u L L U X 8 4 7 W W D T M 7 e f b a y t r q m / 3 j z F + v V x c p S 9 a f 7 7 v O 1 / / Y G W T B O I J V B 7 A v x z f p a L t 9 M / U J G Q Q y z 5 d 5 Y Q O 4 H D 3 4 I 3 + B l 6 i c g 3 k z 1 s G f e c 0 Q G 3 j A r 8 G 8 q P Y 3 y H l M / E e I p 6 a O l G q Z o c g p c x C m P 4 k U / a Q x B D v / 0 Z h q l + V h C G p g R D M e x J z N P L Y 4 3 i A o I Z P y E F 3 5 Q R D g J L x j 5 h R 9 I X M L l 5 e f e s V 8 8 e A J t c P G E l w 2 9 w M / N t Z p G A U M o i i g N l c N B V E b C m g 2 j c F w A j j K F S Z A l i Z 8 O p j 0 E Y x j K 2 X T a g 8 T 7 s o P X v 5 v N l u e M A l x d K K z Z l m 4 p w 6 Z d E Y W j 2 t u 5 a r R Z y S y 3 N p d Z 3 m b R z 6 T M E m u 0 q V t z d t X E f W v m z w + 9 M u l b k / 4 i J 4 G 1 C B Z Z D K z F Q N / m u b e P 0 4 v V F D 3 f Q 3 u 1 Z T D E G B h 4 u D i J 6 w O v F T j 7 Z v 0 N e u k P v Z V 1 5 Q S 9 7 O p t M f u G g o A X X p x N o H g Z Y E S t L v f Q p V 5 X G K 6 s T 8 0 W / r m H r a l x 0 N Y f x x t J P 1 7 1 d l E P Q m I g q O 0 X a s + Q N y 5 3 r c v d p k t N y 0 l m b 7 r y q r q t 8 K y R h 1 O q G q 9 s j w 9 j f 0 B d V r 5 a e T 3 X 7 U X d x 1 5 9 x V 2 9 R l f P v Q s j 7 E X r Y W 6 G 6 j e j r 8 K A D b / V g 1 0 R 0 / v C 9 r 5 o 6 X 1 u e + m A n G R 1 p K 3 W K 2 P u L v T S 1 H G 4 Y G 2 a D k c F A H P J p y P M w s 2 7 p H V j z r + a d + 6 n H u A 2 q M 4 t I o I P Z t L W Z P G s H T / j P I f C U 3 6 M m 5 3 K z U 6 b m w 2 v 8 C e 0 8 A 1 n L 1 + + 9 M s s G n h j o c 6 n a O j l m R A R p o l q H f L Y x / C o E L N e N X V t 2 2 L 6 d 8 U c L f u 3 3 X 7 2 6 n W N 8 B R q + v F 0 6 w 0 B 1 G s 9 B q r C z z X 0 U J d V Q 6 H c Z Y V m t Z X h t e X l Q F S / W S 6 3 k w 6 s s B Y m E 1 1 c R T 4 8 X S 7 a V D 6 c T T g B u / M d Z F M D T W b c w l C t n f Q z I y m B L l Q u S 4 X U Z y l G l a r i 0 6 y x C v 9 I v I x Z k W l c Z D + V P l + l G l W J O j 3 W Q + h Z z O 7 o k W D 9 o n p u 0 y f m M B l A m I G L j O Y 1 X o r w K W A O g 1 d Z k h M 6 D I h M S O X G R E T u U x E z H u X e c 8 G 9 + B S D 9 Q p d p l 4 p r V c J F 4 k M G 6 x k B 4 8 q T P P 7 O I L 7 / 1 Y S G + Q p V 9 I T 5 W y q M k n d Q A 5 W + M l l e / U 9 Z 3 S X T O X y Y j J X S Z n k / j g U h + o U + E y B T H C Z Q R z J 1 1 K U q e x y 4 y J K V 2 m J G b i M h N i H l 3 m k Z g n l 3 l i g / v o U h 9 n p t y y U Y B J O i t s D J R V q E y n 9 c N M H T z 1 w O X I h I q 1 0 G 1 2 w 5 K R L E j K P s E s Q s q A Y B Y e 5 Y D g A Y O B Y B Y X 5 Z D g I R 9 K S D g L i X J E M I u H c k w w C 4 b y P c H v G f x A M I u C M i Y 4 Z n B C c M J H y N a a L 3 J G M J N 0 m R O c M / g D w U z L Z U E w E 3 I p C B Z 8 X w m W f I R s V b h + S 4 K Z e M s J w U y 5 5 S P B T L b l E 8 F P D P 5 I s N X r T g z q y V g / A R a u d q u z 3 g i v 9 Y Q G o 7 7 W M x q M B F t P a T A 6 b D + n w a i x 9 a Q G I 8 n W s x q M L F  t P a z D a b D 2 v k V t 4 Y o N R a f P M t t z C Q x u M X p v l 7 v h 3 K 3 L 3 X L u z u X u O H f v c r X 0 r 3 k 1 X H 4 E / X S B T 7 V r d e c y S 2 F q n 3 Q t l o w N 1 E s o f d R l s s L r E t l Q F c y Q v k H 6 t J q 6 K E G I i h F d i i B C 5 X B Z D Y X q D l 1 1 I E L V h q 4 1 E K E a Q 1 c Y i F B l o e s K R C J 2 + / c G o j J C F x G I U P G g S w d E Y r Y S B k k I S Q 2 S s h U 0 C J U E u i B A h A o B X Q Y g Q s W s z v 2 I U M 7 X G R 8 R 9 i C m E z 1 C l O D L a m f Y v p Q G o W S u U z k i l M J 1 A k e E E r d O 2 4 h Q k a p z N S I f 2 4 5 N 5 6 k K S j / O R 2 r P 9 c 9 a h m W / E o j W h g X p C Y x e W 1 R U 7 C f 9 g e p h L o j I E g g V r n 8 S r G W p J G k D t E S P C O G / D B N R m K j O + i d 1 t v K t p F t P Z T r l M 5 g q y d o W K j a g F q p 1 w K Y 1 V S q 1 L V T p k F q o 0 H B G 7 2 y U P k f E 4 o g j a q E s 3 7 P B o y Y f 2 A J N l R b r 6 U + V D m 0 L V 5 Q t J W o w o x b q L 6 c W a u 8 D t V B 3 B V + t q R J c v U h T p T X b w u U e U w t 1 V l I L N T a h F u r r k V q o r S d q o a 4 + m r L v W G X d x + r W O u W i 3 i j V 6 k S L y C Y F g M 6 v C F F e 1 V k V E c q m O p c i Q j l U Z 1 B E K H P q v I k I 1 X I 6 W S J C S V K n S E Q o N e r E i A g l R J 0 O E a E 0 q J M g I p T 8 d O p D h F K e T n i I U J 2 m s x w i r E T T y Q 0 h S m o 6 p S F C q U w n M k Q o g e n 0 h Q i V Y z p n I X L J X F 8 Z i F K U T l C I U G L S a Q k R S k c 6 G S F C S U i n I E T u m O t 7 A 9 2 z X a Q 0 0 e d Z I u m O q m O 5 h 1 d s A e 0 p o J h O d R L U 0 6 u C W X E X J q D N 6 X Q J q V C / E d 6 G I P Y L Q G m N N t R p h L c 0 F a A Y R u q l K q R B N o j S E L 3 5 4 1 g h Y l h f J 7 O p U O + D L 0 A u c t D P 4 s F 3 u e k / z j A W l 5 s v d V O h f 6 t o E m n l U L / R r i Y n T d G Z C h Y D c t N i V G b K L Y t t 0 c L L b Q t S J M g d i 1 E s y F 2 L U T T I P Y t R P M h 9 i 1 F E y A O L U U z I Q 4 t R V M g j i 1 F c y I 7 F O m z Q x x a k 4 J A n F q P w k K c W o / J P d i 3 W Z Q 7 P L E h B I s 8 t R m E i L y x G g S I v L U Z l n b y y G M W K v L Y Y R Y u 8 s R j F i 7 y 1 G E W M v L M Y i F i l + 5 I C h V 7 h J J A x T 6 h p E 9 x Q C h 7 S y M O C S Z 5 i i N C 6 V A T H U J J m + K Y 0 G P m + I R g k q Y 4 J Z S U K b q E k j D F G a F 0 v o l z Q k m V 4 o J Q 9 g w r L g k m U Y o r Q k m T 4 p p Q O u r E D a E 3 z P E t w S R I c U c o 6 V H c E 2 r l u J N i L Q j 6 4 c I 3 r 2 M e q 4 K R C o K O + 1 i g K s Y N a q F c N 6 m F M t 2 i s l Z V i d t E 4 q m 3 Q y 1 U 0 i 6 1 U E F 7 1 E L l 7 F M L F X N A L R T K I b V Q I E f U Q m F 0 q I W C O G a D Q S G c E I k C O K U W b n y X m e K O n x G J O 3 1 O L d z h C 2 r h x l 5 S C z f 0 i l q 4 k d f U w g 2 8 o R b u 2 y 2 1 c L / u 2 O 1 x o + 7 Z D a v i q y q 8 1 L Y B 3 z Z p a j B 1 u K h A 1 h / q Y T Q b + I U 3 i e Q o G 0 s P C y B v g v k t h 8 I t k Y B q J K c + q u 4 v a y F o w 7 n i E H Q B B Y 0 K C n Q J B Y 0 a C n Q R B V R F V e h 2 j b L X c 7 q O g k Y h B b q S g k Y p B b q W g k Y x B b q a A i q n L H x Q w + w F n C 6 p o F F T g S 6 q o F F V g S 6 r g O q q C j 2 u U f a i T V d W 0 C i t Q N d W Q M W V h b s 1 3 O X w W Q 2 z N 2 q 6 x I J G j Q W 6 y I J G l Q W 6 z I J G n Q W 6 0 I J G p Q W 6 1 I J G r Q W 6 2 I J G t Q W 6 3 A K q t y x 8 V 8 P s D Z k u u Y D X X P g E g Q l I F m P w x u k A i v h J f f o 0 8 K X v h Z B C g b l H t S O B e u + P V S J y t Z s r 0 9 k 0 f z f t F c l U N 3 Q a V F 4 h y a M i w g T o 9 K 8 / Q u w / 6 e S n P y R R N 8 F s 2 f B t v z E Z + R I f 5 N 1 b O J Z d b t m d t Q 0 m y Q Y Q f 9 t E t E E 9 E 9 O a N Z 9 m u p V V t 2 H V G H c F l X G V r n R M y 5 m 9 v W c S x Q Q z u o X b k 0 q k D R J 1 Y q G k V o 9 1 0 x k Q 5 n 4 j 2 R p g a Y d Z o 1 M f w l l 3 s T N e 8 n j s Z r + R / W W o D G 5 o 8 6 M f w d 1 1 J n b w 2 u / o C G o R v M G E n / 4 B W 5 / k T H L i 7 k t 2 M p K o l V D J 7 q b L B 4 W f q L e W Y 0 m W Y E 1 q / C f h P e s 8 / b V M / U B k P 4 y f Z y a L 1 p F j h I Q + i u 0 Z z 2 I Y 2 Z j 3 5 w + 9 z Y x F W L c p + q f J w x 6 S N T X c K o w N k 6 Z t f o c N R u H O n v q O j m S 8 E K 7 F 5 k 3 y E C 5 m 0 Q P U Q 6 D y F 9 t f N C c F U m s 3 v T P p p 2 3 a 7 M W M k t B c e t t n J z o f q + a + 6 H J X F F 5 S z c t h 8 7 b X p Q O 5 V O z a + 4 X 6 j U y n h 2 + C p g L w B N X + C F 4 U e q l W V X k S 3 h c 9 b Z G m V D r k 6 m C M B h 5 2 / h M n M I X w u t n 2 c O q C k v 2 t u c 0 V 4 d 0 V v w e d V 6 E e g T 4 s / d C X X 2 b o T o u j W G k 3 9 m 0 W G r F o p n + d 4 H F J W r q U n 0 q G I P s + X 2 M t T i b 9 A v w H 5 b f f b a y 3 v x / E f M X 1 6 9 W 1 / + w + v r s 9 c r X m 9 X / m f h 0 6 T d L v 1 3 6 c m l 9 6 Y 9 L X y / t L 3 W X r p a C p W L p L 0 t / W / r 7 + l / X / 7 H + z / V / G d P v f V L 1 + f W S 8 2 f 9 3 / 8 D z W F N n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "  A P c Q O U h v W 0 4 3 g o G B / 0 d 6 c U R q + a o = " > A A A y G n i c l V v J c t z I E e V 4 H d P b j H 3 0 B W F K M W O H x C A 1 C t v H 4 b 4 1 y e a + T E s K N D o b D R G b U N V o U h 3 t 7 / D B F / t T f H P 4 6 o t / x G d n V a G Q W W j 0 T F g R E l H v Z S V q e V m Z A K F + H k d C b R z 6 T M E m u 0 q V t z d t X E f W v m z w + 9 M u l b k / 4 i J 4 G 1 C B Z Z D K z F Q N / m u b e P 0 4 v V F D 3 f Q 3 u 1 Z T D E G B h 4 u D i J 6 w O v F T j 7 Z v 0 N e u k P v Z V 1 5 Q S 9 7 O p t M f u G g o A X X p x N o H g Z Y E S t L v f Q p V 5 X G K 6 s T 8 0 W / r m H r a l x 0 N Y f x x t J P 1 7 1 d l E P Q m I g q O 0 X a s + Q N y 5 3 r c v d p k t N y 0 l m b 7 r y q r q t 8 K y R h 1 O q G q 9 s j w 9 j f 0 B d V r 5 a e T 3 X 7 U X d x 1 5 9 x V 2 9 R l f P v Q s j 7 E X r Y W 6 G 6 j e j r 8 K A D b / V g 1 0 R 0 / v C 9 r 5 o 6 X 1 u e + m A n G R 1 p K 3 W K 2 P u L v T S 1 H G 4 Y G 2 a D k c F A H P J p y P M w s 2 7 p H V j z r + a d + 6 n H u A 2 q M 4 t  o E L N e N X V t 2 2 L 6 d 8 U c L f u 3 3 X 7 2 6 n W N 8 B R q + v F 0 6 w 0 B 1 G s 9 B q r C z z X 0 U J d V Q 6 H c Z Y V m t Z X h t e X l Q F S / W S 6 3 k w 6 s s B Y m E 1 1 c R T 4 8 X S 7 a V D 6 c T T g B u / M d Z F M D T W b c w l C t n f Q z I y m B L l Q u S 4 X U Z y l G l a r i 0 6 y x C v 9 I v I x Z k W l c Z D + V P l + l G l W J O j 3 W Q + h Z z O 7 o k W D 9 o n p u 0 y f m M B l A m I G L j O Y 1 X o r w K W A O g 1 d Z k h M 6 D I h M S O X G R E T u U x E z H u X e c 8 G 9 + B S D 9 Q p d p l 4 p r V c J F 4 k M G 6 x k B 4 8 q T P P 7 O I L 7 / 1 Y S G + Q p V 9 I T 5 W y q M k n d Q A 5 W + M l l e / U 9 Z 3 S X T O X y Y j J X S Z n k / j g U h + o U + E y B T H C Z Q R z J 1 1 K U q e x y 4 y J K V 2 m J G b i M h N i H l 3 m k Z g n l 3 l i g / v o U h 9 n p t y y U Y B J O i t s D J R V q E y n 9 c N M H T z 1 w O X I h I q 1 0 G 1 2 w 5 K R L E j K P s E s Q s q A Y B Y e 5 Y D g A Y O B Y B Y X 5 Z D g I R 9 K S D g L i X J E M I u H c k w w C 4 b y P c H v G f x A M I u C M i Y 4 Z n B C c M J H y N a a L 3 J G M J N 0 m R O c M / g D w U z L Z U E w E 3 I p C B Z 8 X w m W f I R s V b h + S 4 K Z e M s J w U y 5 5 S P B T L b l E 8 F P D P 5 I s N X r T g z q y V g / A R a u d q u z 3 g i v 9 Y Q G o 7 7 W M x q M B F t P a T A 6 b D + n w a i x 9 a Q G I 8 n W s x q M L F  t P a z D a b D 2 v k V t 4 Y o N R a f P M t t z C Q x u M X p v l 7 v h 3 K 3 L 3 X L u z u X u O H f v c r X 0 r 3 k 1 X H 4 E / X S B T 7 V r d e c y S 2 F q n 3 Q t l o w N 1 E s o f d R l s s L r E t l Q F c y Q v k H 6 t J q 6 K E G I i h F d i i B C 5 X B Z D Y X q D l 1 1 I E L V h q 4 1 E K E a Q 1 c Y i F B l o e s K R C J 2 + / c G o j J C F x G I U P G g S w d E Y r Y S B k k I S Q 2 S s h U 0 C J U E u i B A h A o B X Q Y g Q s W s z v 2 I U M 7 X G R 8 R 9 i C m E z 1 C l O D L a m f Y v p Q G o W S u U z k i l M J 1 A k e E E r d O 2 4 h Q k a p z N S I f 2 4 5 N 5 6 k K S j / O R 2 r P 9 c 9 a h m W / E o j W h g X p C Y x e W 1 R U 7 C f 9 g e p h L o j I E g g V r n 8 S r G W p J G k D t E S P C O G / D B N R m K j O + i d 1 t v K t p F t P Z T r l M 5 g q y d o W K j a g F q p 1 w K Y 1 V S q 1 L V T p k F q o 0 H B G 7 2 y U P k f E 4 o g j a q E s 3 7 P B o y Y f 2 A J N l R b r 6 U + V D m 0 L V 5 Q t J W o w o x b q L 6 c W a u 8 D t V B 3 B V + t q R J c v U h T p T X b w u U e U w t 1 V l I L N T a h F u r r k V q o r S d q o a 4 + m r L v W G X d x + r W O u W i 3 i j V 6 k S L y C Y F g M 6 v C F F e 1 V k V E c q m O p c i Q j l U Z 1 B E K H P q v I k I 1 X I 6 W S J C S V K n S E Q o N e r E i A g l R J 0 O E a E 0 q J M g I p T 8 d O p D h F K e T n i I U J 2 m s x w i r E T T y Q 0 h S m o 6 p S F C q U w n M k Q o g e n 0 h Q i V Y z p n I X L J X F 8 Z i F K U T l C I U G L S a Q k R S k c 6 G S F C S U i n I E T u m O t 7 A 9 2 z X a Q 0 0 e d Z I u m O q m O 5 h 1 d s A e 0 p o J h O d R L U 0 6 u C W X E X J q D N 6 X Q J q V C / E d 6 G I P Y L Q G m N N t R p h L c 0 F a A Y R u q l K q R B N o j S E L 3 5 4 1 g h Y l h f J 7 O p U O + D L 0 A u c t D P 4 s F 3 u e k / z j A W l 5 s v d V O h f 6 t o E m n l U L / R r i Y n T d G Z C h Y D c t N i V G b K L Y t t 0 c L L b Q t S J M g d i 1 E s y F 2 L U T T I P Y t R P M h 9 i 1 F E y A O L U U z I Q 4 t R V M g j i 1 F c y I 7 F O m z Q x x a k 4 J A n F q P w k K c W o / J P d i 3 W Z Q 7 P L E h B I s 8 t R m E i L y x G g S I v L U Z l n b y y G M W K v L Y Y R Y u 8 s R j F i 7 y 1 G E W M v L M Y i F i l + 5 I C h V 7 h J J A x T 6 h p E 9 x Q C h 7 S y M O C S Z 5 i i N C 6 V A T H U J J m + K Y 0 G P m + I R g k q Y 4 J Z S U K b q E k j D F G a F 0 v o l z Q k m V 4 o J Q 9 g w r L g k m U Y o r Q k m T 4 p p Q O u r E D a E 3 z P E t w S R I c U c o 6 V H c E 2 r l u J N i L Q j 6 4 c I 3 r 2 M e q 4 K R C o K O + 1 i g K s Y N a q F c N 6 m F M t 2 i s l Z V i d t E 4 q m 3 Q y 1 U 0 i 6 1 U E F 7 1 E L l 7 F M L F X N A L R T K I b V Q I E f U Q m F 0 q I W C O G a D Q S G c E I k C O K U W b n y X m e K O n x G J O 3 1 O L d z h C 2 r h x l 5 S C z f 0 i l q 4 k d f U w g 2 8 o R b u 2 y 2 1 c L / u 2 O 1 x o + 7 Z D a v i q y q 8 1 L Y B 3 z Z p a j B 1 u K h A 1 h / q Y T Q b + I U 3 i e Q o G 0 s P C y B v g v k t h 8 I t k Y B q J K c + q u 4 v a y F o w 7 n i E H Q B B Y 0 K C n Q J B Y 0 a C n Q R B V R F V e h 2 j b L X c 7 q O g k Y h B b q S g k Y p B b q W g k Y x B b q a A i q n L H x Q w + w F n C 6 p o F F T g S 6 q o F F V g S 6 r g O q q C j 2 u U f a i T V d W 0 C i t Q N d W Q M W V h b s 1 3 O X w W Q 2 z N 2 q 6 x I J G j Q W 6 y I J G l Q W 6 z I J G n Q W 6 0 I J G p Q W 6 1 I J G r Q W 6 2 I J G t Q W 6 3 A K q t y x 8 V 8 P s D Z k u u Y D X X P g E g Q l I F m P w x u k A i v h J f f o 0 8 K X v h Z B C g b l H t S O B e u + P V S J y t Z s r 0 9 k 0 f z f t F c l U N 3 Q a V F 4 h y a M i w g T o 9 K 8 / Q u w / 6 e S n P y R R N 8 F s 2 f B t v z E Z + R I f 5 N 1 b O J Z d b t m d t Q 0 m y Q Y Q f 9 t E t E E 9 E 9 O a N Z 9 m u p V V t 2 H V G H c F l X G V r n R M y 5 m 9 v W c S x Q Q z u o X b k 0 q k D R J 1 Y q G k V o 9 1 0 x k Q 5 n 4 j 2 R p g a Y d Z o 1 M f w l l 3 s T N e 8 n j s Z r + R / W W o D G 5 o 8 6 M f w d 1 1 J n b w 2 u / o C G o R v M G E n / 4 B W 5 / k T H L i 7 k t 2 M p K o l V D J 7 q b L B 4 W f q L e W Y 0 m W Y E 1 q / C f h P e s 8 / b V M / U B k P 4 y f Z y a L 1 p F j h I Q + i u 0 Z z 2 I Y 2 Z j 3 5 w + 9 z Y x F W L c p + q f J w x 6 S N T X c K o w N k 6 Z t f o c N R u H O n v q O j m S 8 E K 7 F 5 k 3 y E C 5 m 0 Q P U Q 6 D y F 9 t f N C c F U m s 3 v T P p p 2 3 a 7 M W M k t B c e t t n J z o f q + a + 6 H J X F F 5 S z c t h 8 7 b X p Q O 5 V O z a + 4 X 6 j U y n h 2 + C p g L w B N X + C F 4 U e q l W V X k S 3 h c 9 b Z G m V D r k 6 m C M B h 5 2 / h M n M I X w u t n 2 c O q C k v 2 t u c 0 V 4 d 0 V v w e d V 6 E e g T 4 s / d C X X 2 b o T o u j W G k 3 9 m 0 W G r F o p n + d 4 H F J W r q U n 0 q G I P s + X 2 M t T i b 9 A v w H 5 b f f b a y 3 v x / E f M X 1 6 9 W 1 / + w + v r s 9 c r X m 9 X / m f h 0 6 T d L v 1 3 6 c m l 9 6 Y 9 L X y / t L 3 W X r p a C p W L p L 0 t / W / r 7 + l / X / 7 H + z / V / G d P v f V L 1 + f W S 8 2 f 9 3 / 8 D z W F N n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V Y e g S X r W F W G v 9 M s y 7 N h / i 0 R q 2 I E = " > A A A B 7 n i c b V B N S w M x E J 3 U r 1 q / q h 6 9 B I v g q e x K U Y 9 F L x 4 r 2 A 9 o l 5 J N s 2 1 o N h u S b K E s / R F e P C j i 1 d / j z X 9 j 2 u 5 B W x 8 M P N 6 b Y W Z e q A Q 3 1 v O + U W F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T J J q y p o 0  E Y n u h M Q w w S V r W m 4 F 6 y j N S B w K 1 g 7 H 9 3 O / P W H a 8 E Q + 2 a l i Q U y G k k e c E u u k d m 9 C t B r x f r n i V b 0 F 8 D r x c 1 K B H I 1 + + a s 3 S G g a M 2 m p I M Z 0 f U / Z I C P a c i r Y r N R L D V O E j s m Q d R 2 V J G Y m y B b n z v C F U w Y 4 S r Q r a f F C / T 2 R k d i Y a R y 6 z p j Y k V n 1 5 u J / X j e 1 0 W 2 Q c a l S y y R d L o p S g W 2 C 5 7 / j A d e M W j F 1 h F D N 3 a 2 Y j o g m 1 L q E S i 4 E f / X l d d K 6 q v r X 1 d p j r V K / y + M o w h m c w y X 4 c A N 1 e I A G N I H C G J 7 h F d 6 Q Q i / o

3.2. MOVING AVERAGE EQUIPPED GATED ATTENTION

The gated attention mechanism in MEGA adopts the Gated Recurrent Unit (GRU; Cho et al. (2014) ) and Gated Attention Unit (GAU; Hua et al. ( 2022)) as the backbone architectures, with an EMA-based sub-layer embedded into the calculation of the attention matrix. Formally, we first use the output from (5) to compute the shared representation in GAU: X ′ = EMA(X) ∈ R n×d (6) Z = ϕ silu (X ′ W z + b z ) ∈ R n×z where X ′ can be regarded as the updated or contextual input, because it encodes contextual information through EMA. Z is the shared representation with z dimensions, with projection matrix W z ∈ R d×z and bias term b z ∈ R z . ϕ silu is the self-gated activation function (SiLU) (Ramachandran et al., 2017; Elfwing et al., 2018) . Following GAU, the query and key sequences are computed by applying per-dimension scalars and offsets to Z, and the value sequence is from the original X: Q = κ q ⊙ Z + µ q ∈ R n×z (8) K = κ k ⊙ Z + µ k ∈ R n×z (9) V = ϕ silu (XW v + b v ) ∈ R n×v where κ q , µ q , κ k , µ k ∈ R z are the learnable scalars and offsets of queries and keys, respectively. v is the expanded intermediate dimension for the value sequence. The output of attention is: O = f QK T τ (X) + b rel V ∈ R n×v The graphical specification is displayed in Figure 2 (c ). b rel ∈ R n×n is the relative positional bias. We choose b rel from existing approaches, including T5 (Raffel et al., 2020) , RoPE (Su et al., 2021) , TUPE (Ke et al., 2020) and ALiBi (Press et al., 2021) . Subsequently, MEGA introduces the reset gate γ, the update gate φ, and computes the candidate activation output Ĥ: γ = ϕ silu (X ′ W γ + b γ ) ∈ R n×v (12) φ = ϕ sigmoid (X ′ W φ + b φ ) ∈ R n×d (13) Ĥ = ϕ silu (X ′ W h + (γ ⊙ O)U h + b h ) ∈ R n×d The final output Y is computed with the update gate φ: Y = φ ⊙ Ĥ + (1 -φ) ⊙ X ∈ R n×d The graphical architecture of a MEGA sub-layer is visualized in Figure 2 (b) . Laplace Attention Function. As mentioned in Section 2.1, the softmax function is the most common choice for the attention function f (•). So et al. (2021) recently introduced the squared ReLU function f relu 2 (•) via architecture search techniques, which has shown faster convergence speed and competitive generalization performance on language tasks (Hua et al., 2022) . However, one issue of f relu 2 (•) is that neither its range nor its gradient is bounded, leading to unstable model training (see Appendix C.1 for details). To address this issue, we propose the Laplace attention function: f laplace (x; µ, σ) = 0.5 × 1 + erf( x -µ σ √ 2 ) where erf(•) is the error function. µ and σ are two coefficients that we adjust to approximate f relu 2 , yielding µ = 1/2 and σ = 1/4π (The derivations and visualization are provided in Appendix C). Relation to and Differences from GRU, Flash and S4. The computation of the the reset gate γ, the update gate φ, and the candidate activation output Ĥ in (12-14) is reminiscent of GRUs (Cho et al., 2014) . The main difference is that in a GRU the two gates are applied between the hidden states of the current and previous timesteps, while in MEGA they are applied between the outputs from EMA and gated attention sub-layers. In addition, the output gating mechanism in ( 15) is similar to the gated residual connection proposed in Parisotto et al. (2020) ; Xu et al. (2020) . The computation of the shared representation Z, together with the sequences of queries, keys and values in (7-10) are inspired from GAU in Flash (Hua et al., 2022) . MEGA integrates EMA into GAU by computing Z in (7) from the EMA output X ′ rather than the original input X, and combining the GAU output with X ′ for the candidate activation Ĥ in (14). Experimental gains over Flash demonstrate the effectiveness of this design chice ( §4.1). The multi-dimensional damped EMA can be seen as a simplified variant of a state space model. From this perspective, MEGA is also closely related to S4 (Gu et al., 2022a) , a state space model with structured state matrices. S4 leverages the HiPPO framework (Gu et al., 2020) to initialize its low-rank structured state matrices, and the computation of the convolutional kernel in S4 requires complex fast Fourier transformers (FFTs). The EMA sub-layer in MEGA applies diagonalization on the state matrix, which is similar to a concurrent work S4D (Gu et al., 2022b) . The differences between the MEGA EMA sub-layer and S4D are: i) S4D parameterizes state matrix in the complex field while EMA uses real numbers; ii) EMA restricts the diagonal elements in the range of (0, 1); iii) EMA does not rely on the HiPPO framework for parameter initialization.

3.3. MEGA BLOCKS

The MEGA layer (moving average equipped gated attention) is used as a drop-in-replacement for regular attention in Transformer. It is followed by position-wise feed-forward networks (FFNs) and normalization layers to compose one MEGA block. As the gated residual connection has already been included in (15), we omit the original residual connection and directly apply a normalization layer to Y . Concretely, Y = Norm(Mega(X)) Y ′ = Norm(FFN(Y ) + Y ) where Y ′ is the output of the MEGA block. The overall architecture of a MEGA block is shown in Figure 2 2022), without any discussions on its theoretical insights. In this section, we provide theoretical justifications of the expressiveness of single-head gated attention. To facilitate subsequent analysis, we simplify the notations of the multi-head attention. Specifically, we denote the sequences of queries, keys and values as the outputs of input transformations: Q = Q(X), K = K(X), V = V(X) where Q, K, V are three transformations, such as linear projections. Let q ∈ Q = {q 1 , . . . , q n } be a single query vector (q ∈ R d ), and a = A(q, K) denote the corresponding attention weights of q, where A is the attention transformation, i.e. f (•) in (11). For multi-head attention, a common implementation is to split the query into h heads across the model dimension: q = [q (1) , . . . , q (h) ], where q (i) ∈ R d/h , and i ∈ {1, . . . , h} is the query of the i-th head. K and V are split in the same way. The attention weight of the i-th head is a (i) = A(q (i) , K (i) ). Then, the outputs of single-head and multi-head attention are, respectively: O SHA = a T V =    a T V (1) . . . a T V (h)    , O MHA =     a (1) T V (1) . . . a (h) T V (h)     It is straightforward to see that O MHA is more expressive than O SHA , because O MHA leverages h sets of attention weights. In the single-head gated attention, we introduce a gate vector γ = G(X) for each q, and the output of single-head gated attention is O SHGA = O SHA ⊙ γ. The following theorem reveals the equivalence of O SHGA and O MHA w.r.t expressiveness (proof in Appendix B): Theorem 1 Suppose the transformation G is a universal approximator. Then, ∀X, ∃ γ = G(X) s.t. O SHGA = O MHA Theorem 1 indicates that by simply introducing the gate vector, O SHGA is as expressive as O MHA . In practice, G is commonly modeled by a (shallow) neural network, whose universality of approximation has been extensively studied (Hornik et al., 1989; Yarotsky, 2017; Park et al., 2020) .

4. EXPERIMENTS

To evaluate MEGA, we conduct experiments on five benchmark sequence modeling tasks across various data types. All the numbers with ‡ indicate results from the baseline models replicated by us. More detailed descriptions, results and analysis are provided in Appendix D. 2013)), image classification on sequences of pixels (Image; Krizhevsky et al. (2009) ), Pathfinder (Linsley et al., 2018) and its extreme long version (Path-X; Tay et al. (2021) ). These tasks consist of input sequences ranging from 1K to 16K tokens and span across a variety of data types and modalities. Table 2 compares MEGA against several baselines, including Transformer and its efficient variants, and the state-of-the-art S4 models (both version 1 (Gu et al., 2022a) and version 2 (Gu et al., 2022b) ). 2To ensure fair comparison, we adjust the number of layers and model dimensions on each task so that MEGA has similar number of parameters with S4-v1. For each experiment, we report the average over 5 runs with different random seeds. On all the six tasks, MEGA substantially outperforms all the baselines. We also evaluate MEGA-chunk on each task, by setting the chunk size c = 128 for all the tasks, except Path-X where c = 4096. We observe that MEGA-chunk consistently performs well, particularly on the three language tasks. We also examine the speed and memory efficiency of MEGA on the byte-level classification task with the input length of 4K. MEGA-chunk is highly efficient, which is about 5.5 times faster and consumes only 13% as much memory as the vanilla Transformer. In addition, MEGA with full attention field is also much more efficient than Transformer, benefiting from single-head gated attention. Analysis of Multi-dimensional Damped EMA. To demonstrate the effectiveness of the multidimensional damped EMA component in MEGA, we performs ablation studies on two LRA tasksbyte-level text classification (Text) and image classification on sequences of pixels (Image). We train MEGA models with EMA dimension h ∈ {0, 1, 2, 4, 8, 16, 32}, where h = 0 indicates removing the EMA component. From the left figure in Figure 4 , we see that without the EMA component, model accuracy on both the two tasks declines rapidly. Meanwhile, the significant improvements of MEGA with a single dimensional EMA (h = 1) demonstrate the importance of EMA inductive bias. Analysis of Chunk Size. We further analyze the impact of chunk size c on the same two tasks, by varying c ∈ {16, 32, 64, 128, 256, 512, ∞}, where ∞ indicates the original MEGA without chunking. The right figure in Figure 4 shows that image data is more sensitive to chunk size than text data. On the Text task, MEGA-chunk with even a small chunk size c = 16 is able to achieve around 90% accuracy. On the Image task, MEGA-chunk with c = 16 achieves around 75% accuracy, which is still much better than the vanilla Transformer model. Table 3 : Attention functions. Analysis of Attention Functions. Finally, we evaluate performance with different attention functions. Table 3 shows the accuracy of the three attention functions. On text data softmax obtains the best accuracy, while on image data it performs the worst. The laplace function achieves the best accuracy on Image and also competitive result on Text, being consistently better than relu 2 . In the following experiments we use softmax for language tasks and laplace for vision and speech ones.

4.2. RAW SPEECH CLASSIFICATION

To evaluate the capability of MEGA on the long-range modeling of speech signals, we apply MEGA to classify raw speech (with length 16000), rather than using traditional preprocessing (e.g. convert to MFCC features). Following Gu et al. (2022a) , we perform speech classification on the SC10 subset of the Speech Commands dataset (Warden, 2018) . We experiment with the MEGA-chunk variant with c = 1000, since the computation of MEGA and Transformer can not fit in GPU memory. As shown in Table 4 , our MEGA-chunk (base) model with 300K parameters is able to achieve an accuracy of 96.92 that is slightly worse than 97.50 from the state-of-the-art method S4,foot_2 while by adding 0.18M parameters our MEGA-chunk (big) model performs comparably well with S4.

4.3. AUTO-REGRESSIVE LANGUAGE MODELING

We evaluate MEGA on two established language modeling benchmarks -WikiText-103 (Merity et al., 2017) and enwik8 (Hutter, 2006) . WikiText-103 is a word-level language modeling dataset containing 103M training tokens from Wikipedia articles. Following previous work (Baevski & Auli, 2018) , we adopt adaptive softmax and input embeddings and use a vocabulary of 260K tokens. Enwik8 is a character-level language modeling benchmark that has 100M tokens of unprocessed Wikipedia articles and a vocabulary size of about 200. At test time, we split the test data into segments and process each segment sequentially. In Table 5 , we compare with previous top-performing models that are designed to take advantage of longer context, including Transformers (Baevski & Auli, 2018; Al-Rfou et al., 2019) (XFM-adaptive), Transformer-XL (Dai et al., 2019) (XFM-XL) and S4 (Gu et al., 2022a) . On both WikiText-103 and enwik8, we obtain very competitive results, outperforming baselines by a large margin while enjoying much faster (9×) inference speed compared to the Transformer model. MEGA can also naturally achieve length extrapolation at inference time to any sequences that are longer than those seen during training due to the recurrent design of the EMA layer. In addition, we can extrapolate to a longer chunk size for MEGA attention due to the use of rotary positional embeddings for training (Su et al., 2021) . The details are provided in in Appendix D.4.

4.4. NEURAL MACHINE TRANSLATION

To evaluate MEGA on sequence-to-sequence modeling, we conduct experiments on a standard machine translation benchmark, WMT 2016 English-German news translation (WMT'16), consisting of 4.5M sentence pairs of training data, following Ott et al. (2018) . The MEGA models closely follow the architecture of Transformer-base: 6 encoder and decoder layers with model dimension d = 512. Table 7 presents the BLEU scores on the test sets of WMT'16 from two directions: EN→DE and DE→EN. For each experiment, we report the average of both tokenized and SacreBLEU (Post, 2018) scores with 5 different random seeds. MEGA-base significantly outperforms Transformer-base by over 1.1 BLEU. We also report results of MEGA with the Laplace attention function, which slightly but consistently underperforms Softmax.

APPENDIX: MEGA: MOVING AVERAGE EQUIPPED GATED ATTENTION

A EFFICIENT COMPUTATION OF MULTI-DIMENSIONAL DAMPED EMA Note that the computation of the multi-dimensional damped EMAs of different dimensions are entirely independent of each other. Without loss of generality, we set d = 1 and omit the dimension index j in the following formulations. We denote the initial hidden state as h 0 . The multi-dimensional damped EMA defined in (5) can be vectorized into the following formulation: h t = α ⊙ u t + (1 -α ⊙ δ) ⊙ h t-1 y t = η T h t ( ) where α, δ, and η ∈ R h . u t = βx t ∈ R h and h t ∈ R h is the EMA hidden state at timestep t. Let's denote ϕ = 1α ⊙ δ. Then, unrolling the above two equations explicitly yields: h 1 = ϕ ⊙ h 0 + α ⊙ βx 1 h 2 = ϕ 2 ⊙ h 0 + ϕ ⊙ α ⊙ βx 1 + α ⊙ βx 2 . . . y 1 = η T ϕ ⊙ h 0 + η T α ⊙ βx 1 y 2 = η T ϕ 2 ⊙ h 0 + η T ϕ ⊙ α ⊙ βx 1 + η T α ⊙ βx 2 . . . This can be written into a vectorized formula: y t = η T ϕ t ⊙ h 0 + η T ϕ t-1 ⊙ α ⊙ βx 1 + . . . + η T α ⊙ βx t (23) y = K * x + η T ϕ t ⊙ h 0 (24) where * is the convolution transform with kernel K ∈ R n : K = η T (α ⊙ β), η T (ϕ ⊙ α ⊙ β), . . . , η T (ϕ t ⊙ α ⊙ β (25) In the proposed multi-dimensional damped EMA, K can be efficiently computed by the Vandermonde product. With K provided, the output y in (24) can be computed efficiently with FFTs.

B PROOF OF THEOREM 1

Proof We split γ into h heads in the same way as Q, K, and V : 1) . . . γ =    γ ( γ (h)    Then we have O SHGA = a T V ⊙ γ =    a T V (1) ⊙ γ (1) . . . a T V (h) ⊙ γ (h)    To prove Theorem 1, we need to find γ such that i) , ∀i ∈ {1, . . . , h}, where ⊘ is the element-wise divide operation. Since G(X) is a universal approximator and Q, K, V and a are all transformed from X, γ can theoretically recover a (i) T V (i) ⊘ a T V (i) , ∀X. a T V (i) ⊙ γ (i) = a (i) T V (i) ⇐⇒ γ (i) = a (i) T V (i) ⊘ a T V (

C LAPLACE ATTENTION FUNCTION

To approximate the squared ReLU function with the Laplace function in (16), we need to select proper coefficients µ and σ. We derive the values of µ and σ by solving the following two equations at x = √ 2: f relu2 ( √ 2) = f laplace ( √ 2) (26) f ′ relu2 ( √ 2) = f ′ laplace ( √ 2) The Eq. ( 26) delivers µ = 1/2 and Eq. 27 subsequently provides σ = 1/4π. Figure 5 visualizes the two functions.



Keys and values are split in the same way. The S4-v2 used larger model sizes and better-tuned hyper-parameters than S4-v1. We have also experimented with SRU++(Lei, 2021) on Pathfinder but failed to converge on this dataset after tuning hyperparameters. Our S4 number is obtained by running the official S4 code and is a bit worse than the original reported number (98.32), due to different data splits -the order of os.listdir is not deterministic across machines. https://github.com/pytorch/fairseq signature: nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:1.5.1



s s Y B N 1 g l f p 6 B K N g j e Y P A m w Z s M 3 i J4 i 8 H b B G 8 z e I f g H Q b v E r z L 4 D 2 C 9 / j A 9 w n f Z + Y H B B 8 w u E d w j 8 G H B B 8 y + I j g I w Y f E 3 z M h 9 I n v M / M T w g + Y f A p w a c M P i P 4 j M H n B J 8 z + I L g C w Z f E n z J R 3 h F + B U z vy b 4 m s E 3 B N 8 w + J b g W 1 M 3 t 5 6 0 r v L A S I 8 J d Y 0 p X M u P c e u c 2 3 C 5 D c 5 t u t w m 5 7 Z c b o t z 2 y 6 3 z b k d l 9 v h 0 a b V y c h d 3 n H P 5 f Y 4 t + 9 y + 5 w 7 c L k D z v V c r s e 5 Q 5 c 7 5 N y R y x 0 5 k z h 2 y W P e s e 9 y f c 6 d u N w J 5 0 5 d 7 p R z Z y 5 3 5 g z m 3 C X P e c c L l 7 v g 3 K X L X X L u y u W u O H f t c t e c u 3 G 5 G 8 7 d u l w t / U t e D Z c f Q D 9 d 4 F P t a t 2 5 z F K Y 2 S d d i y U T A 9 0 l l D 7 q

r b g G + b N D W Y O l x U I O u P 9 z C a D b z s T S M 5 z i b S w w L I m 2 J + y 6 F w S y S g G s m p j 6 r 7 y 1

e u c 2 3 C 5 D c 5 t u t w m 5 7 Z c b o t z 2 y 6 3 z b k d l 9 v h 0 a b V y c h d 3 n H P 5 f Y 4 t + 9 y + 5 w 7 c L k D z v V c r s e 5 Q 5 c 7 5 N y R y x 0 5 k z h 2 y W P e s e 9 y f c 6 d u N w J 5 0 5 d 7 p R z Z y 5 3 5 g z m 3 C X P e c c L l 7

y b 4 m s E 3 B N 8 w + J b g W 1 M 3 t 5 6 0 r v L A S I 8 J d Y 0 p X M u P c e u c 2 3 C 5 D c 5 t u t w m 5 7 Z c b o t z 2 y 6 3 z b k d l 9 v h 0 a b V y c h d 3 n H P 5 f Y 4 t + 9 y + 5 w 7 c L k D z v V c r s e 5 Q 5 c 7 5 N y R y x 0 5 k z h 2 y W P e s e 9 y f c 6 d u N w J 5 0 5 d 7 p R z Z y 5 3 5 g z m 3 C X P e c c L l 7

r b g G + b N D W Y O l x U I O u P 9 z C a D b z s T S M 5 z i b S w w L I m 2 J + y 6 F w S y S g G s m p j 6 r 7 y 1

r N l e l 8 l r + d 3 R X J T D d 0 G l R e I c m j I s I E 6 P S v P 0 I c P O r k p z 8 k U T f B b N n w b b 8 x G f s S H + T d W z i W f W 7 Z n 7 c N J s m G E H 9 s I t q g n o l p z Z t P M / 3 K q t + w a o w 7 i o d Q W d 7 p R j 3 8 u g c e F j I L x r 5 Q H 9 X 6 E 5 n p R y s o n C E 2 P o b N j U 0 9 y K r L 0 w E M w b E z z R a 7 A g k 8 e q y d a a I Y A v N D s W s d + 3 n s B z C v v 8 r p V c D c e

p s t t c m 7 L 5 b Y 4 t + 1 y 2 z z a t D o Z u c M 7 7 r r c L u f 2 X G 6 P c / s u t 8 + 5 n s v 1 O H f g c g e c O 3 S 5 Q 2 c S R y 5 5 x D v 2 X a 7 P u W O X O + b c i c u d c O 7 U 5 U 6 d w Z y 5 5 B n v e O 5 y 5 5 y 7

7 b 5 b Y 5 t + N y O z z a t D o Z u c s 7 7 r n c H u f 2 X W 6 f c w c u d 8 C 5 r s t 1 O X f o c o e c O 3 K 5 I 2 c S x y 5 5 z D v 2 X K 7 H u R O X O + H c q c u d c u 7 M 5 c 6 c w Z y 7 5 D n v e O F y F 5 y 7 d L l L z l 2 5 3 B X n r l 3 u m

e n P c x J Y i 2 C e x c B a D P R t X n h 7 O L 1 Y T d H z P b R X W w Z D j I a B h 4 u T u D 7 w W o H T r 9 d e o 5 f + 0 F t a U 0 7 Q y 4 7 e F r N v K A h Y 9 u L s A Y q X A c b W y m I P X ep 1 h e H S 2 s R s 4 Z 9 7 2 J o Y B 2 3 9c b y R 9 O M V b w f 1 I C Q G g t p + o f Y M e e N y x 7 r c a b r U t H z I 7 E 2 X X l W 3 F Z 4 1 8 n B K V e O V 7 f F u 7 A + o y 9 I X S 1 / O d F u u + 9 i r L 7 i r L 9 H V C + / c C H v e e p i b o f r N 6 K s w Y M N v 9 W B X x P Q + t 7 3 P W 3 q f 2 V 4 6 I B + y O t J W 6 p U x dx d 6 a e o 4 n L M 2 T Y e j A o C 5 5 N M R Z u F m X d K 6 M e d f z D r 3 U w 9 w G 1 T n F h H B O z N p a z J / 1 o 6 f c Z 5 D 4 S k / x s 1 2 5 W a 7 z c 2 6 V / g P t P A N Z y 9

C L 7 i I 7 w m / J q Z 3 x B 8 w + B b g m 8 Z f E f w n a m b W 0 9 a V 3 l g p M e E u s 4 U r u X H u A 3 O b b r c J u e 2 X G 6 L c 9 s u t 8 2 5 H Z f b 4 d y u y + 3 y a N P q Z O Q e 7 7 j v c v u c O 3 C 5 A 8 4 d u t w h 5 z o u 1 + H c k c s d c e 7 Y 5 Y 6 d S Z y 4 5 A n v 2 H W 5 L u d O X e 6 U c 2 c u d 8 a 5 c 5 c 7 d w Z z 4 Z I X v O O l y 1 1 y 7

7 j P Y B J M e M B g 0 k x 4 y O B D W p y w w 3A 6 V M M j B p N 4 w m M G k 3 7 C E w a T h M I u g + m c D U 8 Z f M q G c s Z w 0 l J 4 z m C S U 3 j B Y F J U e M l g E l V 4 x W D 2 G B x e M 5 y 0 F d 4 w m O Q V 3 j K Y H g j C O w Y b j a n f v s u q f h P 1 e 5 Y + f 8 8 i N g g m f Y l N Q k l e Y o t Q r a 4 X 3 p b + z c d Y g O d 7 A q S H9 4 5 h 4 G 0 v e 3 0 I f I X L U S S 8 h 2 w c D x D C F n h C / 5 4 E K 8 x x 4 a m P e b I Y H a m P Z e A x x 4 p T / w 6 4

y b 4 m s E 3 B N 8 w + J b g W 1 M 3 t 5 6 0 r v L A S I 8 J d Y 0 p X M u P c e u c 2 3 C 5 D c 5 t u t w m 5 7 Z c b o t z 2 y 6 3 z b k d l 9 v h 0 a b V y c h d 3 n H P 5 f Y 4 t + 9 y + 5 w 7 c L k D z v V c r s e 5 Q 5 c 7 5 N y R y x 0 5 k z h 2 y W P e s e 9 y f c 6 d u N w J 5 0 5 d 7 p R z Z y 5 3 5 g z m 3 C X P e c c L l 7 v g 3 K X L X X L u y u W u O H f t c t e c u 3 G 5 G 8 7 d u l w t / U t e D Z c f Q D 9 d 4 F P t a t 2 5 z F K Y 2 S d d i y U T A 9 0 l l D 7 q

p b r B 4 e O o 8 z D G a W i a p G N O 9 s v m / Z 1 k 5 2 q o d b X 2 n I 5 x 0 G o I + i Y 2 t q O a j 8 X p I q B b L W 1 8 v X y 5 U C g 7 P j 8 M M E 8 A o a Z k o c m Z 4 t d G 3 z p S 5 m p v q h n W 1 0 e L Ka t 7 e D y d R + 1 o o e t P r 0 u m 1 8 Z 2 9 5 p Y 1 x 5 O r m j o T

H t u U S l 0 u c y b e d y T W 5 8 F g G I + L m w W y 5 t p O 5 J h c e z m B E 3 T y e L b f wf A a j 7 t Y T G o z E W 8 9 o M D p v P a X B i L 3 1 n A a j + P a T G o z u F 5 / V G B F F F N Q F S 7 J B U b L B A j b Z J H y T g i r Z I n i L w d s E b z N 4 h + A d B u 8 S v M v g P Y L 3 G L x P 8 D 6 D D w g + 4 A M / J P y Q m R 8 R f M T g D s E d B h 8 T f M z g E 4 J P G H x K 8 C k f S p f w L j M / I / i M w e c E n z P 4 g u A L B l 8 S f M n g K 4 K v G H x N 8 D U f 4 Q 3 h N 8 z 8 l u B b B t 8 R f M f g e 4 L v T d 3 c e t K 6 y g M j P S b U D a Z w L T / G b X J u y + W 2 O L f t c t u c 2 3 G 5 H c7 t u t w u 5 / Z c b o 9 H m 1 Y n I / d 5 x w O X O + D c o c s d c u 7 I 5 Y 4 4 1 3 G 5 D u e O X e 6 Y c y c u d + J M 4 t Q l T 3 n H r s t 1 O X f m c m e c O 3 e 5 c 8 5 d u N y F M 5 h L l 7 z k H a 9 c 7 o p z 1 y 5 3 z b k b

x l 5 b 8 G q T k M 5 7 x V + P j J 0 a B + DA + d B J N x k M I k j 3 G I w H Z P h N o N J I e E O g 0 k k 4 S 6 D S S f h H o N J K u E + g 0 k t 4 Q G D S T D h I Y N J M + E R g 4 9 o c c I O w + l Q D Y 8 Z T O I J T x h M + g l P G U w S C r s M p n M 2 P G P w G R v K O c N J S + E F g 0 l O 4 S W D S V H h F Y N J V O E 1 g 9 l j c H j D c N J W e M t g k l d 4 x 2 B 6 I A j v G W w 0 p n 7 7 L q v 6 T d T v W f r 8 P Y v Y J J j 0 J b Y I J X m J b U K 1 u p 5 7 2 / o3 H 2 M B n u 8 J k B 7 e O 4 a B t / P C 6 0 P g K 1 y O I u F N s n E 8 Q A h b 4 A n 9 e x K s M M e F p z 7 m y W J 0 p D 6 W g c c c K 0 7 9 O + D q l 6

U D 6 C y 7 O l G P f y 6 B x 4 W M g t G v l A f 0 P p j m e l H K y i c I T Y + h s 2 N T T 3 I q s v 8 A A b g 2 J l m i 1 2 B B B 4 9 1 s 4 0 U Q y B + U W x a x 3 7 e e w H M K u / y u l U w M x 7 7 l X X 7 g K 7 / X d m d f b b a Y 6 k I 9 i H P 5 0 m e z 7 j m b 7 h

r q 3 9 5 5 P v f f 8 H P / z R j z / 9 y f J P f / b z X / z y s 8 9 / d S 2 y c R HA V Z D F W X H b 9 w X E U Q p X M p I x 3 O Y F + E k / h p v + w 5 b i b 0 o o R J S l l / I p h z e J H 6 b R M A p 8 i d D b X u L L U X 8 4 7 W W D T M 7 e f b a y t r q m / 3 j z F + v V x c p S 9 a f 7 7 v O 1 / / Y G W T B O I J V B 7 A v x z f p a L t 9 M / U J G Q Q y z 5 d 5 Y Q O 4 H D 3 4 I 3 + B l 6 i c g 3 k z 1 s G f e c 0 Q G 3 j A r 8 G 8 q P Y 3 y H l M / E e I p 6 a O l G q Z o c g p c x C m P 4 k U / a Q x B D v / 0 Z h q l + V h C G p g R D M e x J z N P L Y 4 3 i A o I Z P y E F 3 5 Q R D g J L x j 5 h R 9 I X M L l 5 e f e s V 8 8 e A J t c P G E l w 2 9 w M / N t Z p G A U M o i i g N l c N B V E b C m g 2 j c F w A j j K F S Z A l i Z 8 O p j 0 E Y x j K 2 XT a g 8 T 7 s o P X v 5 v N l u e M A l x d K K z Z l m 4 p w 6 Z d E Y W j 2 t u 5 a r R Z y S y 3 N p d Z 3 m

I o I P Z t L W Z P G s H T / j P I f C U 3 6 M m 5 3 K z U 6 b m w 2 v 8 C e 0 8 A 1 n L 1 + + 9 M s s G n h j o c 6 n a O j l m R A R p o l q H f L Y x / C p b r B 4 e O o 8 z D G a W i a p G N O 9 s v m / Z 1 k 5 2 q o d b X 2 n I 5 x 0 G o I + i Y 2 t q O a j 8 X p I q B b L W 1 8 v X y 5 U C g 7 P j 8 M M E 8 A o a Z k o c m Z 4 t d G 3 z p S 5 m p v q h n W 1 0 e L Ka t 7 e D y d R + 1 o o e t P r 0 u m 1 8 Z 2 9 5 p Y 1 x 5 O r m j o T

H t u U S l 0 u c y b e d y T W 5 8 F g G I + L m w W y 5 t p O 5 J h c e z m B E 3 T y e L b f wf A a j 7 t Y T G o z E W 8 9 o M D p v P a X B i L 3 1 n A a j + P a T G o z u F 5 / V G B F F F N Q F S 7 J B U b L B A j b Z J H y T g i r Z I n i L w d s E b z N 4 h + A d B u 8 S v M v g P Y L 3 G L x P 8 D 6 D D w g + 4 A M / J P y Q m R 8 R f M T g D s E d B h 8 T f M z g E 4 J P G H x K 8 C k f S p f w L j M / I / i M w e c E n z P 4 g u A L B l 8 S f M n g K 4 K v G H x N 8 D U f 4 Q 3 h N 8 z 8 l u B b B t 8 R f M f g e 4 L v T d 3 c e t K 6 y g M j P S b U D a Z w L T / G b X J u y + W 2 O L f t c t u c 2 3 G 5 H c7 t u t w u 5 / Z c b o 9 H m 1 Y n I / d 5 x w O X O + D c o c s d c u 7 I 5 Y 4 4 1 3 G 5 D u e O X e 6 Y c y c u d + J M 4 t Q l T 3 n H r s t 1 O X f m c m e c O 3 e 5 c 8 5 d u N y F M 5 h L l 7 z k H a 9 c 7 o p z 1 y 5 3 z b k b

x l 5 b 8 G q T k M 5 7 x V + P j J 0 a B + DA + d B J N x k M I k j 3 G I w H Z P h N o N J I e E O g 0 k k 4 S 6 D S S f h H o N J K u E + g 0 k t 4 Q G D S T D h I Y N J M + E R g 4 9 o c c I O w + l Q D Y 8 Z T O I J T x h M + g l P G U w S C r s M p n M 2 P G P w G R v K O c N J S + E F g 0 l O 4 S W D S V H h F Y N J V O E 1 g 9 l j c H j D c N J W e M t g k l d 4 x 2 B 6 I A j v G W w 0 p n 7 7 L q v 6 T d T v W f r 8 P Y v Y J J j 0 J b Y I J X m J b U K 1 u p 5 7 2 / o3 H 2 M B n u 8 J k B 7 e O 4 a B t / P C 6 0 P g K 1 y O I u F N s n E 8 Q A h b 4 A n 9 e x K s M M e F p z 7 m y W J 0 p D 6 W g c c c K 0 7 9 O + D q l 6

U D 6 C y 7 O l G P f y 6 B x 4 W M g t G v l A f 0 P p j m e l H K y i c I T Y + h s 2 N T T 3 I q s v 8 A A b g 2 J l m i 1 2 B B B 4 9 1 s 4 0 U Q y B + U W x a x 3 7 e e w H M K u / y u l U w M x 7 7 l X X 7 g K 7 / X d m d f b b a Y 6 k I 9 i H P 5 0 m e z 7 j m b 7 h

Figure 2: MEGA -model architecture. (a) shows the overall architecture. (b) illustrates the gated attention sub-layer equipped with EMA, while (c) displays the details of a single-head attention unit.

Figure 3: Illustration of the MEGA-chunk model with two chunks of length 3. in (11) is individually applied to each chunk, yielding linear complexity O(kc 2 ) = O(nc) w.r.t n. However, this method suffers from the critical limitation of losing contextual information from other chunks. Fortunately, the EMA sub-layer in MEGA mitigates this problem by capturing local contextual information near each token, whose outputs are used as the inputs to the attention sub-layer. As a result, the effective context being exploited by chunk-wise attention can go beyond the chunk boundary. Figure 3 illustrates the largest dependency length captured by one MEGA-chunk block.

Figure 4: Ablations on EMA dimension and chunk size.

. ↑) WMT16 (BLEU ↑) WT103 (PPL. ↓) ImageNet (Acc. ↑) SC (Acc. ↑)

(Long Range Arena) Accuracy on the full suite of long range arena (LRA) tasks, together with training speed and peak memory consumption comparison on the Text task with input length of 4K. ‡ indicates results replicated by us. ListOps Text Retrieval Image Pathfinder Path-X Avg. Speed Mem.We begin our experiments with an evaluation on the Long Range Arena (LRA) benchmark recently introduced byTay et al. (2021), which is designed for evaluating sequence models under the longcontext scenario. This benchmark has six tasks: ListOps(Nangia & Bowman, 2018), byte-level text classification (Text;Maas et al. (2011)), byte-level document retrieval(Retrieval; Radev et al. (

ACKNOWLEDGEMENT

We thank the anonymous reviewers for their comments. This material is based on research sponsored by Air Force Research Laboratory (AFRL) under agreement number FA8750-19-1-1000. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation therein.

annex

Published as a conference paper at ICLR 2023 

4.5. IMAGE CLASSIFICATION

To evaluate MEGA on a large-scale image classification task, we conduct experiments on the Imagenet-1k (Deng et al., 2009) dataset, which consists of 1.28M training images and 50K validation images from 1000 classes. Top-1 accuracy on the validation set is reported in Table 6 to assess various models. MEGA obtains about 0.5% accuracy improvement over DeiT-B (Touvron et al., 2021) . We mostly follow DeiT's approach of applying several data augmentation and regularization methods that facilitate the training process, including Cutmix (Yun et al., 2019) , Mixup (Zhang et al., 2017) , stochastic depth (Huang et al., 2016) , repeated augmentation (Hoffer et al., 2020 ), Rand-Augment (Cubuk et al., 2020) , and random erasing (Zhong et al., 2020) . These methods were highly tuned towards optimizing the performance of DeiT, which might be sub-optimal for MEGA. Exploring the optimal data augmentation and regularization methods for MEGA is an interesting direction for future work.

5. RELATED WORK

To incorporate stronger inductive bias into the attention mechanism, one research direction focuses on developing advanced positional encoding methods, including absolute and relative positional embeddings (Vaswani et al., 2017; Huang et al., 2020; Ke et al., 2020) , and relative positional biases (Su et al., 2021; Press et al., 2021) . Another line of research combines the attention mechanism with other neural architectures with intrinsic strong inductive bias, such as convolutional (Gehring et al., 2017; Dai et al., 2021) and recurrence (Dai et al., 2019; Rae et al., 2020; Lei, 2021) . Recently, many advanced variants of Transformer models ('xformers') (Tay et al., 2020; 2021) have emerged to improve the time and memory efficiency. Popular techniques include sparse attention patterns (Parmar et al., 2018; Beltagy et al., 2020; Kitaev et al., 2020) , low-rank approximations of the attention matrix (Wang et al., 2020; Ma et al., 2021) , and approximations through kernelization (Choromanski et al., 2020; Peng et al., 2021) . Although these models demonstrate better asymptotic complexity for long sequences, their efficiency gains are less prominent for moderate length sequences and their performance lags behind Transformers with regular attention. In addition, as EMA and more general state space models such as S4 can be regarded as a convolution transform with kernel size equal to the sequence length, MEGA is also relevant with CNNs with continuous kernels (Romero et al., 2022) .

6. CONCLUSION

We have introduced MEGA, a simple, efficient and effective neural architecture used as a drop-in replacement for regular multi-head attention. By leveraging the classic exponential moving average (EMA) approach, MEGA is capable of incorporating stronger inductive biases into the attention mechanism. Moreover, the EMA approach enables the design of MEGA-chunk, an efficient variant of MEGA with linear complexity. On five sequence modeling tasks across various data types, MEGA achieves impressive improvements over a variety of strong baselines. This leads to a potential direction of future work to apply MEGA for multi-modality modeling.-2 Besides performance improvements, we also investigate the stability of the two attention functions.We conduct experiments on the LRA Pathfinder task with MEGA models with the two functions.Figure 5 presents the accuracy on the validation set across training epochs. We observe that Laplace is much more stable than ReLU 2 .

D EXPERIMENTAL DETAILS D.1 DETAILS OF MEGA IMPLEMENTATION

Uni-directional vs. Bi-directional EMA. In all the experiments, bi-directional EMA is applied when MEGA is used in the encoder mode while uni-directional one is used in the decoder mode with auto-regressive property.Relative Positional Bias. We only used ROPE (Su et al., 2021) for the language modeling tasks because we conducted experiments of length extrapolation to longer attention chunks at inference time D.4. For other tasks, we simply use very simple learnable relative biases, i.e. one learnable parameter for each relative positional distance (e.g. -3, +4).

D.2 LONG RANGE ARENA (LRA)

For all tasks, we closely follow Tay et al. (2020) for details such as data preprocessing, data split, etc. The hyper-parameters of MEGA models on these tasks are listed in Table 8 . . In Table 5 , we report test results that use longer chunk sizes (attention lengths) of 2048 and 4096 for WikiText-103 and enwik8 respectively. MEGA can naturally extrapolate at inference time to sequences longer than those seen during training due to the recurrent design of the EMA layer. That design enables the inputs of each chunk to access the historic context through EMA as illustrated in Figure 3 . On the other hand, due to the use of rotary positional embeddings, attention can be performed on longer chunk sizes at test time than those seen during training. We hope these two types of length extrapolation are clear to readers. We provide the ablation studies on these two types of length extrapolation below, i.e. extrapolation to longer context by increasing input sequence lengths and extrapolation to longer attention lengths by increasing the chunk size.Ablations on context lengths First, we fix the chunk size to be 2048 and vary K within [100, 75, 50, 25, 15, 10, 5] corresponding to maximum context tokens of [2.5K, 3.3K, 4.9K, 9.8K, 16K, 25K, 49K]. We plot the test PPL as we increase the context length in the left of Figure 6 . Although at training time, the maximum context length the model has seen is 6144, MEGA can extrapolate to longer context lengths. The plot shows that PPL decreases as the context length is increased and the improvements saturate when the context length is longer than 25K. This is consistent with the observations in Press et al. (2021) .Ablations on attention chunk sizes Next, we fix the context length to be 25K and increase the chunk size from 512 to 3072. As shown in the right side of Figure 6 , MEGA consistently improves as we increase the attention length although it only uses an attention length of 1024 during training. This contradicts with the findings in Alibi (Press et al., 2021) , which finds that rotary embeddings don't generalize to longer lengths and result in higher PPL. The WMT 2016 English-German dataset contains 4.5M parallel sentence pairs for training. We following the standard setting (Ott et al., 2018) , using Newstest2013 as the validation set and Newstest2014 as the test set. The dataset is pre-processed following (Ma, 2020) , using the scripts from FairSeq package (Ott et al., 2019) . 4 We share the source and target vocabularies within the language pair, with 32K byte pair encoding (BPE) types (Sennrich et al., 2016) . We report the average of both tokenized and SacreBLEU 5 (Post, 2018) . The hyper-parameters of Transformer and MEGA models are listed in Table 10 . 

