DO WE REALLY NEED COMPLICATED MODEL ARCHI-TECTURES FOR TEMPORAL NETWORKS?

Abstract

Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract spatial-temporal information for temporal graph learning. Interestingly, we found that although both RNN and SAM could lead to a good performance, in practice neither of them is always necessary. In this paper, we propose GraphMixer, a conceptually and technically simple architecture that consists of three components: 1 a link-encoder that is only based on multi-layer perceptrons (MLP) to summarize the information from temporal links, 2 a node-encoder that is only based on neighbor mean-pooling to summarize node information, and 3 an MLP-based link classifier that performs link prediction based on the outputs of the encoders. Despite its simplicity, GraphMixer attains an outstanding performance on temporal link prediction benchmarks with faster convergence and better generalization performance. These results motivate us to rethink the importance of simpler model architecture. [Code].

1. INTRODUCTION

In recent years, temporal graph learning has been recognized as an important machine learning problem and has become the cornerstone behind a wealth of high-impact applications Yu et al. (2018) ; Bui et al. (2021) ; Kazemi et al. (2020) ; Zhou et al. (2020) ; Cong et al. (2021b) . Temporal link prediction is one of the classic downstream tasks which focuses on predicting the future interactions among nodes. For example, in an ads ranking system, the user-ad clicks can be modeled as a temporal bipartite graph whose nodes represent users and ads, and links are associated with timestamps indicating when users click ads. Link prediction between them can be used to predict whether a user will click an ad. Designing graph learning models that can capture node evolutionary patterns and accurately predict future links is a crucial direction for many real-world recommender systems. In temporal graph learning, recurrent neural network (RNN) and self-attention mechanism (SAM) have become the de facto standard for temporal graph learning Kumar et al. (2019) ; Sankar et al. (2020) ; Xu et al. (2020) ; Rossi et al. (2020) ; Wang et al. (2020) , and the majority of the existing works focus on designing neural architectures with one of them and additional components to learn representations from raw data. Although powerful, these methods are conceptually and technically complicated with advanced model architectures. It is non-trivial to understand which parts of the model design truly contribute to its success, and whether these components are indispensable. Thus, in this paper, we aim at answering the following two questions: Q1: Are RNN and SAM always indispensable for temporal graph learning? To answer this question, we propose GraphMixer, a simple architecture based entirely on the multi-layer perceptrons (MLPs) and neighbor mean-pooling, which does not utilize any RNN or SAM in its model architecture (Section 3). Despite its simplicity, GraphMixer could obtain outstanding results when comparing it 1 Time t 5 t 4 t 3 x link i,1 x link i,3 x link j,3 Node features < l a t e x i t s h a 1 _ b a s e 6 4 = " x R F v c C A H e 3 s d Z K 6 o 4 9 M L G 0 z y J b M = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a U D b b S b t 0 s w m 7 m 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o b u a 3 x q g 0 j + W T m S T o R 3 Q g e c g Z N V Z 6 H P e 8 X r n i V t 0 5 y C r x c l K B H P V e + a v b j 1 k a o T R M U K 0 7 n p s Y P 6 P K c C Z w W u q m G h P K R n S A H U s l j V D 7 2 f z U K T m z S p + E s b I l D Z m r v y c y G m k 9 i Q L b G V E z 1 M v e T P z P 6 6 Q m v P E z L p P U o G S L R W E q i I n J 7 G / S 5 w q Z E R N L K F P c 3 k r Y k C r K j E 2 n Z E P w l l 9 e J c 2 L q n d V v X y 4 r N R u 8 z i K c A K n c A 4 e X E M N 7 q E O D W A w g G d 4 h T d H O C / O u / O x a C 0 4 + c w x / I H z + Q M L k o 2 n < / l a t e x i t > v 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " c 5 s R + 1 E A F 9 v c Q u 4 v G y A + Q 6 W A K E I = " > A A A C A X i c b V B N S 8 N A E N 3 4 W e t X 1 Y v g J V g E T y W R o h 6 L X j x W s B / Q x r D Z T t q l m 0 3 Y n U h L q B f / i h c P i n j 1 X 3 j z 3 7 j 9 O G j r g 4 H H e z P M z A s S w T U 6 z r e 1 t L y y u r a e 2 8 h v b m 3 v 7 B b 2 9 u s 6 T h W D G o t F r J o B 1 S C 4 h B p y F N B M F N A o E N A I + t d j v / E A S v N Y 3 u E w A S + i X c l D z i g a y S 8 c t i O K v S D M B i P f v W 8 j D D C T c Q d G f q H o l J w J 7 E X i z k i R z F D 1 C 1 / t T s z S C C Q y Q b V u u U 6 C X k Y V c i Z g l G + n G h L K + r Q L L U M l j U B 7 2 e S D k X 1 i l I 4 d x s q U R H u i / p 7 I a K T 1 M A p M 5 / h e P e + N x f + 8 V o r h p Z d x m a Q I k k 0 X h a m w M b b H c d g d r o C h G B p C m e L m V p v 1 q K I M T W h 5 E 4 I 7 / / I i q Z + V 3 P N S + b Z c r F z N 4 s i R I 3 J M T o l L L k i F 3 J A q q R F G H s k z e S V v 1 p P 1 Y r 1 b H 9 P W J W s 2 c 0 D + w P r 8 A X f / l 4 8 = < / l a t e x i t > x node 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 w 4 U 2 n B C p z F X C f j J r 7 f A s 5 K m E 0 M = " > A A A C Y H i c b V F N T x s x E P U u p Q 0 B S q A 3 e r G I k K g q R b s R K h x R u f R U U a k B p G x Y e Z 1 Z M P h j Z c + m R G b / Z G 8 9 9 N J f U i d E K g 2 M Z O n p v T f j 8 X N R S e E w S X 5 F 8 c q r 1 d d v W m v t 9 Y 3 N t 1 u d 7 Z 1 z Z 2 r L Y c C N N P a y Y A 6 k 0 D B A g R I u K w t M F R I u i r v T m X 4 x A e u E 0 d 9 x W s F I s W s t S s E Z B i r v / M g U w 5 u i 9 P d N 3 r / K E O 7 R a z O G h n 6 k W W k Z 9 2 n j H + Y m z q T / 2 h x M 8 v 6 H h 4 Z m r l a Z N l I o g S 7 3 k / w 2 E 3 r Z F 2 z / x t 8 + H Z 9 3 u k k v m R d 9 D t I F 6 J J F n e W d n 9 n Y 8 F q B R i 6 Z c 8 M 0 q X D k m U X B J T T t r H Z Q M X 7 H r m E Y o G Y K 3 M j P A 2 r o f m D G t D Q 2 H I 1 0 z j 7 t 8 E w 5 N 1 V F c M 7 2 d c v a j H x J G 9 Z Y H o + 8 0 F W N o P n j R W U t K R o 6 S 5 u O h Q W O c h o A 4 1 a E X S m / Y S F W D H / S D i G k y 0 9 + D s 7 7 v f R T 7 / D b Y f f k 8 y K O F n l P 9 s g B S c k R O S F f y B k Z E E 5 + R y v R R r Q Z / Y l b 8 V a 8 / W i N o 0 X P O / J f x b t / A Y o w u R o = < / l a t e x i t > x node 2 + 1 |N (v2)| X v Linear < l a t e x i t s h a 1 _ b a s e 6 4 = " Q 8 d 2 q G k W R h h P v n V v q G j D 5 4 C k P / s = " > A A A B 7 3 i c b V D J S g N B E K 2 J W 4 x b 1 K O X x i B 4 k D A j c T k G v X i M Y B Z I h q G n 0 5 M 0 6 V n s r h H C k J / w 4 k E R r / 6 O N / / G T j I H T X x Q 8 H i v i q p 6 f i K F R t v + t g o r q 2 v r G 8 X N 0 t b 2 z u 5 e e f + g p e N U M d 5 k s Y x V x 6 e a S x H x J g q U v J M o T k N f 8 r Y / u p 3 6 7 S e u t I i j B x w n 3 A 3 p I B K B Y B S N 1 E H P O S P o X X j l i l 2 1 Z y D L x M l J B X I 0 v P J X r x + z N O Q R M k m 1 7 j p 2 g m 5 G F Q o m + a T U S z V P K B v R A e 8 a G t G Q a z e b 3 T s h J 0 b p k y B W p i I k M / X 3 R E Z D r c e h b z p D i k O 9 6 E 3 F / 7 x u i s G 1 m 4 k o S Z F H b L 4 o S C X B m E y f J 3 2 h O E M 5 N o Q y J c y t h A 2 p o g x N R C U T g r P 4 8 j J p n V e d y 2 r t v l a p 3 + R x F O E I j u E U H L i C O t x B A 5 r A Q M I z v M K b 9 W i 9 W O / W x 7 y 1 Y O U z h / A H 1 u c P y / u P K w = = < / l a t e x i t > t 1 , t 5 < l a t e x i t s h a 1 _ b a s e 6 4 = " / f B k R L v L 0 G W 7 0 f e 4 G 8 + O x Y C 3 6 F I = " > A A A B 7 3 i c b V D L S g N B E O y N r x h f U Y 9 e B o P g Q c K u B v U Y 9 O I x g n l A s i y z k 9 l k y O z D m V 4 h L P k J L x 4 U 8 e r v e P N v n C R 7 0 M S C h q K q m + 4 u P 5 F C o 2 1 / W 4 W V 1 b X 1 j e J m a W t 7 Z 3 e v v H / Q 0 n G q G G + y W M a q 4 1 P N p Y h 4 E w V K 3 k k U p 6 E v e d s f 3 U 7 9 9 h N X W s T R A 4 4 T 7 o Z 0 E I l A M I p G 6 q B 3 c U b Q q 3 n l i l 2 1 Z y D L x M l J B X I 0 v P J X r x + z N O Q R M k m 1 7 j p 2 g m 5 G F Q o m + a T U S z V P K B v R A e 8 a G t G Q a z e b 3 T s h J 0 b p k y B W p i I k M / X 3 R E Z D r c e h b z p D i k O 9 6 E 3 F / 7 x u i s G 1 m 4 k o S Z F H b L 4 o S C X B m E y f J 3 2 h O E M 5 N o Q y J c y t h A 2 p o g x N R C U T g r P 4 8 j J p n V e d y 2 r t v l a p 3 + R x F O E I j u E U H L i C O t x B A 5 r A Q M I z v M K b 9 W i 9 W O / W x 7 y 1 Y O U z h / A H 1 u c P z Y m P L A = = < / l a t e x i t > t 3 , t 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " V f Q l b j z 1 4 o E v J + w N q J l C E m c z K x A = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l K U Y 9 F L x 4 r 2 l p o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H C o O t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U N n G q G W + x W M a 6 E 1 D D p V C 8 h Q I l 7 y S a 0 y i Q / D E Y 3 8 z 8 x y e u j Y j V A 0 4 S 7 k d 0 q E Q o G E U r 3 W O / 1 i 9 X 3 K o 7 B 1 k l X k 4 q k K P Z L 3 / 1 B j F L I 6 6 Q S W p M 1 3 M T 9 D O q U T D J p 6 V e a n h C 2 Z g O e d d S R S N u / G x + 6 p S c W W V A w l j b U k j m 6 u + J j E b G T K L A d k Y U R 2 b Z m 4 n / e d 0 U w y s / E y p J k S u 2 W B S m k m B M Z n + T g d C c o Z x Y Q p k W 9 l b C R l R T h j a d k g 3 B W 3 5 5 l b R r V e + i W r + r V x r X e R x F O I F T O A c P L q E B t 9 C E F j A Y w j O 8 w p s j n R f n 3 f l Y t B a c f O Y Y / s D 5 / A E K C o 2 m < / l a t e x i t > t 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " x R F v c C A H e 3 s d Z K 6 o 4 9 M L G 0 z y J b M = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a U D b b S b t 0 s w m 7 m 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o b u a 3 x q g 0 j + W T m S T o R 3 Q g e c g Z N V Z 6 H P e 8 X r n i V t 0 5 y C r x c l K B H P V e + a v b j 1 k a o T R M U K 0 7 n p s Y P 6 P K c C Z w W u q m G h P K R n S A H U s l j V D 7 2 f z U K T m z S p + E s b I l D Z m r v y c y G m k 9 i Q L b G V E z 1 M v e T P z P 6 6 Q m v P E z L p P U o G S L R W E q i I n J 7 G / S 5 w q Z E R N L K F P c 3 k r Y k C r K j E 2 n Z E P w l l 9 e J c 2 L q n d V v X y 4 r N R u 8 z i K c A K n c A 4 e X E M N 7 q E O D W A w g G d 4 h T d H O C / O u / O x a C 0 4 + c w x / I H z + Q M L k o 2 n < / l a t e x i t > v 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " p d 6 C G + f g X j P a 9 T r I / + S v i f D 5 V x Y = " > A A A B 6 n i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H Y J U Y 9 E L x 4 x y i O B D Z k d e m H C 7 O x m Z p a E E D 7 B i w e N 8 e o X e f N v H G A P C l b S S a W q O 9 1 d Q S K 4 N q 7 7 7 e Q 2 N r e 2 d / K 7 h b 3 9 g 8 O j 4 v F J U 8 e p Y t h g s Y h V O 6 A a B Z f Y M N w I b C c K a R Q I b A W j u 7 n f G q P S P J Z P Z p K g H 9 G B 5 C F n 1 F j p c d y r 9 I o l t + w u Q N a J l 5 E S Z K j 3 i l / d f s z S C K V h g m r d 8 d z E + F O q D G c C Z 4 V u q j G h b E Q H 2 L F U 0 g i 1 P 1 2 c O i M X V u m T M F a 2 p C E L 9 f f E l E Z a T 6 L A d k b U D P W q N x f / 8 z q p C W / 8 K Z d J a l C y 5 a I w F c T E Z P 4 3 6 X O F z I i J J Z Q p b m 8 l b E g V Z c a m U 7 A h e K s v r 5 N m p e x d l a s P 1 V L t N o s j D 2 d w D p f g w T X U 4 B 7 q 0 A A G A 3 i G V 3 h z h P P i v D s f y 9 a c k 8 2 c w h 8 4 n z 8 N F o 2 o < / l a t e x i t > v 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " b 2 t E e K l W W v / D N O 1 w X N 9 n z f 0 5 O 4 M = " > A A A B 6 n i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H a V q E e i F 4 8 Y 5 Z H A h s w O v T B h d n Y z M 0 t C C J / g x Y P G e P W L v P k 3 D r A H B S v p p F L V n e 6 u I B F c G 9 f 9 d n J r 6 x u b W / n t w s 7 u 3 v 5 B 8 f C o o e N U M a y z W M S q F V C N g k u s G 2 4 E t h K F N A o E N o P h 3 c x v j l B p H s s n M 0 7 Q j 2 h f 8 p A z a q z 0 O O p e d o s l t + z O Q V a J l 5 E S Z K h 1 i 1 + d X s z S C K V h g m r d 9 t z E + B O q D G c C p 4 V O q j G h b E j 7 2 L Z U 0 g i 1 P 5 m f O i V n V u m R M F a 2 p C F z 9 f f E h E Z a j 6 P A d k b U D P S y N x P / 8 9 q p C W / 8 C Z d J a l C y x a I w F c T E Z P Y 3 6 X G F z I i x J Z Q p b m 8 l b E A V Z c a m U 7 A h e M s v r 5 L G R d m 7 K l c e K q X q b R Z H H k 7 g F M 7 B g 2 u o w j 3 U o A 4 M + v A M r / D m C O f F e X c + F q 0 5 J 5 s 5 h j 9 w P n 8 A D p q N q Q = = < / l a t e x i t > v 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " h O 3 2 0 i B a 8 s E m b X N N i l Q x E q L d r 0 Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k U 9 V j 0 4 r G i / Y A 2 l M 1 2 0 y 7 d b M L u p F B C f 4 I X D 4 p 4 9 R d 5 8 9 + 4 b X P Q 1 g c D j / d m m J k X J F I Y d N 1 v Z 2 1 9 Y 3 N r u 7 B T 3 N 3 b P z g s H R 0 3 T Z x q x h s s l r F u B 9 R w K R R v o E D J 2 4 n m N A o k b w W j u 5 n f G n N t R K y e c J J w P 6 I D J U L B K F r p c d y r 9 k p l t + L O Q V a J l 5 M y 5 K j 3 S l / d f s z S i C t k k h r T 8 d w E / Y x q F E z y a b G b G p 5 Q N q I D 3 r F U 0 Y g b P 5 u f O i X n V u m T M N a 2 F J K 5 + n s i o 5 E x k y i w n R H F o V n 2 Z u J / X i f F 8 M b P h E p S 5 I o t F o W p J B i T 2 d + k L z R n K C e W U K a F v Z W w I d W U o U 2 n a E P w l l 9 e J c 3 L i n d V q T 5 U y 7 X b P I 4 C n M I Z X I A H 1 1 C D e 6 h D A x g M 4 B l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E Q H o 2 q < / l a t e x i t > v 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " V x 1 7 v p U u O e 1 M y m v 9 g b 2 X 5 B r X I d I = " > A A A B 6 n i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H Y N P o 5 E L x 4 x y i O B D Z k d e m H C 7 O x m Z p a E E D 7 B i w e N 8 e o X e f N v H G A P C l b S S a W q O 9 1 d Q S K 4 N q 7 7 7 e T W 1 j c 2 t / L b h Z 3 d v f 2 D 4 u F R Q 8 e p Y l h n s Y h V K 6 A a B Z d Y N 9 w I b C U K a R Q I b A b D u 5 n f H K H S P J Z P Z p y g H 9 G + 5 C F n 1 F j p c d S 9 7 B Z L b t m d g 6 w S L y M l y F D r F r 8 6 v Z i l E U r D B N W 6 7 b m J 8 S d U G c 4 E T g u d V G N C 2 Z D 2 s W 2 p p B F q f z I / d U r O r N I j Y a x s S U P m 6 u + J C Y 2 0 H k e B 7 Y y o G e h l b y b + 5 7 V T E 9 7 4 E y 6 T 1 K B k i 0 V h K o i J y e x v 0 u M K m R F j S y h T 3 N 5 K 2 I A q y o x N p 2 B D 8 J Z f X i W N i 7 J 3 V a 4 8 V E r V 2 y y O P J z A K Z y D B 9 d Q h X u o Q R 0 Y 9 O E Z X u H N E c 6 L 8 + 5 8 L F p z T j Z z D H / g f P 4 A E a K N q w = = < / l a t e x i t > v 5 Temporal graph (now) (past) < l a t e x i t s h a 1 _ b a s e 6 4 = " Q 8 d 2 q G k W R h h P v n V v q G j D 5 4 C k P / s = " > A A A B 7 3 i c b V D J S g N B E K 2 J W 4 x b 1 K O X x i B 4 k D A j c T k G v X i M Y B Z I h q G n 0 5 M 0 6 V n s r h H C k J / w 4 k E R r / 6 O N / / G T j I H T X x Q 8 H i v i q p 6 f i K F R t v + t g o r q 2 v r G 8 X N 0 t b 2 z u 5 e e f + g p e N U M d 5 k s Y x V x 6 e a S x H x J g q U v J M o T k N f 8 r Y / u p 3 6 7 S e u t I i j B x w n 3 A 3 p I B K B Y B S N 1 E H P O S P o X X j l i l 2 1 Z y D L x M l J B X I 0 v P J X r x + z N O Q R M k m 1 7 j p 2 g m 5 G F Q o m + a T U S z V P K B v R A e 8 a G t G Q a z e b 3 T s h J 0 b p k y B W p i I k M / X 3 R E Z D r c e h b z p D i k O 9 6 E 3 F / 7 x u i s G 1 m 4 k o S Z F H b L 4 o S C X B m E y f J 3 2 h O E M 5 N o Q y J c y t h A 2 p o g x N R C U T g r P 4 8 j J p n V e d y 2 r t v l a p 3 + R x F O E I j u E U H L i C O t x B A 5 r A Q M I z v M K b 9 W i 9 W O / W x 7 y 1 Y O U z h / A H 1 u c P y / u P K w = = < / l a t e x i t > t 1 , t 5 < l a t e x i t s h a 1 _ b a s e 6 4 = " x R F v c C A H e 3 s d Z K 6 o 4 9 M L G 0 z y J b M = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a U D b b S b t 0 s w m 7 m 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o b u a 3 x q g 0 j + W T m S T o R 3 Q g e c g Z N V Z 6 H P e 8 X r n i V t 0 5 y C r x c l K B H P V e + a v b j 1 k a o T R M U K 0 7 n p s Y P 6 P K c C Z w W u q m G h P K R n S A H U s l j V D 7 2 f z U K T m z S p + E s b I l D Z m r v y c y G m k 9 i Q L b G V E z 1 M v e T P z P 6 6 Q m v P E z L p P U o G S L R W E q i I n J 7 G / S 5 w q Z E R N L K F P c 3 k r Y k C r K j E 2 n Z E P w l l 9 e J c 2 L q n d V v X y 4 r N R u 8 z i K c A K n c A 4 e X E M N 7 q E O D W A w g G d 4 h T d H O C / O u / O x a C 0 4 + c w x / I H z + Q M L k o 2 n < / l a t e x i t > v 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " p d 6 C G + f g X j P a 9 T r I / + S v i f D 5 V x Y = " > A A A B 6 n i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H Y J U Y 9 E L x 4 x y i O B D Z k d e m H C 7 O x m Z p a E E D 7 B i w e N 8 e o X e f N v H G A P C l b S S a W q O 9 1 d Q S K 4 N q 7 7 7 e Q 2 N r e 2 d / K 7 h b 3 9 g 8 O j 4 v F J U 8 e p Y t h g s Y h V O 6 A a B Z f Y M N w I b C c K a R Q I b A W j u 7 n f G q P S P J Z P Z p K g H 9 G B 5 C F n 1 F j p c d y r 9 I o l t + w u Q N a J l 5 E S Z K j 3 i l / d f s z S C K V h g m r d 8 d z E + F O q D G c C Z 4 V u q j G h b E Q H 2 L F U 0 g i 1 P 1 2 c O i M X V u m T M F a 2 p C E L 9 f f E l E Z a T 6 L A d k b U D P W q N x f / 8 z q p C W / 8 K Z d J a l C y 5 a I w F c T E Z P 4 3 6 X O F z I i J J Z Q p b m 8 l b E g V Z c a m U 7 A h e K s v r 5 N m p e x d l a s P 1 V L t N o s j D 2 d w D p f g w T X U 4 B 7 q 0 A A G A 3 i G V 3 h z h P P i v D s f y 9 a c k 8 2 c w h 8 4 n z 8 N F o 2 o < / l a t e x i t > v 2 Link features t 1 t 2 t 3 t 4 t 5 < l a t e x i t s h a 1 _ b a s e 6 4 = " z w v 9 / b O l m o J k t B d H Y C 6 V Z V W y U X g = " > A A A C L X i c h V D J S g N B E O 2 J W 4 z b q E c v g 0 F I I I S Z E J d j U A 8 e I 5 g F k j j 0 d H q S J j 0 L 3 T W S M I w f 5 M V f E c F D R L z 6 G 3 a W g y a C D x o e r 1 5 V V z 0 n 5 E y C a Y 6 1 1 M r q 2 v p G e j O z t b 2 z u 6 f v H 9 R l E A l C a y T g g W g 6 W F L O f F o D B p w 2 Q 0 G x 5 3 D a c A Z X k 3 r j g Q r J A v 8 O R i H t e L j n M 5 c R D E q y 9 e u 2 h 6 H v u P E w s W O r U E r u 2 0 C H E K t 5 g y Q H t p U v P P 5 j O c 3 b e t Y s m l M Y y 8 S a k y y a o 2 r r r + 1 u Q C K P + k A 4 l r J l m S F 0 Y i y A E U 6 T T D u S N M R k g H u 0 p a i P P S o 7 8 f T a x D h R S t d w A 6 G e D 8 Z U / d k R Y 0 / K k e c o 5 2 R x u V i b i H / V W h G 4 F 5 2 Y + W E E 1 C e z j 9 y I G x A Y k + i M L h O U A B 8 p g o l g a l e D 9 L H A B F T A G R W C t X j y M q m X i t Z Z s X x b z l Y u 5 3 G k 0 R E 6 R j l k o X N U Q T e o i m q I o C f 0 g s b o X X v W 3 r Q P 7 X N m T W n z n k P 0 C 9 r X N y X k q U Y = < / l a t e x i t > x link 1,2 (t 1 ), x link 1,2 (t 5 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " N 1 b Q F 1 e q D f m D x C p l z s J m D B d g C j c = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m k q M e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m T j V j D d Z L G P d C a j h U i j e R I G S d x L N a R R I 3 g 7 G t z O / / c S 1 E b F 6 x E n C / Y g O l Q g F o 2 i l B + y 7 / X L F r b p z k F X i 5 a Q C O R r 9 8 l d v E L M 0 4 g q Z p M Z 0 P T d B P 6 M a B Z N 8 W u q l h i e U j e m Q d y 1 V N O L G z + a n T s m Z V Q Y k j L U t h W S u / p 7 I a G T M J A p s Z 0 R x Z J a 9 m f i f 1 0 0 x v P Y z o Z I U u W K L R W E q C c Z k 9 j c Z C M 0 Z y o k l l G l h b y V s R D V l a N M p 2 R C 8 5 Z d X S e u i 6 l 1 W a / e 1 S v 0 m j 6 M I J 3 A K 5 + D B F d T h D h r Q B A Z D e I Z X e H O k 8 + K 8 O x + L 1 o K T z x z D H z i f P w c C j a Q = < / l a t e x i t > t 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " z 8 1 i x 6 Q e H P 3 t 6 9 N C 0 T O p 7 j 1 U n K c = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m T j V j D d Z L G P d C a j h U i j e R I G S d x L N a R R I 3 g 7 G t z O / / c S 1 E b F 6 x E n C / Y g O l Q g F o 2 i l B + z X + u W K W 3 X n I K v E y 0 k F c j T 6 5 a / e I G Z p x B U y S Y 3 p e m 6 C f k Y 1 C i b 5 t N R L D U 8 o G 9 M h 7 1 q q a M S N n 8 1 P n Z I z q w x I G G t b C s l c / T 2 R 0 c i Y S R T Y z o j i y C x 7 M / E / r 5 t i e O 1 n Q i U p c s U W i 8 J U E o z J 7 G 8 y E J o z l B N L K N P C 3 k r Y i G r K 0 K Z T s i F 4 y y + v k t Z F 1 a t V L + 8 v K / W b P I 4 i n M A p n I M H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W g t O P n M M f + B 8 / g A Q G o 2 q < / l a t e x i t > t 6 < l a t e x i t s h a 1 _ b a s e 6 4 = " z 8 1 i x 6 Q e H P 3 t 6 9 N C 0 T O p 7 j 1 U n K c = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 for v 1 ) and each temporal link has its link features (e.g., x link 1,2 (t 1 ), x link 1,2 (t 5 ) link features between v 1 , v 2 at t 1 , t ). For scenarios without node or link features, we use all-zero vectors instead. M P N 6 b Y W Z e k E h h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m T j V j D d Z L G P d C a j h U i j e R I G S d x L N a R R I 3 g 7 G t z O / / c S 1 E b F 6 x E n C / Y g O l Q g F o 2 i l B + z X + u W K W 3 X n I K v E y 0 k F c j T 6 5 a / e I G Z p x B U y S Y 3 p e m 6 C f k Y 1 C i b 5 t N R L D U 8 o G 9 M h 7 1 q q a M S N n 8 1 P n Z I z q w x I G G t b C s l c / T 2 R 0 c i Y S R T Y z o j i y C x 7 M / E / r 5 t i e O 1 n Q i U p c s U W i 8 J U E o z J 7 G 8 y E J o z l B N L K N P C 3 k r Y i G r K 0 K Z T s i F 4 y y + v k t Z F 1 a t V L + 8 v K / W b P I 4 i n M A p n I M H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W g t O P n M M f + B 8 / g A Q G o 2 q < / l a t against baselines that are equipped with the RNNs and SAM. In practice, it achieves state-of-the-art performance in terms of different evaluation metrics (e.g., average precision, AUC, Recall@K, and MRR) on real-world temporal graph datasets, with the even smaller number of model parameters and hyper-parameters, and a conceptually simpler input structure and model architecture (Section 4). Q2: What are the key factors that lead to the success of GraphMixer? We identify three key factors that contribute to the success of GraphMixer: 1 The simplicity of GraphMixer's input data and neural architecture. Different from most deep learning methods that focus on designing conceptually complicated data preparation techniques and technically complicated neural architectures, we choose to simplifying the neural architecture and utilize a conceptually simpler data as input. Both of which could lead to a better model performance and better generalization (Section 4.4). 2 A time-encoding function that encodes any timestamp as an easily distinguishable input vector for GraphMixer. Different from most of the existing methods that propose to learn the time-encoding function from the raw input data, our time-encoding function utilizes conceptually simple features and is fixed during training. Interestingly, we show that our fixed time-encoding function is more preferred than the trainable version (used by most previous studies), and could lead to a smoother optimization landscape, a faster convergence speed, and a better generalization (Section 4.2); 3 A link-encoder that could better distinguish temporal sequences. Different from most existing methods that summarize sequences using SAM, our encoder module is entirely based on MLPs. Interestingly, our encoder can distinguish temporal sequences that cannot be distinguished by SAM, and it could generalize better due to its simpler neural architecture and lower model complexity (Section 4.3). To this end, we summarize our contributions as follows: 1 We propose a conceptually and technically simple architecture GraphMixer; 2 Even without RNN and SAM, GraphMixer not only outperforms all baselines but also enjoys a faster convergence and better generalization ability; 3 Extensive study identifies three factors that contribute to the success of GraphMixer. 4 Our results could motivate future research to rethink the importance of the conceptually and technically simpler method.

2. PRELIMINARY AND EXISTING WORKS

Preliminary. Figure 1 is an illustration on the temporal graph. Our goal is to predict whether two nodes are connected at a specific timestamp t 0 based on all the available temporal graph information happened before that timestamp. For example, to predict whether v 1 , v 2 are connected at t 0 , we only have access to the graph structure, node features, and link features with timestamps from t 1 to t 6 . Related works. Most of the temporal graph learning methods are conceptually and technically complicated with advanced neural architectures. It is non-trivial to fully understand the algorithm details without looking into their implementations. Therefore, we select the four most representative and most closely-related methods to introduce and compare them in more details. • JODIE Kumar et al. ( 2019) is a RNN-based method. Let us denote x i (t) as the embedding of node v i at time t, x link ij (t) as the link feature between v i , v j at time t, and m i as the timestamp that v i latest interact with other node. JODIE pre-processes and updates the representation of each node via RNNs (is it just one RNN or multiple RNNs). More specifically, when an interaction between v i , v j happens at time t, JODIE updates the temporal embedding using RNN by x i (t) = RNN x i (m i ), x j (m j ), x link ij (t), tm i . Then, the dynamic embedding of node v i at time t 0 is computed by h i (t 0 ) = (1 + (t 0m i )w) • x i (m i ). Finally, the prediction on any node pair at time t 0 is computed by MLP([h i (t 0 ) || h j (t 0 )]), where [•||•] is the concatenate operation and MLP(x) is applying 2-layer MLP on x.



Figure1: (Left) Temporal graph with nodes v 1 , . . . , v 5 , per-link timestamps t 1 , . . . , t 6 indicate when two nodes interact. For example, v 1 , v interact at t , t . (Right) Each node has its node features (e.g., x node

