PROTEIN STRUCTURE REPRESENTATION LEARNING THROUGH ORIENTATION-AWARE GRAPH NEURAL NETWORKS

Abstract

By folding to particular 3D structures, proteins play a key role in living beings. To learn meaningful representation from a protein structure for downstream tasks, not only the global backbone topology but the local fine-grained orientational relations between amino acids should also be considered. In this work, we propose the Orientation-Aware Graph Neural Networks (OAGNNs) to better sense the geometric characteristics in protein structure (e.g. inner-residue torsion angles, inter-residue orientations). Extending a single weight from a scalar to a 3D vector, we construct a rich set of geometric-meaningful operations to process both the classical and SO(3) representations of a given structure. To plug our designed perceptron unit into existing Graph Neural Networks, we further introduce an equivariant message passing paradigm, showing superior versatility in maintaining SO(3)-equivariance at the global scale. Experiments have shown that our OAGNNs have a remarkable ability to sense geometric orientational features compared to classical networks. OAGNNs have also achieved state-of-theart performance on various computational biology applications related to protein 3D structures.

1. INTRODUCTION

Built from a sequence of amino-acid residues, a protein performs its biological functions by folding to a particular conformation in 3D space. Therefore, untilizing such 3D structures accurately is the key for downstream analysis. While we have witnessed remarkable progress in protein structure predictions (Rohl et al., 2004; Källberg et al., 2012; Baek et al., 2021; Jumper et al., 2021) , another thread of tasks with protein 3D structures as input starts to draw a great interest, such as function prediction (Hermosilla et al., 2020; Gligorijević et al., 2021) , decoy ranking (Lundström et al., 2001; Kwon et al., 2021; Wang et al., 2021) , protein docking (Duhovny et al., 2002; Shulman-Peleg et al., 2004; Gainza et al., 2020; Sverrisson et al., 2021) , and driver mutation identification (Lefèvre et al., 1997; Antikainen & Martin, 2005; Li et al., 2020; Jankauskaitė et al., 2019) . Most existing works in modeling protein structures directly borrow models designed for other applications, including 3D-CNNs (Ji et al., 2012) in computer vision, Transformers (Vaswani et al., 2017) ) from natural language processing, and GNNs (Kipf & Welling, 2016) in data mining. Though compatible with general objects, these models have overlooked the subtleties in the fine-grained geometries, which are much more essential in protein structures. For instance, given an amino acid in the protein structure, as shown in Figure 1 , the locations of four backbone atoms (carbon, nitrogen, and oxygen) determine a local skeleton, and different residues interact with each other through performing specific orientations between their local frames, either of which have important impacts on the protein structure and its function (Nelson et al., 2008) . Recent attempts in building geometric-aware neural networks mainly focus on baking 3D rigid transformations into network operations, leading to the area of SO(3)-invariant and equivariant networks. One representative work is the Vector Neuron Network (VNN) (Deng et al., 2021) , which achieves SO(3)-equivariance on point clouds by generalizing scalar neurons to 3D vectors. Another work is the GVP-GNN (Jing et al., 2021) that similarly vectorizes hidden neurons in GNN and demonstrates better prediction accuracy on protein design and quality evaluation tasks. However, Figure 1 : Overview (a) Each amino acid has its own rigid backbone with four heavy atoms, and can be represented by both lists of scalar and vector features. (b) Tasks associated with the protein 3D structure. Graph-level tasks consider the whole protein structures, and Node-level tasks operate on specific residues. these two models can only adopt linear combinations of input vectors, which significantly limits their modeling capability. A simple example is that, given two input vector features v 1 and v 2 , the outputs w 1 v 1 + w 2 v 2 through one linear layer is constrained in the 2D plane spanned by v 1 , v 2 even after applying their scalar-product non-linearities. That is, VNN-based models are limited in perceiving orientational features, which have been proven crucial for proteins to perform their functions and interact with other partners (e.g. inner-residue torsion angles, inter-residue orientations) (Nelson et al., 2008; Voet & Voet, 2010; Xu & Berger, 2006; Alford et al., 2017) . To achieve more sensitive geometric orientation awareness, we propose a Directed Weight ( ⃗ W) perceptrons by extending not only the hidden neurons but also the weights from scalars to 3D vectors, naturally saturating the entire network with 3D structure information in the Euclidean space. Directed weights support a set of geometric-meaningful operations on both the vector neurons (vectorlist features) and the classical (scalar-list) latent features, and perform flexible non-linear integration of the hybrid scalar-vector features. As protein structures are naturally attributed proximity graphs, we introduce a new Equivariant Message Passing Paradigm on protein graphs, to connect the ⃗ Wperceptrons with the graph learning models by using rigid backbone transformations for each amino acid, which provides a versatile framework for bringing the biological suitability and flexibility of the GNN architecture. To summarize, our key contributions include: • We propose a new network unit based on the Directed Weights for capturing fine-grained geometric relations, especially for the subtle orientational details in proteins. • We construct an Equivariant Message Passing paradigm based on protein graphs. • Our overall framework, the Orientation-Aware Graph Neural Networks, is versatile in terms of compatibility with existing deep graph learning models, making them biologically suitable with minimal modifications to existing GNN models.

2. RELATED WORK

Representation learning on protein 3D structure. Early approaches rely on hand-crafted features extracted and statistical methods to predict function annotations (Schaap et al., 2001; Zhang & Zhang, 2010) . Deep learning has been found to achieve success then. 3D CNNs are first proposed to process protein 3D structures by scanning atom-level features relying on multiple 3D voxels. One of the representative works (Derevyanko et al., 2018) adopts a 3D CNN-based model for assessing the quality of the predicted structures. 3D CNNs also shed light on other tasks such as interface prediction (Townshend et al., 2019; Amidi et al., 2018) . People also extend them to spherical convolutions (Gainza et al., 2020; Sverrisson et al., 2021; Hermosilla Casajus et al., 2021) , to the Fourier space (Zhemchuzhnikov et al., 2022) and the 3D Voronoi Tessellation space (Igashov et al., 2021) . Graph Convolutional Networks (Kipf & Welling, 2016) have also been adopted to capture geometric and biochemical interactions between residues (Ying et al., 2018; Gao & Ji, 2019; Fout, 2017) , and have been shown to achieve great performance on function prediction (Li et al., 2021) , protein design (Strokach et al., 2020) and binding prediction (Vecchio et al., 2021) . Recently, transformer-based methods (Vaswani et al., 2017) have a trend to replace conventional methods in other bioinformatics tasks (Ingraham et al., 2019; Baek et al., 2021; Jumper et al., 2021; Cao et al., 2021) . Equivariant neural networks. Equivariance is an important property, for generalizing to unseen conditions in geometric learning. (Cohen & Welling, 2016; Weiler et al., 2018; Köhler et al., 2020; Satorras et al., 2021; Thomas et al., 2018; Fuchs et al., 2020; Anderson et al., 2019; Gasteiger et al., 2019b; Batzner et al., 2021; Eismann et al., 2021) . Methods such as Tensor Field Network (Thomas et al., 2018) and Cormorant (Anderson et al., 2019) , have been developed to generate irreducible representations for achieving rotation equivariance in 3D. In comparison to Tensor Filed Network with complex tensor products, the Vector Neuron Network (VNN) achieves rotation equivariance in a much simpler way, which generalizes the values of hidden neurons from scalars to 3D vectors (Deng et al., 2021) . GVP-GNN has also been introduced for learning protein representations by featuring geometric factors as vectors (Jing et al., 2020) . Message passing in GNNs for vector representations using equivariant features have also been explored (Schütt et al., 2021; Luo et al., 2022) . However, to guarantee rotation equivariance, they can only linearly combine 3D vectors, in essence, limiting their geometric representing capacity. Another line of methods condition filters on invariant scalar features for maitaining equivariance (Schütt et al., 2017; Gasteiger et al., 2019a; Liu et al., 2021) .

3. DIRECTED WEIGHT PERCEPTRON

Vector-Tensor MatMul

MatMul

Vector-Vector Cross Product Vector Features

Scalar features

< l a t e x i t s h a 1 _ b a s e 6 4 = " F d O 3 X h r f 2 V Z 4 I h T Z W 2 J 0 Q w x + X k g = " > A A A C D n i c b Z D L S s N A F I Y n 9 V b r L S q u 3 A w W o Y K U R I q 6 L H b j s o K 9 Q B v C Z D J p h 8 5 M w s x E K K H v 4 C u 4 1 b 0 7 c e s r u P V J n L Y R t P W H A x / / O Y d z + I O E U a U d 5 9 M q r K y u r W 8 U N 0 t b 2 z u 7 e / b + Q V v F q c S k h W M W y 2 6 A F G F U k J a m m p F u I g n i A S O d Y N S Y 9 j s P R C o a i 3 s 9 T o j H 0 U D Q i G K k j e X b R 2 G l 4 f c R S 4 b o H P 7 Q m W + X n a o z E 1 w G N 4 c y y N X 0 7 a 9 + G O O U E 6 E x Q 0 r 1 X C f R X o a k p p i R S a m f K p I g P E I D 0 j M o E C f K y 2 b v T + C p c U I Y x d K U 0 H D m / t 7 I E F d q z A M z y Z E e q s X e 1 P y 3 F / C F y z q 6 9 j I q k l Q T g e e H o 5 R  B H c N p N j C k k m D N x g Y Q l t T 8 D v E Q S Y S 1 S b B k Q n E X I 1 i G 9 k X V v a z W 7 m r l + k 0 e T x E c g x N Q A S 6 4 A n V w C 5 q g B T D I w B N 4 B i / W o / V q v V n v < l a t e x i t s h a 1 _ b a s e 6 4 = " x R F v c C A H e 3 s d Z K 6 o 4 9 M L G 0 z y J b M = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a U D b b S b t 0 s w m 7 m 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o b u a 3 x q g 0 j + W T m S T o R 3 Q g e c g Z N V Z 6 H P e 8 X r n i V t 0 5 y C r x c l K B H P V e + a v b j 1 k a o T R M U K 0 7 n p s Y P 6 P K c C Z w W u q m G h P K R n S A H U s l j V D 7 2 f z U K T m z S p + E s b I l D Z m r v y c y G m k 9 i Q L b G V E z 1 M v e T P z P 6 6 Q m v P E z L p P U o G S L R W E q i I n J 7 G / S 5 w q Z E R N L K F P c 3 k r Y k C r K j E 2 n Z E P w l l 9 e J c 2 L q n d V v X y 4 r N R u 8 z i K c A K n c A 4 e X E M N 7 q E O D W A w g G d 4 h T d H O C / O u / O x a C 0 4 + c w x / I H z + Q M L k o 2 n < / l a t e x i t > v1 < l a t e x i t s h a 1 _ b a s e 6 4 = " p d 6 C G + f g X j P a 9 T r I / + S v i f D 5 V x Y = " > A A A B 6 n i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H Y J U Y 9 E L x 4 x y i O B D Z k d e m H C 7 O x m Z p a E E D 7 B i w e N 8 e o X e f N v H G A P C l b S S a W q O 9 1 d Q S K 4 N q 7 7 7 e Q 2 N r e 2 d / K 7 h b 3 9 g 8 O j 4 v F J U 8 e p Y t h g s Y h V O 6 A a B Z f Y M N w I b C c K a R Q I b A W j u 7 n f G q P S P J Z P Z p K g H 9 G B 5 C F n 1 F j p c d y r 9 I o l t + w u Q N a J l 5 E S Z K j 3 i l / d f s z S C K V h g m r d 8 d z E + F O q D G c C Z 4 V u q j G h b E Q H 2 L F U 0 g i 1 P 1 2 c O i M X V u m T M F a 2 p C E L 9 f f E l E Z a T 6 L A d k b U D P W q N x f / 8 z q p C W / 8 K Z d J a l C y 5 a I w F c T E Z P 4 3 6 X O F z I i J J Z Q p b m 8 l b E g V Z c a m U 7 A h e K s v r 5 N m p e x d l a s P 1 V L t N o s j D 2 d w D p f g w T X U 4 B 7 q 0 A A G A 3 i G V 3 h z h P P i v D s f y 9 a c k 8 2 c w h 8 4 n z 8 N F o 2 o < / l a t e x i t > v2 < l a t e x i t s h a 1 _ b a s e 6 4 = " r 7 1 G 5 b f v 7 Z + t z 9 k 6 H J I J F Y 4 X Q O M = " > A A A B + 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R S q s u i G 5 c V 7 A P a U C b T S T N 0 Z h J m J k I J / Q W 3 u n c n b v 0 Y t 3 6 J k z Y L b T 1 w 4 X D O v Z z L C R L O t D V N M J n i C R 1 Y K r G g 2 s 8 W v 8 7 R h V X G K I y V H W n Q Q v 1 9 k W G h 9 U w E d l N g E + l V L x f / 9 Q K x k m z C G z 9 j M k k N l W Q Z H K Y c m R j l R a A x U 5 Q Y P r M E E 8 X s 7 4 h E W G F i b F 0 V W 4 q 3 W s E 6 6 V 7 V v W C W Y w a 4 Y 2 O R m F S O C B N J W M c 7 r 4 K 0 = " > A A A B / H i c b V B N S 8 N A F H y p X 7 V + V T 1 6 W S y C p 5 J I U Y 8 F L x 4 r m L b S h r L Z b N q l u 9 m w u x F K q H / B q 9 6 9 i V f / i 1 d / i d s 2 B 2 0 d e D D M v M c 8 J k w 5 0 8 Z 1 v 5 z S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a m u Z K U J 9 I r l U 3 R B r y l l C f c M M p 9 1 U U S x C T j v h + G b m d x 6 p 0 k w m 9 2 a S 0 k D g Y c J i R r C x k t + X k T S D a s 2 t u 3 O g V e I V p A Y F W o P q d z + S J B M 0 M Y R j r X u e m 5 o g x 8 o w w u m 0 0 s 8 0 T T E Z 4 y H t W Z p g Q X W Q z 5 + d o j O r R C i W y k 5 i 0 F z 9 f Z F j o f V E h H Z T Y D P S y 9 5 M / N c L x V K y i a + D n C V p Z m h C F s F x x p G R a N Y E i p i i x P C J J Z g o Z n 9 H Z I Q V J s b 2 V b G l e M s V r J L 2 R d 2 7 r D f u G r X m Q 1 F P G U 7 g F M 7 B g y t o w i 2 0 w A c C D J 7 h B V 6 d J + f N e X c = " > A A A B / H i c b V B N S 8 N A F H y p X 7 V + V T 1 6 W S y C p 5 J I U Y 8 F L x 4 r m L b S h r L Z b N q l u 9 m w u x F K q H / B q 9 6 9 i V f / i 1 d / i d s 2 B 2 0 d e D D M v M c 8 J k w 5 0 8 Z 1 v 5 z S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a m u Z K U J 9 I r l U 3 R B r y l l C f c M M p 9 1 U U S x C T j v h + G b m d x 6 p 0 k w m 9 2 a S 0 k D g Y c J i R r C x k t + X k T S D a s 2 t u 3 O g V e I V p A Y F W o P q d z + S J B M 0 M Y R j r X u e m 5 o g x 8 o w w u m 0 0 s 8 0 T T E Z 4 y H t W Z p g Q X W Q z 5 + d o j O r R C i W y k 5 i 0 F z 9 f Z F j o f V E h H Z T Y D P S y 9 5 M / N c L x V K y i a + D n C V p Z m h C F s F x x p G R a N Y E i p i i x P C J J Z g o Z n 9 H Z I Q V J s b 2 V b G l e M s V r J L 2 R d 2 7 r D f u G r X m Q 1 F P G U 7 g F M 7 B g y t o w i 2 0 w A c C D J 7 h B V 6 d J + f N e X c + F q s l p 7 g 5 h j 9 w P n 8 A R 0 6 V h A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "  V f 1 h j b m S H 9 w O w 4 k I a w b U N i d C / m 0 = " > A A A C B X i c b V A 9 S w N B E N 3 z M 8 a v U 0 s t F o M Q m 3 A n Q S 2 D N p Y R z A f k Q t j b 2 y R L 9 m 6 P 3 T k x X N L Y + F d s L B S x 9 T / Y + W / c J F d o 4 o O B x 3 s z z M z z Y 8 E 1 O M 6 3 t b S 8 s r q 2 n t v I b 2 5 t 7 + z a e / t 1 L R N F W Y 1 K I V X T J 5 o J H r E a c B C s G S t G Q l + w h j + 4 n v i N e 6 Y 0 l 9 E d D G P W D k k v 4 l 1 O C R i p Y x 9 5 w B 4 g 1 b w X S h 6 M c d E b Y Y 8 G E r A 3 O u 3 Y B a f k T I E X i Z u R A s p Q 7 d h f X i B p E r I I q C B a t 1 w n h n Z K F H A q 2 D j v J Z r F h A 5 I j 7 U M j U j I d D u d f j H G J 0 Y J c F c q U x H g q f p 7 I i W h 1 s P Q N 5 0 h g b 6 e 9 y b i f 1 4 r g e 5 l O + V R n A C L 6 G x R N x E Y J J 5 E g g O u G A U x N I R Q x c 2 t m P a J I h R M c H k T g j v / 8 i K p n 5 X c 8 1 L 5 t l y o X G V x 5 N A h O k Z F 5 K I L V E E 3 q I p q i K J H 9 I x e 0 Z v 1 Z L 1 Y 7 9 b H r H X J y m Y O 0 B 9 Y n z / 2 R V m s F c J f a C l M k C j V 7 x q + v H L A 0 h Q i a o U h 3 b S t D N q E T O B E w K 3 V R B Q t m Q 9 q G j a U R

ReLU(•)

Output features < l a t e x i t s h a 1 _ b a s e 6 4 = " L 1 U A s L 3 4 E M 2 f t d j C y v m 5 N o b 3 5 N E = " > A A A C G X i c b V C 7 T s M w F H X K q 5 R X g J H F o k J i q h K o o G O l L o w F 0 Y f U h M p 2 n d a q 7 U S 2 g 1 R F X f k J f o E V d j b E y s T K l + C 0 H a B w J E t H 5 9 y r c 3 1 w w p k 2 n v f p F F Z W 1 9 Y 3 i p u l r e 2 d 3 T 1 3 / 6 C t 4 1 Q R 2 i I x j 1 U X I 0 0 5 k 7 R l m O G 0 m y i K B O a 0 g 8 e N 3 O / c U 6 V Z L G / N J K G h Q E P J I k a Q s V L f h Q E W W X s K A y Z h I J A Z Y Q x v 7 r J G Y J i g G p 5 P + 2 7 Z q 3 g z w L / E X 5 A y W K D Z d 7 + C Q U x S Q a U h H G n d 8 7 3 E h B l S h h F O p 6 U g 1 T R B Z I y G t G e p R D Y n z G Y / m c I T q w x g F C v 7 p I E z 9 e d G h o T W E 4 H t Z H 6 s X v Z y 8 V 8 P i 6 V k E 9 X C j M k k N V S S e X C U c m h i m N c E B 0 x R Y v j E E k Q U s 7 d D M k I K E W P L L N l S / O U K / p L 2 W c W / q F S v q + V 6 b V F P E R y B Y 3 A K f H A J 6 u A K N E E L E P A A n s A z e H E e n V f n z X m f j x a c x c 4 h + A X n 4 x t T K K A i < / l a t e x i t > V 2 R C⇥3 < l a t e x i t s h a 1 _ b a s e 6 4 = " m O h Q b I m 7 o Y g 3 T P l 6 J C Y Y D a B o I l c = " > A A A C E X i c b V D L S g M x F M 3 4 r P U 1 6 k Z w E y y C q z I j R b s s d O O y i n 1 A Z y x J m m l D k 8 y Q Z I Q y 1 J / w F 9 z q 3 p 2 4 9 Q v c + i V m 2 i 6 0 9 c C F w z n 3 c u 8 9 O O F M G 8 / 7 c l Z W 1 9 Y 3 N g t b x e 2 d 3 b 1 9 9 + C w p e N U E d o k M Y 9 V B y N N O Z O 0 a Z j h t J M o i g T m t I 1 H 9 d x v P 1 C l W S z v z D i h o U A D y S J G k L F S z z 0 O s M j 0 B A Z M w k A g M 8 Q Y 3 t 5 n 9 U n P L X l l b w q 4 T P w 5 K Y E 5 G j 3 3 O + j H J B V U G s K R 1 l 3 f S 0 y Y I W U Y 4 X R S D F J N E 0 R G a E C 7 l k o k q A 6 z 6 Q c T e G a V P o x i Z U s a O F V / T 2 R I a D 0 W 2 H b m R + p F L x f / 9 b B Y 2 G y i a p g x m a S G S j J b H K U c m h j m 8 c A + U 5 Q Y P r Y E E c X s 7 Z A M k U L E 2 B C L N h R / M Y J l 0 r o o + 5 f l y k 2 l V K v O 4 y m A E 3 A K z o E P r k A N X I M G a A I C H s E z e A G v z p P z 5 r w 7 H 7 P W F W c + c w T + w P n 8 A Z a C n R 4 = < / l a t e x i t > s 2 R C < l a t e x i t s h a 1 _ b a s e 6 4 = " S 0 l / M x M h 2 0 d 6 z z F 1 q X M M p C X O f M g = " > A A A C M X i c d Z D N S g M x E M e z f t b 6 V f X o J V g E T 2 V b W / U o 9 O K x i r W C u y 7 Z N N s G s 8 m S z C p l 2 V f x J X w F r 3 r v T b z 6 E m b b C i o 6 E D L 8 / j P M z D 9 M B D f g u m N n b n 5 h c W m 5 t F J e X V v f 2 K x s b V 8 Z l W r K u l Q J p a 9 D Y p j g k n W B g 2 D X i W Y k D g X r h X f t Q u / d M 2 2 4 k p c w S p g f k 4 H k E a c E L A o q J 5 6 y s u a D I R C t 1 U P m x Q S G Y Z T 1 8 q C R Y 4 9 L P C V h d p H f Z m 0 P e M w M P s y D S t W t t V w b D e z W i r / Z w j P S w v U J c d 0 q m k U n q L x 7 f U X T m E m g g h h z U 3 c T 8 D O i g V P B 8 r K X G p Y Q e k c G 7 M a m k t g 5 f j a 5 M M f 7 l v R x p L R 9 E v C E f u / I S G z M K A 5 t Z b G t + a 0 V 8 E 8 t j H 9 N h u j E z 7 h M U m C S T g d H q c C g c G E f 7 n P N K I i R T Q j V 3 O 6 O 6 Z B o Q s G a X L a m f F 2 O / 0 + u G r X 6 U a 1 5 3 q y e N m b 2 l N A u 2 k M H q I 6 O 0 S k 6 Q x 3 U R R Q 9 o m f 0 g l 6 d J 2 f s v D n v 0 9 I 5 Z 9 a z g 3 6 E 8 / E J H W i r j A = = < / l a t e x i t > ! W2 2 R C⇥3 < l a t e x i t s h a 1 _ b a s e 6 4 = " v d s Q 0 B R O I t 6 5 B l W W y O K 6 2 c 8 p m h M = " > A A A C H 3 i c b V C 7 T s M w F H X K q 5 R X g J H F o u I x V U l V A W O l L o w F 0 Y f U h M p x n d a q 7 U S 2 g 1 R F + Q N + g l 9 g h Z 0 N s X b l S 3 D a D t B y J E v n n n u v z v U J Y k a V d p y p V V h b 3 9 j c K m 6 X d n b 3 9 g / s w 6 O 2 i h K J S Q t H L J L d A C n C q C A t T T U j 3 V g S x A N G O s G 4 k f c 7 T 0 Q q G o k H P Y m J z 9 F Q 0 J B i p I 3 U t 8 8 9 j v Q o C N N O B j 0 q 4 L w M 0 v v s M W 1 c e J p y o m A j 6 9 t l p + L M A F e J u y B l s E C z b 3 9 7 g w g n n A i N G V K q 5 z q x 9 l M k N c W M Z C U v U S R G e I y G p G e o Q M b H T 2 f / y e C Z U Q Y w j K R 5 Q s O Z + n s j R V y p C Q / M Z H 6 u W u 7 l 4 r + 9 g C 8 5 6 / D G T 6 m I E 0 0 E n h u H C Y M 6 g n l Y c E A l w Z p N D E F Y U n M 7 x C M k E d Y m 0 p I J x V 2 O Y J W 0 q x X 3 q l K 7 q 5 X r 1 U U 8 R X A C T s E l c M E 1 q I N b 0 A Q t g M E z e A V v 4 N 1 6 s T 6 s T + t r P l q w F j v H 4 A + s 6 Q / K y a M L < / l a t e x i t > W 2 R C 0 ⇥C < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 z K f O B N S x d A d D 5 T F P o r 4 5 2 u X 9 c = " > A A A C P H i c d V B N S w M x E M 3 6 b f 2 q e v Q S L K K n s l t b 9 V j o x W M V 2 w p u L d k 0 2 w a z y Z L M K m X Z H + S f 8 C 9 4 1 a t 4 E 6 + e z b Y V V H Q g 5 P H e D G / m B b H g B l z 3 2 Z m Z n Z t f W F x a L q y s r q 1 v F D e 3 2 k Y l m r I W V U L p y 4 A Y J r h k L e A g 2 G W s G Y k C w T r B T S P X O 7 d M G 6 7 k B Y x i 1 o 3 I Q P K Q U w K W 6 h U b v r K y 5 o M h E K 3 V X e p H B I Z B m H a y n p d h n 0 s 8 Y Y L 0 P L t O G / v Y B x 4 x g x t f 4 D D r F U t u u e b a q m C 3 n P / V G p 4 y N e y N G d c t o W k 1 e 8 U X v 6 9 o E j E J V B B j r j w 3 h m 5 K N H A q W F b w E 8 N i Q m / I g F 1 Z K I n 1 6 a b j Y z O 8 Z 5 k + D p W 2 T w I e s 9 8 n U h I Z M 4 o C 2 5 k v b n 5 r O f m n F k S / n C E 8 6 a Z c x g k w S S f G Y S I w K J w n i f t c M w p i Z A G h m t v d M R 0 S T S j Y v A s 2 l K / L 8 f + g X S l 7 R + X q W b V U r 0 z j W U I 7 a B c d I A 8 d o z o 6 R U 3 U Q h T d o 0 f 0 h J 6 d B + f V e X P e J 6 0 z z n R m G / 0 o 5 + M T 4 j 6 v Q Q = = < / l a t e x i t > ! W1 2 R C 0 ⇥C⇥3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 t 6 1 w u 6 D h D f i 3 m K Z c 8 A A d a z P P 4 0  = " > A A A C I H i c b V C 7 T s M w F H X K q 5 R X g J H F o k J l q p K q A s Z K X R g L o g + p K Z X t O q 1 V x 4 l s B 6 m K 8 g n 8 B L / A C j s b Y o S R L 8 F p M 0 D L k S y d e + 6 9 O t c H R 5 w p 7 T i f V m F t f W N z q 7 h d 2 t n d 2 z + w D 4 8 6 K o w l o W 0 S 8 l D 2 M F K U M 0 H b m m l O e 5 G k K M C c d v G 0 m f W 7 D 1 Q q F o o 7 P Y v o I E B j w X x G k D b S 0 K 5 4 A d I T 7 C f d F H p M w E W J k 9 v 0 P m l W o K d Z Q B V s p k O 7 7 F S d O e A q c X N S B j l a Q / v b G 4 U k D q j Q h C O l + q 4 T 6 U G C p G a E 0 7 T k x Y p G i E z R m P Y N F c j 4 D J L 5 h 1 J 4 Z p Q R 9 E N p n t B w r v 7 e S F C g 1 C z A Z j K 7 V y 3 3 M v H f H g 6 W n L V / N U i Y i G J N B V k Y + z G H O o R Z W n D E J C W a z w x B R D J z O y Q T J B H R J t O S C c V d j m C V d G p V 9 6 J a v 6 m X G 7 U 8 n i I 4 A a f g H L j g E j T A N W i B N i D g E T y D F / B q P V l v 1 r v 1 s R g t W P n O M f g D 6 + s H L r y j N Q = = < / l a t e x i t > W 2 R C 0 ⇥C < l a t e x i t s h a 1 _ b a s e 6 4 = " v d s Q 0 B R O I t 6 5 B l W W y O K 6 2 c 8 p m h M = " > A A A C H 3 i c b V C 7 T s M w F H X K q 5 R X g J H F o u I x V U l V A W O l L o w F 0 Y f U h M p x n d a q 7 U S 2 g 1 R F + Q N + g l 9 g h Z 0 N s X b l S 3 D a D t B y J E v n n n u v z v U J Y k a V d p y p V V h b 3 9 j c K m 6 X d n b 3 9 g / s w 6 O 2 i h K J S Q t H L J L d A C n C q C A t T T U j 3 V g S x A N G O s G 4 k f c 7 T 0 Q q G o k H P Y m J z 9 F Q 0 J B i p I 3 U t 8 8 9 j v Q o C N N O B j 0 q 4 L w M 0 v v s M W 1 c e J p y o m A j 6 9 t l p + L M A F e J u y B l s E C z b 3 9 7 g w g n n A i N G V K q 5 z q x 9 l M k N c W M Z C U v U S R G e I y G p G e o Q M b H T 2 f / y e C Z U Q Y w j K R 5 Q s O Z + n s j R V y p C Q / M Z H 6 u W u 7 l 4 r + 9 g C 8 5 6 / D G T 6 m I E 0 0 E n h u H C Y M 6 g n l Y c E A l w Z p N D E F Y U n M 7 x C M k E d Y m 0 p I J x V 2 O Y J W 0 q x X 3 q l K 7 q 5 X r 1 U U 8 R X A C T s E l c M E 1 q I N b 0 A Q t g M E z e A V v 4 N 1 6 s T 6 s T + t r P l q w F j v H 4 A + s 6 Q / K y a M L < / l a t e x i t > W 2 R C 0 ⇥C < l a t e x i t s h a 1 _ b a s e 6 4 = " C i x o 7 5 5 / t U n g O I 9 z a G d X R f j q m B I = " > A A A C I H i c b V D L T g I x F O 3 4 R H y h L t 0 0 E o M r M q N E W Z K w c Y l G H g k z k r Y U a G g 7 k 7 Z j Q i b z C f 6 E v + B W 9 + 6 M S 1 3 6 J R a Y h Y I n a X J y z r 0 5 t w d H n G n j u p / O y u r a + s Z m b i u / v b O 7 t 1 8 4 O G z p M F a E N k n I Q 9 X B S F P O J G 0 a Z j j t R I o i g T l t 4 3 F 9 6 r c f q N I s l H d m E t F A o K F k A 0 a Q s V K v U P K x S F r p f T J K o c 8 k 9 A U y I 4 y T W y v V S 9 A 3 T F A N L 9 J e o e i W 3 R n g M v E y U g Q Z G r 3 C t 9 8 P S S y o N I Q j r b u e G 5 k g Q c o w w m m a 9 2 N N I 0 T G a E i 7 l k p k c 4 J k 9 q E U n l q l D w e h s k 8 a O F N / b y R I a D 0 R 2 E 5 O 7 9 W L 3 l T 8 1 8 N i I d k M q k H C Z B Q b K s k 8 e B B z a E I 4 b Q v 2 m a L E 8 I k l i C h m b 4 d k h B Q i x n a a t 6 V 4 i x U s k 9 Z 5 2 b s s V 2 4 q x V o 1 q y c H j s E J O A M e u A I 1 c A 0 a o A k I e A T P 4 A W 8 O k / O m / P u f M x H V 5 x s 5 w j 8 g f P 1 A 0 N l o 0 U = < / l a t e x i t > V h 2 R C 0 ⇥3 < l a t e x i t s h a 1 _ b a s e 6 4 = " w P R p n M N n g H O b B Z / m R f L v R D S W i o s = " > A A A C S X i c d V B N a x N B G J 5 N q 9 b U a t S j l 8 F Q 2 l 6 W 3 W 2 y D U K h 4 K X H C C Y p d M M y O 5 l N h s 7 s L D P v q m H Y v + W f 8 B c U P B X q v b f S k 5 O P o h b 7 w g s P z / N + P l k p u I E g u P Q a G 5 t P n j 7 b e t 7 c f r H z 8 l X r 9 Z u h U Z W m b E C V U P o s I 4 Y J X r A B c B D s r N S M y E y w U X b x c a G P v j B t u C o + w 7 x k Y 0 m m B c 8 5 J e C o t N V P M m m H d W o T Y N / A U m 3 q e g 9 / O M Y 4 k Q R m W W 5 H 9 X 6 i 3 A j N p z M g W q u v 9 o + U R j V O g E t m 8 G r Q Q d p q B 3 4 U R L 3 4 E A d + t x v F Y c + B o 2 4 n j k M c + s E y 2 m g d / b R 1 l U w U r S Q r g A p i z H k Y l D C 2 R A O n g t X N p D K s J P S C T N m 5 g w V x y 8 Z 2 + X m N d x 0 z w b n S L g v A S / b v D k u k M X O Z u c r F z e a h t i D / q 2 X y w W b I e 2 P L i 7 I C V t D V 4 r w S G B R e 2 I o n X D M K Y u 4 A o Z q 7 2 z G d E U 0 o O P O b z p T 7 z / H j Y B j 5 Y e x 3 P n X a J 9 H a n i 3 0 D r 1 H + y h E R + g E n a I + G i C K v q O f 6 B r 9 8 n 5 4 N 9 6 t d 7 c q b X j r n r f o n 2 h s / A Z S 3 b S T < / l a t e x i t > V 0 crs := W( ! W2 ⇥ V ) < l a t e x i t s h a 1 _ b a s e 6 4 = " h Y 8 3 B 3 S O K i j J h 0 X Y K + V p 4 2 4 I r j A = " > A A A C I X i c d V D L S g M x F M 3 4 r P V V d e k m W E R X w 0 z t C 0 E o u H F Z w T 7 A K S W T Z j S Y z A z J H b E M 8 w v + h L / g V v f u x J 2 4 8 0 t M H 4 I W P R A 4 n H N v 7 r 3 H j w X X 4 D j v 1 t z 8 w u L S c m 4 l v 7 q 2 v r F Z 2 N p u 6 y h R l L V o J C L V 9 Y l m g o e s B R w E 6 8 a K E e k L 1 v F v T k d + 5 5 Y p z a P w A o Y x 6 0 l y F f K A U w J G 6 h c O P V + m 7 a y f e s D u I D X f Z N k B P j 7 B n i R w 7 Q d p J 5 t W F I q O X X K O 3 F I F O 3 a l U n P L d U N q l X K 1 W s W u 7 Y x R R F M 0 + 4 V P b x D R R L I Q q C B a X 7 p O D L 2 U K O B U s C z v J Z r F h N 6 Q K 3 Z p a E g k 0 7 1 0 f F G G 9 4 0 y w E G k z A s B j 9 W f H S m R W g + l b y p H e + p Z b y T + 6 f l y Z j I E 9 V 7 K w z g B F t L J 4 C A R G C I 8 i g s P u G I U x N A Q Q h U 3 u 2 N 6 T R S h Y E L N m 1 C + L 8 f / k 3 b J d q t 2 + b x c b J S m 8 e T Q L t p D h 8 h F N d R A Z 6 i J W o i i e / S I n t C z 9 W C 9 W K / W 2 6 R 0 z p r 2 7 K B f s D 6 + A N O w p M c = < / l a t e x i t > V 0 lin := WV f a 3 3 X O O / u o F t h I g 6 I u r b R R 3 w = " > A A A C O 3 i c d V B N b 9 N A E F 0 X K C H Q E u D I Z U W E 2 p N l O 1 8 t E l J E L x y L R J J K d W S t N + t k 1 V 2 v t T s G o p X / D 3 + C v 8 A V u L a 3 i m v v X a e p 1 F b t k 0 Z 6 e j N P M / P S Q n A D Q f D P 2 3 j 0 + M n m 0 8 a z 5 v M X W 9 s v W 6 9 e j 4 0 q N W U j q o T S R y k x T P C c j Y C D Y E e F Z k S m g k 3 S k 4 O 6 P / n G t O E q / w r L g k 0 l m e c 8 4 5 S A k 5 L W p z i V 1 l S J j Y H 9 A D t T U F U 7 + M N H H C t n 0 3 y + A K K 1 + m 5 j S W C R Z n Z S J W G F Y + o m c e 0 d V 0 m r H f i D / d 5 g r 4 s D v x d E n U H k S D + I 9 v s d H P r B C m 2 0 x m H S O o 1 n i p a S 5 U A F M e Y 4 D A q Y W q K B U 8 G q Z l w a V h B 6 Q u b s 2 N G c S G a m d v V r h d 8 7 Z Y Y z p V 3 l g F f q T Y c l 0 p i l T N 1 k f b G 5 2 6 v F e 3 u p v L M Z s r 2 p 5 X l R A s v p 1 e K s F B g U r o P E M 6 4 Z B b F 0 h F D N 3 e 2 Y L o g m F F z c T R f K 9 e f 4 Y T K O / L D v d 7 9 0 2 8 N o H U 8 D v U X v 0 C 4 K 0 Q A N 0 W d 0 i E a I o p / o N / q D / n q / v D P v 3 P t / N b r h r T 1 v 0 C 1 4 F 5 c a V r A j < / l a t e x i t > s 0 dot := ! W1 • V < l a t e x i t s h a 1 _ b a s e 6 4 = " W + i A C 8 / G L 6 n D n E B U E g 8 4 6 P U D T O g = " > A A A C M 3 i c d V B N S y N B E O 3 R V W P 8 i n r c S 7 N B d k / D Z I y J B h Y E L 3 t 0 Y Z M I T g g 9 n Z 6 k S f f 0 0 F 2 z a 2 j m v / g n / A t 7 1 a t 4 E 6 / 7 H 7 Y n Z m E V f V D w e K + K q n p x J r i B I L j 3 l p Y / r K y u V d a r G 5 t b 2 z u 1 3 b 2 e U b m m r E u V U P o i J o Y J n r I u c B D s I t O M y F i w f j w 9 K / 3 + T 6 Y N V + k P m G V s I M k 4 5 Q m n B J w 0 r H W i W N p e M b Q R s C u w g i d Q F J 9 x 5 y s u D V N E y k 1 r P p 4 A 0 V r 9 s p E k M I k T 2 y + K Y a 0 e + O 2 T o / Z x E w f + U R A e t k N H W k F 4 0 j r E D T + Y o 4 4 W O B / W H q O R o r l k K V B B j L l s B B k M L N H A q W B F N c o N y w i d k j G 7 d D Q l k p m B n f 9 Y 4 A O n j H C i t K s U 8 F z 9 f 8 I S a c x M x q 6 z P N G 8 9 k r x T S + W r z Z D c j y w P M 1 y Y C l 9 X p z k A o P C Z Y B 4 x D W j I G a O E K q 5 u x 3 T C d G E g o u 5 6 k L 5 9 z l + n / R C v 9 H y m 9 + b 9 d N w E U 8 F f U S f 0 B f U Q G 1 0 i r 6 h c 9 R F F F 2 j 3 + g W 3 X k 3 3 o P 3 6 D 0 9 t y 5 5 i 5 l 9 9 A L e n 7 9 V l 6 1 l < / l a t e x i t > A protein is modeled as a KNN-graph G = (V, E) where each node u ∈ V corresponds to one amino acid, characterized by a scalar-vector tuple V 0 lift := s ! W < l a t e x i t s h a 1 _ b a s e 6 4 = " A E u 2 t 5 I Y d D 9 j S u B F 7 S j R L U 7 t j D Q = " > A A A C A X i c b V C 7 S g N B F L 0 b X z G + o p Y 2 g 0 G w C r s S N G X A x j K C e W C y h t n J b D J k Z n a Z m R X C s p W / Y K u 9 n d j 6 J b Z + i Z N k C 0 0 8 c O F w z r 2 c y w l i z r R x 3 S + n s L a + s b l V 3 C 7 t 7 O 7 t H 5 Q P j 9 o 6 S h S h L R L x S H U D r C l n k r Y M M 5 x 2 Y 0 W x C D j t B J P r m d 9 5 p E q z S N 6 Z a U x 9 g U e S h Y x g Y 6 X 7 f i D S d v a Q j r N B u e J W 3 T n Q K v F y U o E c z U H 5 u z + M S C K o N I R j r X u e G x s / x c o w w m l W 6 i e a x p h M 8 I j 2 L J V Y U O 2 n 8 4 8 z d G a V I Q o j Z U c a N F d / X 6 R Y a D 0 V g d 0 U 2 I z 1 s j c T / / U C s Z R s w r q f M h k n h k q y C A 4 T j k y E Z n W g I V O U G D 6 1 B B P F 7 O + I j L H C x N j S S r Y U b 7 m C V d K + q H q X 1 d p t r d K o 5 / U U 4 Q R O 4 R w 8 u I I G 3 E A T W k B A w j O 8 w K v z 5 L w 5 7 8 7 H Y r X g 5 D f H 8 A f O 5 w 9 n U J e 6 < / l a t e x i t > V h < l a t e x i t s h a 1 _ b a s e 6 4 = " A E u 2 t 5 I Y d D 9 j S u B F 7 S j R L U 7 t j D Q = " > A A A C A X i c b V C 7 S g N B F L 0 b X z G + o p Y 2 g 0 G w C r s S N G X A x j K C e W C y h t n J b D J k Z n a Z m R X C s p W / Y K u 9 n d j 6 J b Z + i Z N k C 0 0 8 c O F w z r 2 c y w l i z r R x 3 S + n s L a + s b l V 3 C 7 t 7 O 7 t H 5 Q P j 9 o 6 S h S h L R L x S H U D r C l n k r Y M M 5 x 2 Y 0 W x C D j t B J P r m d 9 5 p E q z S N 6 Z a U x 9 g U e S h Y x g Y 6 X 7 f i D S d v a Q j r N B u e J W 3 T n Q K v F y U o E c z U H 5 u z + M S C K o N I R j r X u e G x s / x c o w w m l W 6 i e a x p h M 8 I j 2 L J V Y U O 2 n 8 4 8 z d G a V I Q o j Z U c a N F d / X 6 R Y a D 0 V g d 0 U 2 I z 1 s j c T / / U C s Z R s w r q f M h k n h k q y C A 4 T j k y E Z n W g I V O U G D 6 1 B B P F 7 O + I j L H C x N j S S r Y U b 7 m C V d K + q H q X 1 d p t r d K o 5 / U U 4 Q R O 4 R w 8 u I I G 3 E A T W k B A w j O 8 w K v z 5 L w 5 7 8 7 H Y r X g 5 D f H 8 A f O 5 w 9 n U J e 6 < / l a t e x i t > V h < l a t e x i t s h a 1 _ b a s e 6 4 = " F D 5 R + E X 3 7 O 0 w w g q U B X i E 4 C h E Q v 8 = " > A A A C F 3 i c b V D L S g M x F M 3 U V 6 2 v q k t B g k V 0 V W a k a J e F b l x W s Q / o j C V J M 2 1 o k h m S j F C G 2 f k T / o d J Q o Q p s k 4 p H q Y K Q p Z 5 I 2 D T O c d m J F k c C c t v G o P v H b D 1 R p F s k 7 M 4 5 p I N B A s p A R Z K z U K x 7 7 W K Q 6 u 0 + H G f S Z h L 5 A Z o h x e m u l + l n W K 5 b c s j s F X C b e n J T A H I 1 e 8 d v v R y Q R V B r C k d Z d z 4 1 N k C J l G O E 0 K / i J p j E i I z S g X U s l E l Q H 6 f S P D J 5 a p Q / D S N m S B k 7 V 3 x M p E l q P B b a d k z P 1 o j c R / / W w W N h s w m q Q M h k n h k o y W x w m H J o I T k K C f a Y o M X x s C S K K 2 d s h G S K F i L F R F m w o 3 m I E y 6 R 1 U f Y u s 0 = s h + ! WV • V h < l a t e x i t s h a 1 _ b a s e 6 4 = " P X a m e L + B f N G v H P X a 8 Z K p s t 5 7 n W I = " > A A A C Q X i c b V C 7 T s M w F H V 4 U 1 4 F R h a L C l G W K k E I W J C Q W B h B o g + p K Z X j O o 1 V O 4 7 s G 1 A V 5 Z f 4 C X 6 B h Q E G N j b E y o L T d o D C l S w d n Y f u 9 Q k S w Q 2 4 7 r M z M z s 3 v 7 C 4 t F x a W V 1 b 3 y h v b j W M S j V l d a q E 0 q 2 A G C Z 4 z O r A Q b B W o h m R g W D N Y H B R 6 M 0 7 p g 1 X 8 Q 0 M E 9 a R p B / z k F M C l u q W L / 1 A Z o 1 8 H 5 / h a g F N f h t h X 9 m I 5 v 0 I i N b q P v M l g S g I s 2 a e d 7 O x K z + w r p 4 C P M 7 f R t 1 y x a 2 5 o 8 F / g T c B F T S Z q 2 7 5 z e 8 p m k o W A x X E m L b n J t D J i A Z O B c t L f m p Y Q u i A 9 F n b w p h I Z j r Z 6 M c 5 3 r N M D 4 d K 2 x c D H r E / E x m R x g x l Y J 3 F 7 W Z a K 8 h / t U B O b Y b w t J P x O E m B x X S 8 O E w F B o W L O n G P a 0 Z B D C 0 g V H N 7 O 6 Y R 0 Y S C L b 1 k S / G m K / g L G o c 1 7 7 h 2 d H 1 U O T + c 1 L O E d t A u q i I P n a B z d I m u U B 1 R 9 I C e 0 A t 6 d R 6 d d + f D + R x b Z 5 x J Z h v 9 G u f r G / c b s f g = < / l a t e x i t > V 0 = (s h ! Ws) V h < l a t e x i t s h a 1 _ b a s e 6 4 = " U e H E f i 8 I Z X 6 N B Z S V B X K U Z M K t i E I = " > A A A C H X i c b V D L T g I x F O 3 4 R H y h L t 0 0 o t E V m V G i L E n Y u E Q j j 4 Q Z S V s K N L S d S d s x I Z P 5 A X / C X 3 C r e 3 f G r X H r l 9 g B F g q e p M n J O f f m 3 B 4 c c a a N 6 3 4 5 S 8 s r q 2 v r u Y 3 8 5 t b 2 z m 5 h b 7 + p w 1 g R 2 i A h D 1 U b I 0 0 5 k 7 R h m O G 0 H S m K B O a 0 h U e 1 z G 8 9 U K V Z K O / M O K K B Q A P J + o w g Y 6 V u 4 d j H I m m m p 9 B n E v o C m S H G y W 1 6 n 9 S s Z J i g G l 6 k 3 U L R L b k T w E X i z U g R z F D v F r 7 9 X k h i Q a U h H G n d 8 d z I B A l S h h F O 0 7 w f a x o h M k I D 2 r F U I p s T J J P f p P D E K j 3 Y D 5 V 9 0 s C J + n s j Q U L r s c B 2 M r t X z 3 u Z + K + H x V y y 6 V e C h M k o N l S S a X A / 5 t C E M K s K 9 p i i x P C x J Y g o Z m + H Z I g U I s Y W m r e l e P M V L J L m e c m 7 L J V v y s V q Z V Z P D h y C I 3 A G P H A F q u A a 1 E E D E P A I n s E L e H W e n D f n 3 f m Y j i 4 5 s 5 0 D 8 A f O 5 w 8 Y 3 a G Q < / l a t e x i t > V 0 2 R C 0 ⇥3 < l a t e x i t s h a 1 _ b a s e 6 4 = " h N j b O s N K J b Z M s S + g o 8 B A J 6 A S V X U = " > A A A C F H i c b V C 7 T g J B F J 3 1 i f h C L W k m E o M V 2 T V E K U l o L N H I I 2 F X M j M M M G F m d j M z a 0 I 2 W / g T / o K t 9 n b G 1 t 7 W L 3 G A L R Q 8 y U 1 O z r k 3 9 9 6 D I 8 6 0 c d 0 v Z 2 1 9 Y 3 N r O 7 e T 3 9 3 b P z g s H B 2 3 d R g r Q l s k 5 K H q Y q Q p Z 5 K 2 D D O c d i N F k c C c d v C k M f M 7 D 1 R p F s o 7 M 4 1 o I N B I s i E j y F i p X y j 6 W C Q 6 L U O f S e g L Z M Y Y J 7 f p f d I o p / 1 C y a 2 4 c 8 B V 4 m W k B D I 0 + 4 V v f x C S W F B p C E d a 9 z w 3 M k G C l G G E 0 z T v x 5 p G i E z Q i P Y s l U h Q H S T z J 1 J 4 Z p U B H I b K l j R w r v 6 e S J D Q e i q w 7 Z y d q Z e 9 m f i v h 8 X S Z j O s B Q m T U W y o J I v F w 5 h D E 8 J Z Q n D A F C W G T y 1 B R D F 7 O y R j p B A x N s e 8 D c V b j m C V t C 8 q 3 m W l e l M t 1 W t Z P D l Q B K f g H H j g C t T B N W i C F i D g E T y D F / D q P D l v z r v z s W h d c 7 K Z E / A H z u c P 8 L 6 e Y g = = < / l a t e x i t > s 0 2 R C 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " A E u 2 t 5 I Y d D 9 j S u B F 7 S j R L U 7 t j D Q = " > A A A C A X i c b V C 7 S g N B F L 0 b X z G + o p Y 2 g 0 G w C r s S N G X A x j K C e W C y h t n J b D J k Z n a Z m R X C s p W / Y K u 9 n d j 6 J b Z + i Z N k C 0 0 8 c O F w z r 2 c y w l i z r R x 3 S + n s L a + s b l V 3 C 7 t 7 O 7 t H 5 Q P j 9 o 6 S h S h L R L x S H U D r C l n k r Y M M 5 x 2 Y 0 W x C D j t B J P r m d 9 5 p E q z S N 6 Z a U x 9 g U e S h Y x g Y 6 X 7 f i D S d v a Q j r N B u e J W 3 T n Q K v F y U o E c z U H 5 u z + M S C K o N I R j r X u e G x s / x c o w w m l W 6 i e a x p h M 8 I j 2 L J V Y U O 2 n 8 4 8 z d G a V I Q o j Z U c a N F d / X 6 R Y a D 0 V g d 0 U 2 I z 1 s j c T / / U C s Z R s w r q f M h k n h k q y C A 4 T j k y E Z n W g I V O U G D 6 1 B B P F 7 O + I j L H C x N j S S r Y U b 7 m C V d K + q H q X 1 d p t r d K o 5 / U U 4 Q R O 4 R w 8 u I I G 3 E A T W k B A w j O 8 w K v z 5 L w 5 7 8 7 H Y r X g 5 D f H 8 A f O 5 w 9 n U J e 6 < / l a t e x i t > V h < l a t e x i t s h a 1 _ b a s e 6 4 = " C U o F Z R e i p e Y a / Y P p e Q W B t j I f z W g = " > A A A C A X i c b V C 7 S g N B F L 0 b X z G + o p Y 2 g 0 G w C r s S N G X A x j K C e W C y h t n J b D J k Z n a Z m R X C s p W / Y K u 9 n d j 6 J b Z + i Z N k C 0 0 8 c O F w z r 2 c y w l i z r R x 3 S + n s L a + s b l V 3 C 7 t 7 O 7 t H 5 Q P j 9 o 6 S h S h L R L x S H U D r C l n k r Y M M 5 x 2 Y 0 W x C D j t B J P r m d 9 5 p E q z S N 6 Z a U x 9 g U e S h Y x g Y 6 X 7 f i B S n T 2 k 4 2 x Q r r h V d w 6 0 S r y c V C B H c 1 D + 7 g 8 j k g g q D e F Y 6 5 7 n x s Z P s T K M c J q V + o m m M S Y T P K I 9 S y U W V P v p / O M M n V l l i M J I 2 Z E G z d X f F y k W W k 9 F Y D c F N m O 9 7 M 3 E f 7 1 A L C W b s O 6 n T M a J o Z I s g s O E I x O h W R 1 o y B Q l h k 8 t w U Q x + z s i Y 6 w w M b a 0 k i 3 F W 6 5 g l b Q v q t 5 l t X Z b q z T q e T 1 F O I F T O A c P r q A B N 9 C E F h C Q 8 A w v 8 O o 8 O W / O u / O x W C 0 4 + c 0 x / I H z + Q O V p Z f X < / l a t e x i t > s h < l a t e x i t s h a 1 _ b a s e 6 4 = " C U o F Z R e i p e Y a / Y P p e Q W B t j I f z W g = " > A A A C A X i c b V C 7 S g N B F L 0 b X z G + o p Y 2 g 0 G w C r s S N G X A x j K C e W C y h t n J b D J k Z n a Z m R X C s p W / Y K u 9 n d j 6 J b Z + i Z N k C 0 0 8 c O F w z r 2 c y w l i z r R x 3 S + n s L a + s b l V 3 C 7 t 7 O 7 t H 5 Q P j 9 o 6 S h S h L R L x S H U D r C l n k r Y M M 5 x 2 Y 0 W x C D j t B J P r m d 9 5 p E q z S N 6 Z a U x 9 g U e S h Y x g Y 6 X 7 f i B S n T 2 k 4 2 x Q r r h V d w 6 0 S r y c V C B H c 1 D + 7 g 8 j k g g q D e F Y 6 5 7 n x s Z P s T K M c J q V + o m m M S Y T P K I 9 S y U W V P v p / O M M n V l l i M J I 2 Z E G z d X f F y k W W k 9 F Y D c F N m O 9 7 M 3 E f 7 1 A L C W b s O 6 n T M a J o Z I s g s O E I x O h W R 1 o y B Q l h k 8 t w U Q x + z s i Y 6 w w M b a 0 k i 3 F W 6 5 g l b Q v q t 5 l t X Z b q z T q e T 1 F O I F T O A c P r q A B N 9 C E F h C Q 8 A w v 8 O o 8 O W / O u / O x W C 0 4 + c 0 x / I H z + Q O V p Z f X < / l a t e x i t > s h < l a t e x i t s h a 1 _ b a s e 6 4 = " C U o F Z R e i p e Y a / Y P p e Q W B t j I f z W g = " > A A A C A X i c b V C 7 S g N B F L 0 b X z G + o p Y 2 g 0 G w C r s S N G X A x j K C e W C y h t n J b D J k Z n a Z m R X C s p W / Y K u 9 n d j 6 J b Z + i Z N k C 0 0 8 c O F w z r 2 c y w l i z r R x 3 S + n s L a + s b l V 3 C 7 t 7 O 7 t H 5 Q P j 9 o 6 S h S h L R L x S H U D r C l n k r Y M M 5 x 2 Y 0 W x C D j t B J P r m d 9 5 p E q z S N 6 Z a U x 9 g U e S h Y x g Y 6 X 7 f i B S n T 2 k 4 2 x Q r r h V d w 6 0 S r y c V C B H c 1 D + 7 g 8 j k g g q D e F Y 6 5 7 n x s Z P s T K M c J q V + o m m M S Y T P K I 9 S y U W V P v p / O M M n V l l i M J I 2 Z E G z d X f F y k W W k 9 F Y D c F N m O 9 7 M 3 E f 7 1 A L C W b s O 6 n T M a J o Z I s g s O E I x O h W R 1 o y B Q l h k 8 t w U Q x + z s i Y 6 w w M b a 0 k i 3 F W 6 5 g l b Q v q t 5 l t X Z b q z T q e T 1 F O I F T O A c P r q A B N 9 C E F h C Q 8 A w v 8 O o 8 O W / O u / O x W C 0 4 + c 0 x / I H z + Q O V p Z f X < / l a t e x i t > h A = " > A A A C O n i c d V D L T h s x F P X Q 8 g q v t C z Z W I 0 Q r K J J S I A l F Z s u o S I E i Q m R 7 X g S K 3 6 M 7 D u g y J r v 6 U / 0 F 9 h S s W V X d c s H 1 J M E q U V w J c t H 5 9 y r c + + h m R Q O 4 v g x W v j w c X F p e W W 1 s r a + s b l V / f T 5 0 p n c M t 5 h R h p 7 R Y n j U m j e A Q G S X 2 W W E 0 U l 7 9 L x a a l 3 b 7 l 1 w u g L m G S 8 p 8 h Q i 1 Q w A o H q V 7 8 m J s h W D E d A r D V 3 P l E E R j T 1 3 a L v E 6 q 8 K 4 o C J 0 L j m U D 9 9 + L G n + 7 h B I T i D h 8 U / W o t r r f j U E 0 c 1 8 u / 1 c Z z p o 0 b U y a O a 2 h e Z / 3 q U z I w L F d c A 5 P E u e t G n E H P E w u C S V 5 U k t z x j L A x G f L r A D U J P j 0 / P b X A u 4 E Z 4 N T Y 8 D T g K f v v h C f K u Y m i o b P c 1 7 3 W S v J N Q I D k 5 5 n l R F H J J 3 Q 1 L P X J F b d O G P 0 R 1 h m f K r L Q I h W M Q K B m j X e J C b I V i y U Q a 8 2 1 T x S B J U 3 9 p J j 5 h C o / L o o C J 0 L j r U D 9 h + L S D 1 s J C M U d H r Z w h Y 6 L W a M Z t / t x q C 6 O 2 + X f 6 + O K 6 e P O h o n j J q r q b N a 4 T e a G 5 Y p r Y J I 4 d 9 G J M 5 h 6 Y k E w y Y t 6 k j u e E b Y i C 3 4 R o C b B Z + o 3 N x f 4 M D B z n B o b n g a 8 Y f + c 8 E Q 5 t 1 Y 0 d J a L u / t a S f 5 T o + q e M 6 S n U y 9 0 l g P X b G u c 5 h K D w W W g e C 4 s Z y D X A R B m R d g d s y W x h E G I v R 5 C + X 0 5 / j 8 Y d 9 u d N + 3 e + 1 5 z 0 K 3 i 2 U W v 0 G t 0 h D r o B A 3 Q W 3 S G R o i h T + g L + o Z u o s / R 9 + h H d L d t r U X V z E v 0 V 0 U / f w H y C r L O < / l a t e x i t > ! WV 2 R C 0 ⇥C 0 ⇥3 h u = (s u , V u ) with s u ∈ R C , V u ∈ R C×3 , and the edges are constructed spatially by querying its k-nearest neighbours in the space. The edge features E = {e ij } i̸ =j , which represent edge connected node i and j, are also multi-channel scalarvector tuples. For simplicity, we set the channel numbers for scalar and vector features as the same, but they can be different. In practice, scalar, vector features are constructed based on a given protein structure and are kept separate. To make them compatible, the channel numbers of scalars are larger than the vectors. Details can be found in the Appendix Section B.1.

3.1. DIRECTED WEIGHTS

Classical neural networks only consider scalar-list features of the form s ∈ R C , with each layer transforming them with a weight matrix W ∈ R C ′ ×C and a bias term b ∈ R C ′ : s ′ = Ws + b (1) Although has been proved to be a universal approximator, such a layer has intrinsically no geometric interpretation in the 3D shape space, and even the simplest 3D rigid transformation on the input can result in unpredictable changes of network outputs. Recent attempts have lifted neuron features from scalar-lists to vector-lists V ∈ R C×3 with a linear operation mapping to Deng et al., 2021; Jing et al., 2020) : V ′ ∈ R C ′ ×3 ( V ′ = WV (2) Under this construction, SO(3)-actions in the latent space are simply matrix multiplications, which establishes a clear 3D Euclidean structure. However, to maintain input-output consistency under rigid transformations (e.g. rotation and translation), their operations are still limited to classical linear combinations weighted by W. To define operations that are more adaptive to the geometrically meaningful features, we introduce the Directed Weights, that we can define any tensor with the last dimension of 3 as a directed weight matrix, for example, ⃗ W ∈ R C ′ ×C×3 , seen as a stack of geometric meaningful vectors lying in the 3D Euclidean Space, and can be acted by SE(3) group from the right.

3.2. ⃗ W-OPERATORS

With weights and features both equipped with 3D vector representations, we can design a set of geometric operators □ (e.g. • or ×) with learnable ⃗ W as parameters in neural networks, which can operate on both scalar-list features s ∈ R C and vector-list features V ∈ R C×3 ⃗ W□s, ⃗ W□V (3) Geometric vector operations. Let W ∈ R C ′ ×C be a conventional scalar weight matrix, ⃗ W 1 ∈ R C ′ ×C×3 and ⃗ W 2 ∈ R C×3 be directed weight tensors. Beyond linear combinations, We can define two operations that leverage geometric information: s ′ dot (V ; ⃗ W 1 ) = ⃗ W 1 • V ∈ R C ′ (4) V ′ crs (V ; W, ⃗ W 2 ) = W( ⃗ W 2 × V ) ∈ R C ′ ×3 Here s ′ dot transforms C vector features to C ′ scalars using dot-product with directed weights, detailedly, s ′ i = j,k ⃗ W i,j,k 1 V j,k , which explicitly measures angles between vectors, and this operator brings network the ability to accurately sense the orientational features. In V ′ crs , a vector crosses a directed weight before being projected onto the hidden space. While the output of plain linear combinations between two vectors v 1 and v 2 can only lie in the plane w 1 v 1 + w 2 v 2 , cross-product in Equation 5 creates a new direction outside the plane, which is crucial in 3D modelling. For instance, the side-chain angles of a given residue could be largely determined by its C α and C β vectors, but may lie on the direction perpendicular to the space constructed by the two vectors. Scalar lifting. In addition, a directed weight ⃗ W ∈ R C×3 can lift scalars to vectors by adopting the following operation, the ij th entry of s ⃗ W is s i ⃗ W ij V ′ lift (s; ⃗ W) = s ⃗ W ∈ R C×3 This maps each scalar to a particular vector, enabling inverse transformations or information bottlenecks from R to R 3 . An intuition example is that, we can map a scalar representing the distance between two amino acids to a vector pointing from one amino acid to the other, leading to more biological meaningful representations for the protein fragment. Linear combinations. In the end, we keep the linear combination operations with scalar weights for both scalar and vector features: s ′ lin (s; W) = Ws ∈ R C ′ (7) V ′ lin (V ; W) = WV ∈ R C ′ ×3 (8) While these operators enables a more flexible network design, equivariance to rigid transformations is broken if considering more complex functions beyond linear combinations. We will introduce a globally equivariant paradigm to tackle this issue in Section 4.

3.3. ⃗ W-PERCEPTRON UNIT

Now we combine all the ⃗ W-operators together, assembling them into what we call Directed Weight Perceptrons ( ⃗ W-perceptrons). A ⃗ W-perceptron unit is a function mapping from u = (s, V ) to another scalar-vector tuple u ′ = (s ′ , V ′ ), which can be stacked as network layers to form multilayer perceptrons. A single unit comprises three modules in a sequential way: the Directed Linear module, the Non-Linearity module, and the Directed Interaction module (Figure 2 ). Directed linear module. The output hidden representations derived from the set of operations introduced previously are concatenated and projected into another hidden space of scalars and vectors separately: s h = W s [s ′ dot , s ′ lin ] ∈ R C ′ (9) V h = W V [V ′ crs , V ′ lift , V ′ lin ] ∈ R C ′ ×3 (10) Here W s ∈ R C ′ ×(C ′ +C ′ ) and W V ∈ R C ′ ×(C ′ +C+C ′ ) are scalar weight matrices. In other word, the five separate updating functions allow transformation from scalar to scalar (Equation 7), scalar to vector (Equation 6), vector to scalar (Equation 4), and vector to vector (Equation 8, Equation 5), boosting the model's capability to reason in 3D space. Non-linearity module. We then apply the non-linearity module to the hidden representation (s h , V h ). Specifically, we apply standard ReLU non-linearity (Nair & Hinton, 2010) to the scalar components. For the vector representations, following (Weiler et al., 2018; Jing et al., 2020; Schütt et al., 2021) , we compute a sigmoid activation on the L2-norm of each vector and multiply it back to the vector entries accordingly: s h ← ReLU(s h ), v h ij ← v h ij • sigmoid(∥v h i ∥ 2 ) where v h i ∈ R 3 are the vector columns in V h and v h ij are their entries. Directed interaction module. Finally we introduce the Directed Interaction module, integrating the hidden features s h and V h into the output tuple (s ′ , V ′ ) s ′ = s h + ⃗ W V • V h ∈ R C ′ (12) V ′ = (s h ⃗ W s ) ⊙ V h ∈ R C ′ ×3 Here ⃗ W V , ⃗ W s are directed weight matrices with sizes C ′ × C ′ × 3 and C ′ × 3, respectively, ⊙ denotes element-wise multiplication for two matrices. This module establishes a connection between the scalar and vector feature components, facilitating feature blending. Specifically, Equation 12dynamically determines how much the output should rely on scalar and vector representations, and Equation 13 weights a list of vectors using the scalar features as attention scores.

4. ORIENTATION-AWARE GRAPH NEURAL NETWORKS

To achieve SO(3)-equivariance without the loss of modeling capacity, we introduce an equivariant message passing paradigm on protein graphs, to easily plug our versatile ⃗ W-Perceptron into any existing graph learning framework, which makes network architectures free from equivariant constraints (Section 4.1). The integrated models, called Orientation-Aware Graph Neural Networks, can not only accurately model the essential orientational features but also maintain the rotation equivariance efficiently. We also design multiple variants in analogy to other graph neural networks Section 4.2.

4.1. EQUIVARIANT MESSAGE PASSING ON PROTEINS

A function f : R 3 → R 3 is SO(3)-equivariant (rotation-equivariant) if any rotation matrix R ∈ R 3×3 applied to the input vector x leads to the same transformation on the output f (x): Rf (x) = f (Rx) Such equivariant property can be achieved on a protein graph with local orientations, which is naturally defined from its biological structure (Ingraham et al., 2019) . Specifically, each amino acid

5. EXPERIMENTS

To demonstrate the basic rationale in perceiving angular features of our ⃗ W-Perceptron, we design a synthetic task (Section 5.1). We then conduct experiments on multiple benchmarks, including node-level tasks: Residue Identification (RES) (Section 5.2), Computational Protein Design (CPD) (Section 5.3), and graph-level tasks: Model Quality Assessment (MQA) (Section 5.4). Ablation studies are also included (Section 5.5). More details are documented in the Appendix Section B. Datasets. We test the performance of different perceptron designs on a synthetic binary classification dataset with angular. We sample a random unit vector v m ∈ R 3 as the anchor vector. Then each vector v i in the space is labeled positive if its angle between the anchor vector a mi ≤ π/4, and negative otherwise. We generate 100, 000 random vectors with 50% positive samples and 50% negative samples. For a fair comparison, we restrict the number of parameters and hyperparameters of different models to approximately the same. Results. As shown in Figure 3 , in general, our ⃗ W-Perceptron (DWP) converges faster than other models and achieves the lowest validation loss. We also notice that the Geometric Vector Perceptron (GVP) cannot converge given a sufficient number of epochs due to the poor capability of perceiving angulars by only using linear combinations of vector features. In comparison to GVP, Vector Neural Network (VNN) performs better based on its non-linear ReLU operation for vector features. Simple Multi-Layer Perceptron (MLP) can also capture useful information according to its universal approximation ability. The superior performance demonstrates that our ⃗ W-Perceptron has a better ability on perceiving potential geometric features in space.

5.2. RESIDUE IDENTIFICATION

Datasets. Residue Identification (RES) aims to predict the identity of particular masked amino acid based on its local surrounding structure (Torng & Altman, 2017) . We download 1000, 000 (10 6 ) substructures from the ATOM3D project (Townshend et al., 2021) , which are originally derived from 574 proteins from the PDB (Berman et al., 2000) . The entire dataset is split into training, validation, and testing datasets with the ratio of 80%, 10%, and 10%. There are no two proteins with similar structures in the test and non-test datasets based on the CATH 4.2 topology class at the domain level (Orengo et al., 1997) . Metrics. We use classification accuracy to evaluate model performance. Baselines. The 3D-CNN encodes the positions of the atoms in a voxelized 3D volume. We also compare GNN variants such as GCN , GIN and GAT with our OA-GCN, OA-GIN and OA-GAT models. We also include GVP-GNN as well.

Results

As shown in Table 1 , GCN, GIN and GAT almost cannot accurately identify the type of amino acids, suggesting the lack of ability of conventional GNN models to capture 3D geometric information. 3D-CNN performs better due to the power of the voxelization technique. Our models outperform other models, suggesting OA-GNNs could help represent protein substructure geometries better. 

5.3. COMPUTATIONAL PROTEIN DESIGN

Datasets. Computational Protein Design (CPD) predicts the native protein sequence of a given backbone structure. Specifically, we focus on two databases: CATH 4.2 (Ingraham et al., 2019) organizes proteins in a hierarchical structure (Orengo et al., 1997) . Following prior works, there are 18, 024 protein chains in the training set, 608 in the validation set, and 1, 120 in the test. TS50 dataset (Li et al., 2014 ) is a relatively old benchmark consisting of only 50 protein structures widely used in biology communities.foot_0 . Metrics. Native Sequence Recovery Rate is adopted to evaluate the prediction performance, which compares the predicted sequence with the ground truth sequence at each position and calculates the proportion of the correctly recovered amino acids (Li et al., 2014) . The perplexity score (Jelinek et al., 1977) evaluates whether a model could assign a high likelihood to the test sequences (Ingraham et al., 2019) . Baselines. For the CATH 4.2 dataset, we compare our models with other state-of-the-art models, including GVP-GNN, Structured Transformer and Structured GNN (Ingraham et al., 2019) . We compare multiple machine learning approaches in the TS50 dataset, including 3D CNN-based methods (ProDConn (Zhang et al., 2020) , DenseCPD (Qi & Zhang, 2020) ), sequential models (Wang's model Wang et al. (2018) , SPIN (Li et al., 2014 ), SPIN2 (O'Connell et al., 2018) ) and GNN-based method (Strokach et al., 2020) . We also compare with classical statistic methods (Schaap et al., 2001; Cheng et al., 2019) . Results. As shown in Table 3 , our models achieve significant improvement over other methods on the CATH dataset. This result also suggests that the the short-chain and single-chain subsets are more challenging, which is consistent with the result in (Ingraham et al., 2019) . As shown in Table 5 , our models also gain superior performance. These results suggest that adopting OA-GNNs is a more effective way of designing valid protein sequences.

5.4. MODEL QUALITY ASSESSMENT

Datasets. Model quality assessment (MQA) evaluates the quality of predicted protein structures (Kwon et al., 2021) . The goal is to fit a model approximating the numerical metric used to compare the predicted 3D structure with the native 3D structure. For this task, We randomly split the targets in CASP5-CASP10 (Moult et al., 2014) and sample 50 decoys for each target for generating the training and validation sets and use the CASP11 stage 2 proteins as the test set to ensure no similar structures are involved. In total, there are 508 proteins for training, 56 for validation, and 85 targets with 150 decoys for the test. Metrics. We regress the global GDT TS score (Zemla, 2003) , which evaluates the quality of decoys. We adopt the correlation between the predicted and real GDT TS values to measure the performance. We use three statistical correlation metrics: Pearson's correlation r, Spearman's ρ, and Kendall's τ . We calculate the correlation for each target and average all the correlations over all the targets. We also calculate the Global correlations by taking the union of all decoy sets without considering the targets (Pagès et al., 2019) . Baselines. VoroMQA (Olechnovič & Venclovas, 2017 ) leverages potential model with protein contact maps, while RWplus (Zhang & Zhang, 2010) relies on physical energy terms. SBROD (Karasikov et al., 2019) uses hand-crafted features. Proq3D (Uziela et al., 2017) employs a FCNs for regression. 3DCNN (Derevyanko et al., 2018) and Ornate (Pagès et al., 2019) apply 3D CNNs for extracting meaningful features. GraphQA (Baldassarre et al., 2021) , DimeNet (Gasteiger et al., 2019b) and GVP-GNN are the most similar models as ours which adopted GNN-based methods on protein graphs. Results. As shown in table 4, our models outperform other methods on both average and global metrics except for Spearman's correlation ρ where we achieve the second-best, demonstrating our models can not only evaluate the quality for the same protein but also works well across different proteins in a more general way.

5.5. ABLATION STUDIES

In the end, we conduct ablation experiments to study the contribution of different modules of our model on the tasks of CPD (table 3, 5) and RES (table 2 ). More specifically, based on OA-GCN, we replace all the directed weights with the regular linear layer with scalar weights (No DW), or remove the Interaction Module (No Int), or break the equivariance by canceling coordinate transformations in the step of message passing (No Equi). In general, all three components of our model are important since removing each of them results in a significant performance drop. According to the performance gain, the directed weights are the most important module for the CPD task, as the orientational features are essential for the protein to fold to a given structure. For Residue Identity, however, equivariance message passing is the most important component, indicating the necessity of rotation equivariance in learning the representation of local protein structures.

6. CONCLUSION

For the first time, we generalize the scalar weights in neural networks to 3D directed vectors for better perceiving geometric features like orientations and angles, which are essential in learning representations from protein 3D structures. Based on the directed weights ⃗ W, we provide a toolbox of operators as well as a ⃗ W-Perceptron Unit encouraging efficient scalar-vector feature interactions. We also enforce global SO(3)-equivariance on a message passing paradigm using the local orientations naturally defined by the rigid property of each amino acid residue, which can be used for designing more powerful networks. Our integrated Orientation-Aware Graph Neural Networks achieve comparably better performances on multiple biological tasks in comparison with existing state-of-the-art methods. Limitations and future work. While our method shows better awareness to fine geometric details, there remains a huge space for non-linearity designs upon geometric representations. In the future, we will extend our framework to other 3D learning tasks like small molecules or point clouds by learning local rotation transformations and using directed vector weights.

A PROOF OF ROTATION EQUIVARIANCE

A function f taking a 3D vector x ∈ R 3 as input is rotation equivariance, if applying ant rotation matrix R ∈ R 3×3 on x leads to the same transformations of the output f (x). Formally, f : R 3 → R 3 is rotation equivariance by fulfilling: R(f (x)) = f (Rx) For notation consistency, we consider each row of the matrix to be an individual 3D vector and use right-hand matrix multiplication here. When performing equivariant message passing on protein graph for node i with local rotation matrix O i , we first transform the vector representations of its neighbors from global reference to its local reference, that is, for a particular neighbor node j with vector feature V j ∈ R C×3 , apply our DWNN layers f , and transform the updated features back to the global reference. We set only one neighbor node u j for u i for simplicity.The updated vector representation V i for node i is V i = [f (V j O T i )]O i If we apply a global rotation matrix R to all vectors in the global frame, the local rotation matrix O i will be transformed to O ′ i = O i R, and V j to be V ′ j = V j R, now the output V ′ i is V ′ i = [f (V ′ j O ′ T i )]O ′ i (27) = [f (V j RR T O T i )]O i R (28) = [f (V j O T i )]O i R And if instead, we directly apply the rotation matrix to original output V i , we got V ′ i = [f (V j O T i )]O i R (30) = V i R In other words, rotating the input leads to the same transformations to output, so we can preserve rotation equivariance for vector representations in this way. Note that scalar features remain invariant when applying global rotation, so our message-passing paradigm with global and local coordinate transformations is rotation equivariance.

B EXPERIMENT DETAILS

We represent a protein 3D structure as an attributed graph, with each node and edge attached with scalar and vector features that are geometric-aware. We implement our DWNN in the equivariant message passing manner, with 3 layer Directed Weight Perceptrons for all tasks.

B.1 PROTEIN FEATURES

In this paper, we use an attributed graph G = (V, E) to represent the protein structure, where each node corresponds to one particular amino acid in the protein with edges connecting its k-nearest neighbors. Here we set k = 30. The node features V = {v 1 , .., v N } and edge features E = {e ij } i̸ =j are both multi-channel scalar-vector tuples with scalar features like distances and dihedral angles and vector features like unit vectors representing particular orientations. A node v i represents the i-th residue in the protein with scalar and vector features describing its geometric and chemical properties if available. Therefore, a node in this graph may have multichannel scalar-vector tuples (s i , V i ), s i ∈ R 6 or R 26 , V i ∈ R 3×3 as its initial features. • Scalar Feature. The {sin, cos} • {ψ, ω, ϕ}. Here {ψ, ω, ϕ} are dihedral angles computed from its four backbone atom positions, Cα i-1 , N i , Cα i , N i+1 . • Scalar Feature. A one-hot representation of residue if the identity is available. • Vector Feature. The unit vectors in the directions of Cα i+1 -Cα i and Cα i-1 -Cα i . • Vector Feature. The unit vector in the direction of Cβ i -Cα i corresponds to the side-chain directions. The edge e ij connecting the i-th residue and the j-th residue also has multi-channel scalar-vector tuples as its feature (s ij , V ij ), s ij ∈ R 34 , V ij ∈ R 1×3 • Scalar Feature. The encoding of the distance ∥Cα j -Cα i ∥ using 16 Gaussian radial basis functions with centers spaced between 0 to 20 angstroms. • Scalar Feature. The positional encoding of ji corresponding the relative position in the protein sequence. • Scalar Feature: The contact signal describes if the two residues contact in the space,1 if ∥Cα j -Cα j ∥ ≤ 8 and 0 otherwise. • Scalar Feature. The H-bond signal describes if there may be a H-bond between the two nodes calculated by backbone distance. • Vector Feature. The unit vector in the direction of Cα j -Cα i .

B.2 MODEL DETAILS

Synthetic Task. We uniformly sample points on the sphere and the ratio of positive and negative samples is 1:1. We consider the length of each vector as one channel scalar feature ∈ R and the vector itself as one channel vector feature ∈ R 1×3 . The VNN and GVP models in this experiment are set to 3 layers. And our DWP is only 1 layer, consisting 1 Directed Linear Module, 1 NonLinear Module, and 1 Directed Interaction Module. We also train a 3-layer MLP by concatenating the scalar and vector feature as a four-channel scalar feature ∈ R 4 as input. Specifically, the parameter counts of VNN, GVP, MLP, DWP is 24,28,24 and 24. We use Binary CrossEntropy Loss for this two-class classification task. For MLP, the scalar and vector features are concatenated along the channel, which means the feature of each point is a vector in R 4 . For GVP and VNN, the input scalar and vector are represented by separate neural network components. For all models we trained in the protein-related task, we use 128 scalar channels and 32 vector channels for each node's hidden representations, 64 scalar channels, and 16 vector channels for each edge's hidden representations. There is 4 message passing updations in implemented models, where each passing layer consists of 3 stacked DWPs. In total there are about 2.9 million parameters of our models. CPD. We train our model in a BERT-style recovery target (Strokach et al., 2020) , more specifically, we dynamically mask 85% residues of the whole protein and let the model recover them using structure information from its neighbors using CrossEntropy loss. During testing, we mask all the residues and recover the whole protein sequence in one forward pass, we also try to iteratively predict one residue per forwarding pass. MQA. Because MQA is a regression task, we train our model using MSE loss. When testing, we compute the different types of correlations between predicted scores and ground truth.

RES.

Residue identity requires the model to classify the center residue from a local substructure of the protein. We construct a subgraph of each local substructure according to Section B.1. To guarantee equitable comparison, we retrain all the methods based on our protein graph from scratch.

B.3 TRAINING DETAILS

For the synthetic task, we train each model for 1000 epochs with a learning rate of 1e -3 and plot the training loss for evaluation. We train our models with learning rate 3e -4 for CPD and 2e -4 for MQA, a dropout rate of 10% (Srivastava et al., 2014) , and Layernorm paradigm. We train all the models on NVIDIA Tesla V100 for 100 epochs for each task. Our model uses 4-layer message passing, where each message passing consists of 3-layer DWP. The total model consists of about 2.9 million parameters. equivariance. Our proposed framework focuses on its computation in the latent space together with our vector-based feature representations, while many existing works only apply rotations to the primal space of explicit geometric/physical properties (e.g. simple cartesian coordinates). It also provides better flexibility over message-passing operators by eliminating the requirements for specific feature-update arithmetics -this also serves as an adaptor for plugging our designed feature representations into many existing GNN models while guaranteeing equivariance. Among the EGNNs, we could not agree that our model is a simplified model from others. Specifically, some of these EGNNs could be considered a special case of our framework. For instance, E(n) GNN (Satorras et al., 2021) combines node coordinates weighted by hidden representations, which could be recognized as linearly updating of simple vector features (coordinate) and unit matrix transformations (in analogy to our back and forth transformations). Additionally, to the best of our knowledge, there is no previous work that integrates vector-based features and directed weighted neural networks into local-global transformation frameworks. We apply transformations on hidden scalar-vector features back and forth for better representations under equivariance, which turns out to show decent performances in the protein learning tasks.

D.3 COMPARISON TO OTHER BACKBONE-ORIENTATION MODELS

Using sin, cos . . . ϕ, ψ, ω as features are popular techniques in protein learning tasks (Ingraham et al., 2019; Jing et al., 2021; Ganea et al., 2021) , but our models can have a more powerful ability to make use of these features. The meaning of Orientation-Aware GNNs is not to use torsional angles as the node and edge features, but for reasoning on them. Our real contribution is to design a new type of neural network, which has good capability in sensing these geometric features (e.g. angles, orientations) as well as capturing complex interactions. The synthetic experiment gives an intuitive example of our claims, where our designed directed weight perceptrons have more powerful in sensing and handling angular-related characteristics. Here we share an even simpler example: Assume the input contains two 3D vectors v 1 and v 2 described by their coordinates [x 1 , y 1 , z 1 , x 2 , y 2 , z 2 ], the problem we solved is that how should we let the model know the angle between v 1 and v 2 . The most straightforward intuition is that if the weights are directed vectors, then we could count on the inner products between weights and input vectors to capture the angle information. Furthermore, if one wants to retrieve more complex geometric objects, e.g. normal vectors, dihedral angles, or parallel surfaces based on input coordinates, our employed cross-product operation can easily solve that. As we use scalar-vector tuple hidden representations for nodes and edges, and directed weights with geometric-meaningful operators are learnable, our models could perform more complex geometric reasoning ability and modeling capability beyond angles, especially approximating nonlinear functions on vector representations. The most straightforward intuition is that if the weights are directed vectors, then we could count on the inner products between weights and input vectors to capture the angle information. Furthermore, if one wants to retrieve more complex geometric objects, e.g. normal vectors, dihedral angles, or parallel surfaces based on input coordinates, our employed cross-product operation can easily solve that. As we use scalar-vector tuple hidden representations for nodes and edges, and directed weights with geometric-meaningful operators are learnable, our models could perform more complex geometric reasoning ability and modeling capability beyond angles, especially approximating nonlinear functions on vector representations. The motivation and function of residue orientations are also different from prior works. Instead of using the angles (orientations) as input features, residue backbone orientations define the coordinate frame of the whole system and are used in the equivariant message passing part for coordinate transformation. Rigid orientations play the role of bridging local and global coordinates for equivariance in our message passing part. E EXPLANATIONS TO SOME MODULES E.1 ABOUT THE s ′ dot For the s ′ dot features, it can be interpreted as the summation of the channel-wise dot product between the input features (channels of vectors) and the directed weights (a matrix of vectors, so a 3-dimensional tensor). Let us consider the i-th number in the output feature, it can be formulated as: s ′ i = jk W ijk v jk = j (w ij ) T v j Notice that both are 3D vectors, **the dot product can essentially reflect the cosine value between them after normalization**. Essentially, our method utilizes the dot product operation between the directed weights and the input coordinate features, and more importantly, the weight vectors are learnable. We also recommend you pay attention to other operations we introduced for reasoning in complex geometric relations and encouraging efficient scalar-vector feature interactions. For instance, the cross-product operation can create new orthogonal orientations; the interaction modules can facilitate feature blending.

E.2 ABOUT THE SCALR-LIFTING PART

The initial motivation of scalar lifting is to enable scalar-vector interactions, which are not well addressed in previous works (for instance, VNN (Deng et al., 2021) and GVP (Jing et al., 2020) can only perform vector-to-scalar mappings with a very limited choice of operations). Unprojecting from lower dimensions to higher dimensions is indeed non-bijective on the test data themselves, but it is a widely adopted approach like the diluted convolution or the transposed convolution that succeeds in the projection onto higher dimensional features. Mapping from high dimensions to lower space and then projecting back is also a popular technique in FCNs (Kenton & Toutanova, 2019; Jumper et al., 2021) , Generative Models (Wang et al., 2014) , and Manifold Learning, which can be regarded as feature compression and information bottleneck. Therefore, though not unique, the learnable directed weights endow the model with the expressiveness to recover useful information hopefully after training on many data. More intuitively, the scalar lifting operator generates fixed directional vectors with different lengths during inference, which can potentially provide more information during the next round of feature processing and facilitate information flow between scalar and vector features. The scalar lifting module serves to pass scalar information to the vector representation in our model. Though we give a geometric interpretation of the intuition, our directed weight is learnable, the operation of the model is complex and its final geometric meaning is non-linear, which is difficult to define simply. The function of the mapping is learned according to the needs of the downstream task and is not limited to reconstruction or compression. From the experimental results, the network we designed does achieve excellent performance, demonstrating our strong geometric perception and representation capabilities.



As there is no canonical training set for TS50, we follow prior methods to use protein sequences with less than 30% similarity with the TS50 test set from the CATH training set for training(Jing et al., 2020;Qi & Zhang, 2020)



8 9 G C l e 8 c g j + y P r 4 B I 5 W b N Q = = < / l a t e x i t > d(C↵, C↵)

H H d L 6 e 0 s b m 1 v V P e r e z t H x w e V Y 9 P u j p O F a E d E v N Y 9 Q O s K W e S d g w z n P Y T R b E I O O 0 F 0 7 v c 7 z 1 R p V k s H 8 0 s o b 7 A E 8 l C R r D J p W E S s V G 1 5 t b d B d A 6 8 Q p S g w L t U f V 7 O I 5 J K q g 0 h G O t B 5 6 b G D / D y j D C 6 b w y T

a 9 8 d C o t W 6 L e s p w B u d w C R 5 c Q w v u o Q 0 d I B D B M 7 z A q z N 3 3 p x 3 5 2 O 5 W n K K m 1 P 4 A + f z B 2 E e l O 4 = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " H

+ F q s l p 7 g 5 h j 9 w P n 8 A R 0 6 V h A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 E 8 0 o k e Q O 5 A 7 d q h H W E W G x d q j J j s = " > AA A B / X i c b V C 7 S g N B F L 0 b X z G + o p Y 2 g 0 G w C r s i a h m w s Y x g H p I s Y X Y y m 4 y Z x z I z K 4 Q l + A u 2 2 t u J r d 9 i 6 5 c 4 S b b Q x A M X D u f c y 7 m c K O H M W N / / 8 g o r q 2 v r G 8 X N 0 t b 2 z u 5 e e f + g a V S q C W 0 Q x Z V u R 9 h Q z i R t W G Y 5 b S e a Y h F x 2 o p G 1 1 O / 9 U i 1 Y U r e 2 X F C Q 4 E H k s W M Y O u k Z lc l P D W 9 c s W v + j O g Z R L k p A I 5 6 r 3 y d 7 e v S C q o t I R j Y z q B n 9 g w w 9 o y w u m k 1 E 0 N T T A Z 4 Q H t O C q x o C b M Z t 9 O 0 I l T + i h W 2 o 2 0 a K b + v s i w M G Y s I r c p s B 2 a R W 8 q / u t F Y i H Z x l d h x m S S W i r J P D h O O b I K T a t A f a Y p s X z s C C a a u d 8 R G W K N i X W F l V w p w W I F y 6 R 5 V g 0 u q u e 3 5 5 X a f V 5 P E Y 7 g G E 4 h g E u o w Q 3 U o Q E E H u A Z X u D V e / L e v H f v Y 7 5 a 8 P K b Q / g D 7 / M H M 8 6 W C w = = < / l a t e x i t > Bias Scale < l a t e x i t s h a 1 _ b a s e 6 4 = " H C W Y w a 4 Y 2 O R m F S O C B N J W M c 7 r 4 K 0

5 g 4 < / l a t e x i t > sigmoid(k • k) < l a t e x i t s h a 1 _ b a s e 6 4 = " O + F 2 x Z V X S 3 t r 0 R 3 h j T 2 4 + u H C i 9 Q = " > A A A B + 3 i c b V D L T g J B E J z F F + I L 8 e h l I z H B C 9 k 1 R D 0 S v X j w g M Y F E t i Q 2 d l e m D D 7 y E y v g W z 4 F S 8 e N M a r P + L N v 3 F 4 H B S s p J N K V X e 6 u 7 x E c I W W 9 W 3 k 1 t Y 3 N r f y 2 4 W d 3 b 3 9 g + J h q a n i V D J w W C x i 2 f a o A s E j c J C j g H Y i g Y a e g J Y 3 v J n 6 r S e Q i s f R I 4 4 T c E P a j 3 j A G U U t 9 Y q l L s I I s w e 4 c y a V L v N j P O s V y 1 b

D U G 4 2 u 3 1 i n m r F N 4 N Y 6 o r Q n K m / J z I a K j U O P d 0 Z U h y o Z W 8 q / u d 1 U g y u 3 I x H S Y o Q s f m i I B U m x u Y 0 C N P n E h i K s S a U S a 5 v N d m A S s p Q x 1 X Q I d j L L 6 + S 5 n n V v q j W 7 m v l + v U i j j w 5 J i e k Q m x y S e r k l j S I Q x g Z k W f y S t 6 M i f F i v B s f 8 9 a c s Z g 5 I n 9 g f P 4 A o + y U M A = = < / l a t e x i t >

< l a t e x i t s h a 1 _ b a s e 6 4 = " l S B + H K s 3 2 D e u f 6 O B d m P I 5 t C S y p Y = " > A A A C I X i c d V D L S g M x F M 3 4 t r 6 q L t 0 E i + h q y J Q + V B A E N y 4 V 7 A O c U j J p p g 0 m M 0 N y R y z D / I I / 4 S + 4 1 b 0 7 c S f u / B L T W k F F D w Q O 5 9 y b e + 8 J E i k M E P L q T E 3 P z M 7 N L y w W l p Z X V t e K 6 x t N E 6 e a 8 Q a L Z a z b A T V c i o g 3 Q I D k 7 U R z q g L J W 8 H V y c h v X X N t R B x d w D D h H U X 7 k Q g F o 2 C l b n H P D 1 R m 8 m 7 m A 7 + B z H 6 T 5 7 v 4 8 A j 7 i s I g C L N W P q k o l o h b J g d e b R 8 T t 1 q t k 4 O y J f V q p V Y n 2 H P J G C U 0 w V m 3 + O 7 3 Y p Y q H g G T 1 J h L j y T Q y a g G w S T P C 3 5 q e E L Z F e 3 z S 0 s j q r j p Z O O L c r x j l R 4 O Y 2 1 f B H i s f u / I q D J m q A J b O d r T / P Z G 4 p 9 e o H 5 N h n C / k 4 k o S Y F H 7 H N w m E o M M R 7 F h X t C c w Z y a A l l W t j d M R t Q T R n Y U A s 2 l K / L 8 f + k W X a 9 m l s 5 r 5 S O y 5 N 4 F t A W 2 k Z 7 y E N 1 d I x O 0 R l q I I Z u 0 T 1 6 Q I / O n f P k P D s v n 6 V T z q R n E / 2 A 8 / Y B P R e l B w = = < / l a t e x i t > s 0 lin := Ws < l a t e x i t s h a 1 _ b a s e 6 4 = " L g b

J b 3 b s T t y 7 d + i W m j 4 W 2 H r h w O O d e 7 r 0 H x 5 x p 4 7 p f T m 5 l d W 1 9 I 7 9 Z 2 N r e 2 d 0 r 7 h + 0

y 5 W b S q l W n c e T B 0 f g B J w D D 1 y B G r g G D d A E B D y C Z / A C X p 0 n 5 8 1 5 d z 5 m r T l n P n M I / s D 5 / A E N J 6 A X < / l a t e x i t > s h 2 R C 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " A I c T t T yh T j / X R h g K m Z p j t R h 7 N p 4 = " > A A A C Q X i c b V D L S g M x F M 3 U d 3 1 V X b o J F l E Q y o y I u h E E N y 4 V 7 A M 6 t W T S T C c 0 m Q z J H a U M 8 0 v + h L / g x o U u 3 L k T t 2 7 M t F 1 o 9 U D g 3 H P v 4 d 6 c I B H c g O s + O 6 W Z 2 b n 5 h c W l 8 v L K 6 t p 6 Z W O z Y V S q K a t T J Z R u B c Q w w W N W B w 6 C t R L N i A w E a w a D i 6 L f v G P a c B X f w D B h H U n 6 M Q 8 5 J W C l b u X S D 2R m 8 j 1 8 h s f s N s I H 2 F f W o 3 k / A q K 1 u s 9 8 S S A K w q y Z 5 9 2 s m G v k O f Z p T w E e V 7 d R t 1 J 1 a + 4 I + C / x J q S K J r j q V t 7 8 n qK p Z D F Q Q Y x p e 2 4 C n Y x o 4 F S w v O y n h i W E D k i f t S 2 N i W S m k 4 1 + n O N d q / R w q L R 9 M e C R + t O R E W n M U A Z2 s j j d T P c K 8 d 9 e I K c 2 Q 3 j a y X i c p M B i O l 4 c p g K D w k W c u M c 1 o y C G l h C q u b 0 d 0 4 h o Q s G G X r a h e N M R / C W N w 5 p 3 X D u 6 P q q e H 0 7 i W U T b a A f t I w + d o H N 0 i a 5 Q H V H 0 g J 7 Q C 3 p 1 H p 1 3 5 8 P 5 H I + W n I l n C / 2 C 8 / U N 1 x O x 5 g = = < / l a t e x i t >

t e x i t s h a 1 _ b a s e 6 4 = " W 5 B i 3 C D n u C 6 C 2 a A 1 5 W S 6 l m f 3 x f o = " > A A A C M H i c d V B N S w M x E M 3 6 W e t X 1 a O X Y B E 8 l W 1 t 1 a P Q i 8 c q 9 g O 6 t W T T b B v M J k s y q 5 R l f 4 p / w r / g V e 9 6 E j 3 6 K 8 y 2 F V R 0 I O T N e z P M z P M j w Q 2 4 7 o s z N 7 + w u L S c W 8 m v r q 1 v b B a 2 t l t G x Z q y J l V C 6 Y 5 P D B N c s i Z w E K w T a U Z C X 7 C 2 f 1 3 P 9 P Y N 0 4 Y r e Q n j i P V C M p Q 8 4 J S A p f q F Y 0 9 Z W f P h C I j W 6 j b x Q g I j P 0 j a a Y o 9 L v E 0 9 5 O L 9 C q p Y w 9 4 y A w + T P u F o l u q u T Y q 2 C 1 l f 7 W G Z 0 w N l y e M 6 x b R L B r 9 w r s 3 U D Q O m Q Q q i D H d s h t B L y E a O B U s z X u x Y R G h 1 2 T I u h Z K Y u f 0 k s m B K d 6 3 z A A H S t s n A U / Y 7 x 0 J C Y 0 Z h 7 6 t z N Y 1 v 7 W M / F P z w 1 + T I T j p J V x G M T B J p 4 O D W G B Q O H M P D 7 h m F M T Y A k I 1 t 7 t j O i K a U L A e 5 6 0 p X 5 f j / 0 G r U i o f l a r n 1 e J p Z W Z P D u 2 i P X S A y u g Y n a I z 1 E B N R N E d e k C P 6 M m 5 d 5 6 d V + d t W j r n z H p 2 0 I 9 w P j 4 B M p K r E Q = = < / l a t e x i t > ! W 2 R C⇥3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 7 D B A T 3 i R C A P 0 7 g / h o l z h b f k p

j a p X z p A e 9 7 z Q W Q 5 c s 5 l x m k s M B p c 5 4 o G w n I G c B E C Y F W F 3 z E b E E g Y h 7 U o I 5 e V y / D 6 4 b N Y b h / X W e a t 2 0 p z H s 4 J 2 0 B e 0 j x r o C J 2 g b + g M d R B D P 9 A 9 e k C / o p / R U / Q 7 + j N r X Y j m M 9 v o v 4 q e / w I V D q + J < / l a t e x i t > ! Ws 2 R C 0 ⇥3 < l a t e x i t s h a 1 _ b a s e 6 4 = " s Y 7 e j S H s m i b e F F x x f g J U 5 T 8 Y F t Q = " > A A A C Q 3 i c d V D L a h s x F N W 4 j y R O H 0 6 7 7 E b U B G d l x o 7 d Z G n w p n S V l t o O Z B w j y R p b W I 9 B u p N g x P x T f 6 K / U O i q I d v u S r a F a u w p t K G 9 I H Q 4 5 1 7 O v Y d m U j i I 4 6 9 R 7 c H D R 4 9 3 d v f q + 0 + e P n v e O H g x d i a 3 j I + Y k c a e U + K 4 F J q P

Figure 2: Model Details. A 1-layer DWP consists of three following modules. Both the input, hidden and output are tuples. (a) Directed Linear Module applies multiple geometric operations to update scalar and vector features in four different ways with normal and directed weights. (b) Non-Linearity Module employs ReLU and sigmoid functions for scalar and vector features. (c) Directed Interaction Module updates the features by using one another as updating parameters after non-linearity in another way.

Figure 3: a). The synthetic study. The vector in red is the anchor used to calculate the ground truth labels. Vectors falling into the cone are positive samples, whereas outside points are negative. b). The validation Binary Cross Entropy loss of compared methods.

Results of different RES methods on ATOM3D.

Ablation studies.

Results of different CPD methods on the CATH 4.2.

Results of MQA models on CASP 11 stage 2.

Results of structure biology methods on the TS50.

annex

node u i ∈ V has four backbone heavy atom (C u α , C u , N u , O u ), defining a local frame O u as:The local frame O u is a rotation matrix that maps a 3D vector from the local to the global coordinate system. An equivariant message passing paradigm then emerges through transforming node features back and forth between adjacent local frames. Formally, give an amino acid u with hidden representation h = (s, V ), let f l and f g be transformations on the vector feature V from and to the global coordinate system:The message passing update for node u from layer l to layer l + 1 is performed in the following steps, the [ ] notations indicates the concatenation of different hidden representations along the channel-wise dimension:1. Transform the neighbourhood vector representations of u into its local coordinate system O u .2. For each adjacent node v, compute the message m l+1 u on edge (u, v) ∈ E with an multilayer ⃗ W-perceptron F to the aggregated node and edge feature [h l u , h l v , e l uv ] (Equation 18). 3. Update node feature h l u with another multi-layer ⃗ W-perceptron H, and transform it back to the global coordinate system (Equation 19).This paradigm achieves SO(3)-equivariance in a very general sense with no constraints on F, H. The formal proof of equivariance is presented in the Appendix Section A.

4.2. VARIANTS OF ORIENTATION-AWARE GRAPH NEURAL NETWORKS

By integrating the ⃗ W-Perceptrons with equivariant message passing, we propose multiple variants of entire Orientation-Aware Graph Neural Networks to boost the design space for different tasks.OA-GCN. This is the one described and implemented in the previous subsection with Equation 18and Equation 19.OA-GIN. We adopt graph isomorphism operator Xu et al. (2018) with learnable weighing parameter ε for tunable skip connections,OA-GAT. In comparison to GCN and GIN, it is not trivial to incorporate with the Graph Attention Network (GAT) (Veličković et al., 2018) , as residues interact with each other unevenly in proteins.In particular, we define separate attentions for the scalar and vector representationsThe attention scores α s , α v are the softmax values over the inner products of all neighboring sourcetarget pairs defined as the follows:The product for scalars and vectors are standard inner product and Frobenius product, respectively.

B.4 BASELINE DETAILS

We obtain most of our baseline numerical results from other benchmark papers. But because some models haven't been evaluated on some of the datasets we used, we reimplement them following the same model complexity and training hyperparameters as our models.In the synthetic dataset, we download the source code from (Deng et al., 2021) and (Jing et al., 2021) , both of which use CrossEntropy loss. The number of parameters of VNN, GVP, MLP, and DWP is 24, 28, 24, and 24, respectively.For the RES task Table 1 , we implement GCN, GIN, and GAT models, which are trained and evaluated with the same settings as our models. Following the source code from (Townshend et al., 2021; Jing et al., 2021) , we also retrain 3D CNN and GVP-GNN. To make a fair comparison, we keep the model complexity (e.g. hidden dimensions, layer numbers) and hyperparameters (epochs, random seed) of these baselines the same as our models.For the CPD task in Table 3 , we report the result of St-Transformer from (Ingraham et al., 2019) , St-GCN, and GVP-GNN from (Jing et al., 2020) . We modify the source code from Structed-Transformer (Ingraham et al., 2019) to get the result of St-GAT and St-GIN by replacing the original attention layer in the encoder and decoder module with Graph Attention Network and Graph Isomorphism Network. The hidden dimensions and layer numbers are kept the same as our models.In Table 5 , we adopt most of the results from (Jing et al., 2020) except running the St-Transformer model on the TS50 dataset.For MQA in Table 4 , we merge the results of the same task from (Jing et al., 2020) and Structed-Transformer (Ingraham et al., 2019) . As GVP-GNN only reports three of six of our metrics, so we reimplement GVP-GNN and DimeNet on our benchmark. Same as the above model, the hidden dimensions and layer numbers are kept the same as our models.For our ablation models, we concatenate scalar and vector features as input and replace all the directed weights with the regular linear layer with scalar weights to get the No DW model. By removing the Interaction Module lying in the final layer of each perceptron, we obtain the No Int model. We also replace local frames with unity matrices to break the equivariance in the message passing part (No equivariance).In 

C.1 MULTIPLE RUNS ON RES TASK

We conduct different runs on RES dataset for both baseline and our models. As shown in Table 6 , we report the mean and std results of each method across five different random seeds, our models are still doing best among other methods and have stable, robust performance with few deviations. The results have shown that the sequential decoding approach basically caused a degradation of the results, but because we introduced the directed weight and geometric operations, our results are still better than other benchmarks. We have added these results in the supplementary material.C Baselines. We compare with CNN-based models and GNN-based models which learn the protein annotations using 3D structures from scratch. For fair comparison, we also do not include LSTMbased or transformer-based methods, as they all pre-train their models using millions of protein sequences and only fine-tune their models on 3D structures Bepler & Berger (2018) ; Alley et al. (2019) ; Rao et al. (2019) ; Strodthoff et al. (2020) ; Elnaggar et al. (2020) .Results. As shown in Table 8 , our models demonstrate comparable prediction performance in both tasks across different levels of the hierarchy. Classifying the fold and enzyme classes based on the protein structure is one of the key problems in structural biology and our proposed models can be used as new tools to conduct function annotations for new proteins.For FOLD and REACT tasks, all GNN-based baselines use 3D structure information:• Kipf & Welling (2016) and Diehl (2019) construct the input graph using the protein's contact map and use pair-wise distances of residues as edge features. • Baldassarre et al. (2021) use the spatial features including the dihedral angles, surface accessibility, and secondary structure type of each residue's node feature. They also use distances, sequence distances, and bond types for edge features. • Hermosilla Casajus et al. (2021) consider more fine-grained atom-level spatial information, e.g. covalent radius, van der Waals radius, and atom mass, which leads to better performance. • Gligorijević et al. (2021) refer to different types of contact maps and geometric distances for the input of GNN.Despite not using special training techniques, pre-training tasks, and fine-tuning of hyperparameters, our models demonstrate comparable prediction performance in both tasks across different levels of the hierarchy. Especially with the introduction of our directed weight with equivariant message passing, it is a big improvement over the normal GNN. There are several works that explicitly model angles/torsion angles in GNNs. DimeNet (Gasteiger et al., 2019b) uses physical-informed RBF and SBF functions to represent distances and angular information. SphereNet Liu et al. (2021) further employs a 3D meaningful spherical coordinate system to represent molecular torsional angles. However, these methods rely on invariant scalar input features for remaining equivariance, and during convolution, these scalar features (distances, angles) are just used for updating hidden representations and stay the same in the next layer updating.On the technical level, our model also has several differences from other methods when compared with existing studies to consider backbone torsion angles (DimeNet, SpehreNet, etc.), as follows:1. RBF-embedded distances and Trigonometric-embedded angles are adopted in the input feature engineering part (see supplementary material). To better represent the 3D geometries, we also introduce several normal vectors as input node features and edge characteristics. The scalar representations are invariant and vectors are equivariant. But these methods don't consider vector representations and only focus on invariant scalar features.2. We capture fine-grained geometries with the help of vector representations and directed weight perceptrons. And representations are updated layer-by-layer. Our directed weights are learnable to be expandable for complex reasoning capabilities. But these methods only embed angles and distances into edge features during message updating and lack geometry meaningful operations. They also don't have directed weights, which are key contributions to our work.3. We maintain equivariance with the help of local coordinate transformations on vector representations equipped with rigid residue positions. But these methods don't consider equivariant vectors, they only condition on invariant scalars. SphereNet does use local spherical systems, but they are used for calculating relative locations of atoms, seen as a feature engineering part.4. We aim to design a new type of vector neurons for geometric learning and a general message passing paradigm for 3D graphs. We have shown our versatility in any existing graph learning frameworks. However, these methods are designed specifically for molecular graphs and are hard to extend for other 3D point sets.

D.2 COMPARISON TO OTHER EGNN WORKS

Local-global frame transformation is an important technique for 3D object learning (Petrelli & Di Stefano, 2011; Yang et al., 2018; Luo et al., 2022) and is inevitable in the discussions of 3D

