PRECISION COLLABORATION FOR FEDERATED LEARN-ING

Abstract

Inherent heterogeneity of local data distributions, which causes inefficient model learning and significant degradation of model performance, has been a key challenge in Federated Learning (FL). So far, plenty of efforts have focused on addressing data heterogeneity by relying on a hypothetical clustering structure or a consistent information sharing mechanism. However, because of the diversity of the real-world local data, these assumptions may be largely violated. In this work, we argue that information sharing is mostly fragmented in the federated network in reality. More specifically, the distribution overlaps are not consistent but scattered among local clients. We propose the concept "Precision Collaboration" which refers to learning from the informative overlaps precisely while avoiding the potential negative transfer induced by others. In particular, we propose to infer the local data manifolds and estimate the exact local data density simultaneously. The learned manifold aims to precisely identify the overlaps from other clients, and the estimated likelihood allows to generate samples from the manifold in an optimal sampling density. Experiments show that our proposed PCFL significantly overcomes baselines on benchmarks and a real-world clinical scenario.

1. INTRODUCTION

Federated learning (FL) has drawn considerable interest from a variety of disciplines in recent years. FL enables collaborative model learning without the need to access the raw data across different clients, which facilitates real-world scenarios where privacy preservation is crucial, such as finance (Yang et al., 2019) , healthcare (Xu et al., 2021) and criminal justice (Berk, 2012) . While it is common that the data samples in local clients are non-i.i.d., existing research reveals that data heterogeneity could lead to non-guaranteed convergence, inconsistent performance and catastrophic forgetting across different clients (Qu et al., 2022) . Despite the promise of FL, an increasing concern is how to effectively handle data heterogeneity before FL is applied in real-world data scenarios. In view of this challenge, an important direction is personalization. A variety of efforts have been made to explore this direction. For example, Ghosh et al. (2020) proposed to cluster the clients according to their sample distributions and build a customized model for each cluster. However, their hypothesis excludes the possibility of knowledge transfer across clusters. Li et al. (2021b) enhanced personalized model learning by introducing a global regularization term, which assumed that the shared knowledge was consistent across all clients. Considering the diversity of local data, in this paper, we study a more flexible and general scenario where the distribution overlaps could be fragmented as shown in Figure 1 (a). Since the informative and ambiguous data shards exist simultaneously in another client, collaborating with all data could do harm to the model learning. An interesting and challenging problem is how to selectively collaborate with the favorable part of other clients in a privacy-preserving way. In this paper, we put forward the concept "Precision Collaboration" for fragmented information sharing. To begin with, we argue that data heterogeneity comes from inconsistent local data manifolds. In particular, the data manifolds of different local clients could share different overlaps. Maximizing the benefit of collaboration requires a precise utilization of these overlaps. Moreover, local data are usually gathered from the manifold based on a particular density. If we want to generate data from the manifold, a precise distribution density approximation for each client could facilitate model learning. 8 4 9 5 8 7 c e 7 3 Y l 6 l y 3  C R Q I k = " > A A A C 1 H i c j V H L S s N A F D 2 N 7 2 e j L t 0 E i + C q p F L R Z d G N G 0 H B 1 o K t Z T J O 2 6 F 5 k U y E U r s S t / 6 A W / 0 m 8 Q / 0 L 7 w z R v C B 6 I Q k Z e e C N T E 5 N T 0 z O z e / s L i 0 X L R X V h t p l C V c 1 H n k R 0 n T Y 6 n w Z S j q S i p f N O N E s M D z x Z k 3 O N D x s y u R p D I K T 9 U w F u 2 A 9 U L Z l Z w p o j p 2 s R U w 1 e f M H x 2 N L 0 Z y 3 L F L b t k 1 y / k J K j k o I V / H k f 2 E F i 4 R g S N D A I E Q i r A P h p S e c 1 T g I i a u j R F x C S F p 4 g J j z J M 3 I 5 U g B S N 2 Q N 8 e 7 c 5 z N q S 9 z p k a N 6 d T f H o T c j r Y J E 9 E u o S w P s 0 x 8 c x k 1 u x v u U c m p 7 7 b k P 5 e n i s g V q F P 7 F + + D + V / f b o W h S 7 2 T A 2 S a o o N o 6 v j e Z b M d E X f 3 P l U l a I M M X E a X 1 I 8 I c y N 8 6 P P j v G k p n b d W 2 b i L 0 a p W b 3 n u T b D q 7 4 l D b j y f Z w / Q W O 7 X K m W d 0 6 q p d p + P u p Z r G M D W z T P X d R w i G P U z c z v 8 Y B H q 2 F d W z f W 7 b v U K u S e N X x Z 1 t 0 b 6 W O W B A = = < / l a t e x i t > M g < l a t e x i t s h a 1 _ b a s e 6 4 = " R T P Q D 6 U Q x A r / a G q C J h r 4 t d h 2 M Y A = " > A A A C 1 H i c j V H L S s N A F D 2 N r 1 p f U Z d u g k V w V R J R d C m 6 c S N U s A 9 o V S b j t A b z I p k I J X Y l b v 0 B t / p N 4 h / o X 3 h n T E E t o h O S n D n 3 n D t z 7 3 V j 3 0 u l b b + W j I n J q e m Z 8 m x l b n 5 h c c l c X m m m U Z Z w 0 e C R H y V t l 6 X C 9 0 L R k J 7 0 R T t O B A t c X 7 T c 6 0 M V b 9 2 I J P W i 8 F Q O Y n E W s H 7 o 9 T z O J F E X 5 l I 3 Y P K K M z 8 / H p 7 n / e G F W b V r t l 7 W O H A K U E W x 6 p H 5 g i 4 u E Y E j Q w C B E J K w D 4 a U n g 4 c 2 I i J O 0 N O X E L I 0 3 G B I S r k z U g l S M G I v a Z v n 3 a d g g 1 p r 3 K m 2 s 3 p F J / e h J w W N s g T k S 4 h r E 6 z d D z T m R X 7 W + 5 c 5 1 R 3 G 9 D f L X I F x E p c E f u X b 6 T 8 r 0 / V I t H D n q 7 B o 5 p i z a j q e J E l 0 1 1 R N 7 e + V C U p Q 0 y c w p c U T w h z 7 R z 1 2 d K e V N e u e s t 0 / E 0 r F a v 2 v N B m e F e 3 p A E 7 P 8 c 5 D p p b N W e 7 t n O y X d 0 / K E Z d x h r W s U n z 3 M U + j l B H Q 8 / 8 E U 9 4 N p r G r X F n 3 H 9 K j V L h W c W 3 Z T x 8 A O S h l g I = < / l a t e x i t > (d) sample from D i < l a t e x i t s h a 1 _ b a s e 6 4 = " b g 5 M 8 q N P Y 7 5 x S 9 i A O 2 x N 7 b f o 0 Q E = " > A A A C y H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R K p 6 L K o C 3 F V w b S F W i V J p 3 V o X k w m S i n d + A N u 9 c v E P 9 C / 8 M 4 Y Q S 2 i E 5 K c O f e e M 3 P v 9 Z K A p 9 K y X g r G z O z c / E J x s b S 0 v L K 6 V l 7 f a K Z x J n z m + H E Q i 7 b n p i z g E X M k l w F r J 4 K 5 o R e w l j c 8 V v H W L R M p j 6 M L O U p Y N 3 Q H E e 9 z 3 5 V E O S d X Y z 6 5 L l e s q q W X O Q 3 s H F S Q r 0 Z c f s Y l e o j h I 0 M I h g i S c A A X K T 0 d 2 L C Q E N f F m D h B i O s 4 w w Q l 0 m a U x S j D J X Z I 3 w H t O j k b 0 V 5 5 p l r t 0 y k B v Y K U J n Z I E 1 O e I K x O M 3 U 8 0 8 6 K / c 1 7 r D 3 V 3 U b 0 9 3 K v k F i J G 2 L / 0 n 1 m / l e n a p H o 4 1 D X w K m m R D O q O j 9 3 y X R X 1 M 3 N L 1 V J c k i I U 7 h H c U H Y 1 8 r P P p t a k + r a V W 9 d H X / V m Y p V e z / P z f C m b k k D t n + O c x o 0 9 6 p 2 r b p / X q v U j / J R F 7 G F b e z S P A 9 Q x y k a c M i b 4 w G P e D L O j M S 4 M 0 Y f q U Y h 1 2 z i 2 z L u 3 w G y p Z E 4 < / l a t e x i t > M i < l a t e x i t s h a 1 _ b a s e 6 4 = " S b 0 h c j 0 Y K l 0 c 0 7 a 1 c u 9 f 5 Z To realize our proposed precision collaboration, we develop a novel framework named PCFL shown in Figure 1 . We assert that the key to precisely collaborative model learning is identifying and utilizing the distribution overlaps scattered in other clients. These overlaps between clients indeed correspond to a specific data manifold region. We propose to infer the local data manifold to identify the overlaps. While it is hard to learn the local manifold from the insufficient data in local clients directly, we firstly infer the underlying manifold M g of the data from all clients, so that the data from all overlapped distributions are utilized for the manifold inference. Then the local manifold M i ⊂ M g of the i-th client could be determined by local data D i as shown in Figure 1 (b) . C R Q I k = " > A A A C 1 H i c j V H L S s N A F D 2 N 7 2 e j L t 0 E i + C q p F L R Z d G N G 0 H B 1 o K t Z T J O 2 6 F 5 k U y E U r s S t / 6 A W / 0 m 8 Q / 0 L 7 w z R v C B 6 I Q k Z 8 4 9 5 8 7 c e 7 3 Y l 6 l y 3 e e C N T E 5 N T 0 z O z e / s L i 0 X L R X V h t p l C V c 1 H n k R 0 n T Y 6 n w Z S j q S i p f N O N E s M D z x Z k 3 O N D x s y u R p D I K T 9 U w F u 2 A 9 U L Z l Z w p o j p 2 s R U w 1 e f M H x 2 N L 0 Z y 3 L F L b t k 1 y / k J K j k o I V / H k f 2 E F i 4 R g S N D A I E Q i r A P h p S From Figure 1 (c), the local data manifold M i is used to identify the beneficial overlaps from other clients. In particular, if a subset of the data from D j lies on M i , this subset is the overlaps between the i-th and j-th clients. To further boost the local model training, we suggest sampling from M i with an optimal sampling probability estimated from local data as shown in Figure 1 (d) , which effectively mitigates the potential distribution discrepancy. We highlight our key contributions as follows: • While existing research studies FL under certain assumptions about the information sharing, we investigate a more general learning scenario where the data sharing a common distribution is fragmented among local clients; • We achieve a more precise collaboration for the federated network by proposing a framework PCFL. Our framework identifies the meaningful overlaps and excludes ambiguous information from other clients, which avoids potential negative transfer; • PCFL could be used to improve other SOTA algorithms in a plug-and-play way. Empirical experiments corroborate that PCFL significantly outperforms all baselines on a series of benchmark data sets and a real-world clinical data set. 

2.2. PERSONALIZED FEDERATED LEARNING

In addition to reaching a global consensus, personalized model learning also attracts widespread concern in FL community, which may boost the flexibility of learned models when adapting to local distributions (Cui et al., 2022; Li et al., 2021b) . Plentiful research have proposed techniques for a



Figure 1: Overview of our proposed PCFL. (a) Fragmented distribution overlaps exist among clients; (b) learn the global data manifold and determine the local manifold for each client; (c) the data from other clients lie on the local manifold M i are identified as informative overlaps; (d) learn a precise local density for synthetic data generation.

FEDERATED LEARNING AND DATA HETEROGENEITY Recent years have witnessed growing attention to federated learning (McMahan et al., 2017), of which several challenges have been concerning topics including communication efficiency (Konečnỳ et al., 2016), privacy (Agarwal et al., 2018) and data heterogeneity (Karimireddy et al., 2020). While data heterogeneity could cause the lack of convergence and the potential of catastrophic forgetting (Qu et al., 2022), there are researchers aiming to tackle the heterogeneity by learning a global model. For example, Li et al. (2020) propose a proximal term to restrict the local updates to be closer to the initial model. Mohri et al. (2019) seek a fair model performance distribution by maximizing the model performance on any arbitrary target distribution. Li et al. (2021a) develop MOON that corrects local training by maximizing the agreements of representation between local and global models. Instead of pursuing a balanced performance distribution, we are interested in achieving the best performance for each client by precisely learning the shared informative overlaps from others.

