CAFE: CATASTROPHIC DATA LEAKAGE IN FEDER-ATED LEARNING

Abstract

Private training data can be leaked through the gradient sharing mechanism deployed in machine learning systems, such as federated learning (FL). Increasing batch size is often viewed as a promising defense strategy against data leakage. In this paper, we revisit this defense premise and propose an advanced data leakage attack to efficiently recover batch data from the shared aggregated gradients. We name our proposed method as catastrophic data leakage in federated learning (CAFE). Comparing to existing data leakage attacks, CAFE demonstrates the ability to perform large-batch data leakage attack with high data recovery quality. Experimental results on vertical and horizontal FL settings have validated the effectiveness of CAFE in recovering private data from the shared aggregated gradients. Our results suggest that data participated in FL, especially the vertical case, have a high risk of being leaked from the training gradients. Our analysis implies unprecedented and practical data leakage risks in those learning settings.

1. INTRODUCTION

Federated learning (FL) (Chilimbi et al., 2014; Shokri & Shmatikov, 2015) is an emerging machine learning framework where a central server and multiple workers collaboratively train a machine learning model. Most of existing FL methods consider the setting where each worker has data of a different set of subjects but their data share many common features. This setting is also referred to data partitioned or horizontal FL (HFL). Unlike the HFL setting, in many learning scenarios, multiple workers handle data about the same set of subjects, but each has a different set of features. This case arises in financial and healthcare applications (Chen et al., 2020) . In these examples, data owners (e.g., financial institutions and hospitals) have different records of those users in their joint user base, so, by combining their features, they can establish a more accurate model. We refer to this setting as feature-partitioned or vertical FL (VFL). Compared with existing distributed learning paradigms, FL raises new challenges including the heterogeneity of data and the privacy of data (McMahan et al., 2017) . To protect data privacy, only model parameters and the change of parameters (e.g., gradients) are exchanged between server and workers (Li, 2014; Iandola et al., 2015) . Recent works have studied how a malicious worker can embed backdoors or replace the global model in FL (Bagdasaryan et al., 2018; Bhagoji et al., 2019; Xie et al., 2020) . As exchanging gradients is often viewed as privacy-preserving protocols, little attention has been paid to information leakage from public shared gradients and batch identities. In this context, inferring private user data from the gradients has received growing interests (Fredrikson et al., 2015; Hitaj et al., 2017; Melis et al., 2018) . A popular method that was termed deep leakage from gradients (DLG) has been developed in (Zhu et al., 2019) that infers training data in an efficient way without using any generative models or prior information. However, DLG lacks generalizability on model architecture and weight distribution initialization (Geiping et al., 2020) . In Zhao et al. (2020) , an analytical approach has been developed to extract accurate labels from the gradients. Wang et al. (2020) proposed a novel gradient difference as a distance measure to improve recovery accuracy. However, all of them cannot scale up to the large-batch data leakage setting. The contributions of this paper are summarized in the following.

1)

We develop an advanced data leakage attack that we term CAFE to overcome the limitation of current data leakage attacks on FL. CAFE is able to recover large-scale data both in VFL and HFL. 3) The effectiveness and practical risk induced from our data leakage algorithm is justified in the dynamic FL training setting when all parameters in the model are updated every iteration.

2. PRELIMINARY

FL can be categorized into horizontal and vertical FL settings (Kairouz et al., 2019) . In this section, we provide necessary background of FL in this section. Horizontal FL. In HFL, data are distributed among local workers holding the same feature space. Suppose that there are M workers participating in the FL process and the number of samples in the dataset X is N . The dataset is denoted as X := [X 1 , . . . , X m , . . . , X M ] T , where X m ∈ R Nm×p is the local data partitioned to worker m, and p is the dimension of data feature space, N m is the number of data samples partitioned to local worker m, and M m=1 N m = N . Since all local data share the same feature space, each local worker computes the gradients independently and uploads them to the server. The server receives all gradients from each local worker and uses gradient aggregation methods such as FedAvg (Konečnỳ et al., 2016) . Let the parameters of the model as θ and the loss function as L. Then the objective function of HFL can be defined as: min θ 1 N M m=1 L(θ; X m ) with L(θ; X m ) := n∈Nm L(θ; x n ) (1) Vertical FL. Different from HFL, in VFL, each local worker m is associated with a unique set of features. Each data sample x n in dataset X can be written as x n = [x T n1 , . . . , x T nm , . . . , x T nM ] T (2) where x nm ∈ R pm is the data partitioned to worker m and p m is the data dimension in local worker m. The label space {y n } N n=1 can be regarded as a special feature and is partitioned to the server or a certain local worker. Similar to (1), the objective function of VFL can be written as: min θ 1 N N n=1 L(θ; x n1 ; . . . ; x nM )

3. CATASTROPHIC DATA LEAKAGE FROM BATCH GRADIENTS

To realize large-scale data recovery from aggregated gradients, we propose our algorithm named as CAFE: Catastrophic dAta leakage in Federated lEarning. While CAFE can be applied to any type of data, without loss of generality, we use image datasets throughout the paper. We start by providing some intuition on the difficulty of performing large-batch data leakage from aggregated gradients based on the formulation of DLG (Zhu et al., 2019) . Assume that N images are selected as the input for a certain learning iteration. We define the data batch as X = {x n , y n |x n ∈ R H×W ×C , n = 1, 2, . . . , N }, where H, W, C represents the height, the width and the channel number of each image. Likewise, the batched 'recovered data' is denoted by X = {x n , ŷn |x n ∈ R H×W ×C , n = 1, 2, . . . , N }, which have the same dimension as X . Then the objective function is X * = arg min X 1 N N n=1 ∇ θ L(θ, xn, yn) - 1 N N n=1 ∇ θ L(θ, xn, ŷn) 2 Note that in (4), the dimensions of the aggregated gradients is fixed. However, as the N increases, the dimension of X and X rise. When N is sufficiently large, it will be more challenging to find the "right" solution X of (4) corresponding to the ground-truth dataset X . On the other hand, CAFE addresses this large-batch issue by data index alignment for batch data recovery, which can effectively exclude undesired solutions. We discuss a specific example in Appendix A. As a motivating example, Figure 1 compares our proposed attack with DLG on a batch of 40 images. The recovery quality of DLG is far from satisfactory, while CAFE can successfully recover all images in the batch. It is worth noting that because DLG is not effective on large-batch recovery, it is suggested in Zhu et al. (2019) that increasing batch size could be a promising defense. However, the successful recovery of CAFE shows that such defense premise gives a false sense of security in data leakage and the current FL is at risk, as large-batch data recovery can be accomplished.

3.2. CAFE IN VFL

In VFL, the server sends public key to local workers and decides the data index in each iteration of training and evaluation (Yang et al., 2019; Cheng et al., 2019) . During the training process, local workers exchange their intermediate results with others to compute gradients and upload them. Therefore, the server has the access to both the model parameters and their gradients. Notably, CAFE can be readily applied to existing VFL protocols where the batch data index is assigned. Figure 2 gives an overview of CAFE in the VFL setting. The blue part represents a normal VFL paradigm and the red part represents the CAFE attack. Since data are vertically partitioned among different workers, data index alignment turns out to be an inevitable step in the vertical training process, which provides the server (the attacker) an opportunity to control the selected batch data index. Suppose that there are M workers participating FL and the batch size is N . The aggregated gradients can be denoted by Server computes CAFE loss: D(X t ; X t ) and ∇ X t D(X t ; X t ) 13: ∇ θ L(θ, X t ) = 1 N b N b n=1 ∇ θ L(θ, X t n ) with X t n = [x t n1 , x t n2 , . . . , x t nM ]. Server updates the batch data X t with ∇ X t D(X t ; X t )

14:

Server updates the model parameters θ with ∇ θ L(θ, X t ) 15: end for A benign server will perform legitimate computations designed by FL protocol. However, as shown in Figure 2 , a curious server can provide the same legitimate computation as a benign server while simultaneously perform data recovery in a stealthy manner. The server symmetrically generates fake images corresponding to the real ones. Once a batch of original data is selected, the server takes the corresponding fake batch and obtains the fake gradients as ∇ θ L(θ, X t ) = 1 N b N b n=1 ∇ θ L(θ, X t n ) with X t n = [x t n1 , xt n2 , . . . , xt nM ]. Algorithm 1 gives a pseudo code that implements our CAFE attack in VFL cases. The key part in our algorithm is aligning the real data batch indices with the fake ones. We define the squared 2 -norm of the difference between the real and fake aggregated gradients in (7). Since the server has the access to the model parameters, the attacker is able to compute the gradient of fake data from the loss in (7) and optimize the fake data for the purpose of recovering real data.

D(X

t ; X t ) = ∇ θ L(θ, X t ) -∇ θ L(θ, X t ) 2 . (7)

3.3. AUXILIARY REGULARIZERS

In addition to the gradient matching loss in (7), we further introduce two regularization termsinternal representation regularization and total variance (TV) norm. Motivated by (Geiping et al., 2020) , the input vectors of the first fully connected layer can be directly derived from the gradients, we define the real/fake inputs of the first fully connected layer at the tth iteration as Z t / Ẑt ∈ R N ×P and we use 2 norm of their difference as what we call internal representation regularization. To promote the smoothness of the fake images, we assume the TV norm of the real images as a constant, ξ, and compare it with the TV norm of the fake ones, TV( X ). For each image x ∈ R H×W ×C in data batch X t , its TV norm is denoted by TV (x) = c h,w |x h+1,w,c -x h,w,c | + |x h,w+1,c -x h,w,c | . As the result, the loss at the tth iteration D(X t , X t ) can be rewritten as: D(X t ; X t ) = ∇ θ L(θ, X t ) -∇ θ L(θ, X t ) 2 + βT V ( X ) • 1 {T V ( X )-ξ≥0} + γ Z t -Ẑt 2 F , where β and γ are coefficients and 1 {T V ( X )-ξ≥0} is the indicator function. We will provide an ablation study in Section 4.5 to demonstrate the utility of these regularizers. 

3.4. CAFE IN HFL

Similarly, we can apply our CAFE algorithm to HFL as well. Let X t m denote the original batch data taken by local worker m at the tth iteration. The gradients of the parameters at the tth iteration is ∇ θ L(θ, X t ) = 1 M M m=1 ∇ θ L(θ, X t m ) , X t = {X t 1 , X t 2 , . . . , X t m , . . . , X t M }. Symmetrically, we define the batch fake data and fake aggregated gradients as ∇ θ L(θ, X t ) = 1 M M m=1 ∇ θ L(θ, X t m ) , X t = { X t 1 , X t 2 , . . . , X t m , . . . , X t M }. Due to space limitation, we will provide the CAFE algorithm for HFL in Appendix B.

4.1. EXPERIMENT SETUPS AND DATASETS

We conduct experiments on CIFAR-10, CIFAR-100 and Linnaeus 5 datasets in both HFL and VFL settings. All the fake data are initialized uniformly and optimized by the normalized gradient descent method. Our algorithm can recover all the data participating in FL with a relative large batch size (more than 40). Scaling up to our hardware limits, CAFE can leak as many as 2000 images in the VFL setting including 4 workers. Evaluation metrics. To measure the data leakage performance, we introduce peak signal-to-noise ratio (PSNR) value with mean squared error (MSE) defined in ( 11) and ( 12). Higher PSNR value of leaked data represents better performance of data recovery. MSE c (x, x) = 1 HW H i=1 W j=1 [x ijc -xijc ] 2 (11) PSNR(x, x) = 1 C C c=1 20 log 10 (max i,j x ijc ) -10 log 10 (MSE c (x, x)) . ( ) Baseline methods for comparison. We compare CAFE with three other baselines, (i) DLG (Zhu et al., 2019) , (ii) DLG given labels (iDLG) (Zhao et al., 2020) , and (iii) using cosine similarity to compare the real and fake gradients (Geiping et al., 2020) . We implement the original DLG and our CAFE under the same model and optimization methods. We run the DLG on 50 single images respectively and compute the average iterations required to make the PSNR value of a single leaked image above 30. We also compute the expected iteration number per image leakage for our CAFE algorithm. Furthermore, we fix the batch size, and compare the PSNR value obtained by CAFE with that of DLG. We also test the impact of given labels on CAFE by using the techniques in (Zhao et al., 2020) . Moreover, we compare the performance of CAFE under different loss functions: i) replacing the squared 2 norm term with the cosine similarity of two gradients (CAFE with cosine similarity) ii) loss proposed in (Geiping et al., 2020) , which only contains the TV norm regularizer. 

4.2. CAFE IN HFL SETTINGS

In the HFL setting, we use a neural network consisting of 2 convolutional layers and 3 fully connected layers. The number of output channels of the convolutional layers are 64 and 128 respectively. The number of nodes of the first two fully connected layers are 512 and 256. The last layer is the softmax classification layer. We assume that 4 workers are involved in HFL and each of them holds a dataset including 100 images. The batch size of each worker in the training is 10, so there are 40 (10 × 4) images in total participating per iteration. For each experiment, we initialize the fake data using uniform distribution and optimize them for 800 epochs. Figures 4a and 4b show the CAFE loss curves and the PSNR curves on the three datasets in HFL cases. In the loss ratio curve, we set the ratio of current CAFE loss and the initial CAFE loss L(θ,X t ) L(θ,X 0 ) as label y. The PSNR values are always above 35 at the end of each CAFE attacking process, suggesting high data recovery quality (see Figure 1 as an example). Figure 3 shows the attacking process of CAFE on Linnaeus. Under CAFE, PSNR reaches 35 at the 450th epoch where the private training data are completely leaked visually. Comparison with DLG baseline. In Table 1a , we set the batch ratio in CAFE as 0.1 and compare it with DLG under different batch sizes. Clearly, CAFE outperforms DLG thanks to our novel design of large-batch data leakage attack. As shown in Table 1b , DLG cannot obtain satisfactory results when the batch size increases to 40, while CAFE successfully recovers all the images. Due to similarity between iDLG and DLG, the results are in Appendix C. Comparison with cosine similarity. Table 2a shows that the PSNR values are still above 30 if we use cosine similarity instead of 2 norm. The slight drop in PSNR value may result from scaling ambiguity in cosine similarity. There is a performance gap between the loss of CAFE and the loss in Geiping et al. (2020) , which validates the importance of our proposed auxiliary regularizers. Previous works have shown that DLG performs better on an untrained model than a trained one (Geiping et al., 2020) . We also implement CAFE in the 'attacking while learning' mode, in which the FL process is ongoing. When the network is training, the selected batch data and the parameters of the model change every iteration, which may cause the attack loss to diverge. To address this issue, for each real data batch, we compute the real gradients and optimize the corresponding fake data k times. We demonstrate on Linnaeus dataset, set k = 10 and stop CAFE after 1000 iterations (100 epochs). Figure 5 gives the curves of the training loss and the corresponding PSNR value. The PSNR value still can be raised to a relatively high value. It indicates that CAFE can be a practical data leakage attack in a dynamic training environment of FL.

4.5. ABLATION STUDY

We test CAFE under different batch size, batch ratio, and with (without) auxiliary regularizers. PSNR via Batch size. Table 3 shows that the PSNR values still keep above 30 when the batch size increases with fixed number of workers and batch ratio. The result implies that the increasing batch size has little influence on data leakage performance of CAFE. PSNR via Batch ratio. In HFL, 4 workers participate in the learning setting and we fix the amount of data held by each worker as 500. In the VFL case, we implement CAFE on a total of 800 images. In Table 4 , we change the batch ratio from 0.1 to 0.01 while keeping the trained epochs as 800. For both settings, the data leakage performance keeps at the same level. Impact of auxiliary regularizers Table 6 in Appendix D demonstrates the impact of auxiliary regularizers. From Figure 6 , adjusting the threshold ξ prevents images from being over blurred during the reconstruction process. TV norm can eliminate the noisy patterns on the recovered images and increase the PSNR. Images leaked without regularizing the Frobenius norm of the difference between the internal representations Z and Ẑ may lose some details and causes the drop of PSNR.

5. CONCLUSIONS

In this paper, we uncover the risk of catastrophic data leakage in federated learning (CAFE) through an algorithm that can perform large-batch data leakage with high data recovery quality. Extensive experimental results demonstrate that CAFE can recover large-scale private data from the shared aggregated gradients on both vertical and horizontal FL settings, overcoming the batch limitation problem in current data leakage attacks. Our advanced data leakage attack and its stealthy nature suggests practical data privacy concerns in FL and poses new challenges on future defenses.



Figure 1: Illustration of large-batch data leakage on CIFAR-10 from shared gradients in FL

Figure 2: Overview of CAFE in VFL

Figure 3: CAFE on Linnaeus(Epoch: 50, 100, 200, 300, 450, 600, Original data)

Figure 4: CAFE loss ratio and PSNR curves

Figure 5: PSNR and training loss curves

Algorithm 1 CAFE in VFL( regular VFL protocol and CAFE protocol )

CAFE vs DLG

PSNR via Loss

PSNR via Batch size

PSNR via Batch ratio

