A GENERAL DIFFERENTIALLY PRIVATE LEARNING FRAMEWORK FOR DECENTRALIZED DATA Anonymous authors Paper under double-blind review

Abstract

Decentralized consensus learning has been hugely successful that minimizing a finite sum of expected objectives over a network of agents. However, the local communication across neighbouring agents in the network may lead to the leakage of private information. To address this challenge, we propose a general differentially private (DP) learning framework that is applicable to direct and indirect communication networks without a central coordinator. We show that the proposed algorithm retains the performance guarantee in terms of generalization and finite sample performance. We investigate the impact of local privacy-preserving computation on the global DP guarantee. Further, we extend the discussion by adopting a new class of noise-adding DP mechanisms based on generalized Gaussian distributions to improve the utility-privacy trade-offs. Our numerical results demonstrate the effectiveness of our algorithm and its better performance over the state-of-the-art baseline methods in various decentralized settings.

1. INTRODUCTION

Decentralized learning is a process of learning a consensus model using the datasets that are distributed across different agents, such as machines, hospitals, and mobile devices (Shi et al., 2014; Han et al., 2017; Gong et al., 2016; Beyan et al., 2020) . During the process, each local agent (1) keeps its own private data locally; (2) requires no exchange of raw data; and (3) communicates only with its connected agents to train its local model and updates the global parameters directly without a central coordinator. In particular, as medical data are inherently decentralized, i.e., owned or distributed across different institutions, direct sharing or central aggregation of such distributed medical data is increasingly restricted due to either ownership or other regulatory constraints. As a consequence, the advancement of decentralized learning will offer innovative solutions to transform healthcare sectors (Warnat-Herresthal et al., 2021) . Although decentralized learning only requires parallel computation at each local agent and sharing of the estimates or perhaps other intermediate parameters (auxiliary variables) with connected neighbouring agents, past experience has demonstrated the possibility of privacy leakage in the process: the attacker can still recover sensitive information from local communications (e.g., Fredrikson et al. 2015; Shokri et al. 2017) . One defence procedure is to adopt a private variant of the learning algorithm using Differential Privacy (DP) to secure the training process. Very few DP algorithms focus on decentralized learning systems, with the exception of recent works in Xu et al. (2022); Yu et al. (2021a); Huang & Gong (2020) . However, when introducing perturbation into the iterative learning process, these earlier methods only focus on achieving (ϵ, δ)-DP guarantee for each agent. Due to the communications with neighbouring agents during the iterative process, the overall privacy guarantee of the algorithm is no longer (ϵ, δ)-DP. Importantly, it is unclear how one can split the privacy budgets among all the agents in order to achieve a global (ϵ, δ)-DP guarantee for the algorithm when using these earlier methods. Finally, these existing methods consider a standard Gaussian noise-adding mechanism. The added unbounded noise could lead to unstable results, which can severely affect the learning efficiency and degrade the performance of the trained global model (Farokhi, 2022) . This paper aims to provide a unified solution to address these issues and discuss the theoretical guarantees of the proposed algorithm. In the setting of centralized learning for distributed data, a handful number of papers have studied how to integrate privacy-preserving techniques, such as DP, into the training process (Jayaraman et al., 2018; Li et al., 2022a; Guo et al., 2020; Cai et al., 2018; Li et al., 2022b; Huang et al., 2020; Cao et al., 2021) . For example, Jayaraman et al. (2018) proposed DP algorithms for convex problems where ensuring the information obtained from each local model satisfies DP guarantees before being aggregated in a central coordinator; Li et al. (2022b) proposed a unified centralized learning framework to ensure DP guarantees for each local agent for a general class of non-convex problems. However, these algorithms cannot be directly adapted to solve our problem; they focus on the setting where there exists a central coordinator that is responsible for aggregating information obtained from each local agent. There have been very few recent developments in decentralized learning algorithms with DP guarantees. We are aware of only three recent works in the literature. Among them, Huang & Gong ( 2020) is the first one; they proposed a DP Alternating Direction Method of Multipliers (ADMM) algorithm for a wide range of convex learning problems, where they perturb the objective function before solving the minimization associated with the local dataset at each local agent. More recently, Yu et al. (2021a) proposed a DP decentralized stochastic gradient descent (SGD)-based algorithm by perturbing the intermediate parameter updates at each local agent before communicating the perturbed parameter updates with its connected neighbouring agents; Xu et al. ( 2022) proposed a blockchain-enabled decentralized DP learning algorithm through gradient perturbation. However, these gradient-based methods impose restrictions on the objectives, such as smoothness, and therefore have limited application in broader settings. In contrast, we don't restrict ourselves to using gradient descent to find the minimums of the target objective functions. Instead, by using operator theory, we solve the optimization problem by defining a suitable operator or mapping such that the fixed points are the solutions to the original problem; in other words, we consider a broader generic problem of finding a fixed point of averaged iteration of a nonexpansive mapping. Under such an operator theoretical framework, the SGD and ADMM algorithms previously considered in Huang & Gong (2020); Yu et al. (2021a); Xu et al. ( 2022) can be considered special cases of our proposed generic algorithm.

1.2. OUR CONTRIBUTIONS

In this paper, we propose a general framework of a decentralized learning algorithm with DP guarantee, referring to Differentially Private decentralized Krasnosel'skiǐ-Mann iteration (DP-dKM). Our contributions are summarized as follows. 1. Built on the Krasnosel'skiǐ-Mann(KM) iteration (Krasnosel'skii, 1955; Mann, 1953) 3. To the best of our knowledge, we provide the first decentralized learning algorithm that can achieve any desired overall DP guarantee while existing works rely on local DP mechanisms and therefore have no control over the overall privacy cost beforehand. And this is the first work that provides theoretical guarantees on the generalization and finite sample performance of the proposed algorithm. 4. To further optimize the privacy and utility trade-offs, we propose a class of truncated generalized Gaussian noise-adding DP mechanisms, which allows one to achieve significantly higher utility under the same level of DP guarantee. 5. Empirically, we conduct comprehensive experiments to demonstrate that our approach outperforms prior works in various decentralized settings characterized by different communication network diagrams.



We obtain an upper bound of the global sensitivity of the intermediate updates until the fixed iteration step; by injecting enough noises calibrated according to this upper bound, we can achieve a global (ϵ, δ)-DP guarantee without worrying about splitting the total privacy costs among different agents.

