FEDERATED SEMI-SUPERVISED LEARNING WITH DUAL REGULATOR

Abstract

Federated learning emerges as a powerful method to learn from decentralized heterogeneous data while protecting data privacy. Federated semi-supervised learning (FSSL) is even more practical and challenging, where only a fraction of data can be labeled due to high annotation costs. Existing FSSL methods, however, assume independent and identically distributed (IID) labeled data across clients and consistent class distribution between labeled and unlabeled data within a client. In this work, we propose a novel FSSL framework with dual regulator, FedDure, to optimize and customize model training according to specific data distributions of clients. FedDure lifts the previous assumption with a coarse-grained regulator (C-reg) and a fine-grained regulator (F-reg): C-reg regularizes the updating of the local model by tracking the learning effect on labeled data distribution; Freg learns an adaptive weighting scheme tailored for unlabeled instances in each client. We further formulate the client model training as bi-level optimization that adaptively optimize the model in the client with two regulators. Theoretically, we show the convergence guarantee of the dual regulator. Empirically, we demonstrate that FedDure is superior to the existing methods across a wide range of settings, notably by more than 12% on CIFAR-10 and CINIC-10 datasets.

1. INTRODUCTION

Federated learning (FL) is an emerging privacy-preserving machine-learning technique (McMahan et al., 2017) , where multiple clients collaboratively learn a model under the coordination of a central server without exchanging private data. Edge devices like mobile phones have generated and stored a large amount of private data. Centralizing these data could lead to data privacy issues (Voigt & Von dem Bussche, 2017) . Federated learning is a decentralized learning paradigm to leverage these data and has empowered a wide range of applications in many industries, including healthcare (Kaissis et al., 2020; Li et al., 2019 ), consumer products (Hard et al., 2018; Niu et al., 2020) , and public security (Zhuang et al., 2022) . The majority of existing FL works (McMahan et al., 2017; Wang et al., 2020; Li et al., 2021a) assume that the private data in clients are fully labeled, but the assumption is unrealistic in real-world federated applications as annotating data is time-consuming, laborious, and expensive. To remedy these issues, federated semi-supervised learning (FSSL) is proposed to improve model performance with a large amount of unlabeled data on each client. In particular, priors work (Jeong et al., 2021) has achieved competitive performance by exploring the inter-client mutual knowledge, i.e., inter-client consistency loss Jeong et al. (2021) . However, they usually focus on mitigating inter-client heterogeneous data distribution across clients (External Imbalance) and assume that the class distribution between the labeled and unlabeled data is consistent. These assumptions enforce strict requirements of data annotation and would not be practical in many real-world applications. A general case is that labeled data and unlabeled data have different data distribution (Internal Imbalance). For example, photo gallery in mobile phones contains much more unlabeled images and irrelevant samples than the ones that can be labeled manually for image classification task Yang et al. (2011) . Besides, these existing methods require sharing of additional information among clients, which could impose potential privacy risks. Specifically, they transmit models among clients to provide consistency regularization. However, inter-client interactions might open a loophole to unauthorized infringement for privacy risks Chen et al. ( 2019); many reverse-engineering techniques Yin et al. (2020) can even recover the client data from the mutual models, threatening the users' data privacy and security (Yonetani et al., 2017; Wu et al., 2018) . In this paper, we propose a flexible federated semi-supervised learning framework with dual regulator, termed FedDure, to handle the internal and external imbalance problems while accounting for privacy. FedDure explores the adaptive regulators to flexibly update the model parameters in clients; it dynamically adjusts dual regulators to optimize the model training in each client according to the learning process and outcome of the client's specific data distribution. Our framework includes two novel components: Coarse-grained regulator (C-reg) and Fine-grained regulator (Freg). On the one hand, C-reg regularizes the updating of the local model by tracking the learning effect on labeled data distribution. It mitigates the distribution mismatching between labeled and unlabeled data to prevent corrupted pseudo labels and maintain generalization ability. On the other hand, F-reg learns an adaptive weighting scheme tailored for each client; it automatically equips a soft weight for each unlabeled instance to measure its contribution to the training. This scheme automatically adjusts the instance-level weights to strengthen (or weaken) its confidence according to the feedback of F-reg on the labeled data. FedDure utilizes the bi-level optimization strategy to alternately update the local model and dual regulators in clients. We theoretically show that C-reg and F-reg converge with guarantee and empirically demonstrate that FedDure outperforms other methods on various benchmarks. The main contributions of this work are three-fold. (1) We address a more practical scenario of FSSL, where data distribution is different not only across clients (external imbalance) but also between labeled and unlabeled data within a client (internal imbalance). ( 2) We propose a flexible FSSL framework with dual regulator, (FedDure), which designs adaptive regulators to flexibly update the local model according to the learning processes and outcomes on specific data distributions of each client. It does not require sharing of additional information among clients. ( 3) We theoretically analyze and present the convergence of the dual regulator and empirically demonstrate that FedDure is superior to the state-of-the-art FSSL approaches across multiple benchmarks.

2. RELATED WORK

Federated Learning (FL) is an emerging distributed training technique that trains models on decentralized clients and aggregates model updates in a central server (Yang et al., 2019) . It protects data privacy as raw data are always kept locally. FedAvg (McMahan et al., 2017 ) is a pioneering work that aggregates model updates by weighted averaging. Statistical heterogeneity is an important challenge of FL in real-world scenarios, where the data distribution is inconsistent among clients (Li et al., 2020a) . A plethora of works have been proposed to address this challenge with approaches like extra data sharing, regularization, new aggregation mechanisms, and personalization (Zhao et al., 2018; Li et al., 2020b; Zhuang et al., 2020; Li et al., 2021b; Gao et al., 2022) . However, these approaches commonly consider only supervised learning settings and may not be simply applied to scenarios where only a small portion of data is labeled. Numerous studies focus on purely unsupervised federated learning, but they are either application-specific or only learn generic visual representations (Zhuang et al., 2021a; b) ; they do not effectively leverage the small fraction of labeled data that could exist in real-world applications. Our work primarily focuses on federated semi-supervised learning, where a small fraction of data has labels in each client. Semi-Supervised Learning aims to utilize unlabeled data for performance improvements and is usually divided into two popular branches pseudo labeling and consistency regularization. Pseudolabeling methods (Lee et al., 2013; Zou et al., 2022; Pham et al., 2021) usually generate artificial labels of unlabeled data from the model trained by labeled data and apply the filtered high-confidence labels as supervised signals for unlabeled data training. MPL Pham et al. ( 2021) extends the knowledge distillation and meta-learning to SSL by optimizing the teacher model with feedback from the student model. Consistency regularization (Lee et al., 2022; Tarvainen & Valpola, 2017) regularizes the outputs of different perturbed versions of the same input to be consistent. Many works (Sohn et al., 2020; Zhang et al., 2021a; Lee et al., 2022) apply data augmentation as a perturbed strategy for pursuing outcome consistency. FixMatch Sohn et al. (2020) follows the UDA and brings the idea of pseudo-label to model training with unlabeled samples filtered by fixed threshold. Federated Semi-Supervised Learning (FSSL) considers learning models from decentralized clients where a small amount of labeled data resides on either clients or the server Jin et al. (2020) .

