ADVERSARY-AWARE PARTIAL LABEL LEARNING WITH LABEL DISTILLATION

Abstract

To ensure that the data collected from human subjects is entrusted with a secret, rival labels are introduced to conceal the information provided by the participants on purpose. The corresponding learning task can be formulated as a noisy partiallabel learning problem. However, conventional partial-label learning (PLL) methods are still vulnerable to the high ratio of noisy partial labels, especially in a large labelling space. To learn a more robust model, we present Adversary-Aware Partial Label Learning and introduce the rival, a set of noisy labels, to the collection of candidate labels for each instance. By introducing the rival label, the predictive distribution of PLL is factorised such that a handy predictive label is achieved with less uncertainty coming from the transition matrix, assuming the rival generation process is known. Nonetheless, the predictive accuracy is still insufficient to produce an sufficiently accurate positive sample set to leverage the clustering effect of the contrastive loss function. Moreover, the inclusion of rivals also brings an inconsistency issue for the classifier and risk function due to the intractability of the transition matrix. Consequently, an adversarial teacher within momentum (ATM) disambiguation algorithm is proposed to cope with the situation, allowing us to obtain a provably consistent classifier and risk function. In addition, our method has shown high resiliency to the choice of the label noise transition matrix. Extensive experiments demonstrate that our method achieves promising results on the CIFAR10, CIFAR100 and CUB200 datasets.

1. INTRODUCTION

Deep learning algorithms depend heavily on a large-scale, true annotated training dataset. Nonetheless, the costs of accurately annotating a large volume of true labels to the instances are exorbitant, not to mention the time invested in the labelling procedures. As a result, weakly supervised labels such as partial labels that substitute true labels for learning have proliferated and gained massive popularity in recent years. Partial-label learning (PLL) is a special weakly-supervised learning problem associated with a set of candidate labels ⃗ Y for each instance, in which only one true latent label y is in existence. Nonetheless, without an appropriately designed learning algorithm, the limitations of the partial label are evident since deep neural networks are still vulnerable to the ambiguous issue rooted in the partial label problem because of noisy labels Zhou ( 2018 2020) successfully solved the ambiguity problem where there is a set of candidate labels for each instance, and only a true label exists. Apart from the general partial label, we have also seen a variety of partial label generations evolved, simulating different real-life scenarios. The independently and uniformly drawing is the one have seen the most Lv et al. (2020); Feng & An (2019) . The other problem settings include the instance dependent partial label learning, where each partial label set is generated depending on the instance as well as the true label Xu et al. (2021 ). Furthermore, Lv et al. (2020) has introduced label specific partial label learning, where the uniform flipping probability of similar instances differs from dissimilar group instances. Overall, the learning objective of the previous works is all about disambiguation. More specifically, the goal is to design a classifier training with partial labels, aiming to correctly label the testing dataset, hoping the classification performance will be as close as the full supervised learning. On the contrary, there is a lack of discussion on previous works that shed light on the data privacyenhancing techniques in general partial label learning. The privacy risk is inescapable; thus, privacypreserving techniques need to be urgently addressed. Recently, we have seen surging data breach cases worldwide. These potential risks posed by the attacker are often overlooked and pose a detrimental threat to society. For instance, it is most likely for the adversary to learn from stolen or leaked partially labelled data for illegal conduct using the previous proposed partial-label learning methods. Subsequently, it has become an inherent privacy concerns in conventional partial label learning. In this paper, the Adversary-Aware partial label learning is proposed to address and mitigate the ramification of the data breach. In a nutshell, we propose an affordable and practical approach to manually corrupt the collected dataset to prevent the adversary from obtaining high-quality, confidential information meanwhile ensure the trustee has full access to the useful information. However, we have observed that adversary-aware partial label learning possesses some intrinsic learnability issues. Firstly, the intractability is raised from the transition matrix. Secondly, the classifier and risk inconsistency problem has been raised. Hence, we propose an the Adversarial teacher within momentum (ATM)(In section 2.1), adversary-aware loss function equation 19, and a new ambiguity condition equation 1 to counter the issues. Under the adversary-aware partial label problem setting, the rival is added to a candidate set of labels. To achieve that, we extend the original partial label generation equation 2 by factorisation to add the rival Y ′ . Subsequently, we have the adversary-aware partial label generation established as equation 3. Then, we decompose the second equation of equation 3 into the rival embedded intractable transition matrix term Q * and class instance-dependent transition matrix T y,y ′ , which is P(Y ′ = y ′ | Y = y, X = x). In our problem setting, Ty,y ′ , the class instance-independent transition matrix is utilised, which is defined as P(Y ′ = y ′ | Y = y), with the assumption the rival is generated depending only on Y but instance X. Under the assumption, the class instanceindependent transition matrix is simplified and mathematically identifiable. Since all the instances share the same class instance-independent transition matrix in practice, such encryption is more affordable to implement. The rival variable serves as controllable randomness to enhance privacy against the potential adversary and information leakage. In contrast, the previous methods can not guarantee the privacy protection property. However, a fundamental problem has been raised, inclusion of the rival implies an inconsistent classifier according to the adversary-aware label generation equation equation 3. Learning a consistent partial label classifier is vital, but in our problem setting, the consistency classifier may not be obtained due to the intractability of Q * (details are described in section 1.2). As a consequence, the Adversarial teacher within momentum (ATM) is proposed, which is designed to identify the term P( ⃗ Y | Y, Y ′ , X) which is denoted as Q * . The Moco-style dictionary technique He et al. ( 2020) and Wang et al. (2022) have inspired us to explore exploiting the the soft label from instance embedding, leveraging Ty,y ′ to identify or reduce the uncertainty of the Q * due to the property of informational preservation and tractability. Therefore, a consistent partial label learner is obtained if the uncertainty raised from the transition matrix is reduced greatly. Specifically, we transform the inference of label generation in Adversary-Aware PLL as an approximation for the transition matrix Q * . Ultimately, a tractable solution to the unbiased estimate of P( ⃗ Y | Y, Y ′ , X) can be derived. Lastly, we have rigorously proven that a consistent Adversary-Aware PLL classifier can be obtained if P( ⃗ Y | Y, Y ′ , X ) and P(Y ′ | Y ) are approximated accurately according to equation 3. In this work, we are mainly focusing on identifying the transition matrix term P( ⃗ Y | Y, Y ′ , X). The rival is generated manually for privacy enhancement. Thus the P(Y ′ | Y ) is given by design. Overall, our proposed method has not only solved the ambiguity problem in Adversary-Aware PLL but also addressed the potential risks from the data breach by using a rival as the encryption. Our proposed label generation bears some resemblance to local differential privacy Kairouz et al. (2014); Warner (1965) , which aims to randomise the responses. The potential application is to randomise survey responses, a survey technique for improving the reliability of responses to confidential interviews or private questions. Depending on the sophistication of the adversary, our method offers a dynamic mechanism for privacy encryption that is more resilient and flexible to face the potential adversary or privacy risk. By learning from the previous attacks, we can design different levels of protection by adjusting the T term. The main contributions of the work are summarized: • We propose a novel problem setting named adversary-aware partial label learning.



); Patrini et al. (2017); Han et al. (2018). As a result, there have had many partial label learning works (PLL)Cour et al. (2011); Hüllermeier & Beringer (2006); Feng & An (2019); Feng et al. (

