TRANSFERABLE UNLEARNABLE EXAMPLES

Abstract

With more people publishing their personal data online, unauthorized data usage has become a serious concern. The unlearnable examples strategies have been introduced to prevent third parties from training on the data without permission. They add perturbations to the users' data before publishing, so as to make the models trained on the perturbed published dataset invalidated. These perturbations have been generated for a specific training setting and a target dataset. However, their unlearnable effects significantly decrease when used in other training settings or datasets. To tackle this issue, we propose a novel unlearnable strategy based on Class-wise Separability Discriminant (CSD), which boosts the transferability of the unlearnable perturbations by enhancing the linear separability. Extensive experiments demonstrate the transferability of the unlearnable examples crafted by our proposed method across training settings and datasets. The implementation of our method is available at https://github.com/renjie3/TUE.

1. INTRODUCTION

With more people posting their personal data online (intentionally or unintentionally), it has raised the concern that the data might be utilized without the owner's consent to train commercial or malicious machine learning models. While large-scale datasets collected from the Internet like LFW (Huang et al., 2008) , Freebase(Bollacker et al., 2008), and Ms-celeb-1m(Guo et al., 2016) have greatly advanced the development of deep learning, they may contain a certain amount of private data, which has the potential risk of privacy leakage. Thus, growing efforts (Huang et al., 2020; Fowl et al., 2021) have been made to protect data from unauthorized usage by making the data samples "unlearnable" (Huang et al., 2020; Fowl et al., 2021; He et al., 2022) . These methods generate the unlearnable examples by injecting imperceptible "shortcut" perturbation. If the data is used by unauthorized training, the models will be tricked to extract such easy-to-learn shortcut features and ignore the real semantics in the original data (Geirhos et al., 2020) . Consequently, the trained model fails to recognize the user's data during the test phase and the user's data gets protected. As shown in Figure 1 , the practical usage procedure of unlearnable examples (UEs) consists of two stages, i.e., the generation stage and the evaluation stage. Before releasing data, in the generation stage, one can make the data unlearnable by adding perturbations into original data. According to whether the label information is used to generate the perturbations, we divide the UEs into Supervised UEs which is generated with the guidance of the label information, and Unsupervised UEs which is generated without the guidance of label information. In the evaluation stage, the unlearnable version of data is released to the public, and the unauthorized third parties might use different algorithms, like supervised training and unsupervised training, to learn from the data. The UEs aim to provide protection in this stage against unauthorized training and invalidate the models trained on them. However, existing UEs are in lack of training-wise transferability and have weaknesses in datawise transferability in the evaluation stage. First, low training-wise transferability implies that the perturbed samples generated towards one target training setting cannot be transferred to other In this way, the perturbed dataset is still exposed to no protection. Second, insufficient data-wise transferability indicates that the unlearnable effect of perturbations generated for one target dataset will significantly decrease when transferred to other datasets. We have to generate perturbations for each dataset, reducing its flexibility in real-world scenarios. For example, data in various applications such as social media is dynamic or even streaming, which makes it extremely challenging to generate the entire perturbation set when new data is continuously emerging. In this work, we aim to enhance the training-wise and data-wise transferability of unlearnable examples. In detail, our method is motivated by Synthetic Noise (SN) (Yu et al., 2021) , which devises a manually designed linear separable perturbation to generate UEs. Such perturbations do not target specific dataset, thus it has the potential to enhance data-wise transferability. However, SN is manually designed and it is not quantifiable or optimizable. As such, it is impossible to incorporate SN into other optimization processes. Meanwhile, SN lacks training-wise transferability. Therefore, in this work, we propose Class-wise Separability Discriminant (CSD) to generate optimizable linearseparable perturbations. Our framework Transferable Unlearnable Examples with enhanced linear separability can generate UEs with superior training-wise and data-wise transferability.

2. RELATED WORK

Unlearnable Examples. Unlearnable examples are close to availability attack which aims at making the data out of service for training the models by the unauthorized third parties (Muñoz-González et al., 2017) . Several supervised UEs are proposed (Huang et al., 2020; Fowl et al., 2021; Shan et al., 2020) . The vanilla unlearnable examples are crafted by generating and adding unlearnable perturbations into the clean samples (Huang et al., 2020) . Being induced to trust that the perturbation can minimize the loss better than the original image features, the model will pay more attention to the perturbations. Unlearnable Contrastive Learning (UCL) is proposed (He et al., 2022) to generate UEs to protect data from unsupervised learning. Our studies show that both Supervised and Unsupervised UEs lack training-wise transferability as shown in Section 3.2.1 and Appendix A, respectively. Table 1 summarizes the settings of existing methods, where ✗ means ineffective protection. Unsupervised Learning. Recently, unsupervised learning has shown its great potential to learn the representation from unlabeled data. Contrastive learning, one of the popular unsupervised methods, uses the task of instance discrimination to learn the representations. In SimCLR (He et al., 2022) which is the most common contrastive learning method, the positive and negative samples for each instance are created and the task is to discriminate the positive samples and negative samples. Some methods like SimSiam (Chen & He, 2021) and BYOL (Grill et al., 2020) can remove the negative samples and only focus on pushing the representations between positive samples to be similar.

3. PRELIMINARY

In this section, preliminary studies are conducted to explore two types of transferability. We first introduce key notations and definitions, and then show the insufficiency of transferability. Since the majority of UEs are guided by label information, in this work, we focus on Supervised UEs



Figure 1: Two stages of Unlearnable Examples. training settings. As shown in Section 3.2.1, although Error-Minimizing Noise (EMN) (Huang et al., 2020) can protect data from supervised training, we can use unsupervised learning methods, such as Contrastive Learning (Chen et al., 2020a; Chen & He, 2021; Chen et al., 2020b), to extract useful representations from the EMN-protected dataset and obtain high accuracy. In this way, the perturbed dataset is still exposed to no protection. Second, insufficient data-wise transferability indicates that the unlearnable effect of perturbations generated for one target dataset will significantly decrease when transferred to other datasets. We have to generate perturbations for each dataset, reducing its flexibility in real-world scenarios. For example, data in various applications such as social media is dynamic or even streaming, which makes it extremely challenging to generate the entire perturbation set when new data is continuously emerging.

Unlearnable Examples Settings of Existing Approaches

