GENERATING ADVERSARIAL EXAMPLES WITH TASK ORIENTED MULTI-OBJECTIVE OPTIMIZATION

Abstract

Deep learning models, even the-state-of-the-art ones, are highly vulnerable to adversarial examples. Adversarial training is one of the most efficient methods to improve the model's robustness. The key factor for the success of adversarial training is the capability to generate qualified and divergent adversarial examples which satisfy some objectives/goals (e.g., finding adversarial examples that maximize the model losses for simultaneously attacking multiple models). Therefore, multi-objective optimization (MOO) is a natural tool for adversarial example generation to achieve multiple objectives/goals simultaneously. However, we observe that a naive application of MOO tends to maximize all objectives/goals equally, without caring if an objective/goal has been achieved yet. This leads to useless effort to further improve the goal-achieved tasks, while putting less focus on the goal-unachieved tasks. In this paper, we propose Task Oriented MOO to address this issue, in the context where we can explicitly define the goal achievement for a task. Our principle is to only maintain the goal-achieved tasks, while letting the optimizer spend more effort on improving the goal-unachieved tasks. We conduct comprehensive experiments for our Task Oriented MOO on various adversarial example generation schemes. The experimental results firmly demonstrate the merit of our proposed approach. INTRODUCTION



Multi-Objective Optimization (MOO) (Désidéri, 2012) is an optimization problem to find a Pareto optimality that aims to optimize multiple objective functions. In a nutshell, MOO is a natural tool for the aforementioned multi-objective adversarial generations. However, a direct and naive application of MOO to generating robust adversarial examples for multiple models or ensemble of transformations does not work satisfactorily (cf. Appendix E). Concretely, it can be observed that the tasks are not optimized equally. The optimizing process focuses too much on one dominating task and can be trapped easily by it, hence leading to downgraded attack performances. Intuitively, for multi-objective adversarial generations, we can explicitly investigate if an objective or a task achieves or fails to achieve its goal (e.g., the current adversarial example can fool a model successfully or unsuccessfully in multiple models). To avoid some tasks dominating others during the optimization process, we can favour more the tasks that are failing and pay less attention to the tasks that are performing well. For example, in the context of attacking multiple models, we update an adversarial example x a to favor the models that x a has not attacked successfully yet, while trying to maintain the attack capability of x a on the already successful models. In this way, we expect that no task really dominates others and all tasks can be updated equally to fulfill their goals. Bearing this in mind, we propose a new framework named TAsk Oriented Multi-Objective Optimization (TA-MOO) with multi-objective adversarial generations as the demonstrating applications. Specifically, we learn a weight vector (i.e., each dimension is the weight for a task) lying on a simplex corresponding to all tasks. To favor the unsuccessful tasks while maintaining the success of the successful ones, we propose a geometry-based regularization term that represents the distance between the original simplex and a reduced simplex which involves the weight vectors for the currently unsuccessful tasks only. Furthermore, along with the original quadratic term of the standard MOO helping to improve all tasks, minimizing our geometry-based regularization term encourages the weights of the goal-achieved tasks to be as small as possible, while inspiring those for the goal-unachieved ones to have a sum close to 1. By doing so, we aim to focus more on improving the goal-unachieved tasks, while still maintain the performance of goal-achieved tasks. Most related work to ours is Wang et al. (2021) , which considers the worst-case performance across all tasks. However, this original principle reduces the generalizability to other tasks. To mitigate this issue, a specific regularization was proposed to balance all tasks' weights. Our work, which casts an adversarial generation task as a multi-objective optimization problem, is conceptually different from that work, although both methods can be applied to similar tasks. Further discussion about relate work can be found in Appendix A. To summarize, our contributions in this work include: (C1) Ours is among the first works that views an adversarial generation task as a multi-objective optimization problem and comprehensively study the issues when applying multi-objective optimization to tackle this adversarial generation task. (C2) To mitigate the shortcomings of the original MOO (Désidéri, 2012; Sener & Koltun, 2018) when applying to multi-objective adversarial generation, we propose the TA-MOO framework which employs a geometry-based regularization term to favor the unsuccessful tasks, while trying to maintain the performance of the already successful tasks. (C3) We conduct comprehensive experiments for three adversarial generation tasks and one adversarial training task including: (i) attacking multiple models, (ii) learning universal perturbation, (iii) attacking over many data transformations, and (iv) adversarial training on ensemble learning setting. The experimental results show that our TA-MOO outperforms the baselines by a wide margin on the three aforementioned adversarial generation tasks. More importantly, our adversary brings a great benefit on improving adversarial robustness.

2. BACKGROUND

We revisit the background of multi-objective optimization (MOO), which lays the foundation for our task-oriented MOO in the sequel. Given multiple objective functions f (δ) := [f 1 (δ) , ..., f m (δ)] where each f i : R d → R, we aim to find the Pareto optimal solution that simultaneously maximizes all objective functions: max δ f (δ) := [f 1 (δ) , ..., f m (δ)] . While there are a variety of MOO solvers (Miettinen, 2012; Ehrgott, 2005) , in this paper, we adapt from the multi-gradient descent algorithm (MGDA) that was proposed suitably for end-to-end learning by Désidéri (2012) . Specifically, MGDA combines the gradients of individual objectives to a single optimal direction that increases all objectives simultaneously. The optimal direction corresponds to the minimum-norm point that can be found by solving the quadratic programming problem: w * = argmin w∈∆m w T Qw, where ∆ m = π ∈ R m + : π 1 = 1 is the m-simplex and Q ∈ R m×m is the matrix with Q ij = ∇ δ f i (δ) T ∇ δ f j (δ). Finally, the solution of the problem 1 can be found iteratively with each update step δ = δ + ηg where g is the combined gradient g = m i=1 w * i ∇ δ f i (δ) and η > 0 is a sufficiently small learning rate. Furthermore, Désidéri (2012) also proved that by using an appropriate learning rate at each step, we reach the Pareto optimality point δ * at which there exist w ∈ ∆ m such that m i=1 w i ∇ δ f i (δ * ) = 0.

3. OUR PROPOSED METHOD 3.1 TASK ORIENTED MULTI-OBJECTIVE OPTIMIZATION

We now present our TAsk Oriented Multi-Objective Optimization (TA-MOO). We consider the MOO problem in (1) where each task T i (i = 1, ..., m) corresponds to the objective function f i (δ) (i = 1, ..., m). Additionally, assume that given a task T i , we can explicitly observe if this task has currently achieved its goal (e.g., the current adversarial example x can fool successfully the model f i ), which is named a goal-achieved task. We also name a task that has not achieved its goal a goal-unachieved task. Different from the standard MOO, which equally pays equal attention to all tasks, our TA-MOO focuses on improving the currently goal-unachieved tasks, while trying to maintain the performance of the goal-achieved tasks. By this principle, we expect all tasks would be equally improved to simultaneously achieve their goals. To be more precise, we depart from δ 0 and consecutively update in L steps to obtain the sequence δ 1 , δ 2 , ..., δ L that approaches the optimal solution. Considering the t-th step (i.e., 1 ≤ t ≤ L), we currently have δ t and need to update it to obtain δ t+1 . We examine the tasks that have achieved their goals already and denote them as T 1 , T 2 , ..., T s without the loss of generalization. Here we note that the list of goal-achieved tasks is empty if s = 0 and the list of goal-unachieved tasks is empty if s = m. Specifically, to find δ t+1 , we first solve the following optimization problem (OP): w * = argmin w∈∆m w T Qw + λΩ (w) , where Q ∈ R m×m with Q ij = ∇ δ f i (δ t ) T ∇ δ f j (δ t ), λ > 0 is a trade-off parameter, and Ω (w) is a regularization term to let the weights focus more on the goal-unachieved tasks. We next compute the combined gradient g t and update δ t as: g t = m i=1 w * i ∇ δ f i (δ t ) and δ t+1 = δ t + ηg t . The OP in (3) consists of two terms. The first term w T Qw ensures that all tasks are improving, while the second term Ω (w) serves as the regularization to restrict the goal-achieved tasks T 1 , ..., T s by setting the corresponding weights w 1 , ..., w s as small as possible. Before getting into the details of the regularization, we emphasize that to impose the constraint w ∈ ∆ m , we parameterize w = softmax (α) with α ∈ R m and solve the OP in (3) using gradient descent. In what follows, we discuss our proposed geometry-based regularization term Ω (w). Simplex-based regularization. Let S u = β = [β i ] m i=s+1 ∈ R m-s + : m i=s+1 β i = 1 be a simplex w.r.t. the goal-unachieved tasks and S = {0 s } × S u be the extended simplex, where 0 s is the s-dimensional vector of all zeros. We define the regularization term Ω (w) as the distance from w to the extended simplex S: Ω (w) = d (w, S) = min π∈S w -π 2 2 . ( ) Because S is a compact and convex set and w -π The following lemma shows us how to find the projection proj S (w) and evaluate Ω (w). Lemma 1. Sorting w s+1:m into u s+1:m such that u s+1 ≥ u s+2 ≥ ... ≥ u m . Defining ρ = max s + 1 ≤ i ≤ m : u i + 1 i-s 1 - i j=s+1 u j > 0 . Denoting γ = 1 ρ 1 - ρ i=s+1 u i , the projection proj S (w) can be computed as proj S (w) i = 0 1 ≤ i ≤ s max {w i + γ, 0} otherwise Furthermore, the regularization Ω (w) has the form: Ω (w) = s i=1 w 2 i + m i=s+1 (w i -max {w i + γ, 0}) 2 . (5) With further algebraic manipulations, Ω (w) can be significantly simplified as shown in Theorem 1. Theorem 1. The regularization Ω (w) has the following closed-form: Ω (w) = s i=1 w 2 i + 1 m -s 1 - m i=s+1 w i 2 . ( ) The proof of Lemma 1 and Theorem 1 can be found in Appendix B.1. Evidently, the regularization term in Eq. ( 6) in Theorem 1 encourages the weights w 1:s associated with the goal-achieved tasks to be as small as possible and the weights w s+1:m associated with the goal-unachieved tasks to move closer to the simplex S u (i.e., m i=s+1 w i is closer to 1). Parameterized TA-MOO. Algorithm 1 summarizes the key steps of our TA-MOO. We use gradient descent to find solution δ for the OP 1 in L steps and at each iteration we solve the OP in 3 in K steps using gradient descent solver with the parameterization w = softmax (α). To reduce computational cost, at each iteration we reuse the previous solution α and use a few steps K (i.e., K ≤ 10) to get new solution. We then compute the combined gradient g t and finally update δ t to δ t+1 using the combined gradient g t (or sign(g t ) in the case of L ∞ norm). The projecting operation in step 13 is to project δ to a valid space specifying to applications that we introduce hereon. Algorithm 1 Pseudocode for Parameterized TA-MOO. Input: Multi-objective functions f 1:m (δ). δ's solver with L update steps and learning rate η δ . w's Gradient Descent Solver (GD) with K update steps and learning rate η w and variable α. The softmax function denotes by σ. Tradeoff parameter λ. Output: The optimal solution δ * . 1: Initialize δ 0 (e.g., δ 0 ∼ U(-, )). 2: Initialize α 0 = [α i 0 ] m i=1 with α i 0 = 1/m. 3: for t = 0 to L -1 do 4: Collect list of tasks' gradients {∇ δ f i (δ t )} m i=1 . 5: Compute Q with Q ij = ∇ δ f i (δ t ) T ∇ δ f j (δ t ).

6:

Initialize α t+1 = α t 7: Compute the combined gradient for k = 0 to K -1 do 8: Compute L(α t+1 ) = σ(α t+1 ) T Qσ(α t+1 ) + λΩ(σ(α t+1 )). 9: Update α t+1 = α t+1 -η w ∇ α L(α t+1 g t = m i=1 σ(α t+1,i )∇ δ f i (δ t ). 12: Update δ t+1 = δ t + η δ g t . 13: Project δ t+1 to a valid space (specific to domain, e.g., δ ≤ ). 14: end for 15: Output δ * = δ L .

3.2. APPLICATIONS IN ADVERSARIAL GENERATION

Although TA-MOO is a general framework, we in this paper focus on its applications in adversarial generation. Following Wang et al. (2021) , we consider three tasks of generating adversarial examples. Generating adversarial examples for an ensemble model. Considering an ensemble classifier with multiple classification models h 1 , h 2 , ..., h m , where h i (x) ∈ ∆ M = π ∈ R M + : π 1 = 1 with the number of classes M . Given a data sample x, our aim is to find an adversarial example x a = x + δ that can successfully attack all the models. Specifically, we consider a set of tasks each of which, T i , is about whether x + δ can successfully attack model h i , defined as: I argmax 1≤k≤M h i (x + δ, k) = y , where y is the ground truth label of x, I is the indicator function and h i (x, k) returns the probability to predict x to the class k. To find a perturbation δ that can attack successfully all models, we solve the following multi-objective optimization problem: max δ: δ ≤ [f 1 (δ) , ..., f m (δ)] , where f i (δ) = (h i (x + δ) , y) with the loss function which could be the cross-entropy (CE) loss (Madry et al., 2018) , the Kullback-Leibler (KL) loss (Zhang et al., 2019) , or the Carlini-Wagner (CW) loss (Carlini & Wagner, 2017) . Generating universal perturbations. Considering a single classification model h with h (x) ∈ ∆ M and a batch of data samples x 1 , x 2 , ..., x B , we would like to find a perturbation δ with δ ≤ such that x a i = x i + δ, i = 1, ..., B, are adversarial examples. We define the task T i as finding the adversarial example x a i = x i + δ for data sample x i . For each task T i , we can define its goal as finding successfully the adversarial example x a i : I argmax 1≤k≤M h (x a i , k) = argmax 1≤k≤M h (x i , k) . To find the perturbation δ, we solve the following multi-objective optimization problem: max δ: δ ≤ [f 1 (δ) , ..., f m (δ)] , where f i (δ) = (h (x a i ) , y i ) = (h (x i + δ) , y i ) with y i the ground-truth label of x i . Generating adversarial examples against transformations. Considering a single classification model h and m categories of data transformation P 1:m (e.g., rotation, lighting, and translation). Our goal is to find an adversarial attack that is robust to these data transformations. Specifically, given a benign example x, we would like to learn a perturbation δ with δ ≤ that can successfully attack the model after any transformation t i ∼ P i is applied. To formulate as an MOO problem, we consider the task T i as finding the adversarial example x a i = t i (x + δ) with t i ∼ P i . For each task T i , we can define the goal as finding successfully the adversarial example x a i : I argmax 1≤k≤M h (x a i , k) = argmax 1≤k≤M h (x, k) . To find the perturbation δ, we solve the following multi-objective optimization problem: max δ: δ ≤ [f 1 (δ) , ..., f m (δ)] , where f i (δ) = E ti∼Pi [ (h (t i (x + δ)) , y)] with y the ground-truth label of x.

4. EXPERIMENTS

In this section, we provide extensive experiments across four settings: (i) generating adversarial examples for ensemble of models (ENS, Sec 4.1), (ii) generating universal perturbation (UNI, Sec 4.3) , (iii) generating robust adversarial examples against Ensemble of Transformations (EoT, Sec 4.4), and (iv) adversarial training for ensemble of models (AT, Sec 4.2). The details of each setting can be found in Appendix C.

General settings.

Through our experiments, we use six common architectures for the classifier including ResNet18 (He et al., 2016) , VGG16 (Simonyan & Zisserman, 2014) , GoogLeNet (Szegedy et al., 2015) , EfficientNet (Tan & Le, 2019 ), MobileNet Howard et al. (2017 ), and WideResNet Zagoruyko & Komodakis (2016) with the implementation 1 . We evaluate on the full testing set of two benchmark datasets which are CIFAR10 and CIFAR100 (Krizhevsky et al., 2009) . We observed that the attack performance is saturated with standard training models. Therefore, to make the job of adversaries more challenging, we use Adversarial Training with PGD-AT (Madry et al., 2018) to robustify the models and use these robust models as the victim models in our experiments. Evaluation metrics. We use three metrics to evaluate the attack performance including (i) A-All: the Attack Success Rate (ASR) when an adversarial example can achieve goals in all tasks. This is considered as the most important metric to indicate how well one method can achieve in all tasks; (ii)A-Avg: the average Attack Success Rate over all tasks which indicate the average attacking performance; (iii){A-i} K i=1 : Attack Success Rate in each individual task. For reading comprehension purposes, if necessary the highest/second highest performance in each experimental setting is highlighted in Bold/Underline and the most important metric(s) is emphasized in blue color. Baseline methods. We compare our method with the Uniform strategy which assigns the same weight for all tasks and the MinMax method (Wang et al., 2021) which examines only the worstcase performance across all tasks. To increase the generality to other tasks, MinMax requires a regularization to balance between the average and the worst-case performance. We use the same attack setting for all methods: the attack is the L ∞ untargeted attack with 100 steps, step size η δ = 2/255 and perturbation limitation = 8/255. The GD solver in TA-MOO uses 10 steps with learning rate η w = 0.005. Further detail can be found in Appendix C.

4.1. ADVERSARIAL EXAMPLES FOR ENSEMBLE OF MODELS (ENS)

Experimental setting. In our experiment, we use an ensemble of four adversarially trained models: ResNet18, VGG16, GoogLeNet, and EfficientNet. The architecture is the same for both the CIFAR10 and CIFAR100 datasets except for the last layer which corresponds with the number of classes in each dataset. The final output of the ensemble is an average of the probability outputs (i.e., output of the softmax layer). We use three different losses as an object for generating adversarial examples including CE (Madry et al., 2018) , KL (Zhang et al., 2019) , and CW (Carlini & Wagner, 2017) . Results 1: TA-MOO achieves the best performance. Table 1 shows the results of attacking the ensemble model on the CIFAR10 and CIFAR100 datasets. It can be seen that TA-MOO significantly outperforms the baselines and achieves the best performance in all the settings. For example, the improvement over the Uniform strategy is around 10% on both datasets with the CE loss. Comparing to the MinMax method, the biggest improvement is around 3% for CIFAR10 with CW loss and the lowest one is around 0.6% with the KL loss. The improvement can be observed in all the settings, showing the generality of the proposed method. Results 2: When does not MOO work? It can be observed that MOO falls behind all other methods, even compared with the Uniform strategy. Our hypothesis for the failure of MOO is that in the original setting with an ensemble of 4 diverse architectures (i.e., ResNet18, VGG16, GoogLeNet, and EfficientNet) there is one task that dominates the others and makes MOO become trapped (i.e., focusing on improving the dominant task). To verify our hypothesis, we measure the gradient norm ∇ δ f i (δ) corresponding to each model and the final weight w of 1000 samples and report the results in Table 2 . It can be seen that the EfficientNet has a much lower gradient strength, therefore, it has a much higher weight. This explains the highest ASR observed in EfficientNet and the large gap of 19% (56.11% in EfficientNet and 37.05% in GoogLeNet). To further confirm our hypothesis, we provide an additional experiment on a non-diverse ensemble model which consists of 4 individual ResNet18 models. It can be observed that in the non-diverse setting, the gradient strengths are more balanced across models, indicating that no task dominates others. As a result, MOO shows its effectiveness by outperforming the Uniform strategy by 4.3% in A-All. Results 3: The importance of the Task-Oriented regularization. It can be observed from Table 2 that in the diverse setting, TA-MOO has a much lower gap (4%) between the highest ASR (53.4% at EfficientNet) and the lowest one (49.29% at GoogLeNet) compared to MOO ( 19%) . Moreover, while the ASR of EfficientNet is lower by 2.7%, the ASRs of all other architectures have been improved considerably (i.e., 12% in GoogLeNet). This improvement shows the importance of the Task-Oriented regularization, which helps to avoid being trapped by one dominating task, as happened in MOO. For the non-diverse setting, when no task dominates others, TA-MOO still shows its effectiveness when improving the ASR in all tasks by around 5%. The significant improvement can be observed in all settings (except the setting on EfficientNet with the CIFAR10 dataset) as shown in Table 1 , and demonstrates the generality of the Task-Oriented regularization. Result 2: Producing more robust single members. The comparison of average SAR shows that adversarial training with TA-MOO produces more robust single models than PGD-AT does. More specifically, the average robust accuracy (measured by 100% -A-Avg) of TA-MOO-AT is 32.17%, an improvement of 6.06% over PGD-AT in the non-diverse setting, while there is an improvement of 4.66% in the diverse setting. Result 3: Adversarial training with TA-MOO achieves the best robustness. More specifically, on the non-divese setting, TA-MOO-AT achives 38.22% robust accuracy, an improvement of 1% over MinMax-AT and 5.44% over standard PGD-AT. On the diverse setting, the improvement over MinMax-AT and PGD-AT are 0.9% and 4%, respectively. The root of the improvement is the ability to generate stronger adversarial examples in the the sense that they can challenge not only the entire ensemble model but also all single members. These adversarial examples lie in the joint insecure region of members (i.e., the low confidence region of multiple classes), therefore, making the decision boundaries more separate. As a result, adversarial training with TA-MOO produces more robust single models (i.e., lower SAR-Avg) and significantly reduces the transferability of adversarial examples among members (i.e., lower SAR-All). These two conditions explain the best ensemble adversarial robustness achieved by TA-MOO.

4.3. UNIVERSAL PERTURBATION (UNI)

Experimental setting. We follow the experimental setup in Wang et al. (2021) , where the full test set (10k images) is randomly divided into equal-size groups (K images per group). The comparison has been conducted on the CIFAR10 and CIFAR100 datasets, with an adversarially trained ResNet18 model and CW loss. We observed that the ASR-All was mostly zero, indicating that it is difficult to generate a general perturbation for all data points. Therefore, in Table 5 we use ASR-Avg to compare the performances of the methods. More experiments on VGG16 and EfficientNet models can be found in Appendix D.3.

Results.

Table 5 shows the evaluation of generating universal perturbations on the CIFAR10 and CIFAR100 datasets, respectively. K represents the number of images that are using the same perturbation. The larger the value of K, the harder it is to generate a universal perturbation that can be applied successfully to all images. It can be seen that with a small number of tasks (i.e., K=4), MOO and TA-MOO achieve lower performance than the MinMax method. However, with a large number of tasks (i.e, K ≥ 8), MOO and TA-MOO show their effectiveness and achieve the best performance. More specifically, on the CIFAR10 dataset, the improvements of MOO over the Uniform strategy are 5.6%, 4%, 3.2%, and 2.5% with K = 8, K = 12, K = 16, and K = 20, respectively. On the same setting, TA-MOO significantly improves MOO by around 4% in all the K settings and consistently achieves the best performance. Unlike the ENS setting, in the UNI setting, MOO consistently achieves better performance than the Uniform strategy . This improvement can be explained by the fact that in the UNI setting with the same architecture and data transformation, no task dominates the others. There will be a case (a group) when one sample is extremely close to/far from the decision boundary, and hence easier/harder to fool. However, in the entire test set with a large number of groups, the issue of dominating tasks is lessened.

4.4. ROBUST ADVERSARIAL EXAMPLES AGAINST TRANSFORMATIONS (EOT)

Results. Table 6 shows the evaluation on the CIFAR10 and CIFAR100 datasets with 7 common data transformations. It can be observed that (i) MOO has a lower performance than the baselines, (ii) the Task Oriented regularization significantly boosts the performance, and (iii) our TA-MOO method achieves the best performance on both settings and outperforms the MinMax method 0.6% and 1.1% in the CIFAR10 and the CIFAR100 experiments, respectively. The low performance of MOO in observation (i) is again caused by the issue of one task dominating others. In the EoT setting, it is because of the V-vertical flip transformation as shown in Table 6 . Observation (ii) provides another piece of evidence to support the effectiveness of the Task-Oriented regularization for MOO. This regularization boosts the ASRs in all the tasks (except V -the dominant one), increases the average ASR by 5.45% and 2.5% in the CIFAR10 and CIFAR100 experiments, respectively.

5. CONCLUSION

In this paper, we propose Task Oriented Multi-Objective Optimization (TA-MOO), with specific applications to adversarial generation tasks. We develop a geometry-based regularization term to favor the goal-unachieved tasks, while trying to maintain the the goal-achieved tasks. We conduct comprehensive experiments to showcase the merit of our proposed approach on generating adversarial examples and adversarial training. On the other hand, there are acknowledged limitations of our method such as weaknesses of the gradient-based solver and lacking theory on algorithm's convergence which might be target for future works. probabilities of a benign example (Zhang et al., 2019) , or maximizing the CW loss (Carlini & Wagner, 2017) . However, in some contexts, we need to generate adversarial examples maximizing multiple objectives or goals, e.g., attacking multiple models (Pang et al., 2019; Bui et al., 2020) or finding universal perturbations (Moosavi-Dezfooli et al., 2017) . To this end, the work of Wang et al. (2021) examined the worst-case scenario by casting the problem of interest as a min-max problem for finding the weight of each task. However, this principle leads to a problem of lacking generality in other tasks. To mitigate the issue, Wang et al. (2021) proposed a regularization to strike a balance between the average and the worst-case performance. The final optimization was formulated as follow: max δ: δ ≤ min w∈∆m K i=1 w i f i (δ) + γ 2 w -1/K 2 2 , Where f i (v) is the victim model's loss (i.e., cross entropy loss or KL divergence) and γ > 0 is the regularization parameter. The authors used the bisection method (Boyd et al., 2004) with project gradient descent for the inner minimization and project gradient ascent for the outer maximization. There are several major differences in comparison to MOO and TA-MOO methods: (i) In principle, MinMax considers the worst-case performance only while our methods improve performance of all tasks simultaneously. (ii) MinMax weighs the tasks' losses to find the minimal weighted sum loss in its inner minimization, however, as discussed in Section E.4 the higher weighted loss does not directly imply the higher success rate in attacking multi-tasks simultaneously. In contrast, our methods use multi-gradient descent algorithm (Deb, 2011) in order to increase losses of all tasks simultaneously. (iii) The original principle of MinMax leads to the biasing problem to the worst-case task. The above regularization has been used to mitigate the issue, however, it considers all tasks equally. In contrast, our TA-MOO takes goal-achievement status of each task into account and focuses more on the goal-unachieved tasks. While their method also use the multi-objective optimization for generating adversarial examples, there are several major differences to ours. Firstly, their method aims for a generative Deepfake model while our method aims for the standard classification problem which is the most common and important setting in AML. Secondly, we conduct comprehensive experiments to show that a direct and naive application of MOO to adversarial generation tasks does not work satisfactorily because of the gradient dominating problem. Most importantly, we propose the TA-MOO method which employs a geometry-based regularization term to favor the unsuccessful tasks, while trying to maintain the performance of the already successful tasks. We have conducted extensive experiments to show that our TA-MOO consistently achieves the best attacking performance across different settings. We also conducted additional experiments with SOTA multi-task learning methods which are PCGrad (Yu et al., 2020) and CAGrad (Liu et al., 2021a) in Appendix E.5. Compared to these methods, our TA-MOO still achieves the best attack performance thanks to the Task Oriented regularization.

B FURTHER DETAILS OF THE PROPOSED METHOD

B.1 PROOFS Lemma 1. Sorting w s+1:m into u s+1:m such that u s+1 ≥ u s+2 ≥ ... ≥ u m . Defining ρ = max s + 1 ≤ i ≤ m : u i + 1 i-s 1 - i j=s+1 u j > 0 . Denoting γ = 1 ρ 1 - ρ i=s+1 u i , the projection proj S (w) can be computed as proj S (w) i = 0 1 ≤ i ≤ s max {w i + γ, 0} otherwise Furthermore, the regularization Ω (w) has the form: Ω (w) = s i=1 w 2 i + m i=s+1 (w i -max {w i + γ, 0}) 2 . (5) Proof. The proof is based on Wang & Carreira-Perpinán (2013) with modifications. We need to solve the following OP: min π 1 2 w -π 2 2 s.t. :π ≥ 0 π 1 = 1. We note that π 1 = ... = π s = 0. The OP of interest reduces to min πs+1:m 1 2 m i=s+1 (π i -w i ) 2 s.t. :π s+1:m ≥ 0 m i=s+1 π i = 1. Using the Karush-Kuhn-Tucker (KKT) theorem, we construct the following Lagrange function: L (π, γ, β) = 1 2 m i=s+1 (π i -w i ) 2 -γ m i=s+1 π i -1 - m i=s+1 β i π i . Setting the derivative w.r.t. π i to zeros and using the KKT conditions, we obtain: π i -w i -γ -β i = 0, ∀i = s + 1, ..., m m i=s+1 π i =1 β i ≥ 0, π i ≥ 0,β i π i = 0, ∀i = s + 1, ..., m. If π i > 0, β i = 0, hence π i = w i + γ > 0. Otherwise, if π i = 0, w i + γ = -β i ≤ 0. Therefore, w s+1:m has the same order as π s+1:m and we can arrange them as: π s+1 ≥ π s+2 ≥ ... ≥ π ρ > π ρ-1 = ... = π m = 0. u s+1 = w s+1 ≥ u s+2 = w s+2 ≥ .... ≥ u p = w p ≥ u ρ-1 = w ρ-1 ≥ ... ≥ u m = w m ≥ 0. It appears that 1 = m i=s+1 π i = ρ i=s+1 π i = ρ i=s+1 (w i + γ) = ρ i=s+1 w i + (ρ -s) γ. Hence, we gain γ = 1 ρ-s 1 - ρ i=s+1 w i = 1 ρ-s 1 - ρ i=s+1 u i . We now prove that ρ = max s + 1 ≤ i ≤ m : u i + 1 i-s 1 - i j=s+1 u j > 0 . • For i = ρ, we have u ρ + 1 ρ -s   1 - ρ j=s+1 u j   = u ρ + γ = w ρ + γ > 0. • For i < ρ, we have u i + 1 i -s   1 - i j=s+1 u j   = 1 i -s   (i -s)u i + 1 - i j=s+1 u j   = 1 i -s   (i -s)w i + ρ-1 j=s+1 π j - i j=s+1 w j   = 1 i -s   (i -s)w i + ρ-1 j=i+1 π j + i j=s+1 (π j -w j )   = 1 i -s   (i -s) (w i + γ) + ρ-1 j=i+1 π j   = 1 i -s   (i -s)π i + ρ-1 j=i+1 π j   > 0. • For i > ρ, we have u i + 1 i -s   1 - i j=s+1 u j   = 1 i -s   (i -s)u i + 1 - i j=s+1 u j   = 1 i -s   (i -s)w i + ρ-1 j=s+1 π j - i j=s+1 w j   = 1 i -s   (i -s)w i + ρ-1 j=s+1 (π j -w j ) - i j=ρ w j   = 1 i -s   (i -s)w i + (ρ -s -1)γ - i j=ρ w j   = 1 i -s   (ρ -s -1)(w i + γ) + i j=ρ (w i -w j )   ≤ 0. Therefore, ρ = max s + 1 ≤ i ≤ m : u i + 1 i-s 1 - i j=s+1 u j > 0 . Finally, we also have π i = max{w i + γ, 0}, i = s + 1, ..., m and π i = 0, i = 1, ..., s. Theorem 1. The regularization Ω (w) has the following closed-form: Ω (w) = s i=1 w 2 i + 1 m -s 1 - m i=s+1 w i 2 . ( ) Proof. Recall ρ = max s + 1 ≤ i ≤ m : u i + 1 i-s 1 - i j=s+1 u j > 0 . Therefore, ρ = m because we have u m + 1 m -s   1 - m j=s+1 u j   = w m + 1 m -s   1 - m j=s+1 w j   = w m + s j=1 w j m -s > 0. It follows that  γ = 1 m -s 1 - m i=s+1 u i = 1 m -s 1 - m i=s+1 w i ≥ 0. ) i = 0 1 ≤ i ≤ s max {w i + γ, 0} = w i + γ otherwise Ω (w) = s i=1 w 2 i + m i=s+1 (w i -max {w i + γ, 0}) 2 = s i=1 w 2 i + m i=s+1 γ 2 = s i=1 w 2 i + (m -s)γ 2 = s i=1 w 2 i + 1 m -s 1 - m i=s+1 w i 2 .

B.2 ILLUSTRATIONS OF HOW MOO AND TA-MOO WORK

Figure 1 illustrates solutions of MOO and TA-MOO in a scenario of 2 goal-achieved tasks (with corresponding gradients ∇f s 1,2 ) and 2 goal-unachieved tasks (with corresponding gradients ∇f u 1,2 ). As illustrated in the left figure, with standard MOO method, where all the tasks' gradients have been considered regardless their status and the solution associated with the minimal norm is the perpendicular vector as suggested by geometry (Sener & Koltun, 2018) . If considering the goalunachieved tasks only as in the middle case, the MOO solution is the edge case. However, this extreme strategy ignores all the goal-achieved tasks which might lead to the instability. The Task Oriented regularization strikes a balance between the two aforementioned strategies as illustrated in the right figure. The method focuses more on improving the goal-unachieved tasks while spend less effort to maintain the goal-achieved tasks. With λ = 0 the TA-MOO optimal solution is equivalent to the standard MOO optimal solution while it becomes the MOO solution in the case of the goal-unachieved tasks only when λ → ∞.

C EXPERIMENTAL SETTINGS

General settings. Through our experiments, we use six common architectures including ResNet18 (He et al., 2016), VGG16 (Simonyan & Zisserman, 2014) , GoogLeNet (Szegedy et al., 2015) , EfficientNet (B0) (Tan & Le, 2019 ), MobileNet Howard et al. (2017) , and WideResNet (with depth 34 and widen factor 10) Zagoruyko & Komodakis (2016) with the implementation of https: //github.com/kuangliu/pytorch-cifar. We evaluate on the full testing set (10k) of two benchmark datasets which are CIFAR10 and CIFAR100 (Krizhevsky et al., 2009) . More specifically, the two datasets have 50k training images and 10k testing images, respectively, with the same image resolution of 32 × 32 × 3. However, while the CIFAR10 dataset has 10 classes, the CIFAR100 dataset has 100 classes and fewer images per class. Therefore, in general, an adversary is easier to attack a CIFAR100 model than a CIFAR10 one as shown in Table 7 . We observed that the attack performance is saturated with standard training models. Therefore, to make the job of adversaries more challenging, we use Adversarial Training with PGD-AT (Madry et al., 2018) to robustify the models and use these robust models as victim models in our experiments. Specifically, we use the In MinMax (Wang et al., 2021) , we use the same γ = 3 for all settings and use the authors' implementation 2 . Attacking ensemble model settings. In our experiment, we use an ensemble of four adversarially trained models: ResNet18, VGG16, GoogLeNet, and EfficientNet. The architecture is the same for both the CIFAR10 and CIFAR100 datasets except for the last layer which corresponds with the number of classes in each dataset. The final output of the ensemble is an average of the probability outputs (i.e., output of the softmax layer). We use three different losses as an object for generating adversarial examples including Cross Entropy (CE) (Madry et al., 2018) , Kullback-Leibler divergence (KL) (Zhang et al., 2019) , and CW loss (Carlini & Wagner, 2017) . Universal perturbation settings. We follow the experimental setup in Wang et al. (2021) , such that the full test set (10k images) is randomly divided into equal-size groups (K images per group). The comparison has been conducted on the CIFAR10 and CIFAR100 datasets, and CW loss. We use adversarial trained ResNet18, VGG16 and EfficientNet as base models. We observed that the ASR-All was mostly zero, indicating that it is difficult to generate a general perturbation for all data points. Therefore, we use ASR-Avg to compare the performances of the methods. Robust adversarial examples against transformations settings. In our experiment, we use 7 common data transformations including I-Identity, H-Horizontal flip, V-Vertical flip, C-Center crop, B-Adjust brightness, R-Rotation, and G-Adjust gamma. The parameter setting for each transformation has been shown in Table 8 . In the deterministic setting, a transformation has been fixed with one specific parameter, e.g., center cropping with a scale of 0.6 or adjusting brightness with a factor of 1.3. While in the stochastic setting, a transformation has been uniformly sampled from its family, e.g., center cropping with a random scale in range (0.6, 1.0) or adjusting brightness with a random factor in range (1.0, 1.3). The experiment has been conducted on adversarially trained ResNet18 model with the CW loss. represents value for each task (i.e., a sample in a group). w 1 /w 2 represents the weight of the first/second group of K samples, while w represents the the statistic of weight over all groups (mean±std). ∇ δ1 f i (δ 1 ) / ∇ δ2 f i (δ 2 ) represents the gradient norm of the first/second group of K samples, while ∇ δ f i (δ) represents the statistic of gradient norm over all groups (mean±std). I 0 /I 1 represents the indicator function for a successful (1) or unsuccessful (0) task, while I represents the the statistic of successful rate over all groups. Why does MOO work? As shown in Table 11 , MOO consistently achieves better performance than the Uniform strategy (except for the setting with EfficientNet on the CIFAR100 dataset). To find out the reason for the improvement, we investigate the gradient norm ∇ δ f (δ) and weight w for the first, and second groups (as an example) and the average over 100 groups of the testset as shown in Table 12 . It can be seen that in the first and second groups, there are some tasks that have significantly low gradient strengths than other tasks. The gap of the strongest/weakest gradient strength can be a magnitude of 10 6 indicating the domination of one task over others. While this issue can cause the failure as in the ENS setting, however, in the UNI setting, the lowest gradient strengths in each group correspond to unsuccessful tasks (unsuccessful adversarial examples) and vice versa. Recall that we use the multi-gradient descent algorithm to solve MOO, which in principle assigns a higher weight for a weaker gradient vector. Therefore, in the UNI setting, while the dominating issue still exists, fortunately, the result still fits our desired weighting strategy (i.e., higher weight for an unsuccessful task and vice versa). Moreover, when there are a large number of groups (i.e., 100 groups), the issue of dominating tasks is alleviated. The average gradient strength is more balanced as shown in Table 12 . This explains the improvement of MOO over the Uniform strategy in the UNI setting. T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 ∇ δ1 f i (δ 1 )

D.4 ROBUST ADVERSARIAL EXAMPLES AGAINST TRANSFORMATIONS (EOT)

We observed that in EoT with the stochastic setting, adjusting gamma sometimes has the overflow issue resulting in an infinite gradient. Recall that our method using MGDA to solve MOO which relies on the stability of gradient strengths. Therefore, in the case of having infinite gradients, learning weight w is unstable, resulting to lower performance in both MOO and TA-MOO. To overcome the overflow issue, we allocate memory to cache the valid gradient of each task in the previous iteration and replace the infinite value in the current iteration with the valid one in the memory. The storage only requires a tensor with the same shape as the gradient (i.e., as the exact size of the input), therefore, it does not increase the computation resource significantly. As shown in Table 13 , this simple technique helps to improve performance of TA-MOO by 5.3% on both the CIFAR10 and CIFAR100 datasets. It also helps to improve performance of MOO by 0.8% and 4.8%, respectively. Finally, after overcoming the gradient issue, the TA-MOO achieves the best performance on the CIFAR100 dataset and the second best performance on the CIFAR10 dataset (0.4% lower in ASR-All but 0.8% higher in ASR-Avg when comparing to MinMax). This result provides additional evidence of the advantage of our method. 

D.5 GENERATING SPEED COMPARISON AND EXPERIMENTS' STABILITY

Generating Speed Comparison. Table 15 shows the average time to generate one adversarial example in each setting. The results are measured on the CIFAR10 dataset with ResNet18 architecture in the Ensemble of Transformations (EoT) and Universal Perturbation (Uni) settings. We use 1 Titan RTX 24GB for the EoT experiment and 4 Tesla V100 16GB each for the other experiments. It is worth mentioning that our primary focus in this paper is showing the advantage of MOO and the Task-Oriented regularization in generating adversarial examples. Therefore, we did not try to optimize our implementation in terms of generating time. Experiments' Stability. We conduct an experiment with 5 different random seeds to generate adversarial examples for the ENS setting to evaluate the stability of experimental results on choosing of random seed. The experiment is on the CIFAR10 dataset, with an ensemble of 4 architectures including ResNet18, VGG16, GoogLeNet, and EfficientNet. We report mean and variation values in Table 16 . It can be observed that there is a slight variation in attack performances across methods. The variation is small enough compared to the gap between methods (i.e., the biggest variation is 0.32% in SAR-All while the smallest gap is 2.51% between MOO and the Uniform approach), therefore, making the comparison still reliable.

D.6 SENSITIVITY TO HYPER-PARAMETERS

In this section we provide an analytical experiment on the sensitivity of our TA-MOO method to the tradeoff λ. The study has been conducted with the ENS setting with CE loss and the EoT setting with deterministic transformations using ResNet18 architecture. All experiments are on the CIFAR10 dataset. The value of λ is changed from 1 to 1000. It can be observed from Figure 3a (the ENS setting) that (i) increasing λ reduces the performance of dominated task (i.e., ASR on the EfficientNet decreases from 54.49% at λ = 1 to 53.40% at λ = 100) while increases performances of other tasks. In overall, it significantly increases the ASR-All performance of the entire ensemble from 29.14% at λ = 1 to 38.01% at λ = 100. (ii) However, over-high λ (i.e., λ > 200) leads to the drop of performance in all tasks, resulting in a lower overall performance. A similar observation can be seen in the EoT setting in Figure 3b . The attack performance on the dominated task (V-Vertical flipping) decreases from 86.11% at λ = 50 to 83.67% at λ = 200. In contract, in the same range of λ the overall performance increases from 32.85% to 34.36%. The performances of all tasks decrease when using too large λ (i.e., λ > 200). Based on the result of this study, we choose λ = 100 in all the other experiments.

D.7 COMPARISON WITH STANDARD ATTACKS

We conducted an additional comparison on the ENS setting to further confirm the effectiveness of our method over standard adversarial attacks (which consider an entire ensemble as a single model). More specifically, we compare with AutoAttack (Croce & Hein, 2020) , Brendel-Bethge attack (BB) et al., 2018) . For AutoAttack, we use the standard version which includes 4 different attacks. For BB attack, we initialized with the PGD attack with 20 steps. For CW attack, we set the confidence factor to 1.0. We evaluate these attacks on 2 ensemble settings, a diverse (D) ensemble set with 4 different architectures (ResNet18, VGG16, GoogLeNet, and EfficientNet) and a non-diverse (ND) ensemble set with 4 ResNet18 architectures. It can be seen from the Table 17 that our TA-MOO attack consistently achieves the best attack performance, with a significant gap compared to the best standard attack. More specifically, our TA-MOO method achieves 38.01% (SAR-All metric) on the diverse ensemble set, while the second best attack is AutoAttack with 30.71% (a gap of 7.3%). On the non-diverse set, the gap between our TA-MOO and AutoAttack is still notably large at 4%. These standard attacks consider an entire ensemble as a single model, i.e., aim to optimize a single objective given a single ensemble output. Therefore, they cannot guarantee a successful attack on each member. Experimental Setting. We conduct experiments on the ENS setting using the adversarial pretrained models on the RobustBench (Croce et al., 2021) . We use two sets of an ensemble to verify the importance of our task-oriented strategy. The first set is the robust ensemble (RE) set Experimental Results. We report experimental results with different settings in Table 18 , where RE/LE/TAR/UNTAR represents Robust Ensemble/Less-Robust Ensemble/Targeted Attack/Untargeted Attack, respectively. It can be seen that, in the robust ensemble setting (RE-TAR and RE-UNTAR), our MOO achieves a similar performance compared to the baseline, while TA-MOO has a further improvement over MOO. The gap of SAR-All between TA-MOO and the uniform weighting strategy is 0.1% in the targeted attack setting (RE-TAR), while that in the untargeted attack setting is 1.2%. In the less-robust ensemble setting (LE-TAR and LE-UNTAR), the improvement of our methods over the baseline is higher than in the robust ensemble setting. With the gap of SAR-All between TA-MOO and the uniform strategy is 0.38% with the targeted attack setting (LE-TAR), while the gap in the untargeted setting (LE-UNTAR) is 15.22% a significantly higher. While it is acknowledged that the targeted attack is a more common protocol in attacking the ImageNet dataset (Athalye et al., 2018) , however, we believe that our significant improvement on the untargeted attack is still worth noting. We conduct an additional experiment on the EoT setting with the ImageNet dataset and report result in Table 19 . In this experiment, we use the robust pretrained ResNet18 model (model ID: Salman2020Do_R18) as the victim model. We use the standard attack setting, i.e., targeted attack with = 4/255, η = 1/255 with 20 steps. It can be seen that both MOO and TA-MOO could obtain a better attack performance than the uniform strategy. It is a worth noting that, in the experiment on the CIFAR10/CIFAR100 datasets (i.e., Table 6 in the main paper) the dominating issue of the vertical filliping exists and prevents MOO to obtain a better performance. In the ImageNet dataset, the dominating issue is less serious, therefore, explains the improvement of MOO and corroborates our hypothesis on the issue of dominating task. The dominating issue. On one hand, there is the dominating issue that happens in all the three settings. The issue can be recognized by the gap of attack performance among tasks. For example, in Table 21 (i.e., the ENS setting with the diverse ensemble and MOO method), the gap between highest ASR (at EfficientNet) and lowest ASR (at GoogLeNet) is 19%. In the EoT setting, the problem is even worse: The largest gap observed is 53.6% as shown in Table 14 (the highest ASR is 88.19% with Vertical flipping and the lowest ASR is 34.54% with Horizontal flipping in with MOO -D-C10 setting). The dominating issue is also be recognized by the observation that a significant small gradient strength of one task on comparison with other tasks' strength. For example, in Table Table 21 : Attacking Ensemble model with a diverse set D={R-ResNet18, V-VGG16, G-GoogLeNet, E-EfficientNet} and non-diverse set ND={4 ResNets}. w represents the final w of MOO (mean ± std). ∇ δ f i (δ) represents the gradient norm of each model (mean ± std). The table is copied from Table 2 in the main paper for reading comprehension purpose. The root of the dominating issue can be the natural of the setting (i.e., as shown in Table 14 with the EoT setting, when the domination of the Vertical flipping task can be observed in all methods) or because of the MOO solver which is discussed in Section E.3 A-All A-Avg R/R1 V/R2 G/R3 E/R4 D ∇ δ f i (δ) - - Overcoming the dominating issue. On the other hand, if overcoming this issue, MOO can outperform the Uniform strategy. For example, on attacking the non-diverse ensemble model (i.e., 4 ResNets) MOO surpasses the Uniform strategy by 4.3% and 3.5% in the ASR-All and ASR-Avg metrics, respectively. On generating universal perturbations, MOO outperforms the Uniform strategy in most of the settings. As discussed in Section D.4, a simple memory caching trick can helps to overcome the infinite gradient issue and significantly boosts the performance of MOO or TA-MOO. Therefore, we believe that developing a technique to lessen the dominating issue might be a potential extension to further improve the performance. Balancing among goal-unachived tasks. We observed in the EoT setting, the dominating issue is strictly serious when gradients of some tasks are much weaker/stronger than others. It is because of the natural of the transformation operations, therefore, this issue happens regardless status of the tasks. In the set of goal-unachieved tasks' gradients can exist a dominated one, resulting to a much higher weight of the dominated task. Therefore, in order to strike a more balance among goal-unachieved tasks, we apply an additional regularization which minimizes the entropy of goal-unachieved weights H(w) = m i=s+1 -w i log w i . If all tasks have been achieved (i.e., s = m) then the additional regularization will be ignored. This additional regularization helps to improve further 2% in the EoT setting.

E.2 IMPORTANCE OF THE TASK-ORIENTED REGULARIZATION.

In this discussion, we would like to provide more experimental results in the ENS and EoT settings to further emphasize the contribution of the Task-Oriented regularization. Figure 4 shows the ASR of each individual task in the ENS setting with three losses and the EoT setting with ResNet18 architecture and deterministic transformations. As shown in Figure 4a , in the ENS setting, the MOO adversary produces a much higher ASR on the EfficientNet architecture than other architectures with any losses. In contrast, the TA-MOO adversary has a lower ASR on the EfficientNet architecture but a much higher ASR on other architectures. Similar observation can be seen in Figure 4b such that the ASR corresponding to the V-flipping of MOO is slightly higher than that of TA-MOO, however, the ASR on other transformations of MOO is much lower than those of TA-MOO.

E.3 MORE EFFICIENT MOO SOLVERS

Discussions on the weighted-sum method. One of the most common approaches to solve the MOO problem is the scalarizing method, which formulates a single-objective optimization (SOO) such that the optimal solutions to the SOO problem are Pareto optimal solutions to the MOO problem. While this line of approach (e.g., weighted-sum method) is suitable for end-to-end learning such as deep learning, there are several acknowledged weaknesses: (i) the choice of utility function has a large impact on the computational complexity of the resulted SOO problem (Bjornson et al., 2014; Björnson & Jorswieck, 2013) ; (ii) a small change in weights may results in big changes in the combined objective (Caballero et al., 1997) , and vice versa, a huge different weights may produce nearly similar result (Coello Coello, 1999) ; (iii) it does not work well in the case of a non-convex objective space (Deb, 2011) . One of the most common replacement for the weighted-sum method is the constraint method which is applicable to either convex or non-convex problem. Applying a more efficient MOO solver might be one of the potential extensions of this work. Discussions on the gradient descent solver. Inspired by Sener & Koltun (2018) , in this paper we use multi-gradient descent algorithm (Deb, 2011) as an MOO solver which casts the multi-objective problem to a single-objective problem. While Sener & Koltun (2018) project the weight into the desired simplex, we use parameterization with softmax instead. Although this technique is much faster than Frank-Wolfe algorithm, it has some weaknesses that will be addressed in our future work. More specifically, the GD solver with softmax parameterization cannot handle well the edge case which is the root of the dominating issue. The snippet code E.3 provides a minimal example of quadratic optimization problem as similar in MGDA, where the goal is to find w * = argmin w∈∆w 5 i=1 w i g i 2 2 . The solver is the Gradient Solver with softmax parameterization. With input 1 where none of elements dominates others, the solver works quite reasonable with the weights corresponding to 4 first elements are equal and less than the last one (corresponding to bigger strength). With input 2 where g 5 g 1 , the solver still works well where w 1 = 1 corresponding to the minimal strength g 1 = 0.1. However, with input 3 , the solver fails to find a good solution (which should be w = [1, 0, 0, 0] given that input). It is a worth noting that the main goal of this paper is to show the application of Multi-objective Optimization for generating adversarial examples and the impact of the Task-Oriented regularization. Therefore, while the issue of the gradient descent solver is well recognized, we did not take effort to try with a better solver. 1 import torch 2 import torch.nn.functional as F 3 import torch.optim as optim It is broadly accepted that to fool a model, a feasible approach is maximizing the objective loss (i.e., CE, KL, or CW loss), and the higher the loss, the higher attack success rate. While it is true with the same architecture, we found that it does not hold when comparing different architectures. Figure 5 shows the adversarial loss and the attack success rate for each model in the ENS setting. With the CW loss as the adversarial objective, it can be observed that there is a positive correlation between the loss value and the ASR, i.e., the higher the loss, the higher the ASR. For example, with the same adversarial examples, the adversarial loss on EfficientNet is the highest and so is ASR. However, there is no clear correlation observed when using CE and KL losses. Therefore, the higher weighted loss does not directly imply a higher success rate for attacking an ensemble of different architectures. The MinMax method (Wang et al., 2021) which solely weighs the tasks' losses, therefore, does not always achieve a good performance in all the tasks.

E.5 ADDITIONAL EXPERIMENTS WITH MULTI-TASK LEARNING METHODS

It is worth noting again that our work is the first to apply multi-objective optimization to the adversarial generation task for the classification problem. Therefore, the MOO approach should be considered our naive approach instead of a baseline to compare with. However, it is a worth research question that: Might better multi-objective optimizations with a specific constraint to favor diverse Pareto solutions can improve performance in our setting? Therefore, in this section we would like to provide additional experiments with recent multi-task learning methods to explore how better constrained approaches can improve over the naive MOO. We applied two recent multi-task learning methods PCGrad Yu et al. (2020) and CAGrad Liu et al. (2021a) with implementation from their official repositories into our adversarial generation task. For PCGrad we use the mean as the reduction mode. For CAGrad we use parameter α = 0.5 and rescale = 1 as in their default setting. We experiment on attacking ensemble of models setting with two settings, a diverse set D with 4 different architectures including R-ResNet18, V-VGG16, G-GoogLeNet, E-EfficientNet and a non-diverse set ND with 4 ResNet18 models. It can be seen from the Table 22 that in the diverse ensemble setting, the two additional methods significantly outperform the standard MOO method with the improvement gaps of SAR-All around 3% and 5%, respectively. In the non-diverse ensemble setting, while PCGrad achieves a lower performance than the standard MOO method, CAGrad can outperform the MOO method with a 2.7% improvement. The improvement on the diverse set of PCGrad and CAGrad over the standard MOO method is more noticeable than on the non-diverse set. It can be explained by the fact that on the diverse set of model architectures, there is a huge difference in gradients among architectures, therefore, requires a better multi-task learning method to handle the constraint between tasks. On the other hand, on both ensemble settings, our TA-MOO still achieves the best performance, with a huge gap of (5.8%) 7.8% compared to the second best method on the (non) diverse setting. It is because our method can leverage a supervised signal from knowing whether a task is achieved or not to focus on improving unsuccessful tasks. It is a huge advantage compared to unsupervised multi-task learning methods as MOO, PCGrad, and CAGrad.

E.6 CONFLICTING BETWEEN GRADIENTS IN THE ADVERSARIAL GENERATION TASK

In multi-task learning setting, conflicting between gradient is the common issue to tackle with. More specifically, the gradients with respect to the (shared) model parameter of task f i and task f j can have a negative correlation (i.e., cosine similarity between ∇ θ f i (θ, δ) and ∇ θ f j (θ, δ) is negative). However, in the adversarial generation task, we consider the gradient with respect to the input (e.g., ∇ δ f (θ, δ)) to update the adversarial examples. As we explore through empirical experiments, the issue that we to deal with is not the gradient confliction problem but the gradient domination problem. These gradients with respect to the inputs can have a positive correlation but also have a huge difference in their strengths. In this specific challenge, the standard MOO which solely relies on the gradient strengths to calculated the weight for each task is strongly sensitive to the gradient domination problem and in some cases cannot lead to a good solution as discussed in Appendix E.1 To further support our hypothesis, we would like to provide a measurement on the cosine similarity between gradients on different ensemble members on the ENS setting in Table 23 . Each cell (row-ith, column-jth) of the Table reports the cosine similarity between gradient ∇ δ f i (δ) of model ith and gradient ∇ δ f j (δ) of model jth (w.r.t. the same input δ). It can be seen that the gradients between different architectures has the positive correlation instead of negative correlation. On the other hand, as shown in the last row, the gradient norm ∇ δ f i (δ) varies widely among architectures. While this observation is in line with the widely accepted phenomenon about the transferability of adversarial examples, it also does support our motivation to derive the TA-MOO method to improve the standard MOO.

E.7 DISCUSSION ON THE CONVERGENCE OF OUR METHODS

In multi-task learning, the gradient of each task is calculated with respect to the (shared) model parameter (e.g., ∇ θ f (θ, δ)). Therefore, to quantify the convergence of a multi-task learning method, we can measure the gradient norm of the comment gradient direction to quantify the convergence of the model. The gradient norm is expected to be a very small value when the model reaches to the Pareto optimality points. However, in adversarial generation problem, the gradient of each task is calculated with respect to the input (e.g., ∇ δ f (θ, δ)). Therefore, unlike in the multi-task learning, there is a different behavior of gradient in the adversarial generation task. To verify our hypothesis, we measure the gradient norm of the gradient over all attack iterations and visualize in Figure 6 . It can be seen that the gradient norm of all attacks tends to converge to a large value. It is a worth noting that we use projected gradient descent with l ∞ in all attacks. Therefore, in each attack iteration, the amount to update is not the gradient ∇ δ f (θ, δ) but the sign of it scaling with a step size η δ . However, there is still an interesting observation such that MOO and TA-MOO attack have a much lower gradient norm than other attacks. We would like to propose a simple alternative approach to quantify the convergence of our method in the adversarial generation setting. More specifically, we leverage the advantage of the adversarial generation task such that we can access to the label to audit whether the task is successful or not. Therefore, we simply measure the loss and the success attack rate over all attack iterations as shown in Figure 7 . First, we would like to recall the definition of the Pareto optimality. Given m objective function f (δ) [f 1 (δ), ..., f m (δ)], the Pareto optimality δ * of the multi-objective optimization δ * = argmax δ f (δ) if there is no feasible solution δ such that is strictly better than δ * in some tasks (i.e., f i (δ ) > f i (δ * ) for some i) while equally good as δ * in all other tasks (i.e., f j (δ ) = f j (δ * ), j = i). Bear this definition in mind, it can be seen from the loss progress of MOO attack in Figure 7a that (i) from iteration 1st to around iteration 10th all the losses are increased quickly showing that the method optimize efficiently; (ii) after iteration 10th, the loss w.r.t. the EfficientNet model (i.e., model3 in the legend) continually increases while other losses continually decrease. Therefore, any solution after iteration 10th do not dominate each other indicating that the method reaches the Pareto front. On the other hand, it can be seen from Figure 7b that the loss progress of our TA-MOO is more stable. TA-MOO also can optimize to the optimal point efficiently as MOO does, however, after reaching the peak, the losses in all tasks are more stable than those in MOO. This observation indicates that the solutions after the peak point are also in the Pareto front but are more concentrated than those in MOO. It can explain the stability of the success attack rate in TA-MOO in Figure 7b . Comparing across both MOO and TA-MOO at their last iteration shows that while the loss w.r.t. the EfficientNet model (model3) in MOO is a bit higher than that in TA-MOO, these other losses w.r.t. V/G/E models in MOO is lower than those in TA-MOO. This observation indicates that in term of losses, the solutions of MOO and TA-MOO do not dominate each other. However, the solution of TA-MOO is more stable and leads better final attacking performance.

E.8 ADDITIONAL EXPERIMENTS WITH DIFFERENT INITIALIZATIONS FOR MOO

In our method, the default initialization for the weight w is 1/m equally for all tasks. Therefore, one raising valid concern is that Might better initialization can help to boost the performance?. To answer this question, we first find the optimal initital weight by using the weight at the last iteration when running MOO and TA-MOO attacks with the default initialization. For example, as shown in Figure 8a for the ENS setting with diverse architectures, the average weight that MOO assigns for model R/V/G/E converging to 0.15/0.17/0.15/0.53 (set A), respectively. The average weights' distribution learned by TA-MOO is 0.19/0.25/0.19/0.37 (set B), respectively. It is a worth noting that, we consider each set of weights for each data sample separately, and the above weights are just the average over entire testing set (e.g., 10K sample), while the full statistic (mean ± std) of weights can be seen in Table 2 . In order to make the experiment to be more comprehensive with diverse initializations, we use two additional sets including set C=[0.22, 0.23, 0.22, 0.33] and set D=[0.24, 0.25, 0.24, 0.27]. Given these above four weights sets A/B/C/D, we then init the standard MOO with one of these above sets and adjust the learning rate η w with three options 5e-3, 5e-5, 1e-8 and report results in Table 24 . The complete attacking progress can be seen in Figure 8 . It can be seen from Table 24 that better initialization does help to improve the performance of the standard MOO. The best setting is the initialization with set D and η w = 5e-3 achieves 29.53% in A-All metric, a 4.37% improvement over the default MOO initialization. It can be seen from the evolution of the weights in Figure 8c that even initializing with the converged weights (i.e., set A) from the pre-running attack, the weight of each task does not stand still but converges to a different value. It is another different behavior in adversarial generation task compared to the multi-task learning problem. On the other hand, despite of the extensive tuning, the performance of MOO is still far below the TA-MOO approach, with the gap of 8.48% in A-All metric. 



https://github.com/kuangliu/pytorch-cifar https://github.com/wangjksjtu/minmax-adv



is a differentiable and convex function, the optimization problem in (4) has a unique global minimizer Ω (w) = wproj S (w) , where the projection proj S (w) is defined as proj S (w) = argmin π∈S w -π 2 2 .

Recently,Guo et al. (2020) proposed a multi-task adversarial attack and demonstrated on the universal perturbation problem. However, whileWang et al. (2021) and ours can be classified as an iterative optimization-based attack, Guo et al. (2020) requires a generative model in order to generate adversarial examples. While this line of attack is faster than optimization-based attacks at the inference phase, it requires to train a generator on several tasks beforehand. Due to the difference in setting, we do not compare with that work in this paper.More recently, Qiu et al. (2022)  proposed a framework to attack a generative Deepfake model using the multi-gradient descent algorithm in their backpropagation step.

Figure 1: Visualization of standard MOO and TA-MOO solutions in a scenario of 2 goal-achieved tasks (∇f s 1,2 ) and 2 goal-unachieved tasks (∇f u 1,2 ). (left) MOO; (middle) MOO on the set of goal-unachieved tasks only; (right) TA-MOO with a solution focuses more on the goal-unachieved tasks.

Figure 2: Comparison progress of three adversarial training methods. The bigger marker size represents the later epoch. Each point represents the natural accuracy and robust accuracy against PGD-Linf attack on the testing set.

Figure 3: Sensitivity to the parameter λ.

including 3 robust models: ResNet18 (model ID: Salman2020Do_R18 (Salman et al., 2020), robust accuracy 25.32%), ResNet50 (model ID: Salman2020Do_R50 (Salman et al., 2020), robust accuracy 34.96%) and ResNet50 (model ID: Wong2020Fast (Wong et al., 2019), robust accuracy 26.24%). The second set is the less-robust ensemble (LE) which includes 3 models: ResNet18 (model ID: Salman2020Do_R18), ResNet50 (model ID: Salman2020Do_R50) and the standard training ResNet50 (model ID: Standard_R50, robust accuracy 0%). We use both targeted attack and untargeted attack settings, with = 4/255 , and η = 1/255 with 20 steps. We use 5000 images of the validation set to evaluate.

Figure 4: Comparison on the ASR of each individual task. R: ResNet18, V: VGG16, G: GoogLeNet, E: EfficientNet. CE: Cross-entropy loss, KL: Kullback-Leibler divergence, CW: Carnili-Wagner loss

torch.tensor(input_3) 11 alpha = torch.tensor(init_alpha, requires_grad=True) 12 opt = optim.SGD([alpha], lr=1.0) step={}, w={}'.format(step, w.detach().numpy())) 21 22 # Result with input_1 23 # step=19, w=[0.20344244 0.20344244 0.20344244 0.20344244 0.18623024] 24 # Result with input_2 Listing 1: Python example of the Gradient Solver with softmax parameterization E.4 CORRELATION BETWEEN THE OBJECTIVE LOSS AND ATTACK PERFORMANCE.

Figure 5: Loss (left fig) and ASR (right fig) of each task over all attack iterations with the MinMax method. model0/1/2/3 represents R/V/G/E architecture, respectively.

Figure 6: Norm of the gradient ∇ δ f (δ) over all attack iterations. Measure on the diverse set of the ENS setting, with CE loss.

Figure 7: Loss (left fig) and SAR (right fig) of each task over all attack iterations. model0/1/2/3 represents R/V/G/E architecture, respectively. The CW loss is used as the adversaries's objective function.

Figure 8: Weight (left fig) and SAR (right fig) of each task over all attack iterations. model0/1/2/3 represents R/V/G/E architecture, respectively.

).

Evaluation of Attacking Ensemble model on the CIFAR10 and CIFAR100 datasets.

Attacking Ensemble model with a diverse set D={R-ResNet18, V-VGG16, G-GoogLeNet, E-EfficientNet} and non-diverse set ND={4 ResNets}. w represents the final w of MOO (mean ± std). ∇ δ f i (δ) represents the gradient norm of each model (mean ± std).



Evaluation on the Transferability of adversarial examples. Each cell (row-ith, column-jth) reports SAR (higher is better) of adversarial examples from the same source architecture (RME) with an adversary at row-ith to attack an ensemble at column-jth. Each architecture has been denoted by symbols such as R: ResNet18, M: MobileNet, E: EfficientNet, V: VGG16, W: WideResNet. For examples, RME represents for an ensemble of ResNet18, MobileNet and EfficientNet.

Robustness evaluation of Adversarial Training methods on the CIFAR10 dataset. RME represents an ensemble of ResNet18 (R), MobileNet (M) and EfficientNet E), while MobiX3 represents an ensemble of three MobileNets. NAT and ADV measure the natural accuracy and the robust accuracy against PGD-Linf attack (↑the higher the better). Other metrics measure the success attack rate (SAR) of adversarial examples generated by the same PGD-Linf attack on fooling each single member and all members of the ensemble (↓the lower the better).

Evaluation of generating Universal Perturbation on the CIFAR10 and CIFAR100 datasets.

Robust adversarial examples against transformations evaluation. .81 36.23 33.93 87.47 71.05 37.68 40.21 42.12 TA-MOO 31.10 55.26 44.15 41.86 85.19 71.86 45.53 48.70 49.54 C100 Uniform 56.19 76.23 70.43 69.01 87.66 87.36 71.40 74.25 73.47 MinMax 59.75 75.72 70.13 69.26 87.45 86.03 71.54 73.30 72.32 MOO 53.17 74.21 66.96 65.68 89.16 87.03 68.49 71.11 71.06 TA-MOO 60.88 76.71 70.43 69.37 89.11 87.95 71.70 74.73 73.69

Robustness performance of models in the experiments × 10 -4 ) and Cosine Annealing Scheduler to adjust the learning rate with an initial value of 0.1 and train a model in 200 epochs as suggested in the implementation above. We use PGD-AT L ∞ (Madry et al., 2018) with the same setting for both CIFAR10 and CIFAR100 datasets, i.e., perturbation limitation = 8/255, k = 20 steps, and step size η = 2/255.Method settings. In this work, we evaluate all the methods in the untargeted attack setting with L

Data transformation setting. U represents uniform sampling function and p represents probability to excuse a transformation (e.g., flipping). .1  TRANSFERABILITY OF ADVERSARIAL EXAMPLES IN THE ENS SETTINGWe conduct an additional experiment to evaluate the transferability of our adversarial examples. We use an ensemble (RME) of three models: ResNet18, MobileNet, and EfficientNet as a source model and apply different adversaries to generate adversarial examples to this ensemble. We then use these adversarial examples to attack other ensemble architectures (target models), for example, RMEVW is an ensemble of 5 models including ResNet18, MobileNet, EfficientNet, VGG16 and WideResNet. Table9reports the SAR-All metric of transferred adversarial examples, where a higher number indicates a higher success rate of attacking a target model, therefore, also implies a higher transferability of adversarial examples. The first column (heading RME) shows SAR-All when adversarial examples attack the source model (i.e., the whitebox attack setting).The Uniform strategy achieves the lowest transferability. It can be observed from Table9that the Uniform strategy achieves the lowest SAR in the whitebox attack setting. This strategy also has the lowest transferability in attacking other ensembles (except an ensemble RVW). 's transferability drops on dissimilar target models. While MinMax achieves the second-best performance in the whitebox attack setting, its adversarial examples have a low transferability when target models are different from the source model. For example, in the target model RVW where there is only one member of the target model from the source model (RME) (i.e., R or ResNet18), MinMax achieves a 23.75% success rate which is lower than the Uniform strategy by 1.28%. Similar observation can be observed on target models EVW and MVW, where MinMax outperforms the Uniform strategy by just 0.2% and 0.6%, respectively.TA-MOO achieves the highest transferability on a diverse set of ensembles . Our TA-MOO adversary achieves the highest attacking performance on the whitebox attack setting, with a huge gap of 9.24% success rate over the Uniform strategy. Our method also achieves the highest transferability regardless diversity of a target ensemble. More specifically, on target models such as REV, MEV, and RMEV, where members in the source ensemble (RME) are also in the target ensemble, our TA-MOO significantly outperforms the Uniform strategy, with the highest improvement is 5.19% observed on target model RMEV. On the target models EVW and MVW which are less similar to the source model, our method still outperforms the Uniform strategy by 1.46% and 1.65%. The superior performance of our adversary on the transferability shows another benefit of using multi-objective optimization in generating adversarial examples. By reaching the intersection of all members' adversarial regions, our adversary is capable to generate a common vulnerable pattern on an input image shared across architectures, therefore, increasing the transferability of adversarial examples.

Evaluation on the Transferability of adversarial examples. Each cell (row-ith, column-jth) reports SAR (higher is better) of adversarial examples from the same source architecture (RME) with an adversary at row-ith to attack an ensemble at column-jth. Each architecture has been denoted by symbols such as R: ResNet18, M: MobileNet, E: EfficientNet, V: VGG16, W: WideResNet. For examples, RME represents for an ensemble of ResNet18, MobileNet and EfficientNet. The highest/second highest performance is highlighted in Bold/Underline. The table is copied from Table3in the main paper for reading comprehension purpose. MobileNet and EfficientNet (diverse set). To evaluate the adversarial robustness, we compare natural accuracy (NAT) and robust accuracy (ADV) against PGD-Linf attack of these adversarial training methods (the higher the better). We also measure the success attack rate (SAR) of adversarial examples generated by the same PGD-Linf attack on fooling each single member and all members of the ensemble (the lower the better). We use k = 10, = 8/255, η = 2/255 for adversarial training and PGD-Linf with k = 20, = 8/255, η = 2/255 for robustness evaluation. We use SGD optimizer with momentum 0.9 and weight decay 5e-4. Initial learning rate is 0.1 with Cosine Annealing scheduler and train on 100 epochs. The root of the improvement is the ability to generate stronger adversarial examples in the the sense that they can challenge not only the entire ensemble model but also all single members. These adversarial examples lie in the joint insecure region of members (i.e., the low confidence region of multiple classes), therefore, making the decision boundaries more separate. As a result, adversarial training with TA-MOO produces more robust single models (i.e., lower SAR-Avg) and significantly reduces the transferability of adversarial examples among members (i.e., lower SAR-All). These two conditions explain the best ensemble adversarial robustness achieved by TA-MOO.

Robustness evaluation of Adversarial Training methods on the CIFAR10 dataset. RME represents an ensemble of ResNet18 (R), MobileNet (M) and EfficientNet E), while MobiX3 represents an ensemble of three MobileNets. NAT and ADV measure the natural accuracy and the robust accuracy against PGD-Linf attack (↑the higher the better). Other metrics measure the success attack rate (SAR) of adversarial examples generated by the same PGD-Linf attack on fooling each single member and all members of the ensemble (↓the lower the better). The highest/second highest robustness is highlighted in Bold/Underline. The most important metric is emphasized in blue.

Evaluation of generating Universal Perturbation on the CIFAR10 and CIFAR100 datasets.

Evaluation of generating Universal Perturbation (K=8) on the CIFAR10 dataset with ResNet18 architecture and MOO method. {T i } K i=1

Robust adversarial examples against transformations evaluation. The highest/second highest performance is highlighted in Bold/Underline. MOO and TA-MOO represent version with memory to overcome the infinite gradient issue in the stochastic setting.



Average time per sample for generating adversarial example. All experiments are measured on the CIFAR10 dataset, EoT and Uni are with ResNet18 architecture.

Stability of experiments' evaluation on different random seeds. Experiment on the ENS setting, with an ensemble of 4 models: Resnet18, VGG16, GoogleNet and EfficientNet. MOO 37.56 ± 0.32 51.15 ± 0.21 49.37 ± 0.15 52.80 ± 0.45 48.98 ± 0.25 53.24 ± 0.13



Evaluation attacking performance on the ImageNet dataset. RE/LE/TAR/UNTAR represents Robust Ensemble/Less-Robust Ensemble/Targeted Attack/Untargeted Attack, respectively. R18/R50/STD represents robust ResNet18, robust ResNet50 and standard ResNet50 pre-trained model, respectively. The most important metric is emphasized in blue.

Evaluation on the EoT setting with the ImageNet dataset. The most important metric is emphasized in blue.

Evaluation of Attacking Ensemble model on the CIFAR10 (C10) and CIFAR100 (C100) datasets. The highest/second highest performance is highlighted in Bold/Underline. The table is copied from Table1in the main paper for reading comprehension purpose.

be seen that the gradient strength corresponding to the EfficientNet architecture (mean value is 0.98) is much lower than those of other architectures (mean values are at least 4.29). As the result, the weight corresponding to the EfficientNet architecture is much higher than those of others.

Attacking Ensemble model with a diverse set D={R-ResNet18, V-VGG16, G-GoogLeNet, E-EfficientNet} and non-diverse set ND={4 ResNets}.

Correlation between gradients of ensemble members on ENS setting. Each cell (row-ith, column-jth) reports the cosine similarity (mean ± std) between gradient ∇ δ f i (δ) of model ith and gradient ∇ δ f j (δ) of model jth (w.r.t. the same input δ). The last row ∇ δ f i (δ) reports the gradient norm of each model. R: ResNet18, V: VGG16, E: EfficientNet, G: G-GoogLeNet.

Attacking Ensemble model with a diverse set D={R-ResNet18, V-VGG16, G-GoogLeNet, E-EfficientNet}. MOO A/B/C/D is MOO with initial weights from set A/B/C/D, respectively. η w denotes the learning rate to update for the weight w. η w = 5e-3 η w = 5e-5 η w = 1e-8

APPENDIX

The Appendix provides technical and experimental details as well as auxiliary aspects to complement the main paper. Briefly, it contains the following:• Appendix A: Discussion on related work.• Appendix B: Detailed proof and an illustration of our methods.• Appendix C: Detailed description of experimental settings. • Appendix D.6: Additional experiments on sensitivity to hyper-parameters.• Appendix D.7: Additional comparison with standard attacks on attacking performance.• Appendix D.8: Additional experiments on attacking the ImageNet dataset.• Appendix E.1: Additional discussions on the dominating issue and when MOO can work.• Appendix E.2: A summarary on the importance of Task-Oriented regularization.• Appendix E.3: Discussion on the limitation of MOO solver.• Appendix E.4: Discussion on correlation between the objective loss and attack performance.• Appendix E.5: Additional experiments with multi-task learning methods.• Appendix E.6: Discussion on the conflicting between gradients in the adversarial generation task.• Appendix E.7: Discussion on the convergence of our methods.• Appendix E.8: Additional experiments with MOO with different initializations.

A RELATED WORK

Multi-Objective Optimization for multi-task learning. (Désidéri, 2012) proposed a multigradient descent algorithm for multi-objective optimization (MOO) which opens the door for the applications of MOO in machine learning and deep learning. Inspired by Désidéri (2012) , MOO has been applied in multi-task learning (MTL) (Sener & Koltun, 2018; Mahapatra & Rajan, 2020) , few-shot learning (Ye et al., 2021) Generating adversarial examples with single-objective and multi-objective optimizations. Generating qualified adversarial examples is crucial for adversarial training (Madry et al., 2018; Zhang et al., 2019) . Many perturbation based attacks have been proposed, notably FGSM (Goodfellow et al., 2015) , PGD (Madry et al., 2018) , TRADES (Zhang et al., 2019) , CW (Carlini & Wagner, 2017) , BIM (Kurakin et al., 2018) , and AutoAttack (Croce & Hein, 2020) . Most adversarial attacks aim to maximize a single objective, e.g., maximizing the cross-entropy (CE) loss w.r.t. the ground-truth label (Madry et al., 2018) , maximizing the Kullback-Leibler (KL) divergence w.r.t. the predicted

