ADVERSARIAL DETECTOR FOR DECISION TREES EN-SEMBLES USING REPRESENTATION LEARNING Anonymous authors Paper under double-blind review

Abstract

Research on adversarial evasion attacks focuses mainly on neural network models. Among other reasons, this is because of their popularity in certain fields (e.g., computer vision and NLP) and the models' properties, making it easier to search for adversarial examples with minimal input change. Decision trees and tree ensembles are still very popular due to their high performance in fields dominated by tabular data and their explainability. In recent years, several works have defined new adversarial attacks targeting decision trees and tree ensembles. As a result, several papers were published focusing on robust versions of tree ensembles. This research aims to create an adversarial detector for attacks on an ensemble of decision trees. While several previous works have demonstrated the generation of more robust tree ensembles, the process of considering evasion attacks during ensemble generation can affect model performance. We demonstrate a method to detect adversarial samples without affecting either the target model structure or its original performance. We showed that by using representation learning based on the structure of the trees, we achieved better detection rates than the state-ofthe-art technique and better than using the original representation of the dataset to train an adversarial detector. Under review as a conference paper at ICLR 2023 different routes of a tree ensemble. (ii) We designed a pipeline to train and evaluate adversarial detection with reduced possibilities of overfitting or bias. Chen et al. ( 2019) proposed a robust decision trees technique against adversarial evasion attacks. The model training algorithm was changed, as a result of which model itself was changed. As part of the experiment, the new model's accuracy was checked and compared to the non-robust model. Of the eleven datasets tested, seven showed a decrease in accuracy. Our primary motivation for this work is to create a defense layer for a decision tree ensemble against adversarial attacks. Our defense layer does not affect the model itself, allowing the model owner to decide if they want defense applied to their existing system. Secondly, production tree-based models use well-known open-source libraries such as XGBoost (Chen & Guestrin, 2016b), CatBoost (Dorogush et al., 2018), and LightGBM (Ke et al., 2017). These libraries are heavily used, tested, and improved, which is partly why they were chosen in the first place. Currently, as of writing this paper, the above libraries do not contain an official version that is robust against adversarial attacks. Therefore, to add adversarial robustness to a model, it is necessary to use a different third-party version of the model or develop a new one. BACKGROUND We can split the field of adversarial learning into three primary sectors: attacking methods, defending methods, and detectors which aim to detect whether or not a sample is adversarial without changing the model itself. Generating Adversarial Samples. Early work around generating adversarial samples (Goodfellow et al., 2014; Kurakin et al., 2016) used backpropagation to try and discover which input features we should change to maximize the loss function of a model. In the face of more recent attacks, a different loss functions was suggested to find an adversarial sample (Papernot et al., 2016b; Carlini & Wagner, 2017; Cheng et al., 2018) . Other works used concepts from geometry and the location of the boundaries between decision spaces to search for a minimal perturbation for creating an adversarial sample (Moosavi-Dezfooli et al., 2016; Yang et al., 2020) . Because decision tree classifiers are not a continuous space model, earlier backpropagation methods will not work in these cases. Black-box methods, which ignore the internal inner structure of the model, try to approximate the gradients and can generate attacks for decision-trees-based classifiers using multiple queries to find the boundary between different classes in the unknown decision space (Cheng et al., 2018; Chen et al., 2020) . Some relevant white-box techniques focus specifically on the nature of decision trees (Papernot et al., 2016a; Kantchelian et al., 2016; Zhang et al., 2020). Papernot et al. (2016a) defined an algorithm to search for a given sample, the closest leaf in the neighborhood of the original leaf, and perturb the features between them. Kantchelian et al. (2016) formulated a set of equality and inequality constraints based on the tree structure to generate an optimal adversarial sample for tree ensembles using a mixed-integer linear program. Model Defenses Against Adversarial Attacks. Common approach for protecting models is to train a robust model for evasion attacks. Adversarial training (Goodfellow et al., 2014) is one of these methods, with which one can generate adversarial samples and add them to the training data. Other suggested solutions use known techniques with other purposes, such as knowledge distillation, as shown in Papernot et al. (2016c), which used its traits to create a newer model version with smaller gradients to make it more difficult to generate adversarial samples. Another work is Wang et al. (2018) , which used dropout in prediction time to reduce the dependency on specific neurons in a neural network.

1. INTRODUCTION

In recent decades we have seen the introduction of machine learning algorithms in production environments into various fields such as medical imaging (Zhou et al., 2021) , autonomous driving (Huang & Chen, 2020 ) and law enforcement (Vestby & Vestby, 2019) . With the leap in performance of those models and their integration into real-life systems, people began to investigate how to bypass classifiers and defend against those malicious attempts (Dalvi et al., 2004; Lowd & Meek, 2005) . Many papers have addressed examples of adversarial attacks that make small changes that are hard for a human to notice in the inputs of a machine learning model, usually a neural network, so their predictions are wrong. These can be exploited by a malicious actor and used to bypass a model that might, for example, be responsible for a critical classification task affecting people's lives. As a result, various researchers published techniques to detect and defend against adversarial attempts. Most of the research is focused on adversarial attacks targeting neural network models, among other things, because of the nature of their continuous learning space, which allows a gradient ascent process to maximize the model's loss function given a specific input. Thus defenses and detectors mainly target neural network models as well. Tree-based models continue to be very popular, especially for tabular data tasks (Nielsen, 2016 ; Shwartz-Ziv & Armon, 2022; Grinsztajn et al., 2022) , because they usually demand less data and are more interpretable. There are fewer studies on adversarial attacks and defenses affecting decision tree models. Gradient-descent-based methods commonly used in earlier attack models cannot be applied directly to evade decision trees due to the discrete nature of their non-differentiable decisionmaking paths and tree-splitting rules. Unfortunately, this does not mean that decision trees are unaffected by evasion attacks. In this work, we present a detection technique for adversarial evasion attacks against tree-based classifiers, focusing on boosting ensembles. Our main contributions are: (i) We defined a task that allows us to generate sample representations that rely on the distribution of the dataset in the Defending decision tree classifiers combine the structure of the trees with the methods mentioned above. Adversarial boosting was suggested in Kurakin et al. (2016) with an idea similar to adversarial training, and in each boosting round, adversarial samples are created and added to the next round's training data. Other works show how to generate more robust tree models with new optimization formulations while considering evasion attacks and perturbations (Chen et al., 2019; Andriushchenko & Hein, 2019; Calzavara et al., 2020; Vos & Verwer, 2021; 2022) . Adversarial Detectors. There are two main approaches when building an adversarial attack detector: using statistics and hypothesis testing to investigate if there is any difference between the regular samples distribution and the adversarial sample's distribution (Feinman et al., 2017; Grosse et al., 2017; Katzir & Elovici, 2019) , and training a machine learning classifier to act as a detector of adversarial attempts (Metzen et al., 2017; Fidel et al., 2020) . Recent work used the combinations of leaves in tree ensemble predictions, called output configurations, to detect abnormal leaves configuration of a sample compared to a reference dataset to detect an adversarial sample using a defined metric called OC-score (Devos et al., 2022) . We will compare our work to Devos et al. (2022) , which is considered the state-of-the-art in detecting adversarial samples on decision trees at the time of writing this paper.

3.2. PROBLEM FORMULATION

For a given classifier C, a given sample x, with set of features F and a label y in which C(x) = y, an adversarial sample x ′ is defined using an adversarial perturbation δ : x ′ = x + δ. An adversarial sample can be generated by a targeted or untargeted attack. For an untargeted attack we want to find δ that meet with the condition: C(x ′ ) ̸ = y s.t. ||δ|| p < ϵ Which means that C(x) ̸ = C(x ′ ) where the p-norm of δ will be limited by a value ϵ. For a targeted attack we need to define a target class t where t ̸ = y and: C(x ′ ) = t s.t. ||δ|| p < ϵ For p ∈ N 1 (All natural numbers without zero) the p-norm is defined: ||δ|| p := (Σ F i=1 (δ i ) p ) 1 p ) For p = ∞ -measures the largest absolute difference between two features and is defined: ||δ|| ∞ = max xi |x i -x ′ i | Given a decision tree model T and a sample x, our classification task is to detect whether the sample is normal or is an attempted adversarial attack.

4.1. METHOD GENERAL FLOW

Our method consists of 11 main steps: 1. Split the dataset into four different parts for different purposes: S T to train the tree model, S E to train our basic representation model, S D-train to train our adversarial detector and S D-test to evaluate our adversarial detector. 2. Train a tree model. 3. Generate a triplets dataset that will be used to initialize the new representations. This is explained more fully in Subsection 4.2. 4. Train our basic embedding model E. This is explained more fully in Subsection 4.3. 5. Split S D-train and S D-test into two parts. 6. Generate adversarial samples using an attack method A. 7. Generate a new triplet dataset for each of the new sub-datasets. This is explained more fully in Subsection 4.4. 8. Optimize the representations of the new sub-datasets to our new embeddings using E, more details in Subsection 4.4.

9.

Concatenate the new representation of every set to the original ones. 10. Train our adversarial detector. This is explained more fully in Subsection 4.5. 11. Evaluate our adversarial detector. This is explained more fully in Section 5. A detailed visualization sketch shown in Figure 28 in Appendix H, together with a further explanation about the dataset splitting.

4.2. DATASET REPRESENTATION

At the heart of our method is the idea that we want to extract a new representation of a dataset based on the structure of a target tree ensemble model to understand the behavior of normal samples and detect adversarial samples. To extract a meaningful representation, we took inspiration from an embedding process. For each sample, we assign a vector of random numbers with d dimensions, called latent features, that will be optimized using a gradient descent process using a simple feedforward neural network based on the structure and traits of the tree. For a given dataset S and a trees model T we define a new dataset: R T S = {(s i , s j , n k )|s i , s j ∈ S, n k ∈ N T S , s i ∈ n k ∧ s j ∈ n k } Where N T S is the set of all internal nodes of T reached by at least one sample from S, n is a single node, and s is a single sample. Each triplet in R T S contains two samples and a node. Both samples reach the internal node n k in T . We define the supervised task below: f (s i , s j , n k ) = 1 if s i and s j pass to the same child of n k 0 otherwise To collect this new dataset, we take our original dataset and traverse with each of the samples through the different routes in the trees in the ensemble and save the routes aside. During the process, we document which sample passes in which node and create the mapping: M T S (n k ) = S n k Which returns for a given node all the samples which reached it during the above traverse. Then to generate R T S we can use Algorithm 1. Algorithm 1 Generating triplets dataset to train the basic embedding model to fine-tune future representations input : set of nodes N T S , model T , nodes to samples mapping M T S , size of final dataset N output: R T S , labels 1: for i = 1 to N do 2: Sample a random node n k from N T S 3: Sample 2 random samples s i ,s j from M T S (n k ) 4: triplet i ← (s i , s j , n k ) 5: if s i and s j agree on the condition in n k then 6: label i ← 1 7: else 8: label i ← 0 9: end if 10: end for Algorithm 1 returns a list of triplets constructed from 2 samples and one node and a list of labels based on the above logic. For a chosen N, which is the final size we choose for the dataset R T S , we generate the triplets described above. After sampling for a node and relevant samples, in line 5, we check if both samples agree on the condition of the feature threshold in n k , in which case they move to the same child on n k , and the label of the new sample is set to 1. Otherwise, it is set to 0.

4.3. EMBEDDING MODEL

To generate our The basic embedding model is trained using R T S E . We can see in Figure 1 -(a) a general sketch of the architecture we used. During the embedding model-training phase, we use two embedding matrices, one for the samples and one for the nodes, and we optimize the representation with a gradient descent process based on the task defined earlier. Each sample vector is initiated with a random vector of dimension d s and each node with a random vector of dimension d n . The vectors of each sample and the one representing the node are concatenated and passed through a single feed-forward layer with a ReLU activation function and finish in a feed-forward layer with a sigmoid activation function used to evaluate the loss value. The model is trained with a binary cross-entropy loss function and an Adam optimizer. Then, the model weights will be saved to optimize the new sample's representations.

4.4. NEW SAMPLES EMBEDDING

When we want to extract the representation of new samples set S new , we first create a new dataset: R T S E ,Snew = {(s i , s j , n k )|s i ∈ S E , s j ∈ S new , n k ∈ N T Snew , s i ∈ n k ∧ s j ∈ n k } (6) This means the triplet in this new dataset is constructed from one sample from the samples used to train the embedding model (S E ) and another sample from the new sample set whose representation we want to optimize. The labels are set in the same manner as we described before, based on the fact that the two samples align with the feature threshold in the relevant node. We use algorithm 2 to construct this dataset. Algorithm 2 Generating triplets dataset from a new sample set to optimize its new representations input : set of nodes N T Snew , model T , a mapping between nodes to samples which were used to train the embedding model M T S E , mapping between nodes to samples from the new dataset M T Snew , size of final dataset N output: R T S E ,Snew , labels 1: for i = 1 to N do Sample a random samples s j from M T Snew (n k ) 5: triplet i ← (s i , s j , n k ) 6: if s i and s j agree on the condition in n k then 7: label i ← 1 8: else 9: label i ← 0 10: end if 11: end for Similarly to Algorithm 1, in Algorithm 2 we choose N, which is the size we choose for the dataset R T S E ,Snew . In line 3, we take random samples from M T S (n k ), a sample that was used to train the basic representations that reached n k . In line 4, we sample random samples from M T Snew (n k ) which is a sample from the new dataset whose representation we want to optimize. Then, in line 6, we check if both samples agree on the condition of the feature threshold in n k , in which case they move to the same child of n k , and the label of the new sample is set to 1. Otherwise, it is set to 0. Algorithm 2 returns a list of triplets constructed from two samples and one node and a list of labels. Then the weights of the trained embedding model are loaded to the same architecture with all weights frozen (both layers' weights and biases, embedding matrix of the S E samples and embedding matrix of the nodes), and a new unfrozen embedding matrix for the new samples is initialized as described in Figure 1-(b) . The representations are then optimized with a gradient descent process while using R T S E ,Snew . 2017), we trained an adversarial samples detector based on a classifier to try and classify whether a specific sample is a normal sample that came from the original distribution of the input samples of the original dataset or an adversarial sample. We extract four new sub-datasets from the datasets used for training and evaluating the detector. We extract two from each one, a set that stayed as the normal samples, and using the other set, we generate adversarial samples and then throw the original samples away. Afterward, we extracted each dataset's new representations using the process described in Subsection 4.4. We used an XGBoost as the classifier of the detector. As an input to the classifier, we concatenated our new extracted representations to the original features and used them together as the final features set.

5.1. EXPERIMENTAL SETUP

In our experiments, we tested the performance of our method of creating new representations by training an adversarial-evasion-attack detector on 18 datasets, described in Subsection 5.2, with five different attacks, described in Subsection 5.3. We compared our method to OC-score (Devos et al., 2022) , considered the state-of-the-art for that task, and to detector classifiers that we trained on the original representations of the datasets. We tested our method against two tree-based ensembles: We used ROC-AUC and PR-AUC as our metrics. For multi-class datasets, each metric is calculated for each of the different labels in a 1-vs-all manner, and eventually, an average is calculated. Our code is online in githubfoot_0 .

5.2. EVALUATED DATASETS

In our experiments, we tested with 18 classification datasets, some of them binary and others multiclass. The datasets are described in Table 

5.3. EVALUATED ADVERSARIAL ATTACKS

We evaluated our method with untargeted attacks, with 2 different norms -L 2 and L ∞ . For the attack methods, we used four black-box attacks which are relevant to tree-based models: Sign-Opt attack (Cheng et al., 2019) , OPT attack (Cheng et al., 2018) , HopSkipJump attack (Chen et al., 2020) and Cube attack (Andriushchenko & Hein, 2019) . We used one white-box attack specifically for trees-based models: Leaf-Tuple attack (Zhang et al., 2020) . To execute our attacks we used implementation published by Zhang et al. ( 2020) on githubfoot_2 .

5.4. EXPERIMENTS RESULTS

We applied our method and calculated ROC-AUC and PR-AUC for each combination of the norm, attack method, dataset, and model algorithm. We calculated the difference between our method's performance, the OC-score method's performance, and the performance of a detector trained on the original representation. All of the distributions of the differences are shown in Appendix A as boxplots. Due to space constraints, we only show here plots comparing our method to OC-score for L 2 norm, in Figure 2 , for experiments targeting XGBoost tree ensembles and in Figure 3 , for experiments that targeting RandomForest tree ensembles. Each row is a different attack method, each white point is an experiment on a specific dataset, and the red vertical line is a total mean of all the experiments together. The figures show the spread of the differences in ROC-AUC and PR-AUC for each norm and attack combination. The boxplot shows us the different quartiles and the median. As we can see in the figures, the total mean and the median of each section is positive, which means that our metrics yielded better results for the new representation in most of our experiments. The full raw metrics for each one of the experiments are shown in tables in Appendix B for XGBoost and in appendix C for RandomForest. Of our 338 experiments, our method had the best performance in 107 of them and was tied with one of the other methods as the best in 134 other experiments, which means our method was successful in 71.73% of the experiments. We used the Friedman test on the ROC-AUC metric to validate the statistical significance of differences between the evaluated methods and datasets (Demšar, 2006) . The above test is a non-parametric test that does not assume anything about the results distributions and is used when dealing with multiple methods and multiple datasets. The null hypothesis that the 3 methods perform the same was rejected with F F (338, 3) = 288 with p < 0.01. As a second step, we used Nemneyi post-hoc test, which is often used as a second step if it is possible to reject the null hypothesis with a Friedman test and test for superiority between the different methods, and concluded that our new representation method outperforms the OC-score method and the usage of a machine learning detector based on the original representations with p < 0.01.

6. DISCUSSION

As we showed in our experiments, using representation learning to learn a better version of the dataset given a trained model helps to extract better performance when trying to protect our model against adversarial evasion attacks. We showed this on different datasets, different tree-based ensembles, and with different attack methods. An important area for discussion is adversarial attacks on tabular data, which are not as intuitive as adversarial attacks on more continuous inputs such as images or audio. These usually clip the data to a valid range of values, which is not straightforward for tabular data. This issue is not yet heavily researched but was already approached and discussed by Calzavara et al. (2020) and Vos & Verwer (2021) , where for each dataset, they defined a set of rewriting rules and possible budget with which to change the features. We chose not to take up that challenge in this work, but for full context, we added in Appendix D statistics regarding our generated adversarial samples. While analyzing our results, we noticed an interesting outcome of our new representationsgeneration process. We used UMAP (McInnes et al., 2018) to extract 2-dimensional versions of the original representations and our new ones. As we can see in Figure 4 on the left side, the reduction of the original representations, there the adversarial samples are scattered between the normal samples. When looking at the reduced version of our representations on the right, we see that the adversarial samples gather closely to each other and are separated from most of the rest of the normal samples. 

7. CONCLUSIONS AND FUTURE WORK

In this work, we presented a method to train an adversarial samples detector based on new representations of a dataset based on the target model trained on it. We showed that using our new representation shows overall improvement compared to the current state-of-the-art detection of adversarial samples on tree models without changing the original model internals and compared to using the original samples representations to train a detector classifier. As for future work, our new method should also be tested on robust versions of tree ensembles to see if it affects the results. Another direction is to create a unified detector for all attacks together. Our detector method was based on generating adversarial samples for each attack method and training a separate model. However, the normal data behave the same regardless of the attack method, and a new unified approach should be researched. Chong Zhang, Huan Zhang, and Cho-Jui Hsieh. An efficient adversarial attack for tree ensembles. arXiv preprint arXiv:2010.11598, 2020. 

B APPENDIX B -XGBOOST FULL EXPERIMENTS RESULTS

List of tables with the raw experiments metrics for each dataset, attack method, and norm for the XGBoost target experiments. In bold is the method or methods that achieved the highest value. Rows that fill in hyphens are cases where the adversarial sample creation process failed.

C APPENDIX C -RANDOMFOREST FULL EXPERIMENTS RESULTS

List of tables with the raw experiments metrics for each dataset, attack method, and norm for the RandomForest target experiments. In bold is the method or methods that achieved the highest value. Rows that fill in hyphens are cases where the adversarial sample creation process failed. 

D APPENDIX D -ADVERSARIAL SAMPLES INFORMATION

In the following tables, we calculated several statistics about the perturbations generated in our various experiments split by target model type and attack method. Interesting behaviors we noticed: • We can see that for SignOPT and OPT attacks in all of the experiments, a perturbation was applied to all of the features (max features changed is equal to the mean, which is also equal to the number of features for each dataset). • The mean perturbation size for datasets adult and drybean seem very high for attack methods HopSkipJumpAttack, Cube, and LeafTuple. 

E APPENDIX E -DETECTOR ABLATION STUDY

We wanted to investigate how three aspects of our detector affect our metrics results: 1. The dimensions of our new representations (for the samples and for the nodes). 2. The machine learning classifier that was chosen as the final adversarial detector. 3. Key hyperparameters of the final adversarial detector. To do our tests, we took two different experiments and showed their results: • XGBoost target with codrna dataset and OPT attack method. • RandomForest target with sensorless dataset and HopSkipJumpAttack attack method. Each one of the above with the two different norms.

E.1 DIFFERENT CLASSIFIER & HYPERPARAMETERS

We checked three different classifiers as our models for our adversarial evasion attack detector: XGBoost, RandomForest, and K-nearest neighbors (KNN). For the tree ensemble classifiers, we checked how much the number of estimators and their depth impacts the performance of the detector. For the KNN classifier, we tested the results with different values of K.

E.1.1 XGBOOST DETECTOR HYPERPARAMETERS

As we can see from the results in Figures 13, 14, 15, and 16 that in general the number of estimators has a large impact on both ROC-AUC and on PR-AUC until a certain point which is around 50 estimators and above usually there is improvements, but relatively smaller. In most cases, the maximum depth of the trees has a very light impact on the results when the number of estimators is 50 or more. 



https://github.com/anonymous/anonymous https://github.com/chenhongge/RobustTrees/blob/master/data/download data.sh https://github.com/chong-z/tree-ensemble-attack



samples s i from M T S E (n k ) 4:

Figure 1: (a) -General architecture sketch of embedding model. (b) -changes were done to fine-tune the new samples' representations. The elements in red are frozen weights and biases which remain unchanged during a gradient descent process.

Figure 2: XGBoost experiments metrics differences between the new method and OC-score for L 2 norm.

Figure 4: Dimensionality-reduction visualization using UMAP (McInnes et al., 2018). Green points are normal samples, and red points are adversarial samples. On the left, we can see a reduction of the original representations, and on the right, we can see a reduction of the new embedded representations.

Figure 5: XGBoost experiments metrics differences between the new method and OC-score for L 2 norm.

Figure 7: XGBoost experiments metrics differences between the new method and original representation for L 2 norm.

Figure 10: RandomForest experiments metrics differences between the new method and OC-score for L ∞ norm.

Figure 13: Comparing ROC-AUC and PR-AUC for detector based on XGBoost classifier with different hyperparametrs. Target: XGBoost, dataset: codrna, attack method: OPT, norm: 2.

Figure 14: Comparing ROC-AUC and PR-AUC for detector based on XGBoost classifier with different hyperparametrs. Target: XGBoost, dataset: codrna, attack method: OPT, norm: ∞

Figure 21: KNN classifier adversarial detector performance. Attack: OPT -Dataset: codrna.

Figure 22: KNN classifier adversarial detector performance. Attack: HopSkipJumpAttack -Dataset: sensorless

. The top section of Table1describes datasets that were used inChen et al. (2019) as benchmarks for their experiments, already preprocessed and split into training and test sets, and published publicly in a very convenient way on github 2 . The bottom section of Table1describes more datasets that we added, mainly small binary-classification datasets, to have more variety. The datasets vary by the number of samples, features, and classes.

Sign-OPT XGBoost Experiments Results

OPT XGBoost Experiments Results

HSJA XGBoost Experiments Results

Cube XGBoost Experiments Results

Sign-OPT RandomForest Experiments Results

OPT RandomForest Experiments Results

HSJA RandomForest Experiments Results

Cube RandomForest Experiments Results

XGBoost Sign-OPT perturbations statistics

XGBoost OPT perturbations statistics

XGBoost Cube perturbations statistics

XGBoost Leaf-Tuple perturbations statistics

RandomForest Sign-OPT perturbations statistics

RandomForest OPT perturbations statistics

RandomForest HSJA perturbations statistics

RandomForest Cube perturbations statistics

RandomForest Leaf-Tuple perturbations statistics

G APPENDIX G -TREE MODEL'S PERFORMANCE USING LESS DATA

A potential issue in our method is that we reduce the available data used to train the model; thus, the model performance might be affected. As part of our method, we split each dataset we experimented on into several subsets for different roles described in Subsection 4. to the same model trained on more data (X-axis) on the test sets. As we can see, there is a decrease in performance most of the time, but most of the degradation is minor. For a complete description of the numbers in every experiment, refer to Tables 22, 23, 24, 25, 26, 27, 28, 29, 30, 31 . Figure 27 : A comparison between our method models' ROC-AUC on the test set compared to a model trained with the same hyperparameters but with more data which we use in our method to train the representations and detector. 

