FIGHTING FILTERBUBBLES WITH ADVERSARIAL BERT-TRAINING FOR NEWS-RECOMMENDATION

Abstract

Recommender engines play a role in the emergence and reinforcement of filter bubbles. When these systems learn that a user prefers content from a particular site, the user will be less likely to be exposed to different sources or opinions and, ultimately, is more likely to develop extremist tendencies. We trace the roots of this phenomenon to the way the recommender engine represents news articles. The vectorial features modern systems extract from the plain text of news articles are already highly predictive of the associated news outlet. We propose a new training scheme based on adversarial machine learning to tackle this issue . Our experiments show that the features we can extract this way are significantly less predictive of the news outlet and thus offer the possibility to reduce the risk of manifestation of new filter bubbles. We validate our intuitions in a news recommendation task using a recent attention-based recommendation system.

1. INTRODUCTION

Recommender systems are a source of unwanted societal effects like filter bubbles or echo chambers, i.e. situations where users reinforce their own beliefs by only engaging with content that corresponds to their world views (Pariser, 2011) . This effect is also driven by algorithmic news recommendations, for instance on Facebook (Bakshy et al., 2015) . Typically, these automatic recommendations are based on the combination of inferred user preferences and properties of the content: The system learns which properties a user wants to see in the content he consumes. When working with texts, like news articles or social media posts, content properties are often represented by numerical features extracted with large neural network architectures. A prominent example of these neural representations are bidirectional encoder representation from transformers, or BERT (Devlin et al., 2019) , which are for instance applied in Google searchfoot_0 . We investigate the space of feature vectors induced by BERT models in the domain of news articles and ask if the extraction of features is already a source of societal and cultural divides. We discover that the features are predictive of the news source and hence content recommended based on similarities is likely to come from the same source also, thereby possibly promoting the filter bubble effect. For this study we assume that the news outlet is a good proxy feature for the political orientation or world-view of a given news article. Hence, to address this issue, we propose to apply adversarial learning to compute features that are less predictive of their corresponding news source but are still informative of the semantics of the text. Our adversarial training schemes allow for efficient training of BERT models with adversarial heads. We begin this paper by briefly discussing the data set and the BERT model we use in this study. Then we describe our novel adversarial training procedure that trains our model, such that the vectorial features it extracts are less predictive of the news outlet. We evaluate our model in a two-fold validation: First we investigate the quality of our new BERT feature representations in isolation using a large corpus of news articles. Second, we investigate the BERT features in a news recommendation setting using a large dataset of real-world user interactions. We conclude our paper with a discussion of our results.

2. RELATED WORK

Adversarial Training has been used in many applications. It was introduces by Goodfellow et al. (? in the context of generative image modeling, where a generator neural network generates images that are hard to separate from real images for a discriminator network. Since then, this interplay between two adversaries, the generator and the discriminator, has been used in many appolications. Adversarial learning is a popular choice for tackling many problems with machine learning systems. Prominent examples include learning privacy-aware feature representations, that exclude sensitive private information like gender or race from learned representations. An important subdomain is applying these methods to image data in order to modify images in such a way, that it shadows information like race, age or location from the machine learning systems. Recently, Li et al. 2019 have used an adversarial approach to obfuscate privacy-relevant attributes in images using adversarial learning. But also in other data domains, this becomes increasingly important. The recent work of Lia et al. shows that we can obfuscate privacy-relevant information from user data while maintaining all other relevant information in a feature vector. Prior to this, Pittaluga et al. ( 2019) have demonstrated a similar result. In our study, this private attribute is the news outlet. In this study we use the adversarial training to train neural text representations that make it hard to guess their authoring news outlets. Filter Bubbles can also be targeted from the user side. Using techniques from data obfuscation, users can "hide" their true preferences from a system by consuming media from opposing viewpoints. This obfuscation can be done data-driven as demonstrated by and Strucks et al. 2019 , where an algorithm suggests the most effective items to interact with to trick the system.

3. ADVERSARIAL TRAINING OF BERT MODELS

For feature extraction, we rely on a DistilBERT model Sanh et al. ( 2019), a more resource-efficient variant of the original BERT deep network architecture. The model takes a sequence of tokens produced by the sentence-piece tokenizer x 1 , x 2 , . . . x l and, in a sequence of 6 transformer layers (Vaswani et al., 2017) , produces an output sequence z 1 , z 2 , . . . z l where each z i ∈ R 768 . At the training stage, the model is trained to solve masked language modeling task, where 15% of the input tokens are shadowed and we learn to predict the masked inputs x i based on the output vector z i . We use a model by HuggingFace (Wolf et al., 2019) that is pretrained on the Toronto Book Corpus as well as the English Wikipedia . Then we fine-tune this pretrained model on the masked language modeling task using paragraphs from our training data to adapt to the content and style of news articles. This fine-tuned model will act as the baseline in our study. At inference time -following standard practice (e.g. (Xiao, 2018)) -we compute embeddings of the respective news articles by computing the mean embedding z on the output sequence. The BERT model does not process full texts, but only sequences of up to length 512. To mitigate this, we process each paragraph individually, cropping excess tokens, and also average these paragraph vectors into a document vector. This way we reduce the full text, including headline, to a 768dimensional vector that could be fed into a recommendation system. We propose a machine learning model comprised of two parts, in accordance with the terminology commonly used in adversarial learning we call them generator and discriminator. The task of the generator model is to produce vectorial features of the tokenized text that contain little information about the news outlet. The discriminator tries to predict the news outlet of the text. f gen (x 1 , . . . , x l ) = BERT(x 1 , , x l ; Θ) f dis (z 1 , . . . , z l ) = V T σ(l -1 • l ! i=1 W T z i ) Both models act as adversaries: While the discriminator tries to classify feature vectors correctly by learning V and W , the generator learns the BERT parameters Θ to produce features that allow reconstruction of masked tokens while not allowing the discriminator to correctly classify the output. This is captured in the different optimisation functions associated with the models. The generator



https://blog.google/products/search/search-language-understandingbert/

