FIGHTING FILTERBUBBLES WITH ADVERSARIAL BERT-TRAINING FOR NEWS-RECOMMENDATION

Abstract

Recommender engines play a role in the emergence and reinforcement of filter bubbles. When these systems learn that a user prefers content from a particular site, the user will be less likely to be exposed to different sources or opinions and, ultimately, is more likely to develop extremist tendencies. We trace the roots of this phenomenon to the way the recommender engine represents news articles. The vectorial features modern systems extract from the plain text of news articles are already highly predictive of the associated news outlet. We propose a new training scheme based on adversarial machine learning to tackle this issue . Our experiments show that the features we can extract this way are significantly less predictive of the news outlet and thus offer the possibility to reduce the risk of manifestation of new filter bubbles. We validate our intuitions in a news recommendation task using a recent attention-based recommendation system.

1. INTRODUCTION

Recommender systems are a source of unwanted societal effects like filter bubbles or echo chambers, i.e. situations where users reinforce their own beliefs by only engaging with content that corresponds to their world views (Pariser, 2011) . This effect is also driven by algorithmic news recommendations, for instance on Facebook (Bakshy et al., 2015) . Typically, these automatic recommendations are based on the combination of inferred user preferences and properties of the content: The system learns which properties a user wants to see in the content he consumes. When working with texts, like news articles or social media posts, content properties are often represented by numerical features extracted with large neural network architectures. A prominent example of these neural representations are bidirectional encoder representation from transformers, or BERT (Devlin et al., 2019) , which are for instance applied in Google searchfoot_0 . We investigate the space of feature vectors induced by BERT models in the domain of news articles and ask if the extraction of features is already a source of societal and cultural divides. We discover that the features are predictive of the news source and hence content recommended based on similarities is likely to come from the same source also, thereby possibly promoting the filter bubble effect. For this study we assume that the news outlet is a good proxy feature for the political orientation or world-view of a given news article. Hence, to address this issue, we propose to apply adversarial learning to compute features that are less predictive of their corresponding news source but are still informative of the semantics of the text. Our adversarial training schemes allow for efficient training of BERT models with adversarial heads. We begin this paper by briefly discussing the data set and the BERT model we use in this study. Then we describe our novel adversarial training procedure that trains our model, such that the vectorial features it extracts are less predictive of the news outlet. We evaluate our model in a two-fold validation: First we investigate the quality of our new BERT feature representations in isolation using a large corpus of news articles. Second, we investigate the BERT features in a news recommendation setting using a large dataset of real-world user interactions. We conclude our paper with a discussion of our results.



https://blog.google/products/search/search-language-understandingbert/

