CALIBRATION MATTERS: TACKLING MAXIMIZATION BIAS IN LARGE-SCALE ADVERTISING RECOMMENDA-TION SYSTEMS

Abstract

Calibration is defined as the ratio of the average predicted click rate to the true click rate. The optimization of calibration is essential to many online advertising recommendation systems because it directly affects the downstream bids in ads auctions and the amount of money charged to advertisers. Despite its importance, calibration often suffers from a problem called "maximization bias". Maximization bias refers to the phenomenon that the maximum of predicted values overestimates the true maximum. The problem is introduced because the calibration is computed on the set selected by the prediction model itself. It persists even if unbiased predictions are achieved on every datapoint and worsens when covariate shifts exist between the training and test sets. To mitigate this problem, we quantify maximization bias and propose a variance-adjusting debiasing (VAD) meta-algorithm in this paper. The algorithm is efficient, robust, and practical as it is able to mitigate maximization bias problem under covariate shifts, without incurring additional online serving costs or compromising the ranking performance. We demonstrate the effectiveness of the proposed algorithm using a state-of-the-art recommendation neural network model on a large-scale real-world dataset.

1. INTRODUCTION

The online advertising industry has grown exponentially in the past few decades. According to Statista (2022) , the total value of the global internet advertising market was worth USD 566 billion in 2020 and is expected to reach USD 700 billion by 2025. In the online advertising industry, to help advertisers reach the target customers, demand-side platforms (DSPs) try to bid for available ad slots in an ad exchange. A DSP serves many advertisers simultaneously, and ads provided by those advertisers form the DSP's ads candidate pool. From the DSP's perspective, the advertising campaign pipeline executes as follows: (1) The DSP uses data to build machine learning (ML) models for advertisement value estimation. An advertisement's value is often measured by the click-through rate (CTR) or conversion rate. (2) When the ad exchange sends requests in the form of online bidding auctions for some specific ad slots to a DSP, the DSP uses the ML models to predict values for ads in its ads candidate pool. (3) For the bidding requests, the DSP needs to choose the most suitable ads from its ads candidate pool. Therefore, based on the estimated values, the DSP chooses the ad candidates with the highest values and submits corresponding bids to the ad auctions in the ad exchange. (4) For each auction, an ad with the highest bid would win the auction, and would be displayed (i.e., recommended) in this specific ad slot. The ad exchange would charge the winning DSP a certain amount of money based on the submitted bid and the auction mechanism. For the machine learning models in Step (2), besides learning the ranking (i.e. which ads sent to ad exchange), DSPs also need to accurately estimate the value of the chosen ads, because in

