PAIRWISE CONFIDENCE DIFFERENCE ON UNLABELED DATA IS SUFFICIENT FOR BINARY CLASSIFICATION Anonymous

Abstract

Learning with confidence labels is an emerging weakly supervised learning paradigm, where training data are equipped with confidence labels instead of exact labels. Positive-confidence (Pconf) classification is a typical learning problem in this context, where we are given only positive data equipped with confidence. However, pointwise confidence may not be accessible in real-world scenarios. In this paper, we dive into a novel weakly supervised learning problem called confidence-difference (ConfDiff) classification. Instead of pointwise confidence, we are given only unlabeled data pairs equipped with confidence difference specifying the difference in the probabilities of being positive. An unbiased risk estimator is derived to tackle the problem, and we show that the estimation error bound achieves the optimal convergence rate. Extensive experiments on benchmark data sets validate the effectiveness of our proposed approaches in leveraging the supervision information of the confidence difference.

1. INTRODUCTION

Recent years have witnessed the prevalence of deep learning and its successful applications. However, the success is built on the basis of the collection of large amounts of data with unique and accurate labels. In many real-world scenarios, it is often difficult to satisfy such requirements. To circumvent the difficulty, various weakly supervised learning problems have been investigated accordingly, including but not limited to semi-supervised learning (Chapelle et al., 2006; Zhu & Goldberg, 2009; Li & Zhou, 2015; Berthelot et al., 2019) , label-noise learning (Patrini et al., 2017; Han et al., 2018; Li et al., 2021; Wang et al., 2021; Wei et al., 2022 ), positive-unlabeled learning (du Plessis et al., 2014; Su et al., 2021; Yao et al., 2022 ), partial-label learning (Cour et al., 2011; Wang & Zhang, 2020; Wen et al., 2021; Wang et al., 2022; Wu et al., 2022 ), unlabeled-unlabeled learning (Lu et al., 2019; 2020) and similarity-based classification (Bao et al., 2018; Cao et al., 2021b; Bao et al., 2022) . Learning with confidence labels (Ishida et al., 2018; Cao et al., 2021a; b) is another weakly supervised learning paradigm, where we are given training examples with confidence labels instead of exact labels. Positive-confidence (Pconf) classification (Ishida et al., 2018) is a problem setting within this scope, which is aimed at learning a binary classifier from only positive data equipped with confidence (the probability of being positive) without negative data. Pconf classification can alleviate the difficulty when negative data cannot be acquired due to privacy or security issues during the data annotation process. The need to learn from such inexact supervision widely exists in real-world scenarios, such as purchase prediction (Ishida et al., 2018) , user preservation prediction (Ishida et al., 2018) , drivers' drowsiness prediction (Shinoda et al., 2020) , etc. However, the process of collecting large amounts of training examples with pointwise confidence might be actually demanding under many circumstances, since it is tough to describe the probability of being positive for each training example exactly (Shinoda et al., 2020 ). Feng et al. (2021) showed that learning from pairwise comparisons could serve as an alternative strategy given limited pointwise labeling information. Inspired by it, we investigate a more practical problem setting in this paper, where we are given only unlabeled data pairs with confidence difference indicating the difference in the probabilities of being positive. Compared with pointwise confidence, confidence difference can be collected more easily in many real-world scenarios. Take click-through rate prediction in recommender systems (Zhang et al., 2019) for example. The combinations of users and

