SOLVING PARTIAL LABEL LEARNING PROBLEM WITH MULTI-AGENT REINFORCEMENT LEARNING

Abstract

Partial label learning (PLL) deals with classifications when a set of candidate labels instead of the true one is given for each training instance. As a weakly supervised learning problem, the main target of PLL is to discover latent relationships within training samples, and utilize such information to disambiguate noisy labels. Many existing methods choose nearest neighbors of each partially-labeled instance in an unsupervised way such that the obtained instance similarities can be empirically non-optimal and unrelated to the downstream classification task. To address this issue, we propose a novel multi-agent reinforcement learning (MARL) framework which models the connection between each pair of training samples as a reinforcement learning (RL) agent. We use attention-based graph neural network (GNN) to learn the instance similarity, and adaptively refine it using a deterministic policy gradient approach until some pre-defined scoring function is optimized. Different from those two-stage and alternative optimization algorithms whose training procedures are not end-to-end, our RL-based approach directly optimizes the objective function and estimates the instance similarities more precisely. The experimental results show that our method outperforms state-of-the-art competitors with a higher classification accuracy in both synthetic and real examples.

1. INTRODUCTION

PLL, also known as superset label learning (Liu & Dietterich, 2014; 2012; Gong et al., 2017) , has been extensively studied in the past few decades. As a typical weakly-supervised learning problem, PLL assumes that most of training instances are partially labeled and their ground truth labels are unknown. To be more specific, each instance is associated with a small set of candidate labels including the ground truth. PLL has been widely considered in diverse fields, including web mining (Luo & Orabona, 2010) , facial age estimation (Zhang et al., 2016) , photograph captioning (Duygulu et al., 2002; Barnard et al., 2003; Berg et al., 2004; Gallagher & Chen, 2007 ) and image annotation (Cour et al., 2011; Zeng et al., 2013) . It is usually much easier to get blurry labels than acquiring exact ground truths, and accurately labeling each instance is costly and labor-intensive. For example, the natural photographs collected in the real world may contain multiple human faces and are often tagged ambiguously with several potential names in the captions. The goal is to precisely match the persons in each images with the names, and learn a robust classification model which can be generalized to unseen instances. PLL has been widely examined in the past few years. How to distinguish fuzzy labels in training sets and recover their true labels plays an important role in developing efficient and robust PLL methods. One main class of PLL methods, such as LSB-CMM (Liu & Dietterich, 2012 ), M3PL(Yu & Zhang, 2016) and PL-SVM (Nguyen & Caruana, 2008) , directly fits the classifier with traditional machine learning models. These methods ignore the relationships between training instances which leads to unfaithful labelling disambiguation results. Some more advanced methods, such as PL-KNN (Hüllermeier & Beringer, 2006 ), IPAL (Zhang & Yu, 2015) and PL-LEAF (Zhang et al., 2016) , utilize the coorelations between training instances provided by some underlying similarity graph learned in an unsupervised manner. These methods have achieved desirable empirical performance, but still suffer from some common issues. For example, the similarity graph generated via some unsupervised approaches may be non-optimal due to its independence of the main classification task. Another weakness of the graph-based methods is that the label prediction of each test example is voted by its neighbors in the training set which makes the prediction results highly depend on the selection of neighboring instances. Unfortunately, there is usually no guidance of choosing a proper size of the neighboring set in practice. The prediction results can be significantly biased when some unrelated or wrongly-labelled training instances are included as voters. Although some more recent studies such as AGGD (Wang et al., 2019) employ supervised methods to model the instance similarities, the neighbor selection issue has not been fully addressed as they initialize the neighboring set in an unsupervised way and only update the similarity measurements at some specific locations in the subsequent optimization step which has great limitations. On the other hand, these methods need to limit the total number of neighbors and thus some training instances that are useful in improving the predicting accuracy are ignored. Furthermore, the complicated optimization problem proposed by these graph-based methods are difficult to solve in practice. In this work, we propose a novel end-to-end Partial Label learning method using Multi-Agent Reinforcement Learning, called PLRL. Under the MARL setting, each pair of training instances is treated as an individual agent and its action is defined as the similarity measurement between these two instances. All the n training instances build the node set of an underlying similarity graph, and the learned optimal policy precisely quantifies the closeness between any two nodes, which is the edge weight. Unlike traditional two-stage or alternative optimization approaches, our PLRL method directly optimizes a pre-defined score function and adaptively updates the instance similarities in a supervised way which makes the learned similarity graph more related to the main classification task. Specifically, we use GNNs to learn the similarity graph which maximizes a total reward shared by all the n 2 agents. With the similarity graph, an estimated probability distribution over the candidate labels for each test example is obtained through some label propagation strategies. Then we incorporate the estimated probabilities with a kernel ridge regression model to carry out the label disambiguation. The MARL framework together with the classification model is end-to-end trained using policy gradients. The main contributions of this paper are summarized as follows. • We introduce a novel MARL framework to quantify the similarities between training instances and impose no limits on the number of nearest neighbors. • Different from traditional two-stage or alternative optimization methods, we employ an end-to-end approach to jointly implement predictions and labelling disambiguation, which makes the learned instance similarities more related to the main classification task. • We use Policy Gradient to efficiently solve the complicated optimization problem which cannot be easily handled by previous studies. • Experimental results on both synthetic and real datasets show that our method outperforms existing PLL methods with higher classification accuracies in most scenarios, especially in those cases with imbalanced samples.

2. PROBLEM STATEMENT

In PLL, a partially labelled training set D = {(x i , C i ) | 1 ≤ i ≤ n} is given, where x i = (x i1 , ..., x id ) ⊤ ∈ X is a ddimensional instance and C i ⊆ Y is the candidate label set among which only one label is assumed to be valid. Y = {y 1 , y 2 , ..., y q } here denotes a label space with q classes. Let X = [x 1 , ..., x n ] ⊤ ∈ R n×d be the normalized input data matrix. The target of PLL is to induce a multi-class classifier g(•) : X → Y using such fuzzy label information D to precisely classify partially-labelled and unseen instances. In this work, we consider an undirected weighted graph G = (V, W) among instances. V = {x i |1 ≤ i ≤ n} is the set of vertices. W = [w ij ] n×n refers to the non-negative weight adjacency matrix, where w ij ∈ [0, 1] measures how close the two instances x i and x j are to each other with larger value meaning higher correlation. In this work, a novel RL based approach is used to estimate the weighted adjacency matrix W. For each training example (x i , C i ), we aim to generate a normalized real-valued vector f i ∈ R q where each f ij represents the label confidence of the j-th label being the true label of x i . The label confidence vector f i satisfies the following constraints: (i) q j=1 f ij = 1 for any 1 ≤ i ≤ n, (ii) f ij ≥ 0 for any y ij = 1, and (iii) f ij = 0 for any y ij = 0. The second and third constraints indicate the potential ground-truth label resides in the candidate label set. When the label confidence of training set X is generated as F = [f 1 , ..., f n ] ⊤ ∈ R n×q , a classifier g(•) : X → Y can be induced based on this disambiguation results.

