LEARNING TO GENERATE ALL FEASIBLE ACTIONS

Abstract

Several machine learning (ML) applications are characterized by searching for an optimal solution to a complex task. The search space for this optimal solution is often very large, so large in fact that this optimal solution is often not computable. Part of the problem is that many candidate solutions found via ML are actually infeasible and have to be discarded. Restricting the search space to only the feasible solution candidates simplifies finding an optimal solution for the tasks. Further, the set of feasible solutions could be re-used in multiple problems characterized by different tasks. In particular, we observe that complex tasks can be decomposed into subtasks and corresponding skills. We propose to learn a reusable and transferable skill by training an actor to generate all feasible actions. The trained actor can then propose feasible actions, among which an optimal one can be chosen according to a specific task. The actor is trained by interpreting the feasibility of each action as a target distribution. The training procedure minimizes a divergence of the actor's output distribution to this target. We derive the general optimization target for arbitrary f-divergences using a combination of kernel density estimates, resampling, and importance sampling. We further utilize an auxiliary critic to reduce the interactions with the environment. A preliminary comparison to related strategies shows that our approach learns to visit all the modes in the feasible action space, demonstrating the framework's potential for learning skills that can be used in various downstream tasks.

1. INTRODUCTION

Complex tasks can often be decomposed into multiple subtasks, with corresponding skills that solve these subtasks. Learning reusable and transferable skills is an active area of research (Kalashnikov et al. (2021) ; Chebotar et al. (2021) ; Deisenroth et al. (2014) ). However, given a subtask, learning or even defining the corresponding skill is not straightforward. Consider a robotic scenario where a robot is tasked to grasp an object and handle it in downstream tasks. Different downstream tasks can have different optimal grasps if the object has multiple feasible grasping poses. Therefore, a grasping skill cannot be trained based on optimality definitions of individual tasks. However, a grasping algorithm that learned all feasible grasps could support all possible downstream tasks even without explicit knowledge thereof during training. The downstream tasks can then select their respective optimal grasp among the proposed feasible options. Therefore, we consider a skill to be defined by the set of all feasible actions of a subtask. We propose a novel method to train a generative neural network to generate all feasible actions of a subtask by interacting with an environment. The interaction loop is adopted from Contextual Bandit (CB) (Langford et al. (2008) ) and Reinforcement Learning (RL) (Sutton & Barto (2018) ): the environment presents a state for which the actor selects an action, which is tested in the environment, yielding either a success or failure outcome. As in CB, we limit ourselves to one-step interactions as opposed to sequential multi-step interactions common in RL. However, we do not minimize regret, typically done in CB. Instead, we optimize the final policy as in RL. Unlike CB and RL, the approach does not aim to find one optimal solution for a given problem but aims to learn all feasible ones. By interpreting the feasibility of each action given a state as a posterior probability distribution over the actions, a target probability density function (pdf) is defined. The actor is trained to minimize a divergence of its output distribution to this target pdf. The training algorithm in the method proposed can be used with any given f-divergence, including Reverse Kullback-Leibler (RKL), Forward Kullback-Leibler (FKL), and Jensen-Shannon (JS). The possibility to use FKL and JS is instrumental in visiting all the modes of the posterior distribution, as RKL is known to collapse into a single mode (Jerfel et al. (2021) ). The training algorithm presented in this paper uses Kernel Density Estimation (KDE) to estimate the pdf of the actor and Monte Carlo integration with importance sampling to estimate the normalization of the target. The divergences are estimated using samples from a proposal distribution which is a separate KDE based on the original samples of the actor. This resampling step is necessary for convergence, which is discussed in Section 3.3. As interactions with the environment are typically costly, an auxiliary critic network imitating the environment is trained simultaneously. The critic's feasibility estimate of an action is then used to form the target distribution. The learning algorithm has been tested on a planar robotic grasping problem. We test FKL, RKL, and JS divergences and compare them to implementations of maximum entropy (ME) RL and Generative Adversarial Networks (GANs). Besides accuracy, we measure how many grasping modes, i.e., disconnected regions in the action space, are visited by each approach. Generating actions in all grasping modes can ensure that the learned skill is reusable and transferable for various downstream tasks. The contributions of this paper are the following: • Design of a new learning method for generative neural network models to explicitly learn to generate all feasible actions. • Introduction of a novel gradient estimator for f-divergences that takes advantage of KDEs, resampling, and importance sampling. • Application of the proposed learning algorithm to a 2D robotic grasping problem, comparing the proposed gradient estimators for f-divergences with related methods. The rest of this work is structured as follows. Section 2 discusses the related work. Section 3 describes the optimization problem followed by the methodology in Section 4. The evaluation setup is described in Section 5 and the results are presented in Section 6. Section 7 concludes and gives an outlook on future work. & Osindero (2014) ) offer an alternative solution to the posterior sampling problem, as a generator conditioned on a given state can be trained to provide successful actions adversarially. However, the problem analyzed in our paper is not naturally adversarial, as actions that have not yet been tested in the interactive environment should not be implicitly rejected. The discriminator learns to discriminate between tested successful actions from untested ones, providing the generator with inconsistent gradients.



CBs have been successfully applied to several interactive learning problems with discrete action spaces(Langford & Zhang (2007);Agarwal et al. (2014); Foster & Rakhlin (2020);Simchi-Levi  & Xu (2021)). In several cases, the context and action spaces have been embedded in a linear multidimensional action space. The embedding keeps the interaction linear while the action and context embeddings can be non-linear (Chernozhukov et al. (2019); Foster et al. (2020); Zhu et al. (2022)). Recently, there has been an increased interest in extending the approach to continuous action spaces. However, most works are limited to 1D actions (Chernozhukov et al. (2019); Majzoubi et al. (2020); Zhu & Mineiro (2022)). Learning from an interactive environment is also the focus of RL (Sutton & Barto (2018)). Many RL approaches use Actor-Critic (AC) architectures, among which the Soft Actor-Critic (SAC) algorithm (Haarnoja et al. (2018)) is most related to our work. In SAC, the state-action value function of the critic is transformed into an energy-based distribution (Haarnoja et al. (2017)), yielding the target of the actor's distribution. SAC uses RKL as the loss function for the actor, which yields maximum entropy RL. Through a reparameterization trick, which usually uses the family of Gaussians, the RKL is minimized through a direct gradient from the critic.GANs propose a similar architecture to AC, training a generator and discriminator adversarially. This adversarial training is equivalent to minimizing the JS divergence(Goodfellow et al. (2014)) and has been extended to arbitrary f-divergences(Nowozin et al. (2016)). Conditional GANs (Mirza

