DEXDEFORM: DEXTEROUS DEFORMABLE OBJECT MA-NIPULATION WITH HUMAN DEMONSTRATIONS AND DIFFERENTIABLE PHYSICS

Abstract

In this work, we aim to learn dexterous manipulation of deformable objects using multi-fingered hands. Reinforcement learning approaches for dexterous rigid object manipulation would struggle in this setting due to the complexity of physics interaction with deformable objects. At the same time, previous trajectory optimization approaches with differentiable physics for deformable manipulation would suffer from local optima caused by the explosion of contact modes from hand-object interactions. To address these challenges, we propose DexDeform, a principled framework that abstracts dexterous manipulation skills from human demonstration, and refines the learned skills with differentiable physics. Concretely, we first collect a small set of human demonstrations using teleoperation. And we then train a skill model using demonstrations for planning over action abstractions in imagination. To explore the goal space, we further apply augmentations to the existing deformable shapes in demonstrations and use a gradient optimizer to refine the actions planned by the skill model. Finally, we adopt the refined trajectories as new demonstrations for finetuning the skill model. To evaluate the effectiveness of our approach, we introduce a suite of six challenging dexterous deformable object manipulation tasks. Compared with baselines, DexDeform is able to better explore and generalize across novel goals unseen in the initial human demonstrations.

1. INTRODUCTION

The recent success of learning-based approaches for dexterous manipulation has been widely observed on tasks with rigid objects (OpenAI et al., 2020; Chen et al., 2022; Nagabandi et al., 2020) . However, a substantial portion of human dexterous manipulation skills comes from interactions with deformable objects (e.g., making bread, stuffing dumplings, and using sponges). Consider the three simplified variants of such interactions shown in Figure 1 . Folding in row 1 requires the cooperation of the front four fingers of a downward-facing hand to carefully lift and fold the dough. Bun in row 4 requires two hands to simultaneously pinch and push the wrapper. Row 3 shows Flip, an in-hand manipulation task that requires the fingers to flip the dough into the air and deform it with agility. In this paper, we consider the problem of deformable object manipulation with a simulated Shadow Dexterous hand (ShadowRobot, 2013) . The benefits of human-level dexterity can be seen through the lens of versatility (Feix et al., 2015; Chen et al., 2022) . When holding fingers together, the robot hands can function as a spatula to fold deformable objects (Fig. 1 , row 1). When pinching with fingertips, we can arrive at a stable grip on the object while manipulating the shape of the object (Fig. 1 , row 2). Using a spherical grasp, the robot hands are able to quickly squeeze the dough into a

funding

https://sites.google.com/view/

