OPTIMAL NEURAL PROGRAM SYNTHESIS FROM MULTIMODAL SPECIFICATIONS

Abstract

Multimodal program synthesis, which leverages different types of user input to synthesize a desired program, is an attractive way to scale program synthesis to challenging settings; however, it requires integrating noisy signals from the user (like natural language) with hard constraints on the program's behavior. This paper proposes an optimal neural synthesis approach where the goal is to find a program that satisfies user-provided constraints while also maximizing the program's score with respect to a neural model. Specifically, we focus on multimodal synthesis tasks in which the user intent is expressed using combination of natural language (NL) and input-output examples. At the core of our method is a top-down recurrent neural model that places distributions over abstract syntax trees conditioned on the NL input. This model not only allows for efficient search over the space of syntactically valid programs, but it allows us to leverage automated program analysis techniques for pruning the search space based on infeasibility of partial programs with respect to the user's constraints. The experimental results on a multimodal synthesis dataset (STRUCTUREDREGEX) show that our method substantially outperforms prior state-of-the-art techniques in terms of accuracy and explores fewer states during search.

1. INTRODUCTION

In recent years, there has been a revolution in machine learning-based program synthesis techniques for automatically generating programs from high-level expressions of user intent, such as input-output examples (Balog et al., 2017; Chen et al., 2019a; Devlin et al., 2017; Ellis et al., 2019; Kalyan et al., 2018; Shin et al., 2018) and natural language (Yaghmazadeh et al., 2017; Dong & Lapata, 2016; Rabinovich et al., 2017; Yin & Neubig, 2017; Desai et al., 2016; Wang et al., 2018) . Many of these techniques use deep neural networks to consume specifications and then perform model-guided search to find a program that satisfies the user. However, because the user's specification can be inherently ambiguous (Devlin et al., 2017; Yin et al., 2018) , a recent thread of work on multimodal synthesis attempts to combine different types of cues, such as natural language and examples, to allow program synthesis to effectively scale to more complex problems. Critically, this setting introduces a new challenge: how do we efficiently synthesize programs with a combination of hard and soft constraints from distinct sources? In this paper, we formulate multimodal synthesis as an optimal synthesis task and propose an optimal synthesis algorithm to solve it. The goal of optimal synthesis is to generate a program that satisfies any hard constraints provided by the user while also maximizing the score under a learned neural network model that captures noisy information, like that from natural language. In practice, there are many programs that satisfy the hard constraints, so this maximization is crucial to finding the program that actually meets the user's expectations: if our neural model is well-calibrated, a program that maximizes the score under the neural model is more likely to be the user's intended program. Our optimal neural synthesis algorithm takes as input multimodal user guidance. In our setting, we train a neural model to take natural language input that can be used to guide the search for a program consistent with some user-provided examples. Because our search procedure enumerates programs according to their score, the first enumerated program satisfying the examples is guaranteed to be optimal according to the model. A central feature of our approach is the use of a tree-structured neural model, namely the abstract syntax network (ASN) (Rabinovich et al., 2017) , for constructing

