NEURO-SYMBOLIC PROCEDURAL PLANNING WITH COMMONSENSE PROMPTING

Abstract

Procedural planning aims to implement complex high-level goals by decomposition into simpler low-level steps. Although procedural planning is a basic skill set for humans in daily life, it remains a challenge for large language models (LLMs) that lack a deep understanding of the cause-effect relations in procedures. Previous methods require manual exemplars to acquire procedural knowledge from LLMs in the zero-shot setting. However, such elicited pre-trained knowledge in LLMs induces spurious correlations between goals and steps, impairing the model's generalization to unseen tasks. In contrast, this paper proposes a neuro-symbolic procedural PLANner (PLAN) that elicits procedural knowledge from the LLMs with commonsense-infused prompting. To mitigate spurious goal-step correlations, we use symbolic program executors on the latent procedural representations to formalize prompts from external knowledge bases as a causal intervention toward the Structural Causal Model of procedural planning. Both automatic and human evaluations on WikiHow and RobotHow show the superiority of PLAN on procedural planning without further training or manual exemplars. 1 

1. INTRODUCTION

How to make a cup of coffee? As humans, we can easily specify a procedure to solve this task, using our innate ability of commonsense reasoning. However, can we endow machines with the same ability to construct a sequential plan? As depicted in Figure 1 , procedural planning (Pearson, 1996; Zhang et al., 2020b; Huang et al., 2022) aims to decompose a high-level goal (Task: Watch TV) into a sequence of temporally extended steps (Procedural Plan: Step at all five time-steps). We study procedural planning as the conditional text generation problem since it resembles real-world scenarios. Previous approaches (Huang et al., 2022; Ahn et al., 2022) require a small number of carefully written or held-out exemplars to acquire procedural knowledge. However, these manual exemplars evolved from task data are impossible to cover the ever-changing task setups and the flexible dependency relations among goals and steps. In fact, the biased data may cause the model to learn spurious correlations and hinder the model from generalizing well in zero-shot scenarios. Studies in cognitive science show that humans rely on chunking mechanisms (Gobet et al., 2001; Miller, 1956) which turn primitive stimuli into conceptual groups to solve novel and complex problems. Inspired by this, we hypothesize that generalizable procedural planning ability can be achieved by learning cause-effect relations among complex goals and simpler steps using external knowledge. To reveal the cause-effect relations in procedural planning, we devise a Structural Causal Model (SCM) (Peters et al., 2017) , a directed acyclic graph commonly used to describe the causal relationships within a system Pearl (2009). As depicted in Figure 2 , the pre-trained knowledge (D) (e.g., TV and living room is highly correlated) in LLMs confounds (D influences T , S i-1 and S i , resulting in spurious correlations) the system to make biased decisions toward an unreasonable step (e.g., Find

Symbolic Rule

External Knowledge

LLMs LLMs

WikiHow Task How to Surf?

Procedural Plan

Step 1: Practice on the ground first. Step 2: Practice getting up. Step 3: Learn to stand on the board properly. Step 4: Paddle around and get comfortable in the water. Step 5: Talk to more experienced surfers or instructors if you can.

RobotHow Task

Watch TV

Goal Input Goal Input

Task Decomposition Task Decomposition

Procedural Plan

Step 1: Walk to the living room. Step 2: Switch on TV. Step 3: Grab Remote Control. Step 4: Walk to Sofa. Step 5: Sit on Sofa. Television). Thus, we adopt front-door adjustment (definition in Appendix A.3), which utilizes a mediator (P i ) that blocks all directed paths from the cause (T or S i-1 ) to the effect (S i ). In this way, T (or S i-1 ) affects S i by flowing through indirect paths: T (or S i-1 ) affects P i and P i affects S i . And we can identify the causal effects among goals and steps by investigating the indirect effect (Equation 3), which is computed by multiplying the effect of T (or S i-1 ) on P i-1 (Equation 1) with the effect of P i on S i (Equation 2). With the above front-door adjustment, we can mitigate the spurious correlations (e.g., between "television" and "living room") and thus make reasonable decisions on steps (e.g., Find book). Please refer to A.1 for causal preliminaries (including explanation for SCM, confounder, mediator, spurious correlations), and A.3 for the front-door adjustment definition.

Visualization in the Environment

Guided by the above causal analysis of procedural planning, we need to construct the mediator P i and then intervene on task T and prompt P i , which is required to compute the conditional probability in Equation3. As depicted in Figure 3 , we seek to automatically construct commonsense-infused prompts as the mediator P i by concatenating the task, previous steps with commonsense knowledge extracted from external resources (e.g., ConceptNet (Speer et al., 2017) ). First, we modify the goal input by sampling a task-relevant knowledge subgraph (Stage1 in Section 3.1) to implement interventions on T . Then, we modify the prompt by adapting the edge weight to implement interventions on P i (Edge-Wise Adoption of Stage2 in Section 3.1). However, directly incorporating knowledge of graph structure into LLMs leads to the loss of the logical order in eliciting procedural knowledge from LLMs. Thus, we apply symbolic executors (Mao et al., 2019; Yi et al., 2018) that execute the sequential mapping program on latent knowledge representations (e.g., the subevent of). In this way, we transit graph structure knowledge into natural language that preserves procedural structure, such as the sequential order of two low-level steps (Symbolic Structuring of Stage2 in Section 3.1). The procedural prompt P G (e.g, "please get the remote control") is further translated into admissible one PG (e.g., "grab remote control") from available steps in a certain domain (RobotHow or WikiHow in our case). Finally, we utilize the commonsense-infused prompt PG to control the generation of procedural plans in LLMs in a zero-shot setting (Section 3.2). We conducted experiments on RobotHow (Puig et al., 2018) and WikiHow (Koupaee & Wang, 2018) under original and counterfactual situations. Our major contributions can be summarized as: • We develop the first causal framework for procedural planning by 1) defining a temporally extended Structural Causal Model and 2) resolving spurious correlation between high-level goals and low-level steps via front-door adjustment with a prompt-based mediator. • We propose a neuro-symbolic approach to construct commonsense-infused prompts for LLMs to tackle the procedural planning task without manual exemplars or further training. • Extensive evaluations show the superiority of PLAN in terms of reasoning about the causeeffect relations among goals and steps and achieving promising planning ability.



Source code and datasets are publicly available at https://sites.google.com/view/iclr-clap



Figure 1: Two independant procedural planning task examples from RobotHow and WikiHow. PLAN construct commonsense-infused prompt from external knowledge (e.g., ConceptNet) to elicit procedural planning ability of the Large Language Models (LLMs) without training or exemplars.

