NEURAL-SYMBOLIC RECURSIVE MACHINE FOR SYSTEMATIC GENERALIZATION

Abstract

Despite the tremendous success, existing machine learning models still fall short of human-like systematic generalization-learning compositional rules from limited data and applying them to unseen combinations in various domains. We propose Neural-Symbolic Recursive Machine (NSR) to tackle this deficiency. The core representation of NSR is a Grounded Symbol System (GSS) with combinatorial syntax and semantics, which entirely emerges from training data. NSR implements a modular design for neural perception, syntactic parsing, and semantic reasoning, which are jointly learned by a deduction-abduction algorithm. We prove that NSR is expressive enough to model various sequence-to-sequence tasks. Superior systematic generalization is achieved via the inductive biases of equivariance and recursiveness embedded in NSR. In experiments, NSR achieves state-of-the-art performance in three benchmarks from different domains: SCAN for semantic parsing, PCFG for string manipulation, and HINT for arithmetic reasoning. Specifically, NSR achieves 100% generalization accuracy on SCAN and PCFG and outperforms state-of-the-art models on HINT by about 23%. Our NSR demonstrates stronger generalization than pure neural networks due to its symbolic representation and inductive biases. NSR also demonstrates better transferability than existing neural-symbolic approaches due to less domain-specific knowledge required.

1. INTRODUCTION

A remarkable property underlying human intelligence is its systematic compositionality: the algebraic capacity to interpret an infinite number of novel combinations from finite known components (Chomsky, 1957)-"infinite use of finite means" (Chomsky, 1965) . This type of compositionality is central to the human ability to generalize from limited data to novel combinations (Lake et al., 2017) . Recently, several datasets have been proposed to test systematic generalization of machine learning models-SCAN (Lake & Baroni, 2018), PCFG (Hupkes et al., 2020) , CFQ (Keysers et al., 2020), and HINT (Li et al., 2021) , to name a few. While conventional neural networks fail dramatically on these datasets, certain inductive biases have been explored to improve systematic generalization. 2020) introduce a neural-symbolic stack machine to achieve nearly perfect accuracy on SCAN-like datasets. Despite the improved performance, these neural-symbolic methods often require domain-specific knowledge to design non-trivial symbolic components and are difficult to transfer to other domains. To achieve human-like systematic generalization in a wide range of domains, we propose Neural-Symbolic Recursive Machine (NSR), which integrates the joint learning of perception, syntax, and semantics in a principled framework. The core representation of NSR is a Grounded Symbol System (GSS) (see Fig. 1 ), which entirely emerges from training data without domain-specific knowledge. NSR implements a modular design for neural perception, syntactic parsing, and semantic reasoning. Specifically, we first utilize a neural network as the perception module to ground symbols on the raw inputs. Next, the symbols are parsed into a syntax tree of the Grounded Symbol System by a transition-based neural dependency parser (Chen & Manning, 2014) . Finally, we adopt functional programs to realize the semantic meaning of symbols (Ellis et al., 2021) . Theoretically, we show that the proposed NSR is expressive enough to model various sequence-to-sequence tasks. Critically, the inductive biases of equivariance and recursiveness, encoded in each module, enable NSR to break



Csordás et al. (2021); Ontanón et al. (2022) improve Transformers' generalization performance by using relative positional encoding and sharing weights between layers. Chen et al. (

