SIMPLE DEDUCTIVE REASONING TESTS AND NUMER-ICAL DATA SETS FOR EXPOSING LIMITATION OF TO-DAY'S DEEP NEURAL NETWORKS

Abstract

Reasoning is an open problem in the machine learning world today. Deductive reasoning involves storing facts in memory and generation of newer facts over time. The concept of memory, processor and code in deduction systems is fundamentally different from the purpose and formulation of weights in a deep neural network. A majority of the machine learning models are inductive reasoning models including state of the art deep neural networks which are effectively tensor interpolation based models. A step towards realization of memory is through recurrent neural networks and its variants, however the formal representation is not sufficient enough to capture a complex mapping function between input and output patterns. Deep neural networks are positioned to do away with feature engineering which is essentially deductive reasoning methodology. There are existing works in deductive reasoning in neural networks that require learning of syntax, unification and deduction and operate on text data as sequence of tokens. However the performance of deductive reasoning networks is far from perfection which may be either due to syntax or deduction aspects. In this context, we have proposed a suite of completely numeric data sets which do not require parsing as with text data. The 10 data sets are for -(a) selection (3 data sets) -minimum, maximum and top 2nd element in an array of numbers; (b) matching (3 data sets) -duplicate detection, counting and histogram learning; (c) divisibility tests (2 data sets) -divisibility of two numbers and divisibility by 3; (d) representation (2 data sets) -binary representation and parity. Though extremely simple in terms of feature engineering, in all of these tests, simple deep neural networks, random forest and recurrent neural networks have failed with very low accuracies. We propose these as numerical test-bed for testing learning models for deductive reasoning.

1. INTRODUCTION

Deductive reasoning is a branch of artificial intelligence where inferences are represented as assertions or facts over data (Khemani, 2013) . Starting with a set of given facts, the system combines facts based on rules to generate newer facts and update the knowledge store. On the other hand machine learning algorithms employ induction based approaches which are predominantly pattern mapping methods (McClelland et al., 1986) . Fundamentally, in a pipeline of operations, the vectors are arithmetically combined, logically filtered, scaled up or scaled down and mapped to the target vector of interest. A tensor is a more generalization of the vector representation mathematically. However typically even a tensor is internally represented as an array of contiguous storage locations with a data structure indicating dimensions. These tensors undergo a pipeline of transformations minimizing an error function there by mapping a tensor on one side of the pipeline to the tensor on the other side. Deep neural networks have demonstrated their performance almost at the level of human or even better in computer vision and other domains (Bengio et al., 2017) . Although it is promising to see the success of deep neural networks (Dargan et al., 2019) (DNN) there seems to be a popular belief and false opinion that they are suitable for all types of problems. It is important to note here that problem statements solved by DNNs are of mainly of interpolation in nature where tensors are combined along the pipeline to produce the output tensor. The vanilla DNNs are not directly suitable for deductive reasoning type of problem statements. A knowledge base in a deductive reasoning methodology is a storage for facts which are modified over time. For instance, counting number of ones in a binary representation of the input, the current count is a fact and as the algorithm iterates over input, the count value gets updated. Most of the iterations over input, can be represented in a declarative style as first order logic statements such as prolog (Bratko, 2001) . The weight space representation of a deep neural network is not a convenient formulation to capture the facts, unification and inference mechanism as required by a deductive reasoning methodology. However, earlier version of machine learning formulations required feature engineering which itself accommodates for deductive reasoning in the form of outputs of human crafted programs which are added as features. There are on-going research efforts in this direction on modification of neural network architectures to enable them for performing deduction reasoning. A small step in the direction of storage of past information in data is a recurrent neural network formulation and its several variants (Mikolov et al., 2011) . Most of the existing works employ a recurrent neural network based formulation and its variations due to the fundamental need of the notion of memory in a deductive reasoning setting. We have tabulated the observations in the form of a Table 1 . Citation Data Processing (Nangia & Bowman, 2018) Input is a string of list operations and elements such as minimum and maximum. Output is the result of list operations. They have released a ListOps data set. TreeLSTM (Tai et al., 2015b ) (Saxton et al., 2019) Input is a stream of sequences of tokens. A specific sequence is defined as a question. The output is again a sequence of tokens corresponding to answer. They have used an RNN based formulation for a question-answer system. (Wu et al., 2020; Irving et al., 2016; Gauthier et al., 2020; Bansal et al., 2019; Polu & Sutskever, 2020; Evans et al., 2018; Lample & Charton, 2019) Input is a string of tokens corresponding to a truth statements of a theorem. Output is a string of tokens corresponding to proof for the theorem or identification of top few premises. They have used a variation of RNN formulation. Input is a string of tokens corresponding to theorem. Output is a string of tokens corresponding to the first step of the proof. The authors have used Graph Neural Networks (Zhou et al., 2018) .



string of knowledgebase and theorem. Output is a string of proof statements. They also release CoqGym data set of 71K proofs. A TreeLSTM (Tai et al., 2015b) formulation is used. (Huang et al., 2018) Input is a string representation of a theorem. Output is estimation of number of steps required to finish and prediction of the next expression. RNN based formulation. (Piotrowskiet al., 2019)    Input is a string of tokens corresponding to polynomial coefficients in symbolic form before normalization. Output is a normalized equivalent expression.They have used RNN based formulation.(Paliwal et al.,  2020)

