SHALLOW LEARNING IN MATERIO

Abstract

We introduce Shallow Learning In Materio (SLIM) as a resource-efficient method to realize closed-loop higher-order perceptrons. Our SLIM method provides a rebuttal to the Minsky school's disputes with the Rosenblatt school about the efficacy of learning representations in shallow perceptrons. As a proof-of-concept, here we devise a physically-scalable realization of the parity function. Our findings are relevant to artificial intelligence engineers, as well as neuroscientists and biologists.

1. Introduction

How do we best learn representations? We do not yet fully understand how cognition is manifested in any brain, not even in those of a worm (Rankin, 2004) . It is an open question if the shallow brain of a worm is capable of working memory, but if it were then it certainly must depart from the mechanistic models of large-scale brains (Eliasmith et al., 2012) . Nevertheless, worm-brain inspired learning combined with "scalable" deep learning architectures have been employed in self-driving cars (Lechner et al., 2020) . At present, by scalable we refer to TPU-based architectures (Jouppi et al., 2017) trained by gradient-descent (Rumelhart et al., 1986 ). However, one could envision a super-scalable future that is less synthetic and based on self-organized nanomaterial systems (Bose et al., 2015; Chen et al., 2020; Mirigliano et al., 2021) that natively realize higher-order (Lawrence, 2022a) and recurrent neural networks. In this short communication, we shall lay yet another brick towards such a future by providing theoretical arguments. Our perspective on cognitive material systems is illuminated in Figure 1 . Deep learning owes its success to our technological capacity to synthesize massively-parallel and programmable electronic circuits. It is yet to fully exploit Darwinian and Hebbian learning methods that pioneers of the cybernetics movement experimented with by training homeostats (Ashby, 1952) and perceptrons (Rosenblatt, 1961) . The spirit of Darwinian (Stanley et al., 2019) and Hebbian (Scellier & Bengio, 2017) learning continues to be alive, though. Here, we add fuel to that fire by advocating for an in-materio approach. Employing physical systems in their native form for solving computational tasks had gained attention due to the efforts of the 'evolution in materio' community (Miller & Downing, 2002) . The earliest result was by Pask (1960) who grew dendritic metallic threads in a ferrous sulphate solution to function as a sound-frequency discriminator (which he called an ear, quite romantically). Now, more recent efforts are under the banner of physical reservoir computing (Tanaka et al., 2019) for realizing sequential functionality. Here, we will commit to combinational functionality by equilibrium-point logic (Lawrence, 2022b) in material systems realizing closed-loop higher-order perceptrons.

2. Theory

Perceptrons were developed by Rosenblatt and his team, and were trained by a Hebbian learning rule (error-controlled reinforcement) with proven guarantees for convergence. Unfortunately, they started recieving a bad rap after Minsky & Papert (1988) published a proof that 2 N association neurons are required to learn the N -bit parity function. However, this analysis is only applicable if all neurons are threshold logic gates, what Rosenblatt called simple units. Physical neural networks, on the other hand, can natively realize complex units. Hence, we introduce a shallow learning in materio (SLIM) perceptron as depicted in Figure 2 . y x[1] x[2] x[N] w[1] w[2] w[N] s[1] s[2] s[N] s[1] y x[1] x[2] x[N] s[2 N ] s[N+1] w[1] w[N+1] w[2 N ]

Minsky-Papert perceptron SLIM perceptron

Figure 2 : To learn the N -bit parity function, the required number of synaptic weights (depicted as dashed lines) for our SLIM perceptron scales as N instead of the 2 N required for the Minsky-Papert perceptron. This gain in resource-efficiency is possible because the hidden states s 1:N of the SLIM perceptron can compute deep-feedforward functionality by equilibrium-point control. For a proof-of-concept, we commit to a minimally connected recurrent network with physical states s i from i = 1 : N , yielding a state-space model of the form ṡi = x i + F i (s i-1 , s i , s i+1 ),



Figure 1: A typology of cognitive material systems.

