SENSORY RESILIENCE BASED ON SYNESTHESIA

Abstract

Situated cognition depends on accessing environmental state through sensors. Engineering and cost constraints usually lead to limited "pathways" where, for example, a vision sub-system only includes a camera and the software to deal with it. This traditional and rational design style entails any hardware defect on the pathway causes the system to grind to a halt until repair. We propose a "sensoriplexer" as neural component architecture to address this issue, under the common scenario of multiple sensors availability. This component architecture learns to mix and relate pathways, such that an agent facing failure in a sensory sub-system can degrade gracefully and coherently by relying on its other sub-systems. The architecture is inspired by the concept of synesthesia, and relies on statistical coupling between sensor signals. We show the benefit and limitation of the architecture on a simple shape recognition and a more complex emotion recognition scenarios.

1. INTRODUCTION

Situated agents embody various degrees of "SPA loops", where they Sense-Process-Act repeatedly in their environment. The simplest agent senses only one dimension in the environment. Sensing failure prevents it from processing and acting-basically causing it to halt. More complex agents with multiple sensors can sometimes continue operating despite the loss of some sensors, but they cannot usually perform anymore the functions related to the lost sensors. Sensor redundancyfoot_0 is a common solution to ensure continuing operations. Flagship projects like space rovers introduce redundant sensors, such as doubling all "hazcam" on NASA's Curiosity or CNSA's Yutu. Yet engineering and cost constraints block this option in many systems. The loss of most non-critical sensors often means the agent enters a "degraded" mode of operations. Biological agents can compensate to some extent the loss of a sensor, by using the other ones they are endowed with. A blind person can rely on touch to "read" visual cues encoded to Braille, or "hear" a speech signed visually. Impressive technology allows learning to feel sounds on the skin Bach-y Rita (1972) , and by extension for deaf people to "listen" to a conversation by translating sound into tactile patterns on their skin Novich & Eagleman (2015) . The compensation capability appears to rely on brain mechanisms that relate different sensory inputs of the same object. These brain mechanisms also appear common to all, with different degree of expression Cytowic (2018). For example, we perceive an apple via (mainly) vision, touch and smell. We can relate the vision of an apple to its likely smell or touch. Although the brain mechanisms are not entirely identified at the present time, the effect has been named synesthesia, sometimes ideaesthesia Jürgens & Nikolić (2012); Nikolić (2014). This article proposes the sensoriplexer component architecture to model synesthesia-inspired mechanisms in artificial agents. The sensoriplexer (SP) allows an agent learning relations between its "senses" for exploitation in its downstream activities. After presenting work related to resilience and their limits, we present a formal model of SP and a corresponding implementationfoot_1 . We use the implementation to conduct a series of experiments to demonstrate and evaluate the capabilities of systems including SP. The article ends with a discussion of the results, and future work. The problem at hand is of resiliency: A situated agent is resilient if it can continue operating despite a usually pre-defined range of failure modes. The pre-definition covers explicit problems the agent must deal with (e.g. critical safety, guaranteeing a car stops), classes of problems (e.g. liveness properties, absence of deadlocks), and sometimes specifications of behaviours under partial failures (e.g. a car must be steerable and stoppable even if the air conditioner fails). Resiliency is a broad topic, and we frame its meaning for the extent of this article. An important property of resilience framed this way is its extent: We expect a resilient agent to continue operating (as autonomously as possible), yet within compliance, such as safety guarantees. The sensoriplexer addresses resiliency issues in situated agents, using concepts rooted in Neuroscience and Psychology. Other disciplines have inspired alternative approaches, like redundancy: The loss of a sensor triggers a "switch" to a redundant one. Redundancy is a powerful mean to achieve resilience, but it is not always feasible. Aside cost issues incurred by extra sensors, some systems cannot physically accommodate any extra. Typical quadcopter drones with autopilot capabilities (seen here as situated agents) embark a camera, but they often cannot host a second one, whose payload would exceed the lift capabilities. A similar example is on sensors in satellites and space rovers, where weight and available mount space are very constrained: Redundancy is reserved to safety-critical sensors only. In this paper, we do not see SP as a solution to compensate advanced scientific sensors on satellites, but we expect the concept of synesthesia to inspire future sensor designs for such extreme missions. Programmable sensors offer an alternative grounded in Electronics and Chemistry. Programmable packages like the CMUCam (where the sensor is not directly programmed) and sensor chips embedding artificial neural networks like the IMX500 from Sony provide agent designers with the capability to programmatically address resiliency issues CMU (2020); Sony (2020). The trend of "intelligent sensors" is relevant to the problem of resilience, yet their application remains limited under the present understanding of their capabilities. A vision sensor can adapt to lighting issues by switching frequency, but cannot relate to or leverage other signals. On a more theoretical note, we relate this work to the thread of research stemmed from the efficient coding hypothesis, proposed by Barlow, and most notably the statistical representation of signals Barlow (1961) . The hypothesis refers to sensory pathways-conceptual routes from a sense of the environment, through areas of the brain, to their final destinations in the brain-for modeling the processing of sensory "messages". Later work shows a statistical dependency between some pathways Simoncelli (2003) . SP is an adaptation and implementation of these dependencies in artificial agents. These dependencies are concretely modeled as joint distributions over the different pathways, without explicit relation to any architectural candidate area in the brain (we aim at functional equivalence only, at best). Another related theory is practopoiesis from Nikolić Nikolić (2014). This theory of "creation of actions" models general principles of abstractions, and introduces the notion of "traverse" along sensory pathways. Traverses separate representation realms, from what we qualify as low-level raw signals to higher-level representations of these signals. The higher-level representations introduce new semantics, typically meaningless at lower levels. This theory can loosely be compared with multilayer convolutional networks in (for example) character recognition, where the first layers compile low-level glyphs with broad semantics, and last layers capture higher-level concepts of characters. Practopoiesis indicates the number of traverses necessary to achieve a level of representation depends on the task at hand. SP in this article models a tentative traverse from individual sensory paths to an abstract representation of objects represented by the incoming signals. We do not claim SP is a primitive traverse. Its purpose is to model and implement joint representations of objects, and, again, it may not be grounded in any biological apparatus. A final note pertains to the difference with sensor fusion, often critical to the design of systems like self-driving cars Wikipedia (2020). Sensor fusion is closely related to the functions implemented by SP, but they serve different purposes. Sensor fusion aims at decreasing uncertainty about an object, typically its position or some other property. Sensor fusion relies on adding sensors and "fusing" their data into coherent pictures. The addition continues until acceptable uncertainty is achieved for the given target. SP aims at compensating sensor failures by exploiting other sensors, i.e. decreasing uncertainty whenever removing sensors.



Informally, redundancy means here setting multiple sensors on the same environmental object. Two close-by front cameras on a robot are redundant, but a front and rear camera pair is not (they are complementary). Available in the supplementary materials, and on GitLab (private for now)

