BRAINBERT: SELF-SUPERVISED REPRESENTATION LEARNING FOR INTRACRANIAL RECORDINGS

Abstract

We create a reusable Transformer, BrainBERT, for intracranial field potential recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much less data by being pretrained in an unsupervised manner on a large corpus of unannotated neural recordings. Our approach generalizes to new subjects with electrodes in new positions and to unrelated tasks showing that the representations robustly disentangle the neural signal. Just like in NLP where one can study language by investigating what a language model learns, this approach enables investigating the brain by studying what a model of the brain learns. As a first step along this path, we demonstrate a new analysis of the intrinsic dimensionality of the computations in different areas of the brain. To construct BrainBERT, we combine super-resolution spectrograms of neural data with an approach designed for generating contextual representations of audio by masking. In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.

1. INTRODUCTION

Methods that analyze neural recordings have an inherent tradeoff between power and explainability. Linear decoders, by far the most popular, provide explainability; if something is decodable, it is computed and available in that area of the brain. The decoder itself is unlikely to be performing the task we want to decode, instead relying on the brain to do so. Unfortunately, many interesting tasks and features may not be linearly decodable from the brain for many reasons including a paucity of annotated training data, noise from nearby neural processes, and the inherent spatial and temporal resolution of the instrument. More powerful methods that perform non-linear transformations have lower explainability: there is a danger that the task is not being performed by the brain, but by the decoder itself. In the limit, one could conclude that object class is computed by the retina using a CNN-based decoder but it is well established that the retina does not contain explicit information about objects. Self-supervised representation learning provides a balance between these two extremes. We learn representations that are generally useful for representing neural recordings, without any knowledge of a task being performed, and then employ a linear decoder. The model we present here, BrainBERTfoot_0 , learns a complex non-linear transformation of neural data using a Transformer. Using BrainBERT, one can linearly decode neural recordings with much higher accuracy and with far fewer examples than from raw features. BrainBERT is pretrained once across a pool of subjects, and then provides off-the-shelf capabilities for analyzing new subjects with new electrode locations even when data is scarce. Neuroscientific experiments tend to have little data in comparison to other machine learning settings, making additional sample efficiency critical. Other applications, such as brain-computer interfaces can also benefit from shorter training regimes, as well as from BrainBERT's significant performance improvements. In addition, the embeddings of the neural data provide a new means by which to investigate the brain.



https://github.com/czlwang/BrainBERT 1

