CORRELATIVE INFORMATION MAXIMIZATION BASED BIOLOGICALLY PLAUSIBLE NEURAL NETWORKS FOR CORRELATED SOURCE SEPARATION

Abstract

The brain effortlessly extracts latent causes of stimuli, but how it does this at the network level remains unknown. Most prior attempts at this problem proposed neural networks that implement independent component analysis, which works under the limitation that latent causes are mutually independent. Here, we relax this limitation and propose a biologically plausible neural network that extracts correlated latent sources by exploiting information about their domains. To derive this network, we choose the maximum correlative information transfer from inputs to outputs as the separation objective under the constraint that the output vectors are restricted to the set where the source vectors are assumed to be located. The online formulation of this optimization problem naturally leads to neural networks with local learning rules. Our framework incorporates infinitely many set choices for the source domain and flexibly models complex latent structures. Choices of simplex or polytopic source domains result in networks with piecewise-linear activation functions. We provide numerical examples to demonstrate the superior correlated source separation capability for both synthetic and natural sources.

1. INTRODUCTION

Extraction of latent causes, or sources, of complex stimuli sensed by sensory organs is essential for survival. Due to absence of any supervision in most circumstances, this extraction must be performed in an unsupervised manner, a process which has been named blind source separation (BSS) (Comon & Jutten, 2010; Cichocki et al., 2009) . How BSS may be achieved in visual, auditory, or olfactory cortical circuits has attracted the attention of many researchers, e.g. (Bell & Sejnowski, 1995; Olshausen & Field, 1996; Bronkhorst, 2000; Lewicki, 2002; Asari et al., 2006; Narayan et al., 2007; Bee & Micheyl, 2008; McDermott, 2009; Mesgarani & Chang, 2012; Golumbic et al., 2013; Isomura et al., 2015) . Influential papers showed that visual and auditory cortical receptive fields could arise from performing BSS on natural scenes (Bell & Sejnowski, 1995; Olshausen & Field, 1996) and sounds (Lewicki, 2002) . The potential ubiquity of BSS in the brain suggests that there exists generic neural circuit motifs for BSS (Sharma et al., 2000) . Motivated by these observations, here, we present a set of novel biologically plausible neural network algorithms for BSS. BSS algorithms typically derive from normative principles. The most important one is the information maximization principle, which aims to maximize the information transferred from input mixtures to separator outputs under the restriction that the outputs satisfy a specific generative assumption about sources. However, Shannon mutual information is a challenging choice for quantifying information transfer, especially for data-driven adaptive applications, due to its reliance on the joint and conditional densities of the input and output components. This challenge is eased by the independent component analysis (ICA) framework by inducing joint densities into separable forms based on the assumption of source independence (Bell & Sejnowski, 1995) . In particular scenarios, the mutual independence of latent causes of real observations may not be a plausible assumption (Träuble et al., 2021) . To address potential dependence among latent components, Erdogan (2022) recently proposed the use of the second-order statistics-based correlative (log-determinant) mutual information maximization for BSS to eliminate the need for the independence assumption, allowing for correlated source separation. In this article, we propose an online correlative information maximization-based biologically plausible neural network framework (CorInfoMax) for the BSS problem. Our motivations for the proposed framework are as follows: • The correlative mutual information objective function is only dependent on the second-order statistics of the inputs and outputs. Therefore, its use avoids the need for costly higher-order statistics or joint pdf estimates, • The corresponding optimization is equivalent to maximization of correlation, or linear dependence, between input and output, a natural fit for the linear inverse problem, • The framework relies only on the source domain information, eliminating the need for the source independence assumption. Therefore, neural networks constructed with this framework are capable of separating correlated sources. Furthermore, the CorInfoMax framework can be used to generate neural networks for infinitely many source domains corresponding to the combination of different attributes such as sparsity, nonnegativity etc., • The optimization of the proposed objective inherently leads to learning with local update rules. Figure 1 illustrates CorInfoMax neural networks for two different source domain representation choices, which are three-layer neural networks with piecewise linear activation functions.We note that the proposed CorInfoMax framework, beyond solving the BSS problem, can be used to learn structured and potentially correlated representations from data through the maximum correlative information transfer from inputs to the choice of the structured domain at the output.



• CorInfoMax acts as a unifying framework to generate biologically plausible neural networks for various unsupervised data decomposition methods to obtain structured latent representations, such as nonnegative matrix factorization (NMF) (Fu et al., 2019), sparse component analysis (SCA) (Babatas & Erdogan, 2018), bounded component analysis (BCA) (Erdogan, 2013; Inan & Erdogan, 2014) and polytopic matrix factorization (PMF) (Tatli & Erdogan, 2021).

Figure 1: CorInfoMax BSS neural networks for two different canonical source domain representations. x i 's and y i 's represent inputs (mixtures) and (separator) outputs , respectively, W are feedforward weights, e i 's are errors between transformed inputs and outputs, B y , the inverse of output autocorrelation matrix, represents lateral weights at the output. For the canonical form (a), λ i 's are Lagrangian interneurons imposing source domain constraints, A P (A T P ) represents feedforward (feedback) connections between outputs and interneurons. For the canonical form (b), interneurons on the right impose sparsity constraints on the subsets of outputs.

