PEAKS2IMAGE: RECONSTRUCTING FMRI STATISTI-CAL MAPS FROM PEAKS

Abstract

Neuroscience strives to overcome the lack of power due to the small sample size of imaging-based studies. An important step forward has been the creation of large-scale public image repositories, such as NeuroVault. Such repositories allow images to be compared across studies and automatically associated with cognitive terms. Yet, this type of meta-analysis faces a major roadblock: the scarcity and inconsistency of image annotations and metadata. Another resource containing rich annotations is the neuroscientific literature. However it only yields a handful of brain-space coordinates per publication, those of the main activity peaks reported in each study. This has led the community to mostly perform meta-analysis based on these reported coordinates. In this work, we propose Peaks2Image, a neuralnetwork approach to reconstruct continuous spatial representations of brain activity from peak activation tables. Peaks2Image thus associates rich annotations from the neuroscientific literature with dense brain reconstructions. Using those reconstructions, we train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies. We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting.

1. INTRODUCTION

Cognitive neuroscience aims to map cognitive processes onto brain regions. Functional Magnetic Resonance Imaging (fMRI) is one of the most powerful techniques available to identify such associations. This approach measures brain activity while subjects perform cognitive tasks in an MRI scanner, and then contrasts brain signals associated with different mental conditions. Statistical testing of these contrasts then identifies brain regions where the neural activity elicited by the conditions probed is significantly different. Unfortunately, the high cost of data acquisition limits the number of participants and tasks involved in each study (Poldrack et al., 2017) . A small number of participants causes low statistical power and a high proportion of false discoveries (Ioannidis, 2005; Button et al., 2013) . Analyzing a restricted set of cognitive tasks introduces the risk of over-interpreting statistical effects that are not specific to the mental functions under study (Poldrack, 2011) . Meta-analysis consists in aggregating the results of several studies to find effects that are reported consistently (Wager et al., 2007) . It helps overcome the challenge of small sample sizes and uncovers more reliable associations between brain activity and mental function. When feasible, meta-analysis should use the full statistical brain maps produced by the original studies (Salimi-Khorshidi et al., 2009) . Unfortunately, the vast majority of studies does not share the actual brain images. Instead, only the locations of the peaks of activation are reported. These are communicated in the form of tables, in scientific publications, containing stereotactic coordinates -3D coordinates in a standard spatial referential for the brain. This results in a poor representation of brain activity -most of the information contained in the original statistical maps is lost. Meta-analyses that rely on peak activation coordinates reported in publications are called Coordinate-Based Meta-Analyses (CBMA). Recent efforts to openly share full brain images and statistical maps such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017) , could facilitate Image-Based Meta-Analysis (IBMA) -relying on the brain images rather than coordinates. However, annotations available on those large databases are scarce and inconsistent, leading to difficulties to capture the semantics of the cognitive processes associated with images (Menuet et al., 2022) . There is no standard, agreed-upon ontology or vocabulary of mental functions. Moreover, all that is known is the task that the participant is performing, and ascribing mental functions to tasks is difficult (Poldrack et al., 2011) . Therefore, formally describing the mental states that underlie a brain image remains an open problem. Scientific literature and image repositories are therefore complementary: the literature provides rich descriptions of the studied cognitive tasks but poor spatial information, whereas image repositories contain complete representations of brain activity but lack useful annotations. In the present study, we introduce a meta-analysis method that combines the strengths of both. Beyond the challenge of sample size, effects uncovered by individual neuroimaging studies suffer from a lack of specificity. When a cognitive task is studied in isolation, there is no way of knowing if the observed brain activations are specific to the mental functions of interest, or associated with a broader set of mental processes. Authors often over-interpret the observed associations, resulting in the fallacy of unwarranted reverse inference (Poldrack, 2011) . One way of identifying more specific associations between brain activity and mental function is decoding: inferring the mental processes at play, given a brain image of neural activity. Indeed, to discriminate between a wide variety of cognitive states, a decoding model must identify brain regions that characterize each state, rather than brain regions that are merely activated consistently. For the resulting associations to be specific, it is crucial that many and diverse cognitive states are decoded jointly. Due to the difficulty of formalizing the mental states associated with a brain image, high-quality labels are lacking for this supervised task. Performing such large-scale (sometimes called "open-ended") decoding is therefore challenging, and in practice most studies that claim to perform "open-ended" decoding only discriminate a restricted set of cognitive concepts. In this work, Peaks2Image learns to discriminate dozens of cognitive terms on the largest available image repository, NeuroVault. To do so, it leverages both the rich descriptions of mental processes found in the literature and the high-quality neural activity data found in full-brain statistical maps. Peaks2Image reduces the gap between IBMA and CBMA, by reconstructing brain maps from peaks coordinates contained in neuroscientific publications. We leverage brain images from an unlabeled dataset to extract peaks, and train Peaks2Image to reconstruct images from the extracted peaks. We use Peaks2Image to obtain for the first time images associated with neuroscientific studies that only provide stereotactic coordinates. We evaluate whether those reconstructions are relevant by using them for brain image decoding. We associate labels with the studies using some criteria on the termfrequency inverse-document-frequency of the text. We use the decoding architecture from Neural Networks on Dictionaries (NNoD) (Menuet et al., 2022) trained on the neuroscientific corpus. We evaluate the decoding performance against 81 terms from NeuroVault. Peaks2Image succesfully decodes 65 of them on thousands of brain images from the NeuroVault database, without using any supervision from NeuroVault samples during training. While the evaluation has been performed on a limited set of terms, Peaks2Image can decode in a zero-shot setting any term from its vocabulary.

2. RELATED WORK

Automated meta-analysis (Laird et al., 2005; Yarkoni et al., 2011) has risen over the last few years to handle the growth of published neuroscientific studies. Handling neuroscientific concepts properly has emerged as a challenge for CBMA, leading to the use of more complex textual features and models. Dockès et al. (2020) broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. Ngo et al. ( 2021) leveraged language models (Beltagy et al., 2019) to capture term relationships semantically, leading to the encoding of any query. However, CBMA suffers from the drastic information reduction inherent to peak reporting. Using dense images yields more information for meta-analysis (Salimi-Khorshidi et al., 2009) . Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017) , decoding can now be performed across multiple studies (Mensch et al., 2017; Walters et al., 2022) . Nonetheless, annotations associated with those images are often of low quality. Automatic strategies to improve this labeling are necessary to benefit properly from the scale of the data (Poldrack and Yarkoni, 2016) . Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts. Overall, neuroscience meta-analyses have

