DATA AUGMENTATION FOR DEEP LEARNING BASED AC-CELERATED MRI RECONSTRUCTION

Abstract

Deep neural networks have emerged as very successful tools for image restoration and reconstruction tasks. These networks are often trained end-to-end to directly reconstruct an image from a noisy or corrupted measurement of that image. To achieve state-of-the-art performance, training on large and diverse sets of images is considered critical. However, it is often difficult and/or expensive to collect large amounts of training images. Inspired by the success of Data Augmentation (DA) for classification problems, in this paper, we propose a pipeline for data augmentation for image reconstruction tasks arising in medical imaging and explore its effectiveness at reducing the required training data in a variety of settings. We focus on accelerated magnetic resonance imaging, where the goal is to reconstruct an image from a few under-sampled linear measurements. Our DA pipeline is specifically designed to utilize the invariances present in medical imaging measurements as naive DA strategies that neglect the physics of the problem fail. We demonstrate the effectiveness of our data augmentation pipeline by showing that for some problem regimes, DA can achieve comparable performance to the state of the art on the fastMRI dataset while using significantly fewer training data. Specifically, for 8-fold acceleration we achieve performance comparable to the state of the art with only 10% of the training data for multi-coil reconstruction and with only 33% of the training data for single-coil reconstruction. Our findings show that in the low-data regime DA is beneficial, whereas in the high-data regime it has diminishing returns.

1. INTRODUCTION

In magnetic resonance imaging (MRI), an extremely popular medical imaging technique, it is common to reduce the acquisition time by subsampling the measurements, because this reduces cost and increases accessibility of MRI to patients. However, since due to the subsampling, there are fewer equations than unknowns, the signal is not uniquely identifiable from the measurements. To overcome this challenge there has been a flurry of activity over the last decade aimed at utilizing prior knowledge about the signal, in a research area referred to as compressed sensing (Candes et al., 2006; Donoho, 2006) . Classical compressed sensing methods reduce the required number of measurements by utilizing prior knowledge about the images during the reconstruction process, often via a convex regularization that enforces sparsity in an appropriate transformation of the image. More recently, deep learning techniques have been used to enforce much more nuanced forms of prior knowledge (see Ongie et al. (2020) and references therein for an overview). The most successful of these approaches aim to directly learn the inverse mapping from the measurements to the image by training on a large set of training data consisting of signal/measurement pairs. This approach often enables faster reconstruction of images, but more importantly, deep learning techniques yield significantly higher quality reconstructions. This in turn enables fewer measurements further reducing image acquisition times. For instance, in an accelerated MRI competition known as FastMRI Challenge (Zbontar et al., 2018) , all the top contenders used deep learning reconstruction techniques. Contrary to classical compressive sensing approaches, however, deep learning techniques typically rely on large sets of training data often consisting of images along with the corresponding measurement. This is also true about the use of deep learning techniques in other areas such as computer vision and Natural Language Processing (NLP) were superb empirical success has been observed. While large datasets have been harvested and carefully curated by tech companies in areas such as vision and NLP, this is not feasible in many scientific applications including MRI. It is difficult and expensive to collect the necessary datasets for a variety of reasons, including patient confidentiality requirements, cost and time of data acquisition, lack of medical data compatibility standards, and the rarity of certain diseases. A common strategy to reduce reliance on training data in classification tasks is data augmentation. In a classification setting data augmentation consists of adding additional synthetic data obtained by performing invariant alterations to the data (e.g. flips, translations, or rotations) which do not affect the labels. Such data augmentation techniques are commonly used in classification tasks to significantly increase the performance on standard benchmarks such as ImageNet and CIFAR-10. More specific to medical imaging, data augmentation techniques have been successfully applied to registration, classification and segmentation of medical images. More recently, several studies (Zhao et al., 2020b; Karras et al., 2020; Zhao et al., 2020a) have demonstrated that data augmentation can significantly reduce the data needed for GAN training for high quality image generation. In regression tasks such as image reconstruction, however, data augmentation techniques are less common and much more difficult to design in part due to the lack of the aforementioned invariances (e.g., measurements of a rotated image are not the same as measurements from the original image). The goal of this paper is to explore the benefits of data augmentation techniques for accelerated MRI with limited training data. By carefully taking into account the physics behind the MRI acquisition process we design a data augmentation pipeline, which we call MRAugment, that can successfully reduce the amount of training data required. Specifically, our contributions are as follows: • We propose a data augmentation technique tailored to the physics of the MR reconstruction problem. We note that it is not obvious how to perform data augmentation in the context of inverse problems, because by changing an image to enlarge the training set, we do not automatically get a corresponding measurement, contrary to classification problems, where the label is retained. In fact, we demonstrate that naive forms of data augmentation that do not properly take into account the underlying physics do not work. • We demonstrate the effectiveness of MRAugment on a benchmark accelerated MRI data set, specifically on the fastMRI (Zbontar et al., 2018) dataset. For 8-fold acceleration and multi-coil measurements (multi-coil measurements are the standard acquisition mode for clinical practice) we can achieve performance comparable to the state of the art with only 10% of the training data. Similarly, again for 8-fold acceleration and single-coil experiments (an acquisition mode popular for experimentation) MRAugment can achieve the performance of reconstructions methods trained on the entire data set while using only 33% of the training data. • We perform experiments showing that in a low-data regime, where we have only limited training data MRAugment is beneficial, whereas in the high-data regime it has diminishing returns. In particular we demonstrate that in a low data regime (≈ 1% of training data), data augmentation very significantly boosts reconstruction performance and captures diagnostically significant detail that is missed without data augmentation. In a moderate data regime (≈ 10 -33% of training data), MRAugment still achieves significant improvement in reconstruction performance and may help to avoid hallucinations caused by overfitting without the use of DA. • Finally, while we focus on the top performing neural networks, MRAugment can seamlessly integrate with any deep learning model and therefore it can be useful in a variety of MRI problems.

2. BACKGROUND AND PROBLEM FORMULATION

In this section, we provide a brief background on accelerated MRI and formulate the problem. MRI is a medical imaging technique that exploits strong magnetic fields to form images of the anatomy. MRI is a prominent imaging modality in diagnostic medicine and biomedical research because it does not expose patients to ionizing radiation, contrary to competing technologies such as computed and positron emission tomography (CT and PET). However, performing an MR scan is time intensive, which is problematic for the following reasons. First, patients are exposed to long acquisition times in a confined space with high noise levels. Second, long acquisition times induce reconstruction artifacts caused through patient movement,

