LEARNING SPARSE AND LOW-RANK PRIORS FOR IMAGE RECOVERY VIA ITERATIVE REWEIGHTED LEAST SQUARES MINIMIZATION

Abstract

We introduce a novel optimization algorithm for image recovery under learned sparse and low-rank constraints, which we parameterize as weighted extensions of the ℓ p p -vector and S p p Schatten-matrix quasi-norms for 0 < p ≤ 1, respectively. Our proposed algorithm generalizes the Iteratively Reweighted Least Squares (IRLS) method, used for signal recovery under ℓ 1 and nuclear-norm constrained minimization. Further, we interpret our overall minimization approach as a recurrent network that we then employ to deal with inverse low-level computer vision problems. Thanks to the convergence guarantees that our IRLS strategy offers, we are able to train the derived reconstruction networks using a memory-efficient implicit back-propagation scheme, which does not pose any restrictions on their effective depth. To assess our networks' performance, we compare them against other existing reconstruction methods on several inverse problems, namely image deblurring, super-resolution, demosaicking and sparse recovery. Our reconstruction results are shown to be very competitive and in many cases outperform those of existing unrolled networks, whose number of parameters is orders of magnitude higher than that of our learned models. The code is available at this link.

1. INTRODUCTION

With the advent of modern imaging techniques, we are witnessing a significant rise of interest in inverse problems, which appear increasingly in a host of applications ranging from microscopy and medical imaging to digital photography, 2D&3D computer vision, and astronomy (Bertero & Boccacci, 1998 ). An inverse imaging problem amounts to estimating a latent image from a set of possibly incomplete and distorted indirect measurements. In practice, such problems are typical illposed (Tikhonov, 1963; Vogel, 2002) , which implies that the equations relating the underlying image with the measurements (image formation model) are not adequate by themselves to uniquely characterize the solution. Therefore, in order to recover approximate solutions, which are meaningful in a statistical or physical sense, from the set of solutions that are consistent with the image formation model, it is imperative to exploit prior knowledge about certain properties of the underlying image. Among the key approaches for solving ill-posed inverse problems are variational methods (Benning & Burger, 2018) , which entail the minimization of an objective function. A crucial part of such an objective function is the regularization term, whose role is to promote those solutions that fit best our prior knowledge about the latent image. Variational methods have also direct links to Bayesian methods and can be interpreted as seeking the penalized maximum likelihood or the maximum a posteriori (MAP) estimator (Figueiredo et al., 2007) , with the regularizer matching the negative log-prior. Due to the great impact of the regularizer in the reconsturction quality, significant research effort has been put in the design of suitable priors. Among the overwhelming number of existing priors in the literature, sparsity and low-rank (spectral-domain sparsity) promoting priors have received considerable attention. This is mainly due to their solid mathematical foundation and the competitive results they achieve (Bruckstein et al., 2009; Mairal et al., 2014) . Nowdays, thanks to the advancements of deep learning there is a plethora of networks dedicated to image reconstruction problems, which significantly outperform conventional approaches. Nevertheless, they are mostly specialized and applicable to a single task. Further, they are difficult to analyze and interpret since they do not explicitly model any of the well-studied image properties, successfully utilized in the past (Monga et al., 2021) . In this work, we aim to harness the power of supervised learning but at the same time rely on the rich body of modeling and algorithmic ideas that have been developed in the past for dealing with inverse problems. To this end our contributions are: (1) We introduce a generalization of the Iterative Reweighted Least Squares (IRLS) method based on novel tight upper-bounds that we derive. (2) We design a recurrent network architecture that explicitly models sparsity-promoting image priors and is applicable to a wide range of reconstruction problems. (3) We propose a memory efficient training strategy based on implicit back-propagation that does not restrict in any way our network's effective depth.

2. IMAGE RECONSTRUCTION

Let us first focus on how one typically deals with inverse imaging problems of the form: y = Ax + n, where x ∈ R n•c is the multichannel latent image of c channels, that we seek to recover, A : R n•c → R m•c ′ is a linear operator that models the impulse response of the sensing device, y ∈ R m•c ′ is the observation vector, and n ∈ R m•c ′ is a noise vector that models all approximation errors of the forward model and measurement noise. Hereafter, we will assume that n consists of i.i.d samples drawn from a Gaussian distribution of zero mean and variance σ 2 n . Note that despite of the seeming simplicity of this observation model, it is widely used in the literature, since it can accurately enough describe a plethora of practical problems. Specifically, by varying the form of A, Eq. ( 1) can cover many different inverse imaging problems such as denoising, deblurring, demosaicking, inpainting, super-resolution, MRI reconstruction, etc. If we further define the objective function: J (x) = 1 2σ 2 n ∥y -Ax∥ 2 2 + R (x) , where R : R n•c → R + = {x ∈ R|x ≥ 0} is the regularizer (image prior), we can recover an estimate of the latent image x * as the minimizer of the optimization problem: x * = arg min x J (x). Since the type of the regularizer R (x) can significantly affect the reconstruction quality, it is of the utmost importance to employ a proper regularizer for the reconstruction task at hand.

2.1. SPARSE AND LOW-RANK IMAGE PRIORS

Most of the existing image regularizers in the literature can be written in the generic form: R (x) = ℓ i=1 ϕ (G i x) , where G : R n•c → R ℓ•d is the regularization operator that transforms x, G i = M i G, M i = I d ⊗e T i with ⊗ denoting the Kronecker product, and e i is the unit vector of the standard R ℓ basis. Further, ϕ : R d → R + is a potential function that penalizes the response of the d-dimensional transformdomain feature vector, z i = G i x ∈ R d . Among such regularizers, widely used are those that promote sparse and low-rank responses by utilizing as their potential functions the ℓ 1 and nuclear norms (Rudin et al., 1992; Figueiredo et al., 2007; Lefkimmiatis et al., 2013; 2015) . Enforcing sparsity of the solution in some transform-domain has been studied in-depth and is supported both by solid mathematical theory (Donoho, 2006; Candes & Wakin, 2008; Elad, 2010) as well as strong empirical results, which indicate that distorted images do not typically exhibit sparse or low-rank representations, as opposed to their clean counterparts. More recently it has also been advocated that non-convex penalties such as the ℓ p p vector and S p p Schatten-matrix quasi-norms enforce sparsity better and lead to improved image reconstruction results (Chartrand, 2007; Lai et al., 2013; Candes et al., 2008; Gu et al., 2014; Liu et al., 2014; Xie et al., 2016; Kümmerle & Verdun, 2021) . Based on the above, we consider two expressive parametric forms for the potential function ϕ (•), which correspond to weighted and smooth extensions of the ℓ p p and the Schatten matrix S p p quasinorms with 0 < p ≤ 1, respectively. The first one is a sparsity-promoting penalty, defined as: ϕ sp (z; w, p) = 



z, w ∈ R d , (4) while the second one is a low-rank (spectral-domain sparsity) promoting penalty, defined as:ϕ lr (Z; w, p) = r j=1 w j σ 2 j (Z) + γ p 2 , Z ∈ R m×n , w ∈ R r + , with r = min (m, n) . (5)

