A SIMPLE BUT EFFECTIVE AND EFFICIENT GLOBAL MODELING PARADIGM FOR IMAGE RESTORATION

Abstract

Global modelling-based image restoration frameworks (e.g., transformer-like architecture) have gained popularity. Despite the remarkable advancement, their success may be at the cost of model parameters and FLOPs while the intrinsic characteristics (e.g., the task-specific degradation) are ignored. The objective of our work is orthogonal to previous studies and tailors a simple yet effective and efficient global modelling paradigm for image restoration. The key insights which motivate our study are two-fold: 1) Fourier transform is capable of disentangling image degradation and content component, serving as the image degradation prior embedded into image restoration framework; 2) Fourier domain innately embraces global property where each pixel of Fourier space is involved with all spatial pixels. We obey the de facto global modeling rule "spatial interaction + channel evolution" of previous studies. Differently, we customize the core designs: Fourier spatial interaction modeling and Fourier channel evolution. Equipped with the above-mentioned designs, our image restoration paradigm is verified on mainstream image restoration tasks including image de-raining, image enhancement, image de-hazing, and guided image super-resolution. Extensive experiments suggest that our paradigm achieves the competitive performance with fewer computational resources. Our main focus is not to beat previous frameworks but provide an alternative global modeling-based customized image restoration framework with efficient structure. Code will be publicly available.

1. INTRODUCTION

Image restoration aims to recover the latent clear image from its given degraded version. It is a highly ill-posed and challenging issue as there exists infinite feasible results for single degraded image. The representative image restoration tasks include image de-raining, image de-hazing, lowlight enhancement, guided image super-resolution, etc. In the past decades, a mount of research efforts have been devoted to solving the single image restoration problem, which can be classified into two categories: traditional optimization methods and deep learning-based methods (Zhang et al., 2018; Ren et al., 2018; Zhang et al., 2018; Ren et al., 2016b; Fu et al., 2021; Zhang et al., 2020; Liu et al., 2021a) . In terms of traditional image restoration methods, they formulate the image restoration process as an optimization problem and develop various image priors of the expected latent clear image to constrain the solution space, e.g., dark channel prior for image de-hazing (Dark, 2009) , histogram distribution prior for underwater image enhancement (Li et al., 2016) , non-local mean prior for image de-noising (Dixit & Phadke, 2013) , sparse image prior for guided image super-resolution (Kim & Kwon, 2010) as well as the commonly-used local and non-local smooth prior (Chen et al., 2013 ), low-rank prior (Ren et al., 2016a) . However, aforementioned image priors are difficult to develop and these traditional methods involve the iteration optimization, thus consuming the huge computational resources and further hindering their usage. In a word, the common sense is to explore the potential image prior to relieve the optimization difficulty of the ill-posed image restoration. On the line of deep learning-based methods, convolutional neural networks (CNNs) have received widespread attention and achieved promising improvement in image restoration tasks over traditional methods (Liu et al., 2020; Ma et al., 2021; Zhang et al., 2021a; Zhou et al., 2021; 2022b) . More recently, transformer and multi-layer perceptrons (MLPs)-based global modeling paradigms have struck the image restoration field and significantly surpassed the CNN-based methods. Despite the remarkable advancement, they are arbitrarily used for image restoration tasks while ignoring the intrinsic characteristics of specific image restoration task. The success may be owing to the huge cost of computational resources, limiting their practical applications, especially on resource-limited devices. We therefore wonder "Can we provide a customized global modeling image restoration paradigm in a simple but effective and efficient manner?" To this end, motivated by our observations on Fourier transformation for image restoration tasks in Figure 1 , we tailor a simple yet effective and efficient global modelling paradigm, which is orthogonal to previous studies and customized for image restoration. The core insights of our work are two-folder: 1) general image restoration prior: Fourier transform is capable of disentangling image degradation and content component, serving as the image degradation prior embedded into image restoration framework; 2) global modeling: Fourier domain innately embraces global property where each pixel of Fourier space is involved with all spatial pixels. As shown in Figure 2 , the existing global modeling paradigm (e.g., transformer and MLP-Mixer) follow the the de-facto global modeling rule "spatial interaction + channel evolution". Similarly, we obey the rule and customize the core designs: Fourier spatial interaction and Fourier channel evolution. Such designs are different from previous works and provide new insights on global modeling network structures for image restoration. Equipped with the above-mentioned designs, our image restoration paradigm tailed for image restoration is described in Figure 3 . Extensive experiments are conducted on mainstream image restoration tasks including image de-raining, image enhancement, image de-hazing, and guided image super-resolution. Experimental results suggest that our paradigm achieves the competitive performance with fewer computational resources. To emphasize, our main focus is not to beat previous frameworks but provide an alternative global modelling-based customized image restoration framework with efficient structure. Our contributions are summarized as follows: (1) We contribute the first global modeling paradigm for image restoration in a simple but effective and efficient manner. (2) We implicitly embed the



Figure 1: Motivations. Analysis of discrete Fourier transform (DFT) over mainstream image restoration tasks. In (a) and (d), we respectively swap the amplitude component and phase component of a degraded image and its clear version. It can be observed that the degradation effect is transferred, thus indicating that Fourier transform is capable of disentangling image degradation and content component and the degradation mainly lies in the amplitude component. To further verify our observation, we also swap the amplitude component and phase component of a degraded image and an irrelevant image in (b). The degradation is still mainly related to the amplitude component, such as the darkness for image enhancement. Similarly, a low-resolution image and its high-resolution counterpart are different in the amplitude component in (c). These observation motivates us to leverage the Fourier transform as the image degradation prior embedded into image restoration framework. More analysis and results can be found in the Appendix.

