A SIMPLE BUT EFFECTIVE AND EFFICIENT GLOBAL MODELING PARADIGM FOR IMAGE RESTORATION

Abstract

Global modelling-based image restoration frameworks (e.g., transformer-like architecture) have gained popularity. Despite the remarkable advancement, their success may be at the cost of model parameters and FLOPs while the intrinsic characteristics (e.g., the task-specific degradation) are ignored. The objective of our work is orthogonal to previous studies and tailors a simple yet effective and efficient global modelling paradigm for image restoration. The key insights which motivate our study are two-fold: 1) Fourier transform is capable of disentangling image degradation and content component, serving as the image degradation prior embedded into image restoration framework; 2) Fourier domain innately embraces global property where each pixel of Fourier space is involved with all spatial pixels. We obey the de facto global modeling rule "spatial interaction + channel evolution" of previous studies. Differently, we customize the core designs: Fourier spatial interaction modeling and Fourier channel evolution. Equipped with the above-mentioned designs, our image restoration paradigm is verified on mainstream image restoration tasks including image de-raining, image enhancement, image de-hazing, and guided image super-resolution. Extensive experiments suggest that our paradigm achieves the competitive performance with fewer computational resources. Our main focus is not to beat previous frameworks but provide an alternative global modeling-based customized image restoration framework with efficient structure. Code will be publicly available.

1. INTRODUCTION

Image restoration aims to recover the latent clear image from its given degraded version. It is a highly ill-posed and challenging issue as there exists infinite feasible results for single degraded image. The representative image restoration tasks include image de-raining, image de-hazing, lowlight enhancement, guided image super-resolution, etc. In the past decades, a mount of research efforts have been devoted to solving the single image restoration problem, which can be classified into two categories: traditional optimization methods and deep learning-based methods (Zhang et al., 2018; Ren et al., 2018; Zhang et al., 2018; Ren et al., 2016b; Fu et al., 2021; Zhang et al., 2020; Liu et al., 2021a) . In terms of traditional image restoration methods, they formulate the image restoration process as an optimization problem and develop various image priors of the expected latent clear image to constrain the solution space, e.g., dark channel prior for image de-hazing (Dark, 2009) , histogram distribution prior for underwater image enhancement (Li et al., 2016) , non-local mean prior for image de-noising (Dixit & Phadke, 2013) , sparse image prior for guided image super-resolution (Kim & Kwon, 2010) as well as the commonly-used local and non-local smooth prior (Chen et al., 2013 ), low-rank prior (Ren et al., 2016a) . However, aforementioned image priors are difficult to develop and these traditional methods involve the iteration optimization, thus consuming the huge computational resources and further hindering their usage. In a word, the common sense is to explore the potential image prior to relieve the optimization difficulty of the ill-posed image restoration. On the line of deep learning-based methods, convolutional neural networks (CNNs) have received widespread attention and achieved promising improvement in image restoration tasks over traditional methods (Liu et al., 2020; Ma et al., 2021; Zhang et al., 2021a; Zhou et al., 2021; 2022b) . More recently, transformer and multi-layer perceptrons (MLPs)-based global modeling paradigms

