EMBEDDING FOURIER FOR ULTRA-HIGH-DEFINITION LOW-LIGHT IMAGE ENHANCEMENT

Abstract

Ultra-High-Definition (UHD) photo has gradually become the standard configuration in advanced imaging devices. The new standard unveils many issues in existing approaches for low-light image enhancement (LLIE), especially in dealing with the intricate issue of joint luminance enhancement and noise removal while remaining efficient. Unlike existing methods that address the problem in the spatial domain, we propose a new solution, UHDFour, that embeds Fourier transform into a cascaded network. Our approach is motivated by a few unique characteristics in the Fourier domain: 1) most luminance information concentrates on amplitudes while noise is closely related to phases, and 2) a high-resolution image and its low-resolution version share similar amplitude patterns. Through embedding Fourier into our network, the amplitude and phase of a low-light image are separately processed to avoid amplifying noise when enhancing luminance. Besides, UHDFour is scalable to UHD images by implementing amplitude and phase enhancement under the low-resolution regime and then adjusting the high-resolution scale with few computations. We also contribute the first real UHD LLIE dataset, UHD-LL, that contains 2,150 low-noise/normal-clear 4K image pairs with diverse darkness and noise levels captured in different scenarios. With this dataset, we systematically analyze the performance of existing LLIE methods for processing UHD images and demonstrate the advantage of our solution. We believe our new framework, coupled with the dataset, would push the frontier of LLIE towards UHD. The code and dataset are available at https://lichongyi.github.io/UHDFour/.

1. INTRODUCTION

With the advent of advanced imaging sensors and displays, Ultra-High-Definition (UHD) imaging has witnessed rapid development in recent years. While UHD imaging offers broad applications and makes a significant difference in picture quality, the extra pixels also challenge the efficiency of existing image processing algorithms. In this study, we focus on one of the most challenging tasks in image restoration, namely low-light image enhancement (LLIE) , where one needs to jointly enhance the luminance and remove inherent noises caused by sensors and dim environments. Further to these challenges, we lift the difficulty by demanding efficient processing in the UHD regime. Despite the remarkable progress in low-light image enhancement (LLIE) (Li et al., 2021a ), existing methods (Zhao et al., 2021; Wu et al., 2022; Xu et al., 2022) , as shown in Figure 1 , show apparent drawbacks when they are used to process real-world UHD low-light images. This is because (1) most methods (Guo et al., 2020; Liu et al., 2021b; Ma et al., 2022) only focus on luminance enhancement and fail in removing noise; (2) some approaches (Wu et al., 2022; Xu et al., 2022) simultaneously enhance luminance and remove noise in the spatial domain, resulting in the suboptimal enhancement; and (3) existing methods (Wei et al., 2018; Zhao et al., 2021; Wu et al., 2022; Xu et al., 2022) are mainly trained on low-resolution (LR) data, leading to the incompatibility with high-resolution (HR) inputs; and (4) some studies (Xu et al., 2022; Zamir et al., 2022) adopt heavy structures, thus being inefficient for processing UHD images. More discussion on related work is provided in the Appendix. To overcome the challenges aforementioned, we present a new idea for performing LLIE in the Fourier Domain. Our approach differs significantly from existing solutions that process images in the spatial domain. In particular, our method, named as UHDFour, is motivated by our observation of two interesting phenomena in the Fourier domain of low-light noisy images: i) luminance and noise can be decomposed to a certain extent in the Fourier domain. Specifically, luminance would manifest as amplitude while noise is closely related to phase, and ii) the amplitude patterns of images of different resolutions are similar. These observations inspire the design of our network, which handles luminance and noise separately in the Fourier domain. This design is advantageous as it avoids amplifying noise when enhancing luminance, a common issue encountered in existing spatial domain-based methods. In addition, the fact that amplitude patterns of images of different resolutions are similar motivates us to save computation by first processing in the low-resolution regime and performing essential adjustments only in the high-resolution scale. We also contribute the first benchmark for UHD LLIE. The dataset, named UHD-LL, contains 2,150 low-noise/normal-clear 4K UHD image pairs with diverse darkness and noise levels captured in different scenarios. Unlike existing datasets (Wei et al., 2018; Lv et al., 2021; Bychkovsky et al., 2011) that either synthesize or retouch low-light images to obtain the paired input and target sets, we capture real image pairs. During data acquisition, special care is implemented to minimize geometric and photometric misalignment due to camera shake and dynamic environment. With the new UHD-LL dataset, we design a series of quantitative and quantitative benchmarks to analyze the performance of existing LLIE methods and demonstrate the effectiveness of our method. Our contributions are summarized as follows: (1) We propose a new solution for UHD LLIE that is inspired by unique characteristics observed in the Fourier domain. In comparison to existing LLIE methods, the proposed framework shows exceptional effectiveness and efficiency in addressing the joint task of luminance enhancement and noise removal in the UHD regime. (2) We contribute the first UHD LLIE dataset, which contains 2,150 pairs of 4K UHD low-noise/normal-clear data, covering diverse noise and darkness levels and scenes. (3) We conduct a systematical analysis of existing LLIE methods on UHD data.

2. OUR APPROACH

In this section, we first discuss our observations in analyzing low-light images in the Fourier domain, and then present the proposed solution.

2.1. OBSERVATIONS IN FOURIER DOMAIN

Here we provide more details to supplement the observations we highlighted in Sec. 1. We analyze real UHD low-light images in the Fourier domain and provide a concise illustration in Figure 2 . Specifically, (a) Swapping the amplitude of a low-light and noisy (low-noise) image with that of its corresponding normal-light and clear (normal-clear) image produces a normal-light and noisy (normal-noise) image and a low-light and clear (low-clear) image. We show more examples in the Appendix. The result suggests that the luminance and noise can be decomposed to a certain extent in the Fourier domain. In particular, most luminance information is expressed as amplitudes, and



Figure 1: Visual results of state of the arts (Zhao et al. (Zhao et al., 2021), URetinex-Net (Wu et al., 2022), and SNR-Aware (Xu et al., 2022)) pre-trained on an existing low-light image dataset for processing the realworld UHD low-light images. We amplify the brightness of the input UHD low-light images 10 times (top right corner of the first column) to show details and noise. These officially released models were trained using existing paired LR images with mild noise (i.e., the LOL dataset (Wei et al., 2018)). Existing models cannot cope with challenging UHD low-light images well.

