REAL-TIME IMAGE DEMOIR ÉING ON MOBILE DE-VICES

Abstract

Moiré patterns appear frequently when taking photos of digital screens, drastically degrading the image quality. Despite the advance of CNNs in image demoiréing, existing networks are with heavy design, causing redundant computation burden for mobile devices. In this paper, we launch the first study on accelerating demoiréing networks and propose a dynamic demoiréing acceleration method (DDA) towards a real-time deployment on mobile devices. Our stimulus stems from a simple-yet-universal fact that moiré patterns often unbalancedly distribute across an image. Consequently, excessive computation is wasted upon non-moiré areas. Therefore, we reallocate computation costs in proportion to the complexity of image patches. In order to achieve this aim, we measure the complexity of an image patch by designing a novel moiré prior that considers both colorfulness and frequency information of moiré patterns. Then, we restore image patches with higher-complexity using larger networks and the ones with lower-complexity are assigned with smaller networks to relieve the computation burden. At last, we train all networks in a parameter-shared supernet paradigm to avoid additional parameter burden. Extensive experiments on several benchmarks demonstrate the efficacy of our proposed DDA. In addition, the acceleration evaluated on the VIVO X80 Pro smartphone equipped with a chip of Snapdragon 8 Gen 1 shows that our method can drastically reduce the inference time, leading to a real-time image demoiréing on mobile devices.

1. INTRODUCTION

Moiré patterns (Sun et al., 2018; Yang et al., 2017b) describe an artifact of images that in particular appears in television and digital photography. In contemporary society, using mobile phones to take screen pictures has become one of the most productive ways to record information quickly. Nevertheless, moiré patterns occur frequently from the interference between the color filter array (CFA) of camera and high-frequency repetitive signal. The resulting stripes of different colors and frequencies on the captured photos drastically degrade the visual quality. Therefore, developing demoiréing algorithms has received long-time attention in the research community, and yet remains unsolved in particular when running algorithms on mobile devices. Primitive studies on image demoiréing resort to traditional machine learning techniques such as low-rank and sparse matrix decomposition (Liu et al., 2015) and bandpass filters (Yang et al., 2017a) . The rising of convolutional neural networks (CNNs) has vastly boosted the efficacy of image demoiréing (He et al., 2019; Zheng et al., 2020) . However, the improved quantitative performance, such as PSNR (Peak Signal-to-Noise Ratio), comes at the increasing costs of energy and computation. For example, MBCNN (Zheng et al., 2020) eats up 4.22T floating-point operations (FLOPs) in order to restore a 1920×1080 smartphone-taken moiré image. Given that the moiré patterns mostly emerge in mobile photography, such massive computations carry considerable inference latency, preventing a real-time demoiréing experience from the users. Such a handicap could be more serious when it comes to video demoiréing. Therefore, it is of great need to bridge the technology gap between academia and industry. To tackle the aforementioned issue, we initiate the first study on accelerating demoiréing networks towards a real-time deployment on mobile devices. Our motivation arises from empirical actuality in moiré images. As shown in Fig. 1 , an image is often partially contaminated by the moiré pattern. Some areas are filled with intensive moiré stripes, some are much relieved while some are kept away from moiré pollution. It is natural to hand out computation to these moiré centralized areas but less to these diluted areas. In the extreme case, it is needless to cleanse uninfluenced areas. Unfortunately, existing methods (Sun et al., 2018; Zheng et al., 2020) have not distinguished the treatment to the different areas in an image. They not only waste excessive computation on non-moiré areas but also bring about side effects, such as over-whitened image contents. Therefore, reallocating computation costs in compliance with the complexity of a moiré area can be a potential solution to accomplish real-time image demoiréing on mobile devices. Stimulated by the above analysis, we opt to split a whole image into several sub-image patches. To measure the patch moiré complexity, we introduce a novel moiré prior. As can be referred in Fig. 1 , moiré patterns are featured with either high frequency or rich color information. Thus, we define the moiré prior as the product of frequency and color information in a patch. In detail, we model the frequency information by a Gaussian filter and the colorfulness metric is a linear combination of the mean and standard deviation of the pixel cloud in the RGB colour space (Hasler & Suesstrunk, 2003) . Using this prior to measure the moiré complexity, each image patch is then processed by a unique network with its computation costs in proportion to the moiré complexity. In this fashion, larger networks are utilized to restore moiré centralized areas to ensure the recovery quality while smaller networks are leveraged to restore moiré diluted areas to relieve computation burden. Thus, we have a better tradeoff between the image quality and resource requirements on mobile devices. Nevertheless, multiple networks lead to more parameter burden, which also causes deployment pressure due to the short-supply memory on mobile devices. To mitigate this issue, we leverage the supernet paradigm (Yang et al., 2021) to jointly train all networks in a parameter-shared manner. Concretely, we regard the vanilla demoiréing network as a supernet, and weight-shared subnets of different sizes are directly extracted from this supernet to process image patches of different demoiré complexity. During the training phase, each subnet is dynamically trained using corresponding image patches with similar moiré complexity. Consequently, the overall running overhead can be effectively reduced without introduction of any additional parameters. We have conducted extensive experiments for accelerating existing demoiréing networks on the LCDMoiré (Yuan et al., 2019) and FHDMi (He et al., 2020) benchmarks. The results show that our dynamic demoiréing acceleration method, termed DDA, achieves an obvious FLOPs reduction even with PSNR and SSIM increases. For instance, DDA reduces 45.2% FLOPs of the state-of-the-art demoiréing network MBCNN (Zheng et al., 2020) with 0.35 dB PNSR increase. Furthermore, the



Figure 1: Images with moiré patterns. The blue boxes show a blue zoomed in area. Two phenomena can be observed: (1) The moiré complexity varies significantly across different areas of an image. (2) Moiré patterns are mainly characterized by frequency and colorfulness.

availability

//github.com/zyxxmu/DDA.

