REAL-TIME IMAGE DEMOIR ÉING ON MOBILE DE-VICES

Abstract

Moiré patterns appear frequently when taking photos of digital screens, drastically degrading the image quality. Despite the advance of CNNs in image demoiréing, existing networks are with heavy design, causing redundant computation burden for mobile devices. In this paper, we launch the first study on accelerating demoiréing networks and propose a dynamic demoiréing acceleration method (DDA) towards a real-time deployment on mobile devices. Our stimulus stems from a simple-yet-universal fact that moiré patterns often unbalancedly distribute across an image. Consequently, excessive computation is wasted upon non-moiré areas. Therefore, we reallocate computation costs in proportion to the complexity of image patches. In order to achieve this aim, we measure the complexity of an image patch by designing a novel moiré prior that considers both colorfulness and frequency information of moiré patterns. Then, we restore image patches with higher-complexity using larger networks and the ones with lower-complexity are assigned with smaller networks to relieve the computation burden. At last, we train all networks in a parameter-shared supernet paradigm to avoid additional parameter burden. Extensive experiments on several benchmarks demonstrate the efficacy of our proposed DDA. In addition, the acceleration evaluated on the VIVO X80 Pro smartphone equipped with a chip of Snapdragon 8 Gen 1 shows that our method can drastically reduce the inference time, leading to a real-time image demoiréing on mobile devices.

1. INTRODUCTION

Moiré patterns (Sun et al., 2018; Yang et al., 2017b) describe an artifact of images that in particular appears in television and digital photography. In contemporary society, using mobile phones to take screen pictures has become one of the most productive ways to record information quickly. Nevertheless, moiré patterns occur frequently from the interference between the color filter array (CFA) of camera and high-frequency repetitive signal. The resulting stripes of different colors and frequencies on the captured photos drastically degrade the visual quality. Therefore, developing demoiréing algorithms has received long-time attention in the research community, and yet remains unsolved in particular when running algorithms on mobile devices. Primitive studies on image demoiréing resort to traditional machine learning techniques such as low-rank and sparse matrix decomposition (Liu et al., 2015) and bandpass filters (Yang et al., 2017a) . The rising of convolutional neural networks (CNNs) has vastly boosted the efficacy of image demoiréing (He et al., 2019; Zheng et al., 2020) . However, the improved quantitative performance, such as PSNR (Peak Signal-to-Noise Ratio), comes at the increasing costs of energy and computation. For example, MBCNN (Zheng et al., 2020) eats up 4.22T floating-point operations (FLOPs) in order to restore a 1920×1080 smartphone-taken moiré image. Given that the moiré patterns mostly emerge in mobile photography, such massive computations carry considerable inference latency, preventing a real-time demoiréing experience from the users. Such a handicap could

availability

//github.com/zyxxmu/DDA.

