AN OPTIMAL TRANSPORT PERSPECTIVE ON UNPAIRED IMAGE SUPER-RESOLUTION Anonymous

Abstract

Real-world image super-resolution (SR) tasks often do not have paired datasets, which limits the application of supervised techniques. As a result, the tasks are usually approached by unpaired techniques based on Generative Adversarial Networks (GANs), which yield complex training losses with several regularization terms, e.g., content or identity losses. We theoretically investigate optimization problems which arise in such models and find two surprizing observations. First, the learned SR map is always an optimal transport (OT) map. Second, we theoretically prove and empirically show that the learned map is biased, i.e., it does not actually transform the distribution of low-resolution images to high-resolution ones. Inspired by these findings, we propose an algorithm for unpaired SR which learns an unbiased OT map for the perceptual transport cost. Unlike the existing GAN-based alternatives, our algorithm has a simple optimization objective reducing the need for complex hyperparameter selection and an application of additional regularizations. At the same time, it provides a nearly state-of-the-art performance on the large-scale unpaired AIM19 dataset.

1. INTRODUCTION

The problem of image super-resolution (SR) is to reconstruct a high-resolution (HR) image from its low-resolution (LR) counterpart. In many modern deep learning approaches, SR networks are trained in a supervised manner by using synthetic datasets containing LR-HR pairs (Lim et al., 2017, 4.1); (Zhang et al., 2018b, 4.1) . For example, it is common to create LR images from HR with a simple downscaling, e.g., bicubic (Ledig et al., 2017, 3.2) . However, such an artificial setup barely represents the practical setting, in which the degradation is more sophisticated and unknown (Maeda, 2020) . This obstacle suggests the necessity of developing methods capable of learning SR maps from unpaired data without considering prescribed degradations. Contributions. We study the unpaired image SR task and its solutions based on Generative Adversarial Networks (Goodfellow et al., 2014, GANs) and analyse them from the Optimal Transport (OT, see (Villani, 2008)) perspective. 1. We investigate the GAN optimization objectives regularized with content losses, which are common in unpaired image SR methods ( 5, 4). We prove that the solution to such objectives is always an optimal transport map. We theoretically and empirically show that such maps are biased ( 7.1), i.e., they do not transform the LR image distribution to the true HR image distribution. 2. We provide an algorithm to fit an unbiased OT map for perceptual transport cost ( 6.1) and apply it to the unpaired image SR problem ( 7.2). We establish connections between our algorithm and regularized GANs using integral probability metrics (IPMs) as a loss ( 6.2). Our algorithm solves a minimax optimization objective and does not require extensive hyperparameter search, which makes it different from the existing methods for unpaired image SR. At the same time, the algorithm provides a nearly state-of-art performance in the unpaired image SR problem ( 7.2).



Figure 1: Super-resolution of a squirrel using Bicubic upsample, OTS (ours) and DASR (Wei et al., 2021) methods (4×4 upsample, 370×800 crops).

