SPECTRAL SYNTHESIS FOR SATELLITE-TO-SATELLITE TRANSLATION

Abstract

Earth observing satellites carrying multi-spectral sensors are widely used to monitor the physical and biological states of the atmosphere, land, and oceans. These satellites have different vantage points above the Earth and different spectral imaging bands resulting in inconsistent imagery from one to another. This presents challenges in building downstream applications. What if we could generate synthetic bands for existing satellites from the union of all domains? We tackle the problem of generating synthetic spectral imagery for multispectral sensors as an unsupervised image-to-image translation problem with partial labels and introduce a novel shared spectral reconstruction loss. Simulated experiments performed by dropping one or more spectral bands show that cross-domain reconstruction outperforms measurements obtained from a second vantage point. On a downstream cloud detection task, we show that generating synthetic bands with our model improves segmentation performance beyond our baseline. Our proposed approach enables synchronization of multispectral data and provides a basis for more homogeneous remote sensing datasets.

1. INTRODUCTION

Climate change and related environmental issues -including the loss of biodiversity and extreme weather -are listed by the World Economic Forum as the most important risks to our planet (7) . Monitoring the Earth is critical to mitigating these risks, understanding the effects, and making future predictions (38) . Multi-and hyper-spectral satellite-based remote sensing enables global observation of the Earth, allowing scientists to study large-scale system dynamics and inform general circulation models (26). In weather forecasts satellite data initializes the atmospheric state for future predictions. On longer time scales, these data are used to measure the effects of climate change such as land-cover variations, temperature trends, solar radiation levels, and the rate of snow/ice melt. In the coming decades, increased investments from the public and private sectors in satellite-based observations will continue to improve global monitoring, as highlighted in NASA's decadal survey (25). Satellites are designed based on specifications for a given set of applications with fiscal, technological, and physical constraints which limit their temporal, spatial, and spectral resolutions. Geostationary (GEO) satellites rotate with the Earth to stay over a constant position above the equator at a high elevation of 35,786km. This position enables GEO satellites with on-board multi-spectral imagers to take continuous and high-temporal snapshots over large spatial regions and are ideal for monitoring diurnal and fast moving events. Spectral bands measure brightness and radiance intensities of the electromagnetic spectrum at a specified center wavelength and bandwidth. Bands are selected to satisfy defined variables of interest constrained by technological cost and accuracy. Applications of GEO sensors include atmospheric winds measurement (35), tropical cyclone tracking (36), wildfire monitoring (41), and short-term forecasting (24). Multiple GEO satellites are needed to generate global high-temporal resolution datasets to better monitor these events around the world. However, variations in resolutions, sensor uncertainties, and temporal life spans leads to a set of separate datasets which are not consistent, making this process very challenging (26). Developing consistent and homogeneous global datasets would relieve many of these challenges. 29). The Himawari-8 satellite operated by the Japanese Space Agency (cost: $800 million) similarly has 16 bands but swaps a NIR (1.38µm) band for a green channel (0.51µm), enabling the construction of true color images (3). The 1.38µm band is ideal for measuring Cirrus clouds, composed of ice particles in the upper troposphere, a major contributor to regulating the Earth's climate that is not yet well understood (21; 10). Without capturing this band, directly observing Cirrus clouds over Japan, East Asia, and Western Pacific region from Himawari-8 is not possible. Synthetic observations via virtual spectral sensors could be a low-cost solution to improving coverage availability and consistency with current satellites. We present an approach to generate synthetic spectral channels from a multi-domain unpaired satellite dataset. We treat satellites with either dissimilar spectral coverage or varying vantage points as separate spectral sets. In this way, the problem closely resembles that of colorization (40) and image-to-image translation tasks (22; 42; 9) in the case where paired images are not available but with the added complexity of a large number of spectral bands. We use a combination of variational autoencoder (VAE) and generative adverserial network (GAN) (8) architectures adapted to our problem to model a shared latent space, as in unsupervised image-to-image translation(22). Generating synthetic bands is an under-constrained problem that paired with an adverserial loss in high dimensions promotes overfitting. Our approach mitigates these challenges by leveraging a weak supervision signal based on partial overlap in spectral bands between domains. By including a reconstruction loss on overlapping spectral bands between domain pairs we can substantially improve spectral band synthesis. To summarize our contributions, we 1) introduce a shared spectral reconstruction loss to a VAE-GAN architecture for synthetic band generation; 2) test our methodology on real-world scenarios; 3) present and release a test dataset of four hemispheric snapshots from three publicly available geostationary satellites for future research. In the following sections, we will introduce related work in remote sensing and image-to-image translation, describe the architecture, and review experiments. Lastly, we will discuss the implications on this work and conclude with future directions.

2. BACKGROUND

Remote Sensing. Current generation GEO satellites observe 16 spectral bands over large regions every 10-15 minutes at a 0.5-2km resolution. At a sub-optimal 2km, this produces full-disk images of size 5,424×5,424×16 which causes storage constraints while being computationally expensive to process. Physical and statistical models are used to convert these images into more easily interpreted variables such as precipitation, cloud cover, and surface temperature (30). Multiple GEO satellites, currently in orbit, extend the spatial ranges to actively monitoring larger regions. However, differences



Figure 1: (Left) Network architecture for K = 3 satellites. Encoders (E k ), decoders (G k ), and discriminators (D k ) are networks with residual blocks. Losses terms are highlighted in red. (Right) Venn diagram shows how spectral bands can overlap between pairs and multiple satellites. The current generation of GEO satellites are no exception. The GOES-16/17 satellites operated by NASA/NOAA (cost: $11 billion) have a set of 16 imaging bands covering the visible, near-, and thermal-infrared spectral range (29). The Himawari-8 satellite operated by the Japanese Space Agency (cost: $800 million) similarly has 16 bands but swaps a NIR (1.38µm) band for a green channel (0.51µm), enabling the construction of true color images (3). The 1.38µm band is ideal for measuring Cirrus clouds, composed of ice particles in the upper troposphere, a major contributor to regulating the Earth's climate that is not yet well understood (21; 10). Without capturing this band, directly observing Cirrus clouds over Japan, East Asia, and Western Pacific region from Himawari-8 is not possible. Synthetic observations via virtual spectral sensors could be a low-cost solution to improving coverage availability and consistency with current satellites.

Supplementary

Material: https://github.com/anonymous-ai-for-earth/ satellite-to-satellite-translation 

