ISOMETRIC AUTOENCODERS

Abstract

High dimensional data is often assumed to be concentrated on or near a lowdimensional manifold. Autoencoders (AE) is a popular technique to learn representations of such data by pushing it through a neural network with a low dimension bottleneck while minimizing a reconstruction error. Using high capacity AE often leads to a large collection of minimizers, many of which represent a low dimensional manifold that fits the data well but generalizes poorly. Two sources of bad generalization are: extrinsic, where the learned manifold possesses extraneous parts that are far from the data; and intrinsic, where the encoder and decoder introduce arbitrary distortion in the low dimensional parameterization. An approach taken to alleviate these issues is to add a regularizer that favors a particular solution; common regularizers promote sparsity, small derivatives, or robustness to noise. In this paper, we advocate an isometry (i.e., local distance preserving) regularizer. Specifically, our regularizer encourages: (i) the decoder to be an isometry; and (ii) the encoder to be the decoder's pseudo-inverse, that is, the encoder extends the inverse of the decoder to the ambient space by orthogonal projection. In a nutshell, (i) and (ii) fix both intrinsic and extrinsic degrees of freedom and provide a non-linear generalization to principal component analysis (PCA). Experimenting with the isometry regularizer on dimensionality reduction tasks produces useful low-dimensional data representations.

1. INTRODUCTION

A common assumption is that high dimensional data X ⊂ R D is sampled from some distribution p concentrated on, or near, some lower d-dimensional submanifold M ⊂ R D , where d < D. The task of estimating p can therefore be decomposed into: (i) approximate the manifold M; and (ii) approximate p restricted to, or concentrated near M. In this paper we focus on task (i), mostly known as manifold learning. A common approach to approximate the d-dimensional manifold M, e.g., in (Tenenbaum et al., 2000; Roweis & Saul, 2000; Belkin & Niyogi, 2002; Maaten & Hinton, 2008; McQueen et al., 2016; McInnes et al., 2018) , is to embed X in R d . This is often done by first constructing a graph G where nearby samples in X are conngected by edges, and second, optimizing for the locations of the samples in R d striving to minimize edge length distortions in G. Autoencoders (AE) can also be seen as a method to learn low dimensional manifold representation of high dimensional data X . AE are designed to reconstruct X as the image of its low dimensional embedding. When restricting AE to linear encoders and decoders it learns linear subspaces; with mean squared reconstruction loss they reproduce principle component analysis (PCA). Using higher capacity neural networks as the encoder and decoder, allows complex manifolds to be approximated. To avoid overfitting, different regularizers are added to the AE loss. Popular regularizers include sparsity promoting (Ranzato et al., 2007; 2008; Glorot et al., 2011) , contractive or penalizing large derivatives (Rifai et al., 2011a; b), and denoising (Vincent et al., 2010; Poole et al., 2014) . Recent AE regularizers directly promote distance preservation of the encoder (Pai et al., 2019; Peterfreund et al., 2020) . In this paper we advocate a novel AE regularization promoting isometry (i.e., local distance preservation), called Isometric-AE (I-AE). Our key idea is to promote the decoder to be isometric, and the encoder to be its pseudo-inverse. Given an isometric decoder R d → R D , there is no well-defined

