MINIMUM CURVATURE MANIFOLD LEARNING

Abstract

It is widely observed that vanilla autoencoders can have low manifold learning accuracy given a noisy or small training dataset. Recent work has discovered that it is important to regularize the decoder that explicitly parameterizes the manifold, where a neighborhood graph is employed for decoder regularization. However, one caveat of this method is that it is not always straightforward to construct a correct graph. Alternatively, one may consider naive graph-free regularization methods such as minimizing the norm of the decoder's Jacobian or Hessian, but these norms are not coordinate-invariant (i.e. reparametrization-invariant) and hence do not capture any meaningful geometric quantity of the manifold nor result in geometrically meaningful manifold regularization effects. Another recent work called the isometric regularization implicitly forces the manifold to have zero intrinsic curvature, resulting in some geometrically meaningful regularization effects. But, since the intrinsic curvature does not capture how the manifold is embedded in the data space from an extrinsic perspective, the regularization effects are often limited. In this paper, we propose a minimum extrinsic curvature principle for manifold regularization and Minimum Curvature Autoencoder (MCAE), a graphfree coordinate-invariant extrinsic curvature minimization framework for autoencoder regularization. Experiments with various standard datasets demonstrate that MCAE improves manifold learning accuracy compared to existing methods, especially showing strong robustness to noise.

1. INTRODUCTION

Autoencoders are widely used to identify, given a set of high-dimensional data, the underlying lowerdimensional manifold structure and its coordinate space, simultaneously (Kramer, 1991) . The decoder explicitly parameterizes the data manifold as a mapping from a lower-dimensional coordinate space (i.e., latent space) to the high-dimensional data space, and the encoder maps data points to their corresponding coordinates (i.e., latent values). However, vanilla autoencoders trained to reconstruct the given training data often learn manifolds that severely overfit to noisy training data or are wrong in regions where there are fewer data, impairing their manifold learning performances. It has been recently discovered by Lee et al. ( 2021) that autoencoder regularization methods that focus on regularizing the latent space distributions determined entirely by the encoders (Kingma & Welling, 2013; Tolstikhin et al., 2018; Makhzani et al., 2015; Rifai et al., 2011) are not sufficient to learn correct manifolds, yet it is important to properly regularize the decoders that parameterize the manifolds. In (Lee et al., 2021) , neighborhood graphs constructed from data are successfully utilized to regularize the local geometry and connectivity of the manifold, significantly improving the manifold learning accuracy. However, the underlying premise behind this method is that the graph has to be accurate, yet constructing a correct graph may not be always straightforward. There are some graph-free methods such as the denoising autoencoder (Vincent et al., 2010) and reconstruction contractive autoencoder (Alain & Bengio, 2014) that regularize not only an encoder but also a decoder. They can learn manifolds that are robust to noise to some extent, but when the noise level is large, the performance is often less-than-desirable, and they do not always produce correct manifolds, especially in regions where there are fewer data (discussed in more detail in Section 4.2). Since the decoder needs to be regularized, one may come up with some naive regularization strategies such as minimizing the norm of the decoder's Jacobian or Hessian, considering them as mea-

