MINIMUM CURVATURE MANIFOLD LEARNING

Abstract

It is widely observed that vanilla autoencoders can have low manifold learning accuracy given a noisy or small training dataset. Recent work has discovered that it is important to regularize the decoder that explicitly parameterizes the manifold, where a neighborhood graph is employed for decoder regularization. However, one caveat of this method is that it is not always straightforward to construct a correct graph. Alternatively, one may consider naive graph-free regularization methods such as minimizing the norm of the decoder's Jacobian or Hessian, but these norms are not coordinate-invariant (i.e. reparametrization-invariant) and hence do not capture any meaningful geometric quantity of the manifold nor result in geometrically meaningful manifold regularization effects. Another recent work called the isometric regularization implicitly forces the manifold to have zero intrinsic curvature, resulting in some geometrically meaningful regularization effects. But, since the intrinsic curvature does not capture how the manifold is embedded in the data space from an extrinsic perspective, the regularization effects are often limited. In this paper, we propose a minimum extrinsic curvature principle for manifold regularization and Minimum Curvature Autoencoder (MCAE), a graphfree coordinate-invariant extrinsic curvature minimization framework for autoencoder regularization. Experiments with various standard datasets demonstrate that MCAE improves manifold learning accuracy compared to existing methods, especially showing strong robustness to noise.

1. INTRODUCTION

Autoencoders are widely used to identify, given a set of high-dimensional data, the underlying lowerdimensional manifold structure and its coordinate space, simultaneously (Kramer, 1991) . The decoder explicitly parameterizes the data manifold as a mapping from a lower-dimensional coordinate space (i.e., latent space) to the high-dimensional data space, and the encoder maps data points to their corresponding coordinates (i.e., latent values). However, vanilla autoencoders trained to reconstruct the given training data often learn manifolds that severely overfit to noisy training data or are wrong in regions where there are fewer data, impairing their manifold learning performances. Lee et al. (2021) that autoencoder regularization methods that focus on regularizing the latent space distributions determined entirely by the encoders (Kingma & Welling, 2013; Tolstikhin et al., 2018; Makhzani et al., 2015; Rifai et al., 2011) are not sufficient to learn correct manifolds, yet it is important to properly regularize the decoders that parameterize the manifolds. In (Lee et al., 2021) , neighborhood graphs constructed from data are successfully utilized to regularize the local geometry and connectivity of the manifold, significantly improving the manifold learning accuracy. However, the underlying premise behind this method is that the graph has to be accurate, yet constructing a correct graph may not be always straightforward.

It has been recently discovered by

There are some graph-free methods such as the denoising autoencoder (Vincent et al., 2010) and reconstruction contractive autoencoder (Alain & Bengio, 2014) that regularize not only an encoder but also a decoder. They can learn manifolds that are robust to noise to some extent, but when the noise level is large, the performance is often less-than-desirable, and they do not always produce correct manifolds, especially in regions where there are fewer data (discussed in more detail in Section 4.2). Since the decoder needs to be regularized, one may come up with some naive regularization strategies such as minimizing the norm of the decoder's Jacobian or Hessian, considering them as mea-Figure 1 : Left: Two decoders f and f ′ parameterize the same data manifold where the norm of Jacobian of f ′ is smaller than that of f , i.e., ∥J f ∥ > ∥J f ′ ∥. Right: A curve and developable surface embedded in R 3 have zero intrinsic curvatures. sures of the manifold's smoothness. However, these norms do not properly capture any geometric quantity of the manifold because they are not reparametrization-invariant (or coordinate-invariant). As shown in Figure 1 (Left), just by increasing the volume of the latent space without actually changing the manifold, i.e., re-parametrizing the manifold f → f ′ , the above norms can be minimized. Just recently, a coordinate-invariant geometric distortion measure has been introduced to regularize the decoder to be a geometry-preserving mapping, which is called the isometric regularization (LEE et al., 2022) , so that the data space geometry is preserved in the latent space. Minimizing this distortion measure implicitly forces the learned manifold to have zero intrinsic curvature -which only depends on distances measured within the manifold (e.g., a cylinder's side surface has zero intrinsic curvature unlike the spherical surface) -, resulting in some geometrically meaningful manifold regularization effects. The intrinsic curvature, however, does not capture how the manifold lies in the data spacefoot_0 , and thus minimizing the manifold's intrinsic curvature may not be enough to learn correct manifolds. For example, curves and developable surfacesfoot_1 in R 3 always have zero intrinsic curvatures, e.g., Figure 1 (Right) , regardless of how severely they are curved from an extrinsic point of view (Do Carmo, 2016) . The main contribution of this paper is a coordinate-invariant extrinsic curvature minimization framework for autoencoder regularization, which we refer to a Minimum Curvature Autoencoder (MCAE), that is graph-free and effectively improves the manifold learning accuracy given a noisy or small training dataset. Specifically, we develop a coordinate-invariant extrinsic curvature measure of the learned manifold, by investigating how smoothly tangent space changes on the manifold, and use it as a regularization term. To make things more explicit, let M be a manifold of dimension m embedded in R D . Consider a mapping T that maps a point x in M to its tangent space T x M, a linear subspace that has the dimension of m attached at x, i.e., T (x) = T x M. The set of all linear subspaces of dimension m in R D forms a manifold called the Grassmann manifold denoted by Gr(m, R D ) (Bendokat et al., 2020) , and thus the mapping T can be viewed as a mapping between two Riemannian manifolds, i.e., T : M → Gr(m, R D ). By using the Dirichlet energy (Eells & Lemaire, 1978) , a natural smoothness measure of mappings between two Riemannian manifolds defined in a coordinate-invariant way, we formulate an extrinsic curvature measure. We also propose a practical estimation strategy of the curvature measure that can be used for high-dimensional problems, reducing computation costs. Experiments on diverse image and motion capture data confirm that, compared to existing graphfree regularized autoencoders, our MCAE improves manifold learning accuracy for noisy and small training datasets. In particular, our experiments show that even compared to the methods specially designed to be robust to input perturbations such as the DAE (Vincent et al., 2010) and RCAE (Alain & Bengio, 2014) , the MCAE shows comparable or even in some cases significantly higher robust manifold learning performance.



Manifold's intrinsic properties are defined without involving any embedding. A developable surface can be formed by bending or rolling a planar surface without stretching or tearing.

