INVERTIBLE MANIFOLD LEARNING FOR DIMENSION REDUCTION Anonymous

Abstract

It is widely believed that a dimension reduction (DR) process drops information inevitably in most practical scenarios. Thus, most methods try to preserve some essential information of data after DR, as well as manifold based DR methods. However, they usually fail to yield satisfying results, especially in high-dimensional cases. In the context of manifold learning, we think that a good low-dimensional representation should preserve the topological and geometric properties of data manifolds, which involve exactly the entire information of the data manifolds. In this paper, we define the problem of information-lossless NLDR with the manifold assumption and propose a novel two-stage NLDR method, called invertible manifold learning (inv-ML), to tackle this problem. A local isometry constraint of preserving local geometry is applied under this assumption in inv-ML. Firstly, a homeomorphic sparse coordinate transformation is learned to find the lowdimensional representation without losing topological information. Secondly, a linear compression is performed on the learned sparse coding, with the trade-off between the target dimension and the incurred information loss. Experiments are conducted on seven datasets with a neural network implementation of inv-ML, called i-ML-Enc, which demonstrate that the proposed inv-ML not only achieves invertible NLDR in comparison with typical existing methods but also reveals the characteristics of the learned manifolds through linear interpolation in latent space. Moreover, we find that the reliability of tangent space approximated by the local neighborhood on real-world datasets is key to the success of manifold based DR algorithms. The code will be made available soon.

1. INTRODUCTION

In real-world scenarios, it is widely believed that the loss of data information is inevitable after dimension reduction (DR), though the goal of DR is to preserve as much information as possible in the low-dimensional space. In the case of linear DR, compressed sensing (Donoho, 2006) breaks this common sense with practical sparse conditions of the given data. In the case of nonlinear dimension reduction (NLDR), however, it has not been clearly discussed, e.g. what is the structure within data and how to maintain these structures after NLDR? From the perspective of manifold learning, the manifold assumption is widely adopted, but classical manifold based DR methods usually fail to yield good results in the many practical case. Therefore, what is the gap between theoretical and real-world applications of manifold based DR? Here, we give the first detailed discussion of these two problems in the context of manifold learning. We think that a good low-dimensional representation should preserve the topology and geometry of input data, which require the NLDR transformation to be homeomorphic. Thus, we propose an invertible NLDR process, called inv-ML, combining sparse coordinate transformation and local isometry constraint which preserve the property of topology and geometry, to explain the information-lossless NLDR in manifold learning theoretically. We instantiate inv-ML as a neural network called i-ML-Enc via a cascade of equidimensional layers and a linear transform layer. Sufficient experiments are conduct to validate invertible NLDR abilities of i-ML-Enc and analyze learned representations to reveal inherent difficulties of classical manifold learning. Topology preserving dimension reduction. To start, we first make out the theoretical definition of information-lossless DR on a manifold. The topological property is what is invariant under a homeomorphism, and thus what we want to achieve is to construct a homeomorphism for dimension reduction, removing the redundant dimensions while preserving invariant topology. To be more specific, f : M d 0 → R m is a smooth mapping of a differential manifold into another, and if f is a homeomorphism of M d 0 into M d 1 = f (M d 0 ) ⊂ R m , we call f is an embedding of M d 0 into R m . Assume that the data set X = {x j |1 ≤ j ≤ n} sampled from the compact manifold M d 1 ⊂ R m which we call the data manifold and is homeomorphic to M d 0 . For the sample points we get are represented in the coordinate after inclusion mapping i 1 , we can only regard them as points from Euclidean space R m without any prior knowledge, and learn to approximate the data manifold in the latent space Z. According to the Whitney Embedding Theorem (Seshadri & Verma, 2016), M d 0 is can be embedded smoothly into R 2d by a homeomorphism g. Rather than to find the f -1 : M d 1 → M d 0 , our goal is to seek a smooth map h : M d 1 → R s ⊂ R 2d , where h = g • f -1 is a homeomorphism of M d 1 into M d 2 = h(M d 1 ) and d ≤ s ≤ 2d m, and thus the dim(h(X )) = s, which achieves the DR while preserving the topology. Owing to the homeomorphism h we seek as a DR mapping, the data manifold M d 1 is reconstructible via M d 1 = h -1 • h(M d 1 ) , by which we mean h a topology preserving DR as well as information-lossless DR. 1 , and it is represented in the Euclidean space R m after an inclusion mapping i i . We aim to approximate M d 1 from the observed sample x. For the topology preserving dimension reduction methods, it aims to find a homeomorphism g • f -1 to map x into z which is embedded in R s . Geometry preserving dimension reduction. While the topology of the data manifold M d 1 can be preserved by the homeomorphism h discussed above, it may distort the geometry. To preserve the local geometry of the data manifold, the map should be isometric on the tangent space T p M d 1 for every p ∈ M d 1 , indicating that d M d 1 (u, v) = d M d 2 (h(u), h(v)), ∀u, v ∈ T p M d 1 . By Nash's Embedding Theorem (Nash, 1956) , any smooth manifold of class C k with k ≥ 3 and dimension d can be embedded isometrically in the Euclidean space R s with s polynomial in d. Noise perturbation. In the real-world scenarios, sample points are not lied on the ideal manifold strictly due to the limitation of sampling, e.g. non-uniform sampling noises. When the DR method is very robust to the noise, it is reasonable to ignore the effects of the noise and learn the representation Z from the given data. Therefore, the intrinsic dimension of X is approximate to d, resulting in the lowest isometric embedding dimension is larger than s.

2. RELATED WORK

Manifold learning. Most classical linear or nonlinear DR methods aim to preserve the geometric properties of manifolds. The Isomap (Tenenbaum et al., 2000) based methods aim to preserve the global metric between every pair of sample points. For example, McQueen et al. (2016) can be regarded as such methods based on the push-forward Riemannian metric. For the other aspect, LLE (Roweis & Saul, 2000) based methods try to preserve local geometry after DR, whose derivatives like LTSA (Zhang & Zha, 2004) , MLLE (Zhang & Wang, 2007) , etc. have been widely used but usually fail in the high-dimensional case. Recently, based on local properties of manifolds, MLDL (Li et al., 2020) was proposed as a robust NLDR method implemented by a neural network, preserving the local geometry but abandoning the retention of topology. In contrast, our method takes the preservation of both geometry and topology into consideration, trying to maintain these properties of manifolds even in cases of excessive dimension reduction when the target dimension s is smaller than s. Invertible model. From AutoEncoder (AE) (Hinton & Salakhutdinov, 2006) , the fundamental neural network based model, having achieved DR and cut information loss by minimizing the reconstruction loss, some AE based generative models like VAE (Kingma & Welling, 2014) and manifold-based NLDR models like TopoAE (Moor et al., 2020) has emerged. These methods cannot avoid information loss after NLDR, and thus, some invertible models consist of a series of



Figure 1: Illustration of the process of NLDR. The dash line links M d 1 and x means x is sampled from M d1 , and it is represented in the Euclidean space R m after an inclusion mapping i i . We aim to approximate M d 1 from the observed sample x. For the topology preserving dimension reduction methods, it aims to find a homeomorphism g • f -1 to map x into z which is embedded in R s .

