LOCAL DISTANCE PRESERVING AUTO-ENCODERS US-ING CONTINUOUS K-NEAREST NEIGHBOURS GRAPHS

Abstract

Auto-encoder models that preserve similarities in the data are a popular tool in representation learning. In this paper we introduce several auto-encoder models that preserve local distances when mapping from the data space to the latent space. We use a local distance-preserving loss that is based on the continuous k-nearest neighbours graph which is known to capture topological features at all scales simultaneously. To improve training performance, we formulate learning as a constraint optimisation problem with local distance preservation as the main objective and reconstruction accuracy as a constraint. We generalise this approach to hierarchical variational auto-encoders thus learning generative models with geometrically consistent latent and data spaces. Our method provides state-ofthe-art or comparable performance across several standard datasets and evaluation metrics.

1. INTRODUCTION

Auto-encoders and variational auto-encoders (Kingma & Welling, 2014; Rezende et al., 2014) are often used in machine learning to find meaningful latent representations of the data. What constitutes meaningful usually depends on the application and on the downstream tasks, for example, finding representations that have important factors of variations in the data (disentanglement) (Higgins et al., 2017; Chen et al., 2018) , have high mutual information with the data (Chen et al., 2016) , or show clustering behaviour w.r.t. some criteria (van der Maaten & Hinton, 2008) . These representations are usually incentivised by regularisers or architectural/structural choices. One criterion for finding a meaningful latent representation is geometric faithfulness to the data. This is important for data visualisation or further downstream tasks that involve geometric algorithms such as clustering or kNN classification. The data often lies in a small, sparse, low-dimensional manifold in the space it inhabits and finding a lower dimensional projection that is geometrically faithful to it can help not only in visualisation and interpretability but also in predictive performance and robustness (e.g. Karl et al., 2017; Klushyn et al., 2021) . There are several approaches that implement such projections, ISOMAP (Tenenbaum et al., 2000) The approach presented in (Moor et al., 2020) , uses persistent homology computation to define local connectivity graphs over which to preserve local distances. One can choose the dimensionality of the preserved topological features, however, preserving higher-dimensional topological features comes at additional computational cost. In this paper we propose to use the continuous k-nearest neighbours method (Berry & Sauer, 2019) which is based on consistent homology and results in a significantly simpler graph construction method; it is also known to capture topological features at all scales simultaneously. Since AE and VAE methods are usually hard to train and regularise (Alemi et al., 2018; Higgins et al., 2017; Zhao et al., 2018; Rezende & Viola, 2018) , to improve learning we formulate learning as a constraint optimisation with the topological loss as the objective the reconstruction loss as constraint. In addition, we adapt the proposed methods to VAEs with



, LLE(Roweis & Saul, 2000), SNE/t-SNE(Hinton &  Roweis, 2002; van der Maaten & Hinton, 2008; Graving & Couzin, 2020)  and UMAP(McInnes et al.,  2018; Sainburg et al., 2021)  aim to preserve the local neighbourhood structure while topological auto-encoders(Moor et al., 2020), witness auto-encoders(Schönenberger et al., 2020), and (Li et al.,  2021)  use regularisers in auto-encoder models to learn projections that preserve topological features or local distances.

