TENSOR-BASED SKETCHING METHOD FOR THE LOW-RANK APPROXIMATION OF DATA STREAMS

Abstract

Low-rank approximation in data streams is a fundamental and significant task in computing science, machine learning and statistics. Multiple streaming algorithms have emerged over years and most of them are inspired by randomized algorithms, more specifically, sketching methods. However, many algorithms are not able to leverage information of data streams and consequently suffer from low accuracy. Existing data-driven methods improve accuracy but the training cost is expensive in practice. In this paper, from a subspace perspective, we propose a tensor-based sketching method for low-rank approximation of data streams. The proposed algorithm fully exploits the structure of data streams and obtains quasi-optimal sketching matrices by performing tensor decomposition on training data. A series of experiments are carried out and show that the proposed tensor-based method can be more accurate and much faster than the previous work.

1. INTRODUCTION

There are many scenarios that require batch or real-time processing of data streams arising from, e.g., video (Cyganek & Woźniak, 2017; Das, 2021) , signal flow (Cichocki et al., 2015; Sidiropoulos et al., 2017 ), hyperspectral images (Wang et al., 2017; Zhang et al., 2019) and numerical simulations (Zhang et al., 2022; Larcher & Klein, 2019) . A data stream can be seen as an ordered sequence of data continuously generated from one or several distributions (Muthukrishnan, 2005; Indyk et al., 2019) , and the data per time slot can be usually represented as a matrix. Therefore, most of the processing methods of data streams can be considered as operations on matrices, such as matrix multiplications, linear system solutions and low-rank approximation. Wherein, low-rank matrix approximation plays an important role in practical applications, such as independent component analysis (ICA) (Stone, 2002; Hyvärinen, 2013) , principle component analysis (PCA) (Karamizadeh et al., 2020; Jolliffe & Cadima, 2016) , image denoising (Guo et al., 2015; Zhang et al., 2019) . In this work, we consider low-rank approximation of matrices from a data stream. Specifically, let {A d ∈ R m×n } D d=1 be matrices from a data stream D, then the low-rank approximation in D can be described as: min B d ∥A d -B d ∥ F , s.t. rank(B d ) ≤ r, (1.1) where d = 1, 2, • • • , D, ∥ • ∥ F represents the Frobenius norm, and r ∈ Z + is a user-specified target rank.

Related work.

A direct approach to solve problem 1.1 is to calculate the truncated rank-r singular value decomposition (SVD) of A d in turn, and the Eckart-Young theorem ensures that it is the best low-rank approximation (Eckart & Young, 1936) . However, it is too expensive to one by one calculate the truncated rank-r SVD of A d for all d = 1, 2, • • • , D, particularly when m or n is large. To address this issue, many sketching algorithms have emerged such as the SCW algorithm (Sarlos, 2006; Clarkson & Woodruff, 2009; 2017) . Unfortunately, a notable weakness of sketching

