SORTED EIGENVALUE COMPARISON d Eig : A SIMPLE ALTERNATIVE TO d FID

Abstract

For i = 1, 2, let S i be the sample covariance of Z i with n i p-dimensional vectors. First, we theoretically justify an improved Fréchet Inception Distance (d FID ) algorithm that replaces np.trace(sqrtm(S 1 S 2 )) with np.sqrt(eigvals(S 1 S 2 )).sum(). With the appearance of unsorted eigenvalues in the improved d FID , we are then motivated to propose sorted eigenvalue comparison (d Eig ) as a simple alternative: , and λ i j is the j-th largest eigenvalue of S i . Second, we present two main takeaways for the improved d FID and proposed d Eig . (i) d FID : The error bound for computing non-negative eigenvalues of diagonalizable S 1 S 2 is reduced to O(ε)∥S 1 ∥∥S 1 S 2 ∥, along with reducing the run time by ∼ 25%. (ii) d Eig : The error bound for computing non-negative eigenvalues of sample covariance S i is further tightened to O(ε)∥S i ∥, with reducing ∼ 90% run time. Taking a statistical viewpoint (random matrix theory) on S i , we illustrate the asymptotic stability of its largest eigenvalues, i.e., rigidity estimates of O( n). Last, we discuss limitations and future work for d Eig .

1. INTRODUCTION

1 import numpy as np 2 from scipy.linalg import eigvals, eigvalsh 3 4 # The square of improved d FID 5 def dFID(mean1, cov1, mean2, cov2): 6 eigval = eigvals(cov1 @ cov2) 7 # Round computational errors (if exist) 8 # that lead to negative eigenvalues close to 0 9 eigval[eigval < 0] = 0 10 dif = mean1 -mean2 11 res = dif.dot(dif) + np.trace(cov1 + cov2) 12 return res -2 * np.sqrt(eigval).sum() 13 14 # The square of proposed d Eig 15 def dEig(scm1, scm2): In the image domain, it is of great interest to analyze the distribution shift between two collections of data entries (Wiles et al., 2021; Borji, 2019) . On one hand, this is driven by the increasing awareness about the violation of the assumption of 'identical distribution' between training and (real-world) test datasets (Wu et al., 2022b) . As for instance illustrated in the leaderboard of WILDS (Koh et al., 2021; Sagawa et al., 2021) , many algorithms suffer from performance degradation and fail to generalize to heterogeneous testing settings. On the other hand, the importance of assessing distribution shift has been recognized with the rise of generative adversarial nets (GAN) (Goodfellow et al., 2014; Heusel et al., 2017) . The rapid development of GAN variants (Karras et al., 2019; 2020b) urges reliable and accurate metric(s) to assess the discrepancy between generated and real images (Borji, 2019). 16 # Sorted eigenvalues 17 eigval1 = eigvalsh(scm1) 18 eigval1[eigval1 < 0] = 0 19 eigval2 = eigvalsh(scm2) 20 eigval2[eigval2 < 0] = 0 21 dif = np.sqrt(eigval1) -np.sqrt(eigval2) 22 return dif.dot(dif) To objectively assess GAN models, researchers have proposed a plethora of evaluation scores including Inception Score (Salimans et al., 2016) For i = 1, 2, let S i be the sample covariance of Z i = (z i 1 , . . . , z i ni ) with n i p-dimensional vectors. • (d FID ) We articulate the fact that S 1 S 2 is diagonalizable and has non-negative eigenvalues. This allows us to theoretically justify an improved algorithm of d FID , i.e., by replacing the unique principal square root of a matrix with the element-wise square root of its eigenvalues. Therefore, the error bound for computing its eigenvalues is reduced to O(ε)∥S 1 ∥∥S 1 S 2 ∥, reducing the run time by ∼ 25%. • (d Eig ) Since S i is symmetric positive semidefinite, the error bound for computing its nonnegative eigenvalues is further tightened to O(ε)∥S i ∥, along with reducing ∼ 90% run time. From the viewpoint of random matrix theory (RMT), we demonstrate the asymptotically stable behavior of the largest eigenvalues (spikes). 2 THE IMPROVED d FID (Linear Algebra) Notation: Lower case Roman or Greek letters (e.g., s, ϵ, γ, λ) denote scalars, bold lower case letters (e.g., v, z, µ) denote vectors, and bold upper case letters (e.g., Q, S, U , Z, Λ) denote matrices. T is matrix transpose, ∥.∥ is L 2 norm, ≲ denotes asymptotically less than.

2.1. PRINCIPAL SQUARE ROOT OF A MATRIX

Without loss of accuracy, we discuss d FID through the lens of linear algebra. More specifically, scalars, vectors and matrices discussed in the section are deterministic, while a statistical viewpoint on these objects will be later introduced in the proposed d Eig section. For i = 1, 2, let Z i = (z i 1 , . . . , z i ni ) be a collection of n i p-dimensional vectors. For simplicity, we assume sample mean 1 ni ni k=1 z i k = 0 throughout Sec. 2. Accordingly, S i = 1 ni Z i Z T i denotes the sample covariance matrix (SCM) of Z i . We start the discussion with revisiting standard the definition(s) of d FID (Givens & Shortt, 1984) , then we elaborate the properties of principal square root -the key computational challenge of d FID . (1)



+α i



Figure 1: Python codes for the square of improved d FID and proposed d Eig .

Definition 1. Let S i be the SCM of Z i and w.l.o.g. S 1 is non-singular, then we defined FID (S 1 , S 2 ) 2 = Trace(S 1 + S 2 -2(S

, Kernel Inception Distance (d KID )(Bińkowski et al.,  2018), and Precision/Recall(Kynkäänniemi et al., 2019; Sajjadi et al., 2018)  (please also see(Borji,  2019; 2022)  for in-depth review). Among various scores, Fréchet Inception Distance (d FID )(Heusel  et al., 2017)  is arguably the most widely-used metric for benchmarking GAN performance(Parmar  et al., 2022). This is mainly due to the favorable theoretical property of being a mathematical metric(Dowson & Landau, 1982)  and practical property of being well-correlated with perceived image quality(Sajjadi et al., 2018). Meanwhile, Chong & Forsyth (2020) argued that d FID is a biased estimator and Kynkäänniemi et al. (2022) observed its undesirable sensitivity towards fringe features or classes. Despite these weaknesses, d FID currently remains the 'gold standard' for GAN evaluation and continuously attracts broad attention. In a recent study, Mathiasen & Hvilshøj (2020) proposed to compute eigenvalues rather than square root of a matrix as in d FID . We view this as a promising simplification and improvement, nonetheless a precise theoretical analysis has not been performed and therefore becomes the starting point of this paper.The study of random matrix theory (RMT), with an emphasis on understanding the properties of (random) eigenvalues (Paul & Aue, 2014), has brought novel insights in the domain of deep learning(Liao & Couillet, 2018; Pastur, 2022; Baskerville et al., 2022), among which Seddik et al. (2020) analyzed deep learning representations of GAN generated images through the lens of eigenvalues of their sample covariance matrix (SCM). Driven by the need to efficiently quantify the distribution shift between two collections of heterogeneous data entries, we propose to compare sorted eigenvalues (d Eig ) as a simple alternative to d FID . Our contributions are summarized as follows:

