NEURAL NETWORK APPROXIMATION OF LIPSCHITZ FUNCTIONS IN HIGH DIMENSIONS WITH APPLICA-TIONS TO INVERSE PROBLEMS

Abstract

The remarkable successes of neural networks in a huge variety of inverse problems have fueled their adoption in disciplines ranging from medical imaging to seismic analysis over the past decade. However, the high dimensionality of such inverse problems has simultaneously left current theory, which predicts that networks should scale exponentially in the dimension of the problem, unable to explain why the seemingly small networks used in these settings work as well as they do in practice. To reduce this gap between theory and practice, a general method for bounding the complexity required for a neural network to approximate a Lipschitz function on a high-dimensional set with a low-complexity structure is provided herein. The approach is based on the observation that the existence of a linear Johnson-Lindenstrauss embedding A ∈ R d×D of a given high-dimensional set S ⊂ R D into a low dimensional cube [-M, M ] d implies that for any Lipschitz function f : S → R p , there exists a Lipschitz function g : [-M, M ] d → R p such that g(Ax) = f (x) for all x ∈ S. Hence, if one has a neural network which approximates g : [-M, M ] d → R p , then a layer can be added which implements the JL embedding A to obtain a neural network which approximates f : S → R p . By pairing JL embedding results along with results on approximation of Lipschitz functions by neural networks, one then obtains results which bound the complexity required for a neural network to approximate Lipschitz functions on high dimensional sets. The end result is a general theoretical framework which can then be used to better explain the observed empirical successes of smaller networks in a wider variety of inverse problems than current theory allows.

1. INTRODUCTION

At present various network architectures (NN, CNN, ResNet) achieve state-of-the-art performance in a broad range of inverse problems, including matrix completion (Zheng et al., 2016; Monti et al., 2017; Dziugaite & Roy, 2015; He et al., 2017) image-deconvolution (Xu et al., 2014; Kupyn et al., 2018) , low-dose CT-reconstitution (Nah et al., 2017) , electric and magnetic inverse Problems (Coccorese et al., 1994) (seismic analysis, electromagnetic scattering). However, since these problems are very high dimensional, classical universal approximation theory for such networks provides very pessimistic estimates of the network sizes required to learn such inverse maps (i.e., as being much larger than what standard computers can store, much less train). As a result, a gap still exists between the widely observed successes of networks in practice and the network size bounds provided by current theory in many inverse problem applications. The purpose of this paper is to provide a refined bound on the size of networks in a wide range of such applications and to show that the network size is indeed affordable in many inverse problem settings. In particular, the bound developed herein depends on the model complexity of the domain of the forward map instead of the domain's extrinsic input dimension, and therefore is much smaller in a wide variety of model settings. To be more specific, recall in most inverse problems one aims to recover some signal x from its measurement y = F (x). Here y and x could both be high dimensional vectors, or even matrices and tensors, and F , which is called the forward map/operator, could either be linear or nonlinear with various regularity conditions depending on the application. In all cases, however, recovering x from y amounts to inverting F . In other words, one aims want to find the operator F -foot_0 , that sends every measurement y back to the original signal x. Depending on the specific application of interest, there are various commonly considered forms of the forward map F . For example, F could be a linear map from high to low dimensions as in compressive sensing applications; F could be a convolution operator that computes the shifted local blurring of an image as in the image deblurring setting; F could be a mask that filters out the unobserved entries of the data as in the matrix completion application; or F could also be the source-to-solution map of a differential equation as in ODE/PDE based inverse problems. In most of these applications, the inverse operator F -1 does not possess a closed-form expression. As a result, in order to approximate the inverse one commonly uses analytical approaches that involve solving, e.g., an optimization problem. Take the sparse recovery as an example. With the prior knowledge that the true signal x ∈ R n is sparse, one can recover it from the under-determined measurements R m y = Ax with m < n) by solving the optimization problem x = arg min z z 0 , Az = y The inverse of the linear measurement map F (x) = y = Ax when restricted to the low-complexity domain of sparse vectors has an inverse, F -1 (y), that is then the minimizer x above. Note that traditional optimization-based approaches could be extremely slow for large-scale problems (e.g., for n large above). Alternatively, we can approximate the inverse operator by a neural network instead. Amortizing the initial cost of an expensive training stage, the network can later achieve unprecedented speed over time at the test stage leading to better total efficiency over its lifetime. To realize this goal, however, we need to first find a neural network architecture f θ , and train it to approximate F -1 , so that the approximation error max y f θ (y) -F -1 (y) = f θ (y) -x is small. The purpose of this paper is to provide a unified way to give a meaningful estimation of the size of the network that one can use to set up the network in situations where the domain of F is low-complexity as is the case in, e.g., compressive sensing, low-rank matrix completion, deblurring with low-dimensional signal assumptions, etc..

2. RELATED WORK

The expressive power of neural networks is important in applications as a means of both guiding network architecture design choices, as well as for providing confidence that good network solutions exist in general situations. As a result, numerous results about the approximation power has been established in recent years (Zhou, 2020; Petersen & Voigtlaender, 2020; Yarotsky, 2022; 2018; Lin & Jegelka, 2018) . Most results concern the approximation of functions on R D , however, and yield network sizes that increase exponentially with the input dimension D. As a result, the high dimensionality of many inverse problems leads to bounds from most of the existing literature which are too large to explain the observed empirical success of neural approaches in such applications. A similar high-dimensional scaling issue arises in many image classification tasks as well. Motivated by this setting (Chen et al., 2019) refined previous approximation results for ReLU networks, and showed that input data that is close to a low-dimensional manifold leads to network sizes that only grow exponentially with respect to the intrinsic dimension of the manifold. However, this improved bound relies on the data fitting manifold assumption which is quite strong in the inverse problems setting. For example, even the "simple" sparse recovery problem does not have a domain/range that forms a manifold (note that the intersections of s-dimensional subspaces prevent from it being a manifold). Therefore, to study expressive power of networks on inverse problems needs to remove such strict manifold assumptions. Another mild issue with such manifolds results is that the number of neurons also depends on the curvature of the manifold in question which can be difficult to estimate. Furthermore, such curvature dependence is unavoidable for manifold results and needs to be incorporated into any valid bounds. 1 In this paper, we provide another way to estimate the size of the network, by directly using the Guassian width of the data as a measure of its inherent complexity. Our result can therefore be considered generalization of the manifold result discussed above in two ways. First, it applies to more arbitrary data sets with low complexities. And, it also applies to other types of networks besides just feedforward ReLu networks. Both types of generalization are then shown to be useful and applicable to various inverse problems.

3. MAIN RESULTS

We begin by stating a few definitions. We say that a neural network -approximates a function f if the function implemented by the neural network f satisfies f (x) -f (x) ∞ ≤ for all x in the domain of f . We say that a neural network architecture -approximates any function in a function class F if for any function f ∈ F, there exists a choice of edge weights such that the function f implemented by the neural network with that choice of edge weights satisfies f (x) -f (x) ∞ ≤ for all x in the domain of f . Also, for any positive integers d < D, any set S ⊂ R D , and any constant ρ ∈ (0, 1), we say that a matrix A ∈ R d×D is a ρ-JL (Johnson-Lindenstrauss) embedding of S if (1 -ρ) x -x 2 ≤ Ax -Ax 2 ≤ (1 + ρ) x -x 2 for all x, x ∈ S. If we furthermore have A(S) := {Ax : x ∈ S} ⊂ T , we say that A is a ρ-JL embedding of S into T . Intuitively, a ρ-JL embedding of S into R d transforms S from a high-dimensional space to a low-dimensional space without significantly distorting distances between points. Contributions: Existing universal approximation theorems for various types of neural networks are mainly stated for functions defined on an d-dimensional cube. Our main contribution is to generalize these results to functions defined on arbitrary JL-embedable sets, which possibly reside in very high dimensions. We then demonstrate how our result can be applied to inverse problems to obtain a reasonable estimate of the network size. Since our theory is to be applied to general inverse problems, for which we cannot assume anything more than Lipschitz continuous. Hence in this paper, we focus on the class of Lipschitz functions. More explicitly, we show that if there exists a ρ-JL embedding of a high-dimensional set S ⊂ R D into a low-dimensional cube [-M, M ] d , then we can use any neural network architecture which can -approximate L 1-ρ -Lipschitz functions on [-M, M ] d to construct a neural network architecture which can -approximate L-Lipschitz functions on S. To establish this, we show that if there exists ρ-JL embedding A ∈ R d×D of S ⊂ R D into d-dimensions, then for any L-Lipschitz function f : S → R p , there exists a L 1-ρ -Lipschitz function g : [-M, M ] d → R p (where M = sup x∈S Ax ∞ ) such that g(Ax) = f (x) for all x ∈ S. Hence, if we have a neural network which can approximate g : [-M, M ] d → R p , then we can compose it with a neural network which implements the JL embedding A to obtain a neural network which approximates f : S → R p . By pairing JL embedding existence results along with results on approximation of Lipschitz functions by neural networks, we obtain results which bound the complexity required for a neural network to approximate Lipschitz functions on high dimensional sets. We now state our main theorem. Theorem 1. Let d < D be positive integers, and let L, M > 0 and ρ ∈ (0, 1) be constants. Let S ⊂ R D be a bounded subset for which there exists a ρ-JL embedding A ∈ R d×D of S into [-M, M ] d . a) Suppose that any L 1-ρ -Lipschitz function g : [-M, M ] d → R p can be -approximated by a feedforward neural network with at most N nodes, E edges, and L layers. Then, any L-Lipschitz function f : S → R p can be -approximated by a feedforward neural network with at most N + D nodes, E + Dd edges, and L + 1 layers. b) Furthermore, if there exists a single feedforward neural network architecture with at most N nodes, E edges, and L layers that can -approximate any L 1-ρ -Lipschitz function g : [-M, M ] d → R p , then there also exists another feedforward neural network architecture with at most N + D nodes, E +Dd edges, and L+1 layers that can -approximate any L-Lipschitz function f : S → R p . c) Suppose that the ρ-JL embedding is of the form A = M D, where M is a partial circulant matrix, and D is a diagonal matrix with ±1 on its diagonal. Also, suppose that any L 1-ρ -Lipschitz d) Furthermore, if there exists a single convolutional neural network architecture with at most N nodes, P parameters, and L layers that can -approximate any L 1-ρ -Lipschitz function g : [-M, M ] d → R p , then there also exists another convolutional neural network architecture with at most N + 2D nodes, P + 2Dd parameters, and L + 3 layers that can -approximate any L-Lipschitz function f : S → R p . Remark 1. The theorem ensures that the network size for approximating f grows exponentially with the compressed dimension d instead of growing exponentially with the input dimension D. The task now reduces to making the compressed dimension d as small as possible while still ensuring that a ρ-JL embedding of S into [-M, M ] d exists. Remark 2. The theorem is quite general as parts a and b are not restricted to any particular type of network or activation function. In Section 3.3, we provide two corollaries of Theorem 1 that establish the expressive power of the feedforward and convolutional neural networks. Remark 3. If an inverse operator is Lipschitz continuous and there exists a ρ-JL embedding of the set of possible observations S into d dimensions, then the theorem gives us a bound on the complexity of a neural network architecture required to approximate the inverse operator.

3.1. JL EMBEDDINGS, AND COVERING NUMBERS AND GAUSSIAN WIDTH

As the existence of the JL map is a critical assumption of our theorem, in this section, we discuss the sufficient conditions for this assumption to hold. In addition, we also care about the structures of the JL maps, as they will end up being the first layer of the final neural network. For example, if the neural network is of convolution type, we need to make sure that a circulant JL matrix exists. Existence of ρ-JL maps: It is well-known that for finite sets S, the existence of a ρ-JL embedding can be guaranteed by the Johnson-Lindenstrauss Lemma. For sets S with infinite cardinally, the Johnson-Lindenstrauss lemma cannot be directly used. In the following proposition, we extend the Johnson-Lindenstrauss lemma from a finite set of n points to a general set S. Proposition 1. Let ρ ∈ (0, 1). For S ⊆ R D , define U S := x -x x -x 2 : x, x ∈ S s.t. x = x to be the closure of the set of unit secants of S, and N (U S , • 2 , δ) to be the covering number of U S with δ-balls. Then, there exists a set S 1 with |S 1 | = 2N (U S , • 2 , δ) points such that if a matrix A ∈ R d×D is a ρ-JL embedding of S 1 , then A is also a (ρ + 2 A δ)-JL embedding of S. The proposition guarantees that whenever we have a JL-map for finite sets, we can extend it to a JL-map for infinite sets with similar level of complexity measured in terms of the covering numbers. There are many known JL-maps for finite sets that we can extend from, including sub-Gaussian matrix (Matoušek, 2008) , Gaussian circulant matrices with random sign flip (Cheng & Zhang, 2014) , etc. We present some of the related results here. Proposition 2 ( (Matoušek, 2008) ). Let x 1 , . . . , x n ∈ R D . Let ρ ∈ (0, 1 2 ) and β ∈ (0, 1). Let A ∈ R d×D be a random matrix whose entries are i.i.d. from a subgaussian distribution with mean 0 and variance 1. Then, there exists a constant C > 0 depending only on the subgaussian distribution such that if d ≥ Cρ -2 log n β , then 1 √ d A will be a ρ-JL embedding of {x 1 , . . . , x n } with probability at least 1 -β. Proposition 3 (Corollary 1.3 in (Cheng & Zhang, 2014) ). Let x 1 , . . . , x n ∈ R D . Let ρ ∈ (0, 1 2 ), and let d = O(ρ -2 log 1+α n) for some α > 0. Let A = 1 √ d M D where M ∈ R d×D is a random Gaussian circulant matrix and D ∈ R D×D is a random Rademacher diagonal matrix. Then, with probability at least 2 3 1 -(D + d)e -log α n , A is a ρ-JL embedding of {x 1 , . . . , x n }. Note that the α in the proposition can be set to be any positive number making the probability of failure less than 1. Combining the results of Propositions 2 and 3 with Proposition 1, we have the following existence result for the JL map of an arbitrary set S, Proposition 4. Let ρ ∈ (0, 1) be a constant. For S ⊆ R D , let N (U S , • 2 , δ) to be the covering number with δ-balls of the unit secant U S of S defined in Proposition 1. Then a) If D ≥ d ρ -2 log N (U S , • 2 , ρ 4 √ 3D ), then there exists a matrix A ∈ R d×D which is a ρ-JL embedding of S. b) If D ≥ d ρ -2 log(4D + 4d) log N (U S , • 2 , ρ 4 √ 3D ) , then there exists a matrix A ∈ R d×D in the form of M D and of size d × D that works as ρ-JL map for S, where M is a partial circulant matrix and D is a diagonal matrix with ±1 on its diagonal. The above proposition characterizes the compressibility of a set S by a JL-mapping terms of the covering number. Alternatively, one can also characterize it using the Gaussian width. For example, in (Iwen et al., accepted. (See Arxiv 2110.04193)) it is shown using methods from (Vershynin, 2018) that if the set of unit secants of S has a low Gaussian width, then with high probability a subgaussian random matrix with provide a low-distortion linear embedding, and the dimension d required scales quadratically with the Gaussian width of the set of unit secants of S. Proposition 5 (Corollary 2.1 in (Iwen et al., accepted. (See Arxiv 2110.04193))). Let ρ, β ∈ (0, 1) be constants. Let A ∈ R d×D be a matrix whose rows a T 1 , . . . , a T d are independent, isotropic (E[a i a T i ] = I), and subgaussian random vectors. Let S ⊂ R D , and Let ω(U S ) := E sup u∈U S u, z , z ∼ Normal(0, I) to be the Gaussian width of U S . Then, there exists a constant C > 0 depending only on the distribution of the rows of A such that if d ≥ C ρ 2 ω(U S ) + log 2 β 2 , then 1 √ d A is a ρ-JL embedding of S with probability at least 1 -β. In practice, one can use either the Gaussian width (Proposition 5) or the covering number (Proposition 4) to compute the lower bound of d, whichever is more convenient for a specific application.

3.2. UNIVERSAL APPROXIMATOR NEURAL NETWORKS FOR LIPSCHITZ FUNCTIONS ON d-DIMENSIONAL CUBES

In Theorem 1, we showed that with the help of JL, approximation rate of neural networks for functions defined on an arbitrary set S can be derived from their approximation rates for functions defined on the cube [-M, M ] d . In this section, we review known results for the later, so that they can be used in combination of Theorem 1 to provide useful approximation results for network applications to various inverse problems. Specifically, we review two types of universal approximators for functions defined on the cube [-M, M ] d . One is the Feedforward ReLU network and the other is the Resnet type convolution neural network. Feedforward ReLU network: The fully connected feedforward neural network with ReLU activation is known to be a universal approximator of any Lipschitz function on the box [-M, M ] d . Moreover, for such networks, the non-asymptotic approximation error has also been established, allowing us to get an estimate of the network size. The proposition below is a variant of Proposition 1 in (Yarotsky, 2018) , and the proof uses an approximating function that uses the same ideas as in (Yarotsky, 2018) . Proposition 6. Given constants L, M, > 0 and positive integers d and p, there exists a ReLU NN architecture with at most (p+C 1 ) 2 LM √ d + 1 d edges, C 2 2 LM √ d + 1 d +p nodes, and log 2 (d + 1) +2 layers that can -approximate any L-Lipschitz function g : [-M, M ] d → R p . Here, C 1 , C 2 > 0 are universal constants. Also for each edge of the ReLU NN, the corresponding weight is either independent of g, or is of the form g i (x) for some fixed x ∈ [-M, M ] d and coordinate i = 1, . . . , p. Convolutional Neural Network: As many successful network applications on inverse problems results from the use of filters in the CNN architectures (Jin et al., 2017) , we are particularly interested in the expressive power of CNN in approximating the Lipschitz functions. Currently known non-asymptotic results for CNN includes (Zhou, 2020; Petersen & Voigtlaender, 2020; Yarotsky, 2022) , but they are established under stricter assumptions than merely Lipschitz continuous. On the other hand, the ResNet-based CNN with the following architecture has been shown to possess good convergence rate. The following asymptotic result is proved in (Oono & Suzuki, 2019) . We note that the authors of (Oono & Suzuki, 2019) proved a more general result for β-Hölder functions, but we state it for Lipschitz functions, i.e., β = 1. Proposition 7 (Corollary 4 from (Oono & Suzuki, 2019) ). Let f : [-1, 1] d → R be a Lipschitz function. Then, for any K ∈ {2, ..., d}, there exists a CNN f (CNN) with O(N ) residual blocks, each of which has depth O(log N ) and O(1) channels, and whose filter size is at most K, such that CN N σ θ := F C W,b • (Conv σ ω M ,b M + id) • • • • • (Conv σ ω1,b1 + id) • P f -f (CNN) ∞ ≤ O(N -1/d ).

3.3. MAIN RESULTS

We can now combine Propositions 4 , 5, 6 and 7 with our Theorem 1 to obtain theorems bounding the required complexity of a feed-forward/convolutional neural network that can -approximate any L-Lipschitz function on arbitrary sets S ⊂ R D for which a ρ-JL embedding into [-M, M ] d exists. Theorem 2. Let d < D be positive integers, and let L > 0 and ρ ∈ (0, 1) be constants. Let S ⊂ R D be a bounded set and U S be its set of unit secants. Suppose that d min ρ -2 log N (U S , • 2 , ρ 4 √ 3D ), ρ -2 (ω(U S )) 2 , where N (U S , • 2 , ρ 4 √ 3D ) is the covering number and ω(U S ) is the Gaussian width of U S . Then, there exists a ReLU neural network architecture with at most (p + C 1 ) 2 LM √ d (1-ρ) + 1 d + Dd edges, C 2 2 LM √ d (1-ρ) + 1 d + p + D nodes, and log 2 (d + 1) + 3 layers that can -approximate any L-Lipschitz function f : S → R p , where M = sup x∈S Ax ∞ . Our Theorem 3 is a variant on our Theorem 2, but for convolutional neural networks. Theorem 3. Let d < D be positive integers, and let L > 0 and ρ ∈ (0, 1) be constants. Let S ⊂ R D be a bounded set and U S be its set of unit secants. Suppose that d ρ -2 log(4D + 4d) log N (U S , • 2 , ρ 4 √ 3D ). Then, for any L-Lipschitz function f : S → R p , there exists a ResNet type CNN f (CNN) in the form of (1) with O(N ) residual blocks, each of which has a depth O(log N ) and O(1) channels, and whose filter size is at most K such that f -f (CNN) ∞ ≤ O(N -1/d ).

4. APPLICATIONS TO INVERSE PROBLEMS

Now we focus on inverse problems and demonstrate how the main theorems can be used to provide a reasonable estimate of the size of the neural networks needed to solve some classical inverse problems in signal processing. The problems we consider here are sparse recovery, blind deconvolution, and matrix completion. In all the inverse problems, we want to recover some signal x ∈ S from its forward measurement y = F (x), where the forward map F is assumed to be known. The minimal assumption we have to impose on F is the invertibility. Assumption 1 (invertibility of the forward map): Let S be the domain of the forward map F , and Y = F (S) be the range. Assume that the inverse operator F -1 : Y → S exists and is Lipschitz continuous with constant L, F -1 (y 1 ) -F -1 (y 2 ) ≤ L y 1 -y 2 , for all y 1 , y 2 ∈ Y. For any inverse problems satisfying Assumption 1, Theorem 1, 2 and 3 provide ways to estimate the size of the universal approximator networks for the inverse map. When applying the theorems to each problem, we need to estimate the covering number of U Y first. Depending on the problem, one may estimate the covering number either numerically or theoretically. If the domain Y of the inverse map is irregular and discrete, then it may be easier to compute the covering number numerically. If the domain has a nice mathematical structure, then we may be able to estimate it theoretically. Below are three examples of the theoretical estimation. From them, we see that it is quite common for inverse problems to have a small intrinsic complexity, with which Theorems 2 and 3 can significantly reduce the required size of the network from the previously known results. We emphasize that the covering number that the theorems use is the one of the unit secant of Y, which can be much larger than the covering number of Y itself. Sparse recovery: Sparsity is now one of the most commonly used priors in inverse problems as signals in many real applications possess certain level sparsity in some appropriate domain. For simplicity, we consider the strictly sparse signals. Let Σ N s be the set of s-sparse vectors of length N . Assume a sparse vector is measured linearly y = Ax ≡ F (x), the inverse problem amounts to recovering x from y. Now that we want to use a network to approximate the inverse map F -1 : AΣ N s ≡ Y ∈ y → x ∈ Σ N s , and estimate the size of the network through the theorems, we need to estimate the covering number of the unit secant U AΣ N s . Proposition 8. Let U AΣ N s denote the set of unit secants of AΣ N s . Then, we have log N (U AΣ N s , • , δ) s log N δ . then it is known (e.g. (Candes & Plan, 2010) ) that provided that the number of observations |Ω| µ 0 r max{m, n} log 2 max{m, n}, then with overwhelming probability, the inverse map F -1 : Y → X exists and is Lipschitz continuous. Let us denote the set of low-rank matrices satisfying (4) to be C. To estimate the complexity of the inverse map, we compute the covering number of U Y for Y = {Y = P Ω X : X ∈ M m,n r ∩ C}. Proposition 10. Suppose the mask is chosen so that the inverse map F -1 : Y = P Ω X → X is Lipschitz continuous with Lipschitz constant L, then for Y = {P Ω X : X ∈ M m,n r ∩ C}, the logarithm of the covering number of the set of unit secants of Y is bounded by log N (U Y , • 2 , δ) r(m + n) log L δ . Combining this proposition with Theorem 2 and 3, we obtain that there exist neural networks of full connected type or of CNN type having about O( -r(m+n) log(Lnm) ) number of weights, that can solve the blind-deconvolution problem up to an error of .

5. CONCLUSION AND DISCUSSION

The main message of this paper is that when neural networks are used to approximate Lipschitz continuous functions, the size of the network only needs to grow exponentially with respect to the intrinsic complexity of the input set measured using either the Gaussian width or the covering numbers. Therefore, it is more optimistic than the previous estimate that requires the size of the network to grow exponentially with respect to the input dimension. Similar results were derived previously in (Chen et al., 2019) in a more restrictive setting, namely, the input set is assumed to be close to a smooth manifold with a small curvature, and the network type is restricted to the feedforward ReLU networks. In this paper, by utilizing the JL map, we are able to state the result in a very general setting, that does not pose any structural requirement on the inputs set other than that they have a small complexity. In addition, our result holds for many different types of networks -although we only stated it for feedfoward neural networks and the ResNet type of convolutional neural networks, the same idea naturally applies to other types of networks as long as an associated JL-map exists. The estimate we provided for the network size ultimately depends on the complexity of the input set, measured by either the covering number or the Gaussian width of its set of unit secants. The computation of these quantities varies case by case, and in some cases might be rather difficult. This is a possible limitation of the proposed method. Because if the estimation of the input complexity is not tight enough, we may again get a pessimistic bound. Having said that, for most of the classical inverse problems, the covering number and the Gaussian width are not too difficult to calculate. As we demonstrated in Section 4, there are many known properties of them that one can use to facilitate the calculation. And when a training dataset is given, one can even compute the covering number numerically with off-the-shelf algorithms. Finally, although the applications of neural networks to inverse problems are seeing its success. There are much more failed attempts with unclear reasons. One common explanation is that the size of the network in use is not large enough for the targeted applications. Since inverse problems models usually have a much higher intrinsic dimensionality than say image classification models, the required network sizes might also be much larger. The classical universal approximation theorems only guarantees small errors when the network size approaches infinity, therefore is not very helpful in the non-asymptotic regime where we have to choose the network size, which is now known to be critical to good performances. We hope the presented result can give more insight on this matter.



To see why, e.g., curvature dependence is unavoidable, consider any discrete training dataset in a compact ball. There always exists a 1-dimensional manifold, namely a curve, that goes through all the data points. Thus, the mere existence of the 1-dimensional manifold does not mean the data complexity is low. Curvature information and other manifold properties matter as well!



Figure 1: If there exists a ρ-JL embedding of S ⊂ R D into [-M, M ] d , then we can write the target function f : S → R p as f = g • JL where g : [-M, M ] d → R p . So, we can then construct a neural network approximation of f by using a neural network approximation of g and adding a layer to implement the JL embedding.

1) where σ is the activation function, each Conv ωm,bm is an convolution layer with L m filters ω 1 m ,...,ω Lm m stored in ω M and L m bias b 1 m , ..., b Lm m stored in b m . The addition by the identity map, Conv σ ω M ,b M + id, makes it a residual block. F C W,b represents a fully connected layer appended to the final layer of the network. We see that the ResNet-based CNN is essentially a normal CNN with skip connections.

annex

Proof. By definition, the unit secant of Y = AΣ N s is defined aswhich contains all unit vectors that are linear combinations of 2s columns of A. Let T with |T | = 2s be a fixed support set, the covering number of span(A T ) ∩ S m-1 is ( 3 δ ) 2s , so the covering number of U Y is at mostIf the inverse of F exists, such as in the case when A is a Restricted-Isometry-Property matrix, then by Theorem 2 and 3, there exist neural networks of fully connected type or of CNN type with O( -s log N ) number of weights, that can do the sparse recovery up to an error of .Blind deconvolution: Blind deconvolution concerns the recovery of a signal x from its blurry measurements y = k ⊗ x (2) when the kernel k is also unknown. Here ⊗ denotes the convolution operation.Blind-deconvolution is an ill-posed problem due to the existence of a scaling ambiguity between x and k, namely, if (k, x) is a solution, then (αk, 1 α x) with α = 0 is also a solution. To resolve this issue, we focus on recovering the outer product xk T , where x and k here are both columns vectors. The recovery of the outer product xk T from the convolution y = k ⊗ x can be well-posed in various settings (Lee et al., 2015; Ahmed et al., 2013) . For example, (Ahmed et al., 2013) showed that if we assume x = Φu and k = Ψv, where Φ ∈ R N,n (n < N ) is i.i.d. Gaussian matrix and Ψ ∈ R N,m (m < N ) is a matrix of small coherence, then for large enough N , the outer-product xk T can be stably recovered from y in the following sense. For any two signal-kernel pairs (x, k), (x, k) and their corresponding convolutions y, ỹ, we have(3) with some L. When using a neural network to approximate the inverse map by F -1 : y → xk T , we need to estimate the covering number of the unit secant cone of Y = {y = x ⊗ k, x ∈ ΦΣ N s , k ∈ spanΨ}, which is done in the following proposition. Proposition 9. Suppose the inverse map F -1 : y → xk T is Lipschitz continuous with Lipschitz constant L, then for Y = {y = x ⊗ k, x ∈ span(Φ), k ∈ spanΨ}, the logarithm of the covering number of the set of unit secants of Y is bounded byCombining this proposition with Theorem 2 and 3, we obtain that there exist neural networks of full connected type or of CNN type having about O( -max{m,n} log(L(n+m)) ) number of weights, that can solve the blind-deconvolution problem up to an error of .

Matrix completion:

Matrix Completion is a central task in machine learning where we want to recover a matrix from its partially observed entries. It arises from a number of applications including image super resolution (Shi et al., 2013; Cao et al., 2014) , image/video denoising (Ji et al., 2010) , recommender systems (Zheng et al., 2016; Monti et al., 2017) , and gene-expression prediction (Kapur et al., 2016) , etc.. Recently neural network models have achieved state-of-the-art performance (Zheng et al., 2016; Monti et al., 2017; Dziugaite & Roy, 2015; He et al., 2017) , but a general existence result in the non-asymptotic regime is still missing.In this setting, the measurements Y = P Ω X consists of a set of observed entries of the unknown low-rank matrix X, where Ω is the index set of the observed entries and P Ω is the mask that sets all but entries in Ω to 0. Assuming M n,m r is the set of n × m matrices with rank at most r and X ∈ M n,m r . If the mask is random, and the left and right eigenvectors U, V of X are incoherent, in the sense that (U V T ) i,j ≤ µ 1 r nm

