Oblivious Sketching-based Central Path Method for Solving Linear Programming Problems

Abstract

In this work, we propose a sketching-based central path method for solving linear programmings, whose running time matches the state of art results Cohen et al. (2019b); Lee et al. (2019). Our method opens up the iterations of the central path method and deploys an "iterate and sketch" approach towards the problem by introducing a new coordinate-wise embedding technique, which may be of independent interest. Compare to previous methods, the work Cohen et al. (2019b) enjoys feasibility while being nonoblivious, and Lee et al. ( 2019) is oblivious but infeasible, and relies on dense sketching matrices such as subsampled randomized Hadamard/Fourier transform matrices. Our method enjoys the benefits of being both oblivious and feasible, and can use sparse sketching matrix Nelson & Nguyên (2013) to speed up the online matrix-vector multiplication. Our framework for solving LP naturally generalizes to a broader class of convex optimization problems including empirical risk minimization.

1. Introduction

Linear programming is one of the fundamental models widely used in both theory and practice. It has been extensively applied in many fields such as economics Tintner (1955) ; Dorfman et al. (1987 ), operations research Delson & Shahidehpour (1992) , compressed sensing Donoho (2006); Candes et al. (2006) , medical studies Mangasarian et al. (1990; 1995) 2019) only maintains an infeasible update in each iteration and requires the usage of dense sketching matrices, which will ruin the potential sparsity structure of the original linear programs. Thus, a natural question to ask is:



We use O * hides n o(1) and logO(1) (1/δ) factors. In each iteration, we approximate the central path by solving a linear system. Our approach constructs a randomized oblivious system equation which can be solved exactly. While previous work Cohen et al. (2019b) constructs a non-oblivious one, and Lee et al. (2019) doesn't solve the system exactly.



, adversarial deep learningWong & Kolter (2018);Weng et al. (2018), etc., due to its simple and intuitive structure. The problem of solving linear programmings has been studied since the 19-th centurySierksma & Zwols (2015).Consider solving a general linear program in standard form min Ax=b,x≥0 c x of size A ∈ R d×n without redundant constraints. For the generic case d = Ω(n) we considered in this paper, the state of art results take a total running time ofO * (n ω + n 2.5-α/2 + n 2+1/6 ) 1 toobtain a solution of δ accuracy in current matrix multiplication time Cohen et al. (2019b); Lee et al. (2019), where ω is the exponent of matrix multiplication whose current value is roughly 2.373 Williams (2012); Le Gall (2014), and α is the dual exponent of matrix multiplication whose current value is 0.31 Le Gall & Urrutia (2018). The breakthrough work due to Cohen, Lee, and Song Cohen et al. (2019b) improves the long standing running time of O * (n 2.5 ) since 1989 Vaidya (1989). For the current ω and α, Cohen et al. (2019b) algorithm takes O * (n 2.373 ) time. For the current state-of-art results, the work Cohen et al. (2019b) involves a non-oblivious sampling technique, whose sampling set and size changes along the iterations. It avoids the possibilities of implementing expensive calculations in the preprocessing stage and also makes it harder to extend to other classical optimization problems. On the other hand, the work Lee et al. (

annex

Is there an oblivious and feasible algorithm for solving linear programming programs in fast running time (i.e. current matrix multiplication time) ?In this work, we propose a both oblivious and feasible (per iteration) 2 method that solves linear programs in the same running time as the state of art.The algorithm we propose is a sketching-based short step central path method. The classical short step method follows the central path in the interior of the feasible region. It decreases the complementarity gap uniformly by roughly a 1 -1/ √ n factor in each iteration and takes The coordinate-wise embedding we introduce in this work is a distribution of matrices R ∈ R b sketch ×n with b sketch n such that, for any inner product g h between two ndimensional vector g, h ∈ R n , with "high" probability g R Rh approximates g h well. In the case of solving linear programmings, we approximate the calculation of matrix-vector multiplication P h in each iteration by P R Rh through OCE, such that the resulting random vector is close to previous one in each coordinate, i.e., (P R Rh) i ≈ (P h) i for all i ∈ [n]. Combining with lazy update and low-rank update techniques to maintain the query structure P R Rh for any input vector h ∈ R n , we can ensure the new random path is still close to the central path throughout the iterations. Therefore, our method decrease the average running time per iteration while keeping the same number of iterations. Furthermore, the sketching matrix R in our approach can be chosen in an oblivious way since it does not depend on the algorithm updates. Compare to previous work Lee et al. (2019) , our approximation form P R Rh also helps admit a closed form solution in each iteration for solving LP. Thus, our approach takes the advantages of being oblivious and feasible, compared to other state of art results Cohen et al. (2019b) ; Lee et al. (2019) .We state our main result as follows:Theorem 1.1 (Main result, informal). Given a linear program min Ax=b,x≥0 c x with no redundant constraints. Let δ lp denotes precision. Our algorithm takes O(n 2.373 log(n/δ lp )) time to solve this LP.

1.1. Related works

Linear programming. Linear programmings have been studied for nearly a century. One of the first and most popular LP algorithm is the simplex algorithm Dantzig (1947) . Despite it works well in practical small size problems, the simplex algorithm is known to be an exponential time algorithm in the worst case of Klee-Minty cube Klee & Minty (1972) . The first polynomial time algorithm for solving LP is the ellipsoid method Khachiyan (1980) proposed by Khachiyan. Although this algorithm runs in polynomial time in theory, but in practice this algorithm runs much slower than the simplex algorithm. The interior point type of methods Karmarkar (1984) have both polynomial running time in theory and fast and stable performance in practice. In the case of d = Ω(n) considered in this work, Karmarkar's algorithm Karmarkar (1984) takes O * (n 3.5 ) running time. Then it was improved to O * (n 3 ) in the work Renegar (1988); Vaidya (1987) . In 1989, Vaidya further proposed an algorithm that takes a running of O * (n 2.5 ). This result hasn't been improved until recent work due to Cohen, Lee and Song Cohen et al. (2019b) .Sketching. Classical sketching methodology proposed by Clarkson & Woodruff (2013) is the so-called "sketch and solve". The most standard and well-known applications are

