Quadratic optimization with orthogonality constraint: explicit Łojasiewicz exponent and linear convergence of retraction-based line-search and stochastic variance-reduced gradient methods

Quadratic optimization with orthogonality constraint: explicit Łojasiewicz exponent and linear... The problem of optimizing a quadratic form over an orthogonality constraint (QP-OC for short) is one of the most fundamental matrix optimization problems and arises in many applications. In this paper, we characterize the growth behavior of the objective function around the critical points of the QP-OC problem and demonstrate how such characterization can be used to obtain strong convergence rate results for iterative methods that exploit the manifold structure of the orthogonality constraint (i.e., the Stiefel manifold) to find a critical point of the problem. Specifically, our primary contribution is to show that the Łojasiewicz exponent at any critical point of the QP-OC problem is 1 / 2. Such a result is significant, as it expands the currently very limited repertoire of optimization problems for which the Łojasiewicz exponent is explicitly known. Moreover, it allows us to show, in a unified manner and for the first time, that a large family of retraction-based line-search methods will converge linearly to a critical point of the QP-OC problem. Then, as our secondary contribution, we propose a stochastic variance-reduced gradient (SVRG) method called Stiefel-SVRG for solving the QP-OC problem and present a novel Łojasiewicz inequality-based linear convergence analysis of the method. An important feature of Stiefel-SVRG is that it allows for general retractions and does not require the computation of any vector transport on the Stiefel manifold. As such, it is computationally more advantageous than other recently-proposed SVRG-type algorithms for manifold optimization. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Mathematical Programming Springer Journals

Quadratic optimization with orthogonality constraint: explicit Łojasiewicz exponent and linear convergence of retraction-based line-search and stochastic variance-reduced gradient methods

Liu, Huikang; So, Anthony; Wu, Weijie
Mathematical Programming, Volume 178 (2) – Jun 1, 2018
Download PDF
48 pages
Read Article

Loading next page...
 
/lp/springer_journal/quadratic-optimization-with-orthogonality-constraint-explicit-w5cYpPVbCf
Publisher
Springer Journals
Copyright
Copyright © 2018 by Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society
Subject
Mathematics; Calculus of Variations and Optimal Control; Optimization; Mathematics of Computing; Numerical Analysis; Combinatorics; Theoretical, Mathematical and Computational Physics; Mathematical Methods in Physics
ISSN
0025-5610
eISSN
1436-4646
DOI
10.1007/s10107-018-1285-1
Publisher site
See Article on Publisher Site

Abstract

The problem of optimizing a quadratic form over an orthogonality constraint (QP-OC for short) is one of the most fundamental matrix optimization problems and arises in many applications. In this paper, we characterize the growth behavior of the objective function around the critical points of the QP-OC problem and demonstrate how such characterization can be used to obtain strong convergence rate results for iterative methods that exploit the manifold structure of the orthogonality constraint (i.e., the Stiefel manifold) to find a critical point of the problem. Specifically, our primary contribution is to show that the Łojasiewicz exponent at any critical point of the QP-OC problem is 1 / 2. Such a result is significant, as it expands the currently very limited repertoire of optimization problems for which the Łojasiewicz exponent is explicitly known. Moreover, it allows us to show, in a unified manner and for the first time, that a large family of retraction-based line-search methods will converge linearly to a critical point of the QP-OC problem. Then, as our secondary contribution, we propose a stochastic variance-reduced gradient (SVRG) method called Stiefel-SVRG for solving the QP-OC problem and present a novel Łojasiewicz inequality-based linear convergence analysis of the method. An important feature of Stiefel-SVRG is that it allows for general retractions and does not require the computation of any vector transport on the Stiefel manifold. As such, it is computationally more advantageous than other recently-proposed SVRG-type algorithms for manifold optimization.

Journal

Mathematical Programming – Springer Journals

Published: Jun 1, 2018

References

  • Steepest descent algorithms for optimization under unitary matrix constraint
    Abrudan, TE; Eriksson, J; Koivunen, V
  • Convergence of the iterates of descent methods for analytic cost functions
    Absil, P-A; Mahony, R; Andrews, B
  • Optimization Algorithms on Matrix Manifolds
    Absil, P-A; Mahony, R; Sepulchre, R
  • Projection-like retractions on matrix manifolds
    Absil, P-A; Malick, J
  • Learning sparsely used overcomplete dictionaries via alternating minimization
    Agarwal, A; Anandkumar, A; Jain, P; Netrapalli, P
  • Extrema of sums of heterogeneous quadratic forms
    Bolla, M; Michaletzky, G; Tusnády, G; Ziermann, M
  • Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity
    Bolte, J; Danilidis, A; Ley, O; Mazet, L
  • From error bounds to the complexity of first-order descent methods for convex functions
    Bolte, J; Ngyuen, TP; Peypouquet, J; Suter, BW
  • Stochastic gradient descent on Riemannian manifolds
    Bonnabel, S
  • Phase retrieval via Wirtinger flow: theory and algorithms
    Candès, EJ; Li, X; Soltanolkotabi, M
  • Perturbation analyses for the QR factorization
    Chang, X-W; Paige, CC; Stewart, GW
  • On smooth decompositions of matrices
    Dieci, L; Eirola, T
  • Convergence of neural networks for programming problems via a nonsmooth Łojasiewicz inequality
    Forti, M; Nistri, P; Quincampoix, M
  • Matrix Computations
    Golub, GH; Loan, CF
  • A framework of constraint preserving update schemes for optimization on stiefel manifold
    Jiang, B; Dai, Y-H
  • Empirical arithmetic averaging over the compact stiefel manifold
    Kaneko, T; Fiori, S; Tanaka, T
  • Trace optimization and eigenproblems in dimension reduction methods
    Kokiopoulou, E; Chen, J; Saad, Y
  • New fractional error bounds for polynomial systems with applications to Hölderian stability in optimization and spectral theory of tensors
    Li, G; Mordukhovich, BS; Phạm, TS
  • On the estimation performance and convergence rate of the generalized power method for phase synchronization
    Liu, H; Yue, M-C; So, AM-C
  • New error bounds and their applications to convergence analysis of iterative algorithms
    Luo, Z-Q
  • Error bounds for analytic systems and their applications
    Luo, Z-Q; Pang, J-S
  • High Performance Optimization, Volume 33 of Applied Optimization
    Luo, Z-Q; Sturm, JF
  • Error bounds and convergence analysis of feasible descent methods: a general approach
    Luo, Z-Q; Tseng, P
  • Optimization algorithms exploiting unitary constraints
    Manton, JH
  • Convergence to equilibrium for discretizations of gradient-like flows on Riemannian manifolds
    Merlet, B; Nguyen, TN
  • Introductory Lectures on Convex Optimization: A Basic Course
    Nesterov, Y
  • Phase retrieval using alternating minimization
    Netrapalli, P; Jain, P; Sanghavi, S
  • A Riemannian optimization approach to the matrix singular value decomposition
    Sato, H; Iwai, T
  • Convergence results for projected line-search methods on varieties of low-rank matrices via Łojasiewicz inequality
    Schneider, R; Uschmajew, A
  • A generalized solution of the orthogonal Procrustes problem
    Schönemann, PH
  • On two-sided orthogonal Procrustes problems
    Schönemann, PH
  • Hamiltonian and Gradient Flows, Algorithms and Control. Fields Institue Communications
    Smith, ST
  • Moment inequalities for sums of random matrices and their applications in optimization
    So, AM-C
  • Non-asymptotic convergence analysis of inexact gradient methods for machine learning without strong convexity
    So, AM-C; Zhou, Z
  • On perturbation bounds for the QR factorization
    Sun, J
  • Complete dictionary recovery over the sphere I: overview and the geometric picture
    Sun, J; Qu, Q; Wright, J
  • Complete dictionary recovery over the sphere II: recovery by Riemannian trust-region method
    Sun, J; Qu, Q; Wright, J
  • Guaranteed matrix completion via non-convex factorization
    Sun, R; Luo, Z-Q
  • Provable sparse tensor decomposition
    Sun, WW; Lu, J; Liu, H; Cheng, G
  • Convex Functions and Optimization Methods on Riemannian Manifolds, Volume 297 of Mathematics and Its Applications
    Udrişte, C
  • A new convergence proof for the higher-order power method and generalizations
    Uschmajew, A
  • A feasible method for optimization with orthogonality constraints
    Wen, Z; Yin, W
  • Globally convergent optimization algorithms on Riemannian manifolds: uniform framework for unconstrained and constrained optimization
    Yang, Y
  • Near-optimal bounds for phase synchronization
    Zhong, Y; Boumal, N
  • A unified approach to error bounds for structured convex optimization problems
    Zhou, Z; So, AM-C

You’re reading a free preview. Subscribe to read the entire article.

Try 2 weeks free now

DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

or browse the journals available

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create folders to
organize your research

Export folders, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off

Sign up
for free
Start 14 day
Free Trial