.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "api/gallery/optimization/momentum/plot_01_nesterov_acceleration.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_api_gallery_optimization_momentum_plot_01_nesterov_acceleration.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_api_gallery_optimization_momentum_plot_01_nesterov_acceleration.py:


Nesterov acceleration: O(1/k^2) versus O(1/k)
===================================================

On an ill-conditioned convex quadratic, plain gradient descent reduces
the objective gap like :math:`O(1/k)` until the strongly convex regime
kicks in; Nesterov's accelerated gradient, with the same step size,
achieves the optimal first-order rate :math:`O(1/k^2)`.

.. GENERATED FROM PYTHON SOURCE LINES 12-17

.. code-block:: Python

    import matplotlib.pyplot as plt
    import numpy as np

    from mathematicskit.optimization import GradientDescent, NesterovAcceleratedGradient








.. GENERATED FROM PYTHON SOURCE LINES 18-20

f(x) = (x^2 + 0.001 y^2) / 2, step 1/L = 1
-----------------------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 20-39

.. code-block:: Python


    d = np.array([1.0, 1e-3])
    f = lambda x: 0.5 * float(np.sum(d * x * x))
    grad = lambda x: d * x
    x0 = np.array([1.0, 1.0])

    gd = GradientDescent(alpha=1.0, tol=0.0, max_iter=2000).minimize(f, grad, x0)
    nag = NesterovAcceleratedGradient(alpha=1.0, tol=0.0, max_iter=2000).minimize(f, grad, x0)
    print(f"after 2000 iterations: gradient descent f = {gd.fun:.3e}, Nesterov f = {nag.fun:.3e}")

    k = np.arange(1, 2001)
    fig, ax = plt.subplots()
    ax.loglog(k, [f(x) for x in gd.path[1:]], label="gradient descent")
    ax.loglog(k, [f(x) for x in nag.path[1:]], label="Nesterov (1983)")
    ax.loglog(k, 2.0 * np.sum(x0**2) / (k + 1) ** 2, "k--", label=r"bound $2L\|x_0-x^*\|^2/(k+1)^2$")
    ax.set_xlabel("iteration k")
    ax.set_ylabel(r"$f(x_k) - f^*$")
    ax.legend()
    ax.set_title("Accelerated gradient descent")



.. image-sg:: /api/gallery/optimization/momentum/images/sphx_glr_plot_01_nesterov_acceleration_001.png
   :alt: Accelerated gradient descent
   :srcset: /api/gallery/optimization/momentum/images/sphx_glr_plot_01_nesterov_acceleration_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    after 2000 iterations: gradient descent f = 9.140e-06, Nesterov f = 6.232e-11

    Text(0.5, 1.0, 'Accelerated gradient descent')




.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.064 seconds)


.. _sphx_glr_download_api_gallery_optimization_momentum_plot_01_nesterov_acceleration.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_01_nesterov_acceleration.ipynb <plot_01_nesterov_acceleration.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_01_nesterov_acceleration.py <plot_01_nesterov_acceleration.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_01_nesterov_acceleration.zip <plot_01_nesterov_acceleration.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_