.. _sphx_glr_api_gallery_optimization_momentum:

Momentum methods
----------------

Nesterov's accelerated gradient and the Adam optimizer.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="On an ill-conditioned convex quadratic, plain gradient descent reduces the objective gap like O(1/k) until the strongly convex regime kicks in; Nesterov&#x27;s accelerated gradient, with the same step size, achieves the optimal first-order rate O(1/k^2).">

.. only:: html

  .. image:: /api/gallery/optimization/momentum/images/thumb/sphx_glr_plot_01_nesterov_acceleration_thumb.png
    :alt:

  :doc:`/api/gallery/optimization/momentum/plot_01_nesterov_acceleration`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Nesterov acceleration: O(1/k^2) versus O(1/k)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A quadratic whose two coordinates differ in curvature by a factor of 1000. Fixed-step gradient descent must use a step small enough for the stiff coordinate, so it crawls along the flat one. Adam rescales each coordinate by its root-mean-square gradient and moves both at a similar speed.">

.. only:: html

  .. image:: /api/gallery/optimization/momentum/images/thumb/sphx_glr_plot_02_adam_thumb.png
    :alt:

  :doc:`/api/gallery/optimization/momentum/plot_02_adam`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Adam on a badly scaled problem</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /api/gallery/optimization/momentum/plot_01_nesterov_acceleration
   /api/gallery/optimization/momentum/plot_02_adam