.. _sphx_glr_api_gallery_information_theory_entropy:

Entropy
-------

Measures of information: Hartley, Shannon, mutual information, and divergence.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Hartley measured the information in a message of L symbols from an alphabet of n as L\log n: the logarithm makes information add up when messages are concatenated. The plot compares alphabets by the bits each symbol carries, and the number of symbols needed to send one of a million equally likely messages.">

.. only:: html

  .. image:: /api/gallery/information_theory/entropy/images/thumb/sphx_glr_plot_01_hartley_information_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/entropy/plot_01_hartley_information`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Hartley's measure of information (1928)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Shannon replaced Hartley&#x27;s count of messages with an average over their probabilities, H = -\sum p\log_2 p: the expected surprise of the next symbol. The binary entropy function peaks at one bit for a fair coin, and the entropy of English letter frequencies (about 4.2 bits) falls short of Hartley&#x27;s \log_2 26 = 4.7.">

.. only:: html

  .. image:: /api/gallery/information_theory/entropy/images/thumb/sphx_glr_plot_02_shannon_entropy_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/entropy/plot_02_shannon_entropy`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Shannon entropy (1948)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Shannon measured what the output Y of a noisy channel tells about its input X by I(X;Y) = H(X) - H(X\mid Y), the uncertainty removed by observing the output. The information diagram shows the decomposition for a binary symmetric channel, and the curve shows I(X;Y) against the input bias for several noise levels.">

.. only:: html

  .. image:: /api/gallery/information_theory/entropy/images/thumb/sphx_glr_plot_03_mutual_information_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/entropy/plot_03_mutual_information`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Mutual information (1948)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Kullback and Leibler defined D(p\,\|\,q) = \sum p\log(p/q) as the mean information per observation for telling p from q. It is never negative (Gibbs&#x27; inequality) and vanishes only at p = q, but it is not symmetric. It is also the extra bits per symbol paid for coding a p-source with a code built for q.">

.. only:: html

  .. image:: /api/gallery/information_theory/entropy/images/thumb/sphx_glr_plot_04_kullback_leibler_divergence_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/entropy/plot_04_kullback_leibler_divergence`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">The Kullback-Leibler divergence (1951)</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /api/gallery/information_theory/entropy/plot_01_hartley_information
   /api/gallery/information_theory/entropy/plot_02_shannon_entropy
   /api/gallery/information_theory/entropy/plot_03_mutual_information
   /api/gallery/information_theory/entropy/plot_04_kullback_leibler_divergence