.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "api/gallery/information_theory/entropy/plot_02_shannon_entropy.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_api_gallery_information_theory_entropy_plot_02_shannon_entropy.py>`
        to download the full example code or to run this example in your browser via JupyterLite.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_api_gallery_information_theory_entropy_plot_02_shannon_entropy.py:


Shannon entropy (1948)
======================

Shannon replaced Hartley's count of messages with an average over their
probabilities, :math:`H = -\sum p\log_2 p`: the expected surprise of the
next symbol. The binary entropy function peaks at one bit for a fair
coin, and the entropy of English letter frequencies (about 4.2 bits)
falls short of Hartley's :math:`\log_2 26 = 4.7`.

.. GENERATED FROM PYTHON SOURCE LINES 13-22

.. code-block:: Python

    import matplotlib.pyplot as plt
    import numpy as np

    from mathematicskit.information_theory import binary_entropy, entropy, hartley_information

    # English letter frequencies (percent), A-Z.
    english = np.array([8.2, 1.5, 2.8, 4.3, 12.7, 2.2, 2.0, 6.1, 7.0, 0.15, 0.77, 4.0, 2.4, 6.7, 7.5, 1.9, 0.095, 6.0, 6.3, 9.1, 2.8, 0.98, 2.4, 0.15, 2.0, 0.074])
    print(f"English letters: H = {entropy(english):.3f} bits; uniform: {hartley_information(26):.3f} bits")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    English letters: H = 4.176 bits; uniform: 4.700 bits




.. GENERATED FROM PYTHON SOURCE LINES 23-35

.. code-block:: Python

    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4))
    p = np.linspace(0, 1, 401)
    ax1.plot(p, binary_entropy(p))
    ax1.set_xlabel("p")
    ax1.set_ylabel("h(p) (bits)")
    ax1.set_title("Binary entropy function")

    order = np.argsort(english)[::-1]
    ax2.bar([chr(65 + i) for i in order], english[order] / english.sum())
    ax2.axhline(1 / 26, color="k", ls="--", lw=1, label="uniform")
    ax2.set_title(f"English letters: H = {entropy(english):.2f} bits < log2 26 = 4.70")
    ax2.legend()



.. image-sg:: /api/gallery/information_theory/entropy/images/sphx_glr_plot_02_shannon_entropy_001.png
   :alt: Binary entropy function, English letters: H = 4.18 bits < log2 26 = 4.70
   :srcset: /api/gallery/information_theory/entropy/images/sphx_glr_plot_02_shannon_entropy_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    <matplotlib.legend.Legend object at 0x113531e80>




.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.055 seconds)


.. _sphx_glr_download_api_gallery_information_theory_entropy_plot_02_shannon_entropy.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: lite-badge

      .. image:: images/jupyterlite_badge_logo.svg
        :target: ../../../../lite/lab/index.html?path=api/gallery/information_theory/entropy/plot_02_shannon_entropy.ipynb
        :alt: Launch JupyterLite
        :width: 150 px

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_02_shannon_entropy.ipynb <plot_02_shannon_entropy.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_02_shannon_entropy.py <plot_02_shannon_entropy.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_02_shannon_entropy.zip <plot_02_shannon_entropy.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_