.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "api/gallery/communications/information/plot_03_shannon_entropy.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_api_gallery_communications_information_plot_03_shannon_entropy.py>`
        to download the full example code or to run this example in your browser via JupyterLite.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_api_gallery_communications_information_plot_03_shannon_entropy.py:


Shannon's entropy and the binary symmetric channel
==================================================

Claude Shannon's "A mathematical theory of communication" (1948) measured
information by entropy, :math:`H = -\sum p\log_2 p`. For a binary source
:func:`~engineeringkit.communications.entropy_binary` peaks at one bit when
both symbols are equally likely. The same paper defined channel capacity. A
binary symmetric channel that flips each bit with probability :math:`p` has
capacity :math:`1 - H(p)`
(:func:`~engineeringkit.communications.binary_symmetric_capacity`): even a
very noisy channel can carry information reliably at a reduced rate, and a
channel with :math:`p = 1/2` carries nothing.

.. GENERATED FROM PYTHON SOURCE LINES 17-22

.. code-block:: Python

    import matplotlib.pyplot as plt
    import numpy as np

    from engineeringkit.communications import binary_symmetric_capacity, entropy_binary








.. GENERATED FROM PYTHON SOURCE LINES 23-27

Entropy as the limit of lossless compression
--------------------------------------------

The empirical entropy of a long biased bit stream matches :math:`H(p)`.

.. GENERATED FROM PYTHON SOURCE LINES 27-37

.. code-block:: Python


    rng = np.random.default_rng(1948)
    for p in [0.5, 0.1, 0.01]:
        bits = rng.random(200_000) < p
        blocks = bits[: bits.size // 8 * 8].reshape(-1, 8) @ (1 << np.arange(8))
        counts = np.bincount(blocks, minlength=256) / blocks.size
        empirical = -np.sum(counts[counts > 0] * np.log2(counts[counts > 0])) / 8
        print(f"p = {p:4}: H(p) = {entropy_binary(p):.4f} bit, empirical (8-bit blocks) = {empirical:.4f} bit")
        assert np.isclose(empirical, entropy_binary(p), atol=0.01)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    p =  0.5: H(p) = 1.0000 bit, empirical (8-bit blocks) = 0.9991 bit
    p =  0.1: H(p) = 0.4690 bit, empirical (8-bit blocks) = 0.4678 bit
    p = 0.01: H(p) = 0.0808 bit, empirical (8-bit blocks) = 0.0809 bit




.. GENERATED FROM PYTHON SOURCE LINES 38-43

Capacity of a bit-flipping channel
----------------------------------

Mutual information :math:`I = H(Y) - H(Y|X)` over input distributions
peaks at the uniform input, where it equals :math:`1 - H(p)`.

.. GENERATED FROM PYTHON SOURCE LINES 43-59

.. code-block:: Python


    p = np.linspace(0, 1, 201)
    q = np.linspace(0, 1, 201)
    crossover = 0.11
    mutual = entropy_binary(q * (1 - crossover) + (1 - q) * crossover) - entropy_binary(crossover)
    assert np.isclose(mutual.max(), binary_symmetric_capacity(crossover))
    fig, axes = plt.subplots(1, 3, figsize=(11, 3.3))
    axes[0].plot(p, entropy_binary(p))
    axes[0].set(xlabel="P(1)", ylabel="H [bit]", title="Binary entropy")
    axes[1].plot(q, mutual)
    axes[1].set(xlabel="input P(1)", ylabel="I(X;Y) [bit]", title=f"BSC, p = {crossover}")
    axes[2].plot(p, binary_symmetric_capacity(p))
    axes[2].set(xlabel="crossover probability p", ylabel="capacity [bit/use]", title="C = 1 - H(p)")
    fig.tight_layout()

    plt.show()



.. image-sg:: /api/gallery/communications/information/images/sphx_glr_plot_03_shannon_entropy_001.png
   :alt: Binary entropy, BSC, p = 0.11, C = 1 - H(p)
   :srcset: /api/gallery/communications/information/images/sphx_glr_plot_03_shannon_entropy_001.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.242 seconds)


.. _sphx_glr_download_api_gallery_communications_information_plot_03_shannon_entropy.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: lite-badge

      .. image:: images/jupyterlite_badge_logo.svg
        :target: ../../../../lite/lab/index.html?path=api/gallery/communications/information/plot_03_shannon_entropy.ipynb
        :alt: Launch JupyterLite
        :width: 150 px

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_03_shannon_entropy.ipynb <plot_03_shannon_entropy.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_03_shannon_entropy.py <plot_03_shannon_entropy.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_03_shannon_entropy.zip <plot_03_shannon_entropy.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_