.. _sphx_glr_api_gallery_information_theory_channels:

Channels
--------

Channel capacity, the noisy-channel coding theorem, and rate-distortion.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A binary symmetric channel flips each bit with probability p. Shannon showed its capacity is C = 1 - h(p) bits per use, the largest mutual information over input distributions, reached by a fair coin. At p = 0.11 half of every bit survives; at p = 1/2 the output is independent of the input.">

.. only:: html

  .. image:: /api/gallery/information_theory/channels/images/thumb/sphx_glr_plot_01_binary_symmetric_channel_capacity_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/channels/plot_01_binary_symmetric_channel_capacity`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Capacity of the binary symmetric channel (1948)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Shannon proved that every rate below capacity can be sent with error probability as small as desired, and no rate above it can. His proof used random codes: draw 2^{nR} codewords at random and decode to the nearest one. Here the rate is fixed at R = 1/4 and the block length grows: below capacity the error falls toward zero, above capacity it rises toward one.">

.. only:: html

  .. image:: /api/gallery/information_theory/channels/images/thumb/sphx_glr_plot_02_noisy_channel_coding_theorem_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/channels/plot_02_noisy_channel_coding_theorem`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">The noisy-channel coding theorem (1948)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A channel of bandwidth B with Gaussian noise carries at most C = B\log_2(1 + S/N) bits per second. Capacity grows only logarithmically with power, and dividing by the bit rate gives the Shannon limit: no code works below E_b/N_0 = \ln 2, or -1.59 dB.">

.. only:: html

  .. image:: /api/gallery/information_theory/channels/images/thumb/sphx_glr_plot_03_shannon_hartley_theorem_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/channels/plot_03_shannon_hartley_theorem`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">The Shannon-Hartley theorem (1949)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="If some error is allowed, fewer bits are needed. Shannon&#x27;s rate-distortion function R(D) is the fewest bits per symbol that reproduce a source within average distortion D: for a fair coin with Hamming distortion R(D) = 1 - h(D), and for a Gaussian with squared error R(D) = \tfrac12\log_2(\sigma^2/D), 6.02 dB of quality per bit.">

.. only:: html

  .. image:: /api/gallery/information_theory/channels/images/thumb/sphx_glr_plot_04_rate_distortion_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/channels/plot_04_rate_distortion`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Rate-distortion theory (1959)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Most channels have no closed-form capacity. Blahut and Arimoto independently found an alternating maximization that converges to it, with upper and lower bounds that squeeze together at every step. Here it recovers 1 - h(p) for the binary symmetric channel and finds the capacity and skewed optimal input of the Z channel, and \log_2 3 for the noisy typewriter (send every other letter).">

.. only:: html

  .. image:: /api/gallery/information_theory/channels/images/thumb/sphx_glr_plot_05_blahut_arimoto_thumb.png
    :alt:

  :doc:`/api/gallery/information_theory/channels/plot_05_blahut_arimoto`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">The Blahut-Arimoto algorithm (1972)</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /api/gallery/information_theory/channels/plot_01_binary_symmetric_channel_capacity
   /api/gallery/information_theory/channels/plot_02_noisy_channel_coding_theorem
   /api/gallery/information_theory/channels/plot_03_shannon_hartley_theorem
   /api/gallery/information_theory/channels/plot_04_rate_distortion
   /api/gallery/information_theory/channels/plot_05_blahut_arimoto