Skip to main content
Ctrl+K
mathematicskit logo mathematicskit logo
  • Subpackages
  • History
  • Examples
  • API
  • GitHub
  • PyPI
  • Subpackages
  • History
  • Examples
  • API
  • GitHub
  • PyPI

Section Navigation

  • mathematicskit.abstract_algebra
    • Group actions
    • Character tables
    • Error-correcting codes
    • Finite fields
    • Gröbner bases
    • Groups
    • Homomorphisms
    • Polynomial ring arithmetic
    • Group structure
    • Subgroups and cosets
  • mathematicskit.calculus
    • Reverse-mode autodiff
    • Dual numbers
    • Method of exhaustion
    • Finite differences
    • Newton and Leibniz
    • Quadrature
    • Taylor series
  • mathematicskit.combinatorics
    • Counting
    • Combinatorial designs
    • Extremal combinatorics
    • Inclusion-exclusion
    • Matchings
    • Necklaces
    • Integer partitions
    • Pascal’s triangle
    • Integer sequences
    • Special numbers
    • Labeled trees
  • mathematicskit.complex_analysis
    • Conformal maps
    • Contour integrals
    • Domain coloring
    • Holomorphic functions
    • Residues
    • Series and singularities
  • mathematicskit.fractals_chaos
    • Box-counting dimension
    • Cellular automata
    • Fractal curves
    • Aggregation
    • Iterated function systems
    • Lindenmayer systems
    • Lyapunov exponents
    • Mandelbrot and Julia sets
    • Discrete maps
  • mathematicskit.geometry
    • Bézier curves
    • Convex hull
    • Curves and the Frenet-Serret frame
    • Distances between shapes
    • Enclosing circles
    • Point-in-polygon
    • Polygon area
    • Polyhedra
    • Proximity
    • Line simplification
    • Surfaces
    • Voronoi diagrams and Delaunay triangulation
  • mathematicskit.graph_theory
    • Assignment
    • Graph coloring
    • Connected components
    • Eulerian paths
    • Extremal graph theory
    • Hamiltonian cycles
    • Bipartite matching
    • Maximum flow
    • Ranking
    • Heuristic search
    • Shortest paths
    • Minimum spanning tree
    • Spectral graph theory
  • mathematicskit.information_theory
    • Channel codes
    • Channels
    • Entropy
    • Source coding
  • mathematicskit.linalg
    • Cholesky decomposition
    • Classical matrix theory
    • Eigenvalue algorithms
    • Iterative solvers
    • LU decomposition
    • QR decomposition
    • Numerical stability
    • Singular value decomposition
  • mathematicskit.number_theory
    • Continued fractions
    • Chinese Remainder Theorem
    • Diophantine equations
    • Elliptic curves
    • Integer factorization
    • Lattice reduction
    • Modular arithmetic
    • Primality
    • Distribution of primes
    • Quadratic residues
    • Sums of squares
    • Multiplicative functions
    • The zeta function
  • mathematicskit.numerical_analysis
    • Function approximation
    • Runge’s phenomenon and Chebyshev nodes
    • Floating-point arithmetic
    • Interpolation
    • Polynomials
    • Least-squares regression
    • Root finding
    • Cubic splines
  • mathematicskit.ode_dynamics
    • Bifurcation normal forms
    • Chaotic flows
    • Epidemic models
    • Excitable systems
    • Numerical integration
    • Limit cycles
    • The logistic map
    • Phase portraits
    • Poincare sections
    • Population models
    • Fixed-point stability
    • Stiffness and boundary-value problems
    • Synchronization
  • mathematicskit.optimization
    • Simulated annealing
    • Nonlinear conjugate gradient
    • Constrained optimization
    • Direct search
    • Dynamic programming
    • Game theory
    • Gradient descent
    • Nonlinear least squares
    • Linear programming
    • Momentum methods
    • Newton’s method and BFGS
    • One-dimensional search
    • Stochastic approximation
    • Benchmark functions
    • The travelling salesman problem
  • mathematicskit.pde
    • Advection schemes
    • Conservation laws and shocks
    • Poisson and Laplace equations
    • The heat equation
    • Spectral methods
    • Stability: CFL and von Neumann
    • The wave equation
  • mathematicskit.probability
    • Kolmogorov’s axioms
    • Bayesian inference
    • Branching processes
    • Classical problems
    • Continuous distributions
    • Discrete distributions
    • Limit theorems and inequalities
    • Markov chains
    • Markov chain Monte Carlo
    • Monte Carlo integration
    • Queueing theory
    • Stochastic differential equations
    • Random walks and Brownian motion
  • mathematicskit.special_functions
    • Airy functions
    • Bessel functions
    • Convolution
    • Elliptic functions
    • Error function and Fresnel integrals
    • Digital filters
    • Discrete Fourier transform
    • Gamma and beta functions
    • Hypergeometric functions
    • Lambert W function
    • Laplace transform
    • Mathieu functions
    • Orthogonal polynomials
    • Wavelets
    • Z-transform
    • Riemann zeta function
  • mathematicskit.statistics
    • Bootstrap resampling
    • Confidence intervals
    • Correlation
    • Descriptive statistics
    • Hypothesis tests
    • Likelihood
    • Multiple testing
    • Nonparametric tests
    • Principal component analysis
    • Linear regression
    • Shrinkage estimation
    • Time series
  • mathematicskit.topology
    • Complexes
    • Curves
    • Topological data analysis
    • Fixed points
    • Homology
    • Surfaces
  • Examples
  • Examples
  • Entropy

Entropy#

Measures of information: Hartley, Shannon, mutual information, and divergence.

Hartley’s measure of information (1928)

Hartley's measure of information (1928)

Shannon entropy (1948)

Shannon entropy (1948)

Mutual information (1948)

Mutual information (1948)

The Kullback-Leibler divergence (1951)

The Kullback-Leibler divergence (1951)

previous

The Blahut-Arimoto algorithm (1972)

next

Hartley’s measure of information (1928)
Show Source

© Copyright 2026, mathematicskit contributors.

Created using Sphinx 9.1.0.

Built with the PyData Sphinx Theme 0.22.0.