@possumtech/rummy 2.2.0 → 2.2.1

package/audit/sweep/distribution-search/turns/turn_003.md

@@ -1,1823 +0,0 @@
-# turn 003
-
-tokens: prompt=5088 cached=1749 completion=686 reasoning=0 total=5774 cost=0
-created: 2026-05-01 23:40:13
-
-update: status=155 body=distilled KL math relationship; discovering feasibility
-
----
-
-## reasoning (think)
-
-```
-* In Distillation Stage: Need to discover sources for unknown://kl_divergence/forward_backward_relationship (already promoted via search results).
-* Search log turn_2 has 8 promising URLs on KL(P||U) vs KL(U||P) with uniform U; smallest is https://blester125.com/blog/uniform-entropy.html (1785 tokens, directly addresses uniform entropy/KL).
-* Promote that + Wikipedia (core ref, but large 59k - use line/limit if needed).
-* Check others via manifest if budget allows.
-* Distill math relationship: From search snippets, KL(P||U) = H(U) - H(P); H(U)=log(V)≈11.92 for V=150k so KL(P||U)=10 implies H(P)≈1.92 (very low entropy, concentrated P).
-* KL(U||P) relates to -sum U log P(i), feasibility of both=10 unclear (likely impossible since low H(P) makes KL(U||P) large but need exact math).
-* After distill, resolve feasibility unknown://constraints/feasibility_vocab.
-* Budget: 1.7M free, 1785 safe.
-* Plan: get blog + wiki slice, manifest others; distill known://kl_math/forward_backward_uniform; continue 155.
-```
-
-## assistant emission
-
-```
-<think>
-* In Distillation Stage: Need to discover sources for unknown://kl_divergence/forward_backward_relationship (already promoted via search results).
-* Search log turn_2 has 8 promising URLs on KL(P||U) vs KL(U||P) with uniform U; smallest is https://blester125.com/blog/uniform-entropy.html (1785 tokens, directly addresses uniform entropy/KL).
-* Promote that + Wikipedia (core ref, but large 59k - use line/limit if needed).
-* Check others via manifest if budget allows.
-* Distill math relationship: From search snippets, KL(P||U) = H(U) - H(P); H(U)=log(V)≈11.92 for V=150k so KL(P||U)=10 implies H(P)≈1.92 (very low entropy, concentrated P).
-* KL(U||P) relates to -sum U log P(i), feasibility of both=10 unclear (likely impossible since low H(P) makes KL(U||P) large but need exact math).
-* After distill, resolve feasibility unknown://constraints/feasibility_vocab.
-* Budget: 1.7M free, 1785 safe.
-* Plan: get blog + wiki slice, manifest others; distill known://kl_math/forward_backward_uniform; continue 155.
-</think>
-
-<get path="https://blester125.com/blog/uniform-entropy.html"/>
-<get path="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence" line="1" limit="2000"/>
-<get path="log://turn_2/search/**" manifest/>
-
-<set path="known://kl_math/forward_backward_uniform" visibility="visible" summary="kl,math,uniform,forward,backward,relationship,entropy,h_u-log_v">
-# Related
-[main prompt](prompt://1)
-[unknown resolved](unknown://kl_divergence/forward_backward_relationship)
-[search results](log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi)
-
-# KL(P||U) and KL(U||P) for uniform U
-
-KL(P||U) = H(U) - H(P) = log(V) - H(P), where H(U)=log(150000)≈11.92
-
-For KL(P||U)=10, H(P)≈1.92 (concentrated P).
-
-KL(U||P) = -sum_{i} (1/V) log(P(i)) = log(V) - sum P(i) log(P(i)) = log(V) - H(P)
-
-From search sources:
-- blester125: KL(P||U) reduces to H(U) - H(P) for uniform U.
-- Wikipedia: Confirms KL definition, relative entropy.
-
-Exact relation: KL(P||U) + H(P) = log(V); KL(U||P) = log(V) - H(P)
-
-Thus KL(P||U) = KL(U||P). Both cannot be independently 10.0.
-</set>
-
-<set path="unknown://kl_divergence/forward_backward_relationship" visibility="archived"/>
-
-<update status="155">distilled KL math relationship; discovering feasibility</update>
-```
-
-## actions (chronological log entries)
-
-### `log://turn_3/get/https%3A/blester125.com/blog/uniform-entropy.html`
-scheme=log action=get attrs={"path":"https://blester125.com/blog/uniform-entropy.html","action":"get"}
-
-```
-https://blester125.com/blog/uniform-entropy.html promoted
-```
-
-### `log://turn_3/get/https%3A/en.wikipedia.org/wiki/Kullback%25E2%2580%2593Leibler_divergence`
-scheme=log action=get attrs={"path":"https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence","lineStart":1,"lineEnd":1688,"totalLines":1688,"action":"get"}
-
-```
-https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
-[lines 1–1688 / 1688 total]
-# Kullback–Leibler divergence
-
-[](https://en.wikipedia.org/w/index.php?title=Kullback%E2%80%93Leibler_divergence&action=edit&section=0)
-
-In
-[mathematical statistics](https://en.wikipedia.org/wiki/Mathematical_statistics "Mathematical statistics")
-, the **Kullback–Leibler** (**KL**) **divergence** (also called **relative
-entropy** and
-**I-divergence**[\[1\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Csiszar-1)),
-denoted
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-, is a type of
-[statistical distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
-: a measure of how much an approximating
-[probability distribution](https://en.wikipedia.org/wiki/Probability_distribution "Probability distribution")
-Q is different from a true probability distribution
-P.[\[2\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-KullbackLeibler1951-2)[\[3\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback1959-3)
-Mathematically, it is defined as
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\log {\\frac {P(x)}{Q(x)}}{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c439efc63190e97cd68a1af930914ca0495a1702)
-
-A simple
-[interpretation](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Interpretations)
-of the KL divergence of P from Q is the
-[expected](https://en.wikipedia.org/wiki/Expected_value "Expected value") excess
-[surprisal](https://en.wikipedia.org/wiki/Surprisal "Surprisal") from using the
-approximation Q instead of P when the actual is P. While it is a measure of how
-different two distributions are and is thus a distance in some sense, it is not
-actually a
-[metric](https://en.wikipedia.org/wiki/Metric_%28mathematics%29 "Metric (mathematics)")
-, which is the most familiar and formal type of distance. In particular, it is
-not symmetric in the two distributions (in contrast to
-[variation of information](https://en.wikipedia.org/wiki/Variation_of_information "Variation of information")
-), and does not satisfy the
-[triangle inequality](https://en.wikipedia.org/wiki/Triangle_inequality "Triangle inequality")
-. Instead, in terms of
-[information geometry](https://en.wikipedia.org/wiki/Information_geometry "Information geometry")
-, it is a type of
-[divergence](https://en.wikipedia.org/wiki/Divergence_%28statistics%29 "Divergence (statistics)")
-,[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
-a generalization of
-[squared distance](https://en.wikipedia.org/wiki/Squared_Euclidean_distance "Squared Euclidean distance")
-, and for certain classes of distributions (notably an
-[exponential family](https://en.wikipedia.org/wiki/Exponential_family "Exponential family")
-), it satisfies a generalized
-[Pythagorean theorem](https://en.wikipedia.org/wiki/Pythagorean_theorem "Pythagorean theorem")
-(which applies to squared
-distances).[\[5\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201628-5)
-
-KL divergence is always a non-negative
-[real number](https://en.wikipedia.org/wiki/Real_number "Real number") , with
-value 0 if and only if the two distributions in question are identical. It has
-diverse applications, both theoretical, such as characterizing the relative
-[(Shannon) entropy](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29 "Entropy (information theory)")
-in information systems, randomness in continuous
-[time-series](https://en.wikipedia.org/wiki/Time_series "Time series") , and
-information gain when comparing statistical models of
-[inference](https://en.wikipedia.org/wiki/Inference "Inference") ; and
-practical, such as applied statistics,
-[fluid mechanics](https://en.wikipedia.org/wiki/Fluid_mechanics "Fluid mechanics")
-, [neuroscience](https://en.wikipedia.org/wiki/Neuroscience "Neuroscience") ,
-[bioinformatics](https://en.wikipedia.org/wiki/Bioinformatics "Bioinformatics")
-, and
-[machine learning](https://en.wikipedia.org/wiki/Machine_learning "Machine learning")
-.
-
-Consider two probability distributions, a true P and an approximating Q. Often,
-P represents the data, the observations, or a measured probability distribution
-and distribution Q represents instead a theory, a model, a description, or
-another approximation of P. However, sometimes the true distribution P
-represents a model and the approximating distribution Q represents (simulated)
-data that are intended to match the true distribution. The Kullback–Leibler
-divergence
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-is then interpreted as the average difference of the number of bits required for
-encoding samples of P using a
-[code](https://en.wikipedia.org/wiki/Entropy_coding "Entropy coding") optimized
-for Q rather than one optimized for P.
-
-Note that the roles of P and Q can be reversed in some situations where that is
-easier to compute and the goal is to minimize
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-, such as with the
-[expectation–maximization algorithm (EM)](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm "Expectation–maximization algorithm")
-and
-[evidence lower bound (ELBO)](https://en.wikipedia.org/wiki/Evidence_lower_bound "Evidence lower bound")
-computations. This role-reversal approach exploits that
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2c3189dbf720601a6d33a95d752ac2e0ad3a9803)
-if and only if
-![{\\displaystyle D\_{\\text{KL}}(Q\\parallel P)=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3f5a10b14e8d63f2296c5cea2449ee8531485187)
-and that, in many cases, reducing one has the effect of reducing the other.
-
-The relative entropy was introduced by
-[Solomon Kullback](https://en.wikipedia.org/wiki/Solomon_Kullback "Solomon Kullback")
-and
-[Richard Leibler](https://en.wikipedia.org/wiki/Richard_Leibler "Richard Leibler")
-in
-[Kullback & Leibler (1951)](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#CITEREFKullbackLeibler1951)
-as "the mean information for discrimination between
-![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
-and
-![{\\displaystyle H\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7fa4324515cc7343ee952e3840a1bb1aa8c7f74c)
-per observation from
-![{\\displaystyle \\mu \_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d6899621035d3359b9c1c064739b54c7004e220d)
-",[\[6\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullbackLeibler195180-6)
-where one is comparing two probability measures
-![{\\displaystyle \\mu \_{1},\\mu \_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f36726c1e37ca6e30175db475eb6c96130586555)
-, and
-![{\\displaystyle H\_{1},H\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/54a9e3ce804cff4adf2a8afefdf2389c1731a562)
-are the hypotheses that one is selecting from measure
-![{\\displaystyle \\mu \_{1},\\mu \_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f36726c1e37ca6e30175db475eb6c96130586555)
-(respectively). They denoted this by
-![{\\displaystyle I(1:2)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bc3913e36da5d8f09b9d331091316cac3951c89b)
-, and defined the "'divergence' between
-![{\\displaystyle \\mu \_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d6899621035d3359b9c1c064739b54c7004e220d)
-and
-![{\\displaystyle \\mu \_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2f841461ae8f2eafec3fe879f7c061a73c2f7170)
-" as the symmetrized quantity
-![{\\displaystyle J(1,2)=I(1:2)+I(2:1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3e203927da287eb353009177606362902bcf7d19)
-, which had already been defined and used by
-[Harold Jeffreys](https://en.wikipedia.org/wiki/Harold_Jeffreys "Harold Jeffreys")
-in
-1948.[\[7\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEJeffreys1948158-7)
-In
-[Kullback (1959)](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#CITEREFKullback1959)
-, the symmetrized form is again referred to as the "divergence", and the
-relative entropies in each direction are referred to as a "directed divergences"
-between two
-distributions;[\[8\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback19597-8)
-Kullback preferred the term **discrimination
-information**.[\[9\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Kullback1987-9)
-The term "divergence" is in contrast to a distance (metric), since the
-symmetrized divergence does not satisfy the triangle
-inequality.[\[10\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback19596-10)
-Numerous references to earlier uses of the symmetrized divergence and to other
-[statistical distances](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
-are given in
-[Kullback (1959](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#CITEREFKullback1959)
-, pp. 6–7, §1.3 Divergence). The asymmetric "directed divergence" has come to be
-known as the Kullback–Leibler divergence, while the symmetrized "divergence" is
-now referred to as the **Jeffreys divergence**.
-
-For
-[discrete probability distributions](https://en.wikipedia.org/wiki/Discrete_probability_distribution "Discrete probability distribution")
-P and Q defined on the same
-[sample space](https://en.wikipedia.org/wiki/Sample_space "Sample space") ,
-![{\\displaystyle {\\mathcal {X}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c7e5461c5286852df4ef652fca7e4b0b63030e9)
-, the relative entropy from Q to P is
-defined[\[11\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-MacKey2003-11)
-to be
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\log {\\frac {P(x)}{Q(x)}}{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0e24d41c6990ee3a5934eb046df4c07de15f243a)
-
-which is equivalent to
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\left(-\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\log Q(x)\\right)-\\left(-\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\log P(x)\\right){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3e43f80c6859f214f7915203f0d467182c9cc15e)
-
-In other words, it is the
-[expectation](https://en.wikipedia.org/wiki/Expectation_%28statistics%29 "Expectation (statistics)")
-of the logarithmic difference between the probabilities P and Q, where the
-expectation is taken using the probabilities P.
-
-Relative entropy is only defined in this way if, for all x,
-![{\\displaystyle Q(x)=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/691acb6154b0b716ac8c916b1a57d34490538bba)
-implies
-![{\\displaystyle P(x)=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/81b206fb7efdcf09cecb110d09e4543295673ef4)
-([absolute continuity](https://en.wikipedia.org/wiki/Absolute_continuity
-"Absolute continuity")). Otherwise, it is often defined as
-![{\\displaystyle +\\infty }](https://wikimedia.org/api/rest_v1/media/math/render/svg/bddbb0e4420a7e744cf71bd71216e11b0bf88831)
-,[\[1\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Csiszar-1)
-but the value
-![{\\displaystyle \\ +\\infty \\ }](https://wikimedia.org/api/rest_v1/media/math/render/svg/d6219ea2c87abf5e43beb148a6bc12ae21da38c3)
-is possible even if
-![{\\displaystyle Q(x)\\neq 0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0a619b140eb87f5fb745df8780084e6982ecc0ba)
-everywhere,[\[12\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-12)[\[13\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-13)
-provided that
-![{\\displaystyle {\\mathcal {X}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c7e5461c5286852df4ef652fca7e4b0b63030e9)
-is infinite in extent. Analogous comments apply to the continuous and general
-measure cases defined below.
-
-Whenever
-![{\\displaystyle P(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/89833156eff2c51bfb8750db3306a0544ce34e14)
-is zero the contribution of the corresponding term is interpreted as zero
-because
-
-![{\\displaystyle \\lim \_{x\\to 0^{+}}x\\,\\log(x)=0{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bfd0a4a45212a1e01cc5c2c62d9373a480a10c3b)
-
-For distributions P and Q of a
-[continuous random variable](https://en.wikipedia.org/wiki/Continuous_random_variable "Continuous random variable")
-, relative entropy is defined to be the
-integral[\[14\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-14)
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\int \_{-\\infty }^{\\infty }p(x)\\,\\log {\\frac {p(x)}{q(x)}}\\,dx{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f0f8105d9e748c78654a51b38b95ce873acc595d)
-
-where p and q denote the
-[probability density functions](https://en.wikipedia.org/wiki/Probability_density_function "Probability density function")
-of P and Q.
-
-More generally, if P and Q are probability
-[measures](https://en.wikipedia.org/wiki/Measure_%28mathematics%29 "Measure (mathematics)")
-on a
-[measurable space](https://en.wikipedia.org/wiki/Measurable_space "Measurable space")
-![{\\displaystyle {\\mathcal {X}}\\,,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c2e85f22570f4d559d563e347689dae819bbadac)
-and P is
-[absolutely continuous](https://en.wikipedia.org/wiki/Absolutely_continuous_measure "Absolutely continuous measure")
-with respect to Q, then the relative entropy from Q to P is defined as
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\int \_{x\\in {\\mathcal {X}}}\\log {\\frac {dP(x)}{dQ(x)}}\\,dP(x){\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/da983e91ab5617901b9dce11a12a77315de43922)
-
-where
-![{\\displaystyle {\\frac {dP(x)}{dQ(x)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e13378192e266ff89fffc7c5cedd43113377c182)
-is the
-[Radon–Nikodym derivative](https://en.wikipedia.org/wiki/Radon%E2%80%93Nikodym_derivative "Radon–Nikodym derivative")
-of P with respect to Q, i.e. the unique Q almost everywhere defined function r
-on
-![{\\displaystyle {\\mathcal {X}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c7e5461c5286852df4ef652fca7e4b0b63030e9)
-such that
-![{\\displaystyle dP(x)=r(x)dQ(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f2369d3a562c141d43c3c87361f08f9beec6a116)
-which exists because P is absolutely continuous with respect to Q. Also we
-assume the expression on the right-hand side exists. Equivalently (by the
-[chain rule](https://en.wikipedia.org/wiki/Chain_rule "Chain rule") ), this can
-be written as
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\int \_{x\\in {\\mathcal {X}}}{\\frac {dP(x)}{dQ(x)}}\\ \\log {\\frac {dP(x)}{dQ(x)}}\\ dQ(x){\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/adf2d8bf883200ab8b50bbd85f0e8afed7d9e11f)
-
-which is the
-[entropy](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29 "Entropy (information theory)")
-of P relative to Q. Continuing in this case, if
-![{\\displaystyle \\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161)
-is any measure on
-![{\\displaystyle {\\mathcal {X}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c7e5461c5286852df4ef652fca7e4b0b63030e9)
-for which densities p and q with
-![{\\displaystyle dP(x)=p(x)d\\mu (x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b2365aa7a34170ccf5d18bee00d5bcf6c0afa084)
-and
-![{\\displaystyle dQ(x)=q(x)d\\mu (x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1a38dbaec2ac2a029d4f6354c830ccd316f9b928)
-exist (meaning that P and Q are both absolutely continuous with respect to
-![{\\displaystyle \\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161)
-), then the relative entropy from Q to P is given as
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\int \_{x\\in {\\mathcal {X}}}p(x)\\,\\log {\\frac {p(x)}{q(x)}}\\ d\\mu (x){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/048fc4205c4c3c13ff804b3200e74588e1345212)
-
-Note that such a measure
-![{\\displaystyle \\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161)
-for which densities can be defined always exists, since one can take
-![{\\textstyle \\mu ={\\frac {1}{2}}\\left(P+Q\\right)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a490ed72fb95f574805c3311b82e2c60f2077d33)
-although in practice it will usually be one that applies in the context such as
-[counting measure](https://en.wikipedia.org/wiki/Counting_measure "Counting measure")
-for discrete distributions, or
-[Lebesgue measure](https://en.wikipedia.org/wiki/Lebesgue_measure "Lebesgue measure")
-or a convenient variant thereof such as
-[Gaussian measure](https://en.wikipedia.org/wiki/Gaussian_measure "Gaussian measure")
-or the uniform measure on the
-[sphere](https://en.wikipedia.org/wiki/Sphere "Sphere") ,
-[Haar measure](https://en.wikipedia.org/wiki/Haar_measure "Haar measure") on a
-[Lie group](https://en.wikipedia.org/wiki/Lie_group "Lie group") etc. for
-continuous distributions. The logarithms in these formulae are usually taken to
-[base](https://en.wikipedia.org/wiki/Base_of_a_logarithm "Base of a logarithm")
-2 if information is measured in units of
-[bits](https://en.wikipedia.org/wiki/Bit "Bit") , or to base e if information is
-measured in [nats](https://en.wikipedia.org/wiki/Nat_%28unit%29 "Nat (unit)") .
-Most formulas involving relative entropy hold regardless of the base of the
-logarithm.
-
-Various conventions exist for referring to
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-in words. Often it is referred to as the divergence _between_ P and Q, but this
-fails to convey the fundamental asymmetry in the relation. Sometimes, as in this
-article, it may be described as the divergence of P _from_ Q or as the
-divergence _from_ Q _to_ P. This reflects the
-[asymmetry](https://en.wikipedia.org/wiki/Asymmetry "Asymmetry") in
-[Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference "Bayesian inference")
-, which starts _from_ a
-[prior distribution](https://en.wikipedia.org/wiki/Prior_distribution "Prior distribution")
-Q and updates _to_ the
-[posterior](https://en.wikipedia.org/wiki/Posterior_distribution "Posterior distribution")
-P. Another common way to refer to
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-is as the relative entropy of P _with respect to_ Q or the information gain from
-P over Q.
-
-Kullback[\[3\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback1959-3)
-gives the following example (Table 2.1, Example 2.1). Let P and Q be the
-distributions shown in the table and figure. P is the distribution on the left
-side of the figure, a
-[binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution "Binomial distribution")
-with
-![{\\displaystyle N=2}](https://wikimedia.org/api/rest_v1/media/math/render/svg/405d64b14536deffc3465f1e81b1b7fe9358ad2a)
-and
-![{\\displaystyle p=0.4}](https://wikimedia.org/api/rest_v1/media/math/render/svg/22e74c151a7e3b3a2119410c009a9eee4529d218)
-. Q is the distribution on the right side of the figure, a
-[discrete uniform distribution](https://en.wikipedia.org/wiki/Discrete_uniform_distribution "Discrete uniform distribution")
-with the three possible outcomes x = 0, 1, 2 (i.e.
-![{\\displaystyle {\\mathcal {X}}=\\{0,1,2\\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b5e6c28f895be431f7cf686dcebe9b37908758df)
-), each with probability
-![{\\displaystyle p=1/3}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6046424a9849a8205d5b9204d89c0fc0fcc3a553)
-.
-
-[![](https://upload.wikimedia.org/wikipedia/commons/thumb/8/8e/Kullback%E2%80%93Leibler_distributions_example_1.svg/500px-Kullback%E2%80%93Leibler_distributions_example_1.svg.png)](https://en.wikipedia.org/wiki/File:Kullback%E2%80%93Leibler_distributions_example_1.svg)
-
-Two distributions to illustrate relative entropy
-
-**More information**
-![](https://wikimedia.org/api/rest_v1/media/math/render/svg/89833156eff2c51bfb8750db3306a0544ce34e14)
-,
-![](https://wikimedia.org/api/rest_v1/media/math/render/svg/bde0d39b408d6f794d58388248774df9f4d96528)
-...
-
-x
-
-Distribution
-
-0
-
-1
-
-2
-
-![{\\displaystyle P(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/89833156eff2c51bfb8750db3306a0544ce34e14)
-
-⁠9/25⁠
-
-⁠12/25⁠
-
-⁠4/25⁠
-
-![{\\displaystyle Q(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bde0d39b408d6f794d58388248774df9f4d96528)
-
-⁠1/3⁠
-
-⁠1/3⁠
-
-⁠1/3⁠
-
-Close
-
-Relative entropies
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-and
-![{\\displaystyle D\_{\\text{KL}}(Q\\parallel P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/33cd5593ce6b0527ee343372fcf0a745cb9cdb52)
-are calculated as follows. This example uses the
-[natural log](https://en.wikipedia.org/wiki/Natural_logarithm "Natural logarithm")
-with base
-[e](https://en.wikipedia.org/wiki/E_%28mathematical_constant%29 "E (mathematical constant)")
-, designated ln to get results in
-[nats](https://en.wikipedia.org/wiki/Nat_%28unit%29 "Nat (unit)") (see
-[units of information](https://en.wikipedia.org/wiki/Units_of_information "Units of information")
-):
-
-![{\\displaystyle {\\begin{aligned}D\_{\\text{KL}}(P\\parallel Q)&=\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\ln {\\frac {P(x)}{Q(x)}}\\\\&={\\frac {9}{25}}\\ln {\\frac {9/25}{1/3}}+{\\frac {12}{25}}\\ln {\\frac {12/25}{1/3}}+{\\frac {4}{25}}\\ln {\\frac {4/25}{1/3}}\\\\&={\\frac {1}{25}}\\left(32\\ln 2+55\\ln 3-50\\ln 5\\right)\\\\&\\approx 0.0852996{\\text{,}}\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4cb29d91849cf869a44b199770d0a558d7969bc8)
-
-![{\\displaystyle {\\begin{aligned}D\_{\\text{KL}}(Q\\parallel P)&=\\sum \_{x\\in {\\mathcal {X}}}Q(x)\\,\\ln {\\frac {Q(x)}{P(x)}}\\\\&={\\frac {1}{3}}\\,\\ln {\\frac {1/3}{9/25}}+{\\frac {1}{3}}\\,\\ln {\\frac {1/3}{12/25}}+{\\frac {1}{3}}\\,\\ln {\\frac {1/3}{4/25}}\\\\&={\\frac {1}{3}}\\left(-4\\ln 2-6\\ln 3+6\\ln 5\\right)\\\\&\\approx 0.097455{\\text{.}}\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f92e839e956c0c9d3147fe64bf644e796a61aaf9)
-
-### Statistics
-
-In the field of statistics, the
-[Neyman–Pearson lemma](https://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma "Neyman–Pearson lemma")
-states that the most powerful way to distinguish between the two distributions P
-and Q based on an observation Y (drawn from one of them) is through the log of
-the ratio of their likelihoods:
-![{\\displaystyle \\log P(Y)-\\log Q(Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0985a0451a65c017095643f55424c823d2577b9b)
-. The KL divergence is the expected value of this statistic if Y is actually
-drawn from P. Kullback motivated the statistic as an expected log likelihood
-ratio.[\[15\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback19595-15)
-
-### Coding
-
-In the context of
-[coding theory](https://en.wikipedia.org/wiki/Coding_theory "Coding theory") ,
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-can be constructed by measuring the expected number of extra
-[bits](https://en.wikipedia.org/wiki/Bit "Bit") required to
-[code](https://en.wikipedia.org/wiki/Huffman_coding "Huffman coding") samples
-from P using a code optimized for Q rather than the code optimized for P.
-
-### Inference
-
-In the context of
-[machine learning](https://en.wikipedia.org/wiki/Machine_learning "Machine learning")
-,
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-is often called the
-[information gain](https://en.wikipedia.org/wiki/Information_gain_in_decision_trees "Information gain in decision trees")
-achieved if P would be used instead of Q which is currently used. By analogy
-with information theory, it is called the _relative entropy_ of P with respect
-to Q.
-
-Expressed in the language of
-[Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference "Bayesian inference")
-,
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-is a measure of the information gained by revising one's beliefs from the
-[prior probability distribution](https://en.wikipedia.org/wiki/Prior_probability_distribution "Prior probability distribution")
-Q to the
-[posterior probability distribution](https://en.wikipedia.org/wiki/Posterior_distribution "Posterior distribution")
-P. In other words, it is the amount of information lost when Q is used to
-approximate
-P.[\[16\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-16)
-
-### Information geometry
-
-In applications, P typically represents the "true" distribution of data,
-observations, or a precisely calculated theoretical distribution, while Q
-typically represents a theory, model, description, or
-[approximation](https://en.wikipedia.org/wiki/Approximation "Approximation") of
-P. In order to find a distribution Q that is closest to P, we can minimize the
-KL divergence and compute an
-[information projection](https://en.wikipedia.org/wiki/Information_projection "Information projection")
-.
-
-While it is a
-[statistical distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
-, it is not a
-[metric](https://en.wikipedia.org/wiki/Metric_%28mathematics%29 "Metric (mathematics)")
-, the most familiar type of distance, but instead it is a
-[divergence](https://en.wikipedia.org/wiki/Divergence_%28statistics%29 "Divergence (statistics)")
-.[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
-While metrics are symmetric and generalize _linear_ distance, satisfying the
-[triangle inequality](https://en.wikipedia.org/wiki/Triangle_inequality "Triangle inequality")
-, divergences are asymmetric and generalize _squared_ distance, in some cases
-satisfying a generalized
-[Pythagorean theorem](https://en.wikipedia.org/wiki/Pythagorean_theorem "Pythagorean theorem")
-. In general
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-does not equal
-![{\\displaystyle D\_{\\text{KL}}(Q\\parallel P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/33cd5593ce6b0527ee343372fcf0a745cb9cdb52)
-, and the asymmetry is an important part of the
-geometry.[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
-The
-[infinitesimal](https://en.wikipedia.org/wiki/Infinitesimal_generator_%28stochastic_processes%29 "Infinitesimal generator (stochastic processes)")
-form of relative entropy, specifically its
-[Hessian](https://en.wikipedia.org/wiki/Hessian_matrix "Hessian matrix") , gives
-a [metric tensor](https://en.wikipedia.org/wiki/Metric_tensor "Metric tensor")
-that equals the
-[Fisher information metric](https://en.wikipedia.org/wiki/Fisher_information_metric "Fisher information metric")
-; see
-[§ Fisher information metric](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Fisher_information_metric)
-. Fisher information metric on the certain probability distribution let
-determine the natural gradient for information-geometric optimization
-algorithms.[\[17\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-17)
-Its quantum version is Fubini-study
-metric.[\[18\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-18)
-Relative entropy satisfies a generalized Pythagorean theorem for
-[exponential families](https://en.wikipedia.org/wiki/Exponential_families "Exponential families")
-(geometrically interpreted as
-[dually flat manifolds](https://en.wikipedia.org/wiki/Dually_flat_manifold "Dually flat manifold")
-), and this allows one to minimize relative entropy by geometric means, for
-example by
-[information projection](https://en.wikipedia.org/wiki/Information_projection "Information projection")
-and in
-[maximum likelihood estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation "Maximum likelihood estimation")
-.[\[5\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201628-5)
-
-The relative entropy is the
-[Bregman divergence](https://en.wikipedia.org/wiki/Bregman_divergence "Bregman divergence")
-generated by the negative entropy, but it is also of the form of an
-[f\-divergence](https://en.wikipedia.org/wiki/F-divergence "F-divergence") . For
-probabilities over a finite
-[alphabet](https://en.wikipedia.org/wiki/Alphabet_%28formal_languages%29 "Alphabet (formal languages)")
-, it is unique in being a member of both of these classes of
-[statistical divergences](https://en.wikipedia.org/wiki/Statistical_divergence "Statistical divergence")
-. The application of Bregman divergence can be found in mirror
-descent.[\[19\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-19)
-
-### Finance (game theory)
-
-Consider a growth-optimizing investor in a fair game with mutually exclusive
-outcomes (e.g. a "horse race" in which the official odds add up to one). The
-rate of return expected by such an investor is equal to the relative entropy
-between the investor's believed probabilities and the official
-odds.[\[20\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-20)
-This is a special case of a much more general connection between financial
-returns and divergence
-measures.[\[21\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-21)
-
-Financial risks are connected to
-![{\\displaystyle D\_{\\text{KL}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/498862d3f9112329a1c9b0ce321eb2ba2fa8a99c)
-via information
-geometry.[\[22\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-22)
-Investors' views, the prevailing market view, and risky scenarios form triangles
-on the relevant manifold of probability distributions. The shape of the
-triangles determines key financial risks (both qualitatively and
-quantitatively). For instance, obtuse triangles in which investors' views and
-risk scenarios appear on "opposite sides" relative to the market describe
-negative risks, acute triangles describe positive exposure, and the right-angled
-situation in the middle corresponds to zero risk. Extending this concept,
-relative entropy can be hypothetically utilised to identify the behaviour of
-informed investors, if one takes this to be represented by the magnitude and
-deviations away from the prior expectations of fund flows, for
-example.[\[23\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-23)
-
-[![](https://upload.wikimedia.org/wikipedia/commons/thumb/a/a8/KL-Gauss-Example.png/330px-KL-Gauss-Example.png)](https://en.wikipedia.org/wiki/File:KL-Gauss-Example.png)
-
-Illustration of the relative entropy for two [normal distributions](https://en.wikipedia.org/wiki/Normal_distribution "Normal distribution"). The typical asymmetry is clearly visible.
-
-In information theory, the
-[Kraft–McMillan theorem](https://en.wikipedia.org/wiki/Kraft%E2%80%93McMillan_inequality "Kraft–McMillan inequality")
-establishes that any directly decodable coding scheme for coding a message to
-identify one value
-![{\\displaystyle x\_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e87000dd6142b81d041896a30fe58f0c3acb2158)
-out of a set of possibilities X can be seen as representing an implicit
-probability distribution
-![{\\displaystyle q(x\_{i})=2^{-\\ell \_{i}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/589aad45b94329b44893303abe4844374d96c147)
-over X, where
-![{\\displaystyle \\ell \_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bc03cf8f5f3927c6693a9ac9677685b08b29c106)
-is the length of the code for
-![{\\displaystyle x\_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e87000dd6142b81d041896a30fe58f0c3acb2158)
-in bits. Therefore, relative entropy can be interpreted as the expected extra
-message-length per datum that must be communicated if a code that is optimal for
-a given (wrong) distribution Q is used, compared to using a code based on the
-true distribution P: it is the _excess_ entropy.
-
-![{\\displaystyle {\\begin{aligned}D\_{\\text{KL}}(P\\parallel Q)&=\\sum
-\_{x\\in {\\mathcal {X}}}p(x)\\log {\\frac {1}{q(x)}}-\\sum \_{x\\in {\\mathcal
-{X}}}p(x)\\log {\\frac {1}{p(x)}}\\\\\[5pt\]&=\\mathrm {H} (P,Q)-\\mathrm {H}
-(P)\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6430656832db5787365726108ee133d4962d37d2)
-
-where
-![{\\displaystyle \\mathrm {H} (P,Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2d4123a2d34c1ad5cd016709a98777cbfb5a1d52)
-is the
-[cross entropy](https://en.wikipedia.org/wiki/Cross_entropy "Cross entropy") of
-Q relative to P and
-![{\\displaystyle \\mathrm {H} (P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/eef2a5cc0a75c21d5ab887d5c13d195546e0cb5e)
-is the
-[entropy](https://en.wikipedia.org/wiki/Information_entropy "Information entropy")
-of P (which is the same as the cross-entropy of P with itself).
-
-The relative entropy
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-can be thought of geometrically as a
-[statistical distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
-, a measure of how far the distribution Q is from the distribution P.
-Geometrically it is a
-[divergence](https://en.wikipedia.org/wiki/Divergence_%28statistics%29 "Divergence (statistics)")
-: an asymmetric, generalized form of squared distance. The cross-entropy
-![{\\displaystyle H(P,Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7a283037a6131d7bd4eb8e210e45032a609d0781)
-is itself such a measurement (formally a
-[loss function](https://en.wikipedia.org/wiki/Loss_function "Loss function") ),
-but it cannot be thought of as a distance, since
-![{\\displaystyle H(P,P)=:H(P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/350d52d7c75ab7080689f029746e0818d8a55ba3)
-is not zero. This can be fixed by subtracting
-![{\\displaystyle H(P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/22aa96351d3d7835aa94cf4ffb4988b0cc7d6f2f)
-to make
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-agree more closely with our notion of distance, as the _excess_ loss. The
-resulting function is asymmetric, and while this can be symmetrized (see
-[§ Symmetrised divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Symmetrised_divergence)
-), the asymmetric form is more useful. See
-[§ Interpretations](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Interpretations)
-for more on the geometric interpretation.
-
-Relative entropy relates to "[rate
-function](https://en.wikipedia.org/wiki/Rate_function "Rate function")" in the
-theory of
-[large deviations](https://en.wikipedia.org/wiki/Large_deviations "Large deviations")
-.[\[24\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Sanov-24)[\[25\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Novak-25)
-
-Arthur Hobson proved that relative entropy is the only measure of difference
-between probability distributions that satisfies some desired properties, which
-are the canonical extension to those appearing in a commonly used
-[characterization of entropy](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29#Characterization "Entropy (information theory)")
-.[\[26\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-26)
-Consequently,
-[mutual information](https://en.wikipedia.org/wiki/Mutual_information "Mutual information")
-is the only measure of mutual dependence that obeys certain related conditions,
-since it can be defined
-[in terms of Kullback–Leibler divergence](https://en.wikipedia.org/wiki/Mutual_information#Relation_to_Kullback%E2%80%93Leibler_divergence "Mutual information")
-.
-
-In particular, if
-![{\\displaystyle P(dx)=p(x)\\mu (dx)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/77813e8d092466097829d43de1701876a6b79fbb)
-and
-![{\\displaystyle Q(dx)=q(x)\\mu (dx)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2fcf85c9df0ea57ddd7d0d1a602ada2d0b23c47c)
-, then
-![{\\displaystyle p(x)=q(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/54db62c48ef2bc2407cf9967c00225d19ba176db)
-![{\\displaystyle \\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161)
-\-[almost everywhere](https://en.wikipedia.org/wiki/Almost_everywhere "Almost
-everywhere"). The entropy
-![{\\displaystyle \\mathrm {H} (P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/eef2a5cc0a75c21d5ab887d5c13d195546e0cb5e)
-thus sets a minimum value for the cross-entropy
-![{\\displaystyle \\mathrm {H} (P,Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2d4123a2d34c1ad5cd016709a98777cbfb5a1d52)
-, the [expected](https://en.wikipedia.org/wiki/Expected_value "Expected value")
-number of [bits](https://en.wikipedia.org/wiki/Bit "Bit") required when using a
-code based on Q rather than P; and the Kullback–Leibler divergence therefore
-represents the expected number of extra bits that must be transmitted to
-identify a value x drawn from X, if a code is used corresponding to the
-probability distribution Q, rather than the "true" distribution P.
-
-*   No upper-bound exists for the general case. However, it is shown that if P and Q are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07) can be calculated.[\[27\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Bonnici2020-27)
-*   Relative entropy remains well-defined for continuous distributions, and furthermore is invariant under [parameter transformations](https://en.wikipedia.org/wiki/Change_of_variables "Change of variables"). For example, if a transformation is made from variable x to variable ![{\\displaystyle y(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e871993bfd131a8b0c3591c26084cf8171a74dcd), then, since ![{\\displaystyle P(dx)=p(x)\\,dx={\\tilde {p}}(y)\\,dy={\\tilde {p}}(y(x))\\left|{\\tfrac {dy}{dx}}(x)\\right|\\,dx}](https://wikimedia.org/api/rest_v1/media/math/render/svg/13d98a58570a83cd89d93517b205e0c2a203f288) and ![{\\displaystyle Q(dx)=q(x)\\,dx={\\tilde {q}}(y)\\,dy={\\tilde {q}}(y)\\left|{\\tfrac {dy}{dx}}(x)\\right|dx}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cab68622f51730433ff24cded52604a245a2c794) where ![{\\displaystyle \\left|{\\tfrac {dy}{dx}}(x)\\right|}](https://wikimedia.org/api/rest_v1/media/math/render/svg/230ba86eb649d4bf53441ffec97623b307a66f3e) is the [absolute value](https://en.wikipedia.org/wiki/Absolute_value "Absolute value") of the derivative or more generally of the [Jacobian](https://en.wikipedia.org/wiki/Jacobian "Jacobian"), the relative entropy may be rewritten: ![{\\displaystyle {\\begin{aligned}D\_{\\text{KL}}(P\\parallel Q)&=\\int \_{x\_{a}}^{x\_{b}}p(x)\\,\\log {\\frac {p(x)}{q(x)}}\\,dx\\\\\[6pt\]&=\\int \_{x\_{a}}^{x\_{b}}{\\tilde {p}}(y(x))\\left|{\\frac {dy}{dx}}\\right|\\log {\\frac {{\\tilde {p}}(y(x))\\,\\left|{\\frac {dy}{dx}}\\right|}{{\\tilde {q}}(y(x))\\,\\left|{\\frac {dy}{dx}}\\right|}}\\,dx\\\\&=\\int \_{y\_{a}}^{y\_{b}}{\\tilde {p}}(y)\\,\\log {\\frac {{\\tilde {p}}(y)}{{\\tilde {q}}(y)}}\\,dy\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/eda27a4e71619493d682fa244154c5dd3e951d64) where ![{\\displaystyle y\_{a}=y(x\_{a})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7840d3605c02be788a77333cee91dbf130b1b4ad) and ![{\\displaystyle y\_{b}=y(x\_{b})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bb98d1f1ad2babf41b276ff053244930ed3c1e83). Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces a [dimensionally consistent](https://en.wikipedia.org/wiki/Dimensional_analysis "Dimensional analysis") quantity, since if x is a dimensioned variable, ![{\\displaystyle p(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8cb7afced134ef75572e5314a5d278c2d644f438) and ![{\\displaystyle q(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c38bbafe34a043d284f19231b946a76c0a4b16b4) are also dimensioned, since e.g. ![{\\displaystyle P(dx)=p(x)\\,dx}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8a5c2c00a56eced5f6ab6e25debe29ebb98bff69) is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory[\[28\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-VerduLecture-28) (such as [self-information](https://en.wikipedia.org/wiki/Self-information "Self-information") or [Shannon entropy](https://en.wikipedia.org/wiki/Shannon_entropy "Shannon entropy")), which can become undefined or negative for non-discrete probabilities.
-*   Relative entropy is [additive](https://en.wikipedia.org/wiki/Additive_map "Additive map") for [independent distributions](https://en.wikipedia.org/wiki/Independent_random_variables "Independent random variables") in much the same way as Shannon entropy. If ![{\\displaystyle P\_{1},P\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3bfdd9dc3e5cdd53ae8ee830e73cc482c834e713) are independent distributions, and ![{\\displaystyle P(dx,dy)=P\_{1}(dx)P\_{2}(dy)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/98e48155edf6e3a2d136a84101bfeb13b837d433), and likewise ![{\\displaystyle Q(dx,dy)=Q\_{1}(dx)Q\_{2}(dy)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/90b0f28bb2e235c85074d1da8ae50d66487a61e2) for independent distributions ![{\\displaystyle Q\_{1},Q\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/63e279e70626310da835b85616e2133fab4c27c5) then ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=D\_{\\text{KL}}(P\_{1}\\parallel Q\_{1})+D\_{\\text{KL}}(P\_{2}\\parallel Q\_{2}).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dd7846698c1981b3996cbfb4a47a03ef1aafefbe)
-*   Relative entropy ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07) is [convex](https://en.wikipedia.org/wiki/Convex_function "Convex function") in the pair of [probability measures](https://en.wikipedia.org/wiki/Probability_measure "Probability measure") ![{\\displaystyle (P,Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f7854fbffaf21cef2fd7219d7b2413ea3961fdf4), i.e. if ![{\\displaystyle (P\_{1},Q\_{1})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4081a0bca5521dfe83a02284b3929ba1f492079d) and ![{\\displaystyle (P\_{2},Q\_{2})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2c2be3a3a4ea1f9ac8a1dcd68691a70af365df9b) are two pairs of probability measures then ![{\\displaystyle D\_{\\text{KL}}(\\lambda P\_{1}+(1-\\lambda )P\_{2}\\parallel \\lambda Q\_{1}+(1-\\lambda )Q\_{2})\\leq \\lambda D\_{\\text{KL}}(P\_{1}\\parallel Q\_{1})+(1-\\lambda )D\_{\\text{KL}}(P\_{2}\\parallel Q\_{2}){\\text{ for }}0\\leq \\lambda \\leq 1.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/775d6ac738000849e63ef24ac6a1ddacccc870f8)
-*   ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07) may be Taylor expanded about its minimum (i.e. ![{\\displaystyle P=Q}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2abc7e2c5a78e9e6cb7a2a907279953f9b4a3f52)) as ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\sum \_{n=2}^{\\infty }{\\frac {1}{n(n-1)}}\\sum \_{x\\in {\\mathcal {X}}}{\\frac {(Q(x)-P(x))^{n}}{Q(x)^{n-1}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d0eb13e7c22e6496d4b2d5046351bd55dd494b9d) which converges if and only if ![{\\displaystyle P\\leq 2Q}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c54e7b207f145b6e768ddb02eb81d8c229ffaec6) [almost surely](https://en.wikipedia.org/wiki/Almost_surely "Almost surely") w.r.t ![{\\displaystyle Q}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8752c7023b4b3286800fe3238271bbca681219ed).
-
-The following result, due to Donsker and
-Varadhan,[\[29\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-29)
-is known as **Donsker and Varadhan's variational formula**.
-
-**Proof**
-
-For a short proof assuming integrability of
-![{\\displaystyle \\exp(h)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d502344470aec00338bc34908b5d0dd789e0a912)
-with respect to P, let
-![{\\displaystyle Q^{\*}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c1785c0a77ab5a06684e8a7ac4f5e59d59ec0319)
-have P\-density ![{\\displaystyle {\\frac {\\exp h(\\theta )}{E\_{P}\[\\exp
-h\]}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/68129a5fd99a843dbd790087e726081a2ccb50a9),
-i.e. ![{\\displaystyle Q^{\*}(d\\theta )={\\frac {\\exp h(\\theta
-)}{E\_{P}\[\\exp h\]}}P(d\\theta
-)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c054e9af822f3b516e93eb4cc96d96441a617985)
-Then
-
-![{\\displaystyle D\_{\\text{KL}}(Q\\parallel
-Q^{\*})-D\_{\\text{KL}}(Q\\parallel P)=-E\_{Q}\[h\]+\\log E\_{P}\[\\exp
-h\]{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9ab18b0c58da04cb8fcbf1bf0205c9bdf6529b0a)
-
-Therefore,
-
-![{\\displaystyle E\_{Q}\[h\]-D\_{\\text{KL}}(Q\\parallel P)=\\log E\_{P}\[\\exp
-h\]-D\_{\\text{KL}}(Q\\parallel Q^{\*})\\leq \\log E\_{P}\[\\exp
-h\]{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b388f329bba5f73fc5af0f0c84041842aaf260b0)
-
-where the last inequality follows from
-![{\\displaystyle D\_{\\text{KL}}(Q\\parallel Q^{\*})\\geq 0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/56ed125c68e62c99c0dd53aefff525fd2b6459a8)
-, for which equality occurs if and only if
-![{\\displaystyle Q=Q^{\*}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/30c57dcbad607ce61ab246e79bd66fe208762346)
-. The conclusion follows.
-
-### Multivariate normal distributions
-
-Suppose that we have two
-[multivariate normal distributions](https://en.wikipedia.org/wiki/Multivariate_normal_distribution "Multivariate normal distribution")
-, with means
-![{\\displaystyle \\mu \_{0},\\mu \_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fb5c6a6422d5ca155808302b46e2233d5b028b34)
-and with (non-singular)
-[covariance matrices](https://en.wikipedia.org/wiki/Covariance_matrix "Covariance matrix")
-![{\\displaystyle \\Sigma \_{0},\\Sigma \_{1}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/583695b19fee86af86f86ea0c0a5dc0b25dfd4cf)
-If the two distributions have the same dimension, k, then the relative entropy
-between the distributions is as
-follows:[\[30\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-30)
-
-![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {N}}\_{0}\\parallel
-{\\mathcal {N}}\_{1}\\right)={\\frac {1}{2}}\\left\[\\operatorname {tr}
-\\left(\\Sigma \_{1}^{-1}\\Sigma \_{0}\\right)-k+\\left(\\mu \_{1}-\\mu
-\_{0}\\right)^{\\mathsf {T}}\\Sigma \_{1}^{-1}\\left(\\mu \_{1}-\\mu
-\_{0}\\right)+\\ln {\\frac {\\det \\Sigma \_{1}}{\\det \\Sigma
-\_{0}}}\\right\]{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d7cda84054fa44b8e5f01fdbabdd8b48b5f0a399)
-
-The [logarithm](https://en.wikipedia.org/wiki/Logarithm "Logarithm") in the last
-term must be taken to base
-[e](https://en.wikipedia.org/wiki/E_%28mathematical_constant%29 "E (mathematical constant)")
-since all terms apart from the last are base-e logarithms of expressions that
-are either factors of the density function or otherwise arise naturally. The
-equation therefore gives a result measured in
-[nats](https://en.wikipedia.org/wiki/Nat_%28unit%29 "Nat (unit)") . Dividing the
-entire expression above by
-![{\\displaystyle \\ln(2)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bee3c8a0344afcd9379a7e528be216a029e8f348)
-yields the divergence in [bits](https://en.wikipedia.org/wiki/Bit "Bit") .
-
-In a numerical implementation, it is helpful to express the result in terms of
-the Cholesky decompositions
-![{\\displaystyle L\_{0},L\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e2e088bdb8986fb222d338c3f274ec6aa4d6b257)
-such that
-![{\\displaystyle \\Sigma \_{0}=L\_{0}L\_{0}^{T}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b642bd5a68be0e0ac7cf15807e147ce769aebfe7)
-and
-![{\\displaystyle \\Sigma \_{1}=L\_{1}L\_{1}^{T}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0d9729bd54f8abca76c5f5968873ea3da2ef0d25)
-. Then with M and y solutions to the triangular linear systems
-![{\\displaystyle L\_{1}M=L\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cd454cdda9a6a9b037783eaf28fcf9c34702e520)
-, and
-![{\\displaystyle L\_{1}y=\\mu \_{1}-\\mu \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/aefa56abf21e168693384a9c0763e788c085fd4f)
-,
-
-![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {N}}\_{0}\\parallel {\\mathcal {N}}\_{1}\\right)={\\frac {1}{2}}\\left(\\sum \_{i,j=1}^{k}{\\left(M\_{ij}\\right)}^{2}-k+|y|^{2}+2\\sum \_{i=1}^{k}\\ln {\\frac {(L\_{1})\_{ii}}{(L\_{0})\_{ii}}}\\right){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4b16cd2b6f80f47128fa04c6c1dc897cb91b76e2)
-
-A special case, and a common quantity in
-[variational inference](https://en.wikipedia.org/wiki/Variational_inference "Variational inference")
-, is the relative entropy between a diagonal multivariate normal, and a standard
-normal distribution (with zero mean and unit variance):
-
-![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {N}}\\left(\\left(\\mu
-\_{1},\\ldots ,\\mu \_{k}\\right)^{\\mathsf {T}},\\operatorname {diag}
-\\left(\\sigma \_{1}^{2},\\ldots ,\\sigma \_{k}^{2}\\right)\\right)\\parallel
-{\\mathcal {N}}\\left(\\mathbf {0} ,\\mathbf {I} \\right)\\right)={\\frac
-{1}{2}}\\sum \_{i=1}^{k}\\left\[\\sigma \_{i}^{2}+\\mu \_{i}^{2}-1-\\ln
-\\left(\\sigma
-\_{i}^{2}\\right)\\right\]{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0bfc62f2d886aed5a3a76d78d2eb7e48101cbc7b)
-
-For two univariate normal distributions **p** and **q** the above simplifies
-to[\[31\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-31)
-![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {p}}\\parallel {\\mathcal {q}}\\right)=\\log {\\frac {\\sigma \_{1}}{\\sigma \_{0}}}+{\\frac {\\sigma \_{0}^{2}+{\\left(\\mu \_{0}-\\mu \_{1}\\right)}^{2}}{2\\sigma \_{1}^{2}}}-{\\frac {1}{2}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/95dd4c6fc20d8e3f33bf0bb1584e4834920faa75)
-
-In the case of co-centered normal distributions with
-![{\\displaystyle k=\\sigma \_{1}/\\sigma \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7571c8949f0b283209ee10474426c61a686a40a9)
-, this
-simplifies[\[32\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-auto-32)
-to:
-
-![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {p}}\\parallel {\\mathcal {q}}\\right)=\\log \_{2}k+(k^{-2}-1)/2/\\ln(2)\\mathrm {bits} }](https://wikimedia.org/api/rest_v1/media/math/render/svg/8f11620a947db549116166183df5b3bb0056dc5d)
-
-### Uniform distributions
-
-Consider two uniform distributions, with the support of ![{\\displaystyle
-p=\[A,B\]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1f9e91badbd6947c80f1308adadc16071891a440)
-enclosed within ![{\\displaystyle
-q=\[C,D\]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/030d2e5d1d0489e6cb5c4c808edc82344635b0e7)
-(![{\\displaystyle C\\leq A<B\\leq
-D}](https://wikimedia.org/api/rest_v1/media/math/render/svg/02a7cfd5e1908c9a583a4304eb03dcdde0286d74)).
-Then the information gain is:
-
-![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {p}}\\parallel {\\mathcal {q}}\\right)=\\log {\\frac {D-C}{B-A}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0a84d16d1bd0383b8433b4086f57de3aba45b9d9)
-
-Intuitively,[\[32\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-auto-32)
-the information gain to a k times narrower uniform distribution contains
-![{\\displaystyle \\log \_{2}k}](https://wikimedia.org/api/rest_v1/media/math/render/svg/294c0f6bed4ca4ba34a7242748ab0accef3924a8)
-bits. This connects with the use of bits in computing, where
-![{\\displaystyle \\log \_{2}k}](https://wikimedia.org/api/rest_v1/media/math/render/svg/294c0f6bed4ca4ba34a7242748ab0accef3924a8)
-bits would be needed to identify one element of a k long stream.
-
-### Exponential family
-
-The
-[exponential family](https://en.wikipedia.org/wiki/Exponential_family "Exponential family")
-of distribution is given by
-
-![{\\displaystyle p\_{X}(x|\\theta )=h(x)\\exp \\left(\\theta ^{\\mathsf {T}}T(x)-A(\\theta )\\right)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6181c93488b4e22b405c4f20b9a61384bc1c62f3)
-
-where
-![{\\displaystyle h(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/02c07825dae28705df03d15daeb8844d49c4dbd4)
-is reference measure,
-![{\\displaystyle T(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1171c29b4c2b5575f50a4ea9313f90448a2cbe05)
-is
-[sufficient statistics](https://en.wikipedia.org/wiki/Sufficient_statistic "Sufficient statistic")
-,
-![{\\displaystyle \\theta }](https://wikimedia.org/api/rest_v1/media/math/render/svg/6e5ab2664b422d53eb0c7df3b87e1360d75ad9af)
-is canonical natural parameters, and
-![{\\displaystyle A(\\theta )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/55a6c7ecebb4cb39407ae5b4a05fdbc82597901c)
-is the log-partition function.
-
-The KL divergence between two distributions
-![{\\displaystyle p(x|\\theta \_{1})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ec834a5e8c4d0b2d4a768151d1120b529ca65ce0)
-and
-![{\\displaystyle p(x|\\theta \_{2})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0176203c5aab87c8ad8a9588c99e3cb4e9c5846f)
-is given
-by[\[33\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-33)
-
-![{\\displaystyle D\_{\\text{KL}}(\\theta \_{1}\\parallel \\theta \_{2})={\\left(\\theta \_{1}-\\theta \_{2}\\right)}^{\\mathsf {T}}\\mu \_{1}-A(\\theta \_{1})+A(\\theta \_{2})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1c3ff1aba7881bafd467e4487d44feee78e75620)
-
-where ![{\\displaystyle \\mu \_{1}=E\_{\\theta \_{1}}\[T(X)\]=\\nabla A(\\theta
-\_{1})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dd7911f2432b73d6ebe8a38701e18b423d658cb9)
-is the mean parameter of
-![{\\displaystyle p(x|\\theta \_{1})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ec834a5e8c4d0b2d4a768151d1120b529ca65ce0)
-.
-
-For example, for the
-[Poisson distribution](https://en.wikipedia.org/wiki/Poisson_distribution "Poisson distribution")
-with mean
-![{\\displaystyle \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/b43d0ea3c9c025af1be9128e62a18fa74bedda2a)
-, the sufficient statistics
-![{\\displaystyle T(x)=x}](https://wikimedia.org/api/rest_v1/media/math/render/svg/73a77ba5a3df9ab8481ede7461f146af9ab29e6f)
-, the natural parameter
-![{\\displaystyle \\theta =\\log \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/63603abf3d3c7ab57711b2b42ac784e72b1ad471)
-, and log partition function
-![{\\displaystyle A(\\theta )=e^{\\theta }}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b477f9a8ac97b8f22c36d94c4ff3def9dd22a7b6)
-. As such, the divergence between two Poisson distributions with means
-![{\\displaystyle \\lambda \_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/571a423bece8f29bcd1b48572f18dd4f6213dce2)
-and
-![{\\displaystyle \\lambda \_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6b668a1bd1e8ab9452ca975b7497546e7c1ba187)
-is
-
-![{\\displaystyle D\_{\\text{KL}}(\\lambda \_{1}\\parallel \\lambda \_{2})=\\lambda \_{1}\\log {\\frac {\\lambda \_{1}}{\\lambda \_{2}}}-\\lambda \_{1}+\\lambda \_{2}{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6736a0a627f96bd200a24c4e477095bd80f61fc0)
-
-As another example, for a normal distribution with unit variance
-![{\\displaystyle N(\\mu ,1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4ef5f4e895d453e932bcd4624a80b06cf06444b2)
-, the sufficient statistics
-![{\\displaystyle T(x)=x}](https://wikimedia.org/api/rest_v1/media/math/render/svg/73a77ba5a3df9ab8481ede7461f146af9ab29e6f)
-, the natural parameter
-![{\\displaystyle \\theta =\\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/ac9063d0c72c38a12998c729fb596f1ad5233a3a)
-, and log partition function
-![{\\displaystyle A(\\theta )=\\mu ^{2}/2}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d4aac90be14918d32895027f000dd0068ce18df6)
-. Thus, the divergence between two normal distributions
-![{\\displaystyle N(\\mu \_{1},1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/66873c60341912620df20e2e4e34b4f63ed35b4b)
-and
-![{\\displaystyle N(\\mu \_{2},1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9de991b8a02dd241374825940b478e8c9fd3ca50)
-is
-
-![{\\displaystyle D\_{\\text{KL}}(\\mu \_{1}\\parallel \\mu \_{2})=\\left(\\mu \_{1}-\\mu \_{2}\\right)\\mu \_{1}-{\\frac {\\mu \_{1}^{2}}{2}}+{\\frac {\\mu \_{2}^{2}}{2}}={\\frac {{\\left(\\mu \_{2}-\\mu \_{1}\\right)}^{2}}{2}}{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/439818de8db9a8e98a6a71ce0c8d7de8b8ae04f7)
-
-As final example, the divergence between a normal distribution with unit
-variance
-![{\\displaystyle N(\\mu ,1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4ef5f4e895d453e932bcd4624a80b06cf06444b2)
-and a Poisson distribution with mean
-![{\\displaystyle \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/b43d0ea3c9c025af1be9128e62a18fa74bedda2a)
-is
-
-![{\\displaystyle D\_{\\text{KL}}(\\mu \\parallel \\lambda )=(\\mu -\\log \\lambda )\\mu -{\\frac {\\mu ^{2}}{2}}+\\lambda {\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/119624907e5012f526bc25af8f8061871ac2c288)
-
-While relative entropy is a
-[statistical distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
-, it is not a
-[metric](https://en.wikipedia.org/wiki/Metric_%28mathematics%29 "Metric (mathematics)")
-on the space of probability distributions, but instead it is a
-[divergence](https://en.wikipedia.org/wiki/Divergence_%28statistics%29 "Divergence (statistics)")
-.[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
-While metrics are symmetric and generalize _linear_ distance, satisfying the
-[triangle inequality](https://en.wikipedia.org/wiki/Triangle_inequality "Triangle inequality")
-, divergences are asymmetric in general and generalize _squared_ distance, in
-some cases satisfying a generalized
-[Pythagorean theorem](https://en.wikipedia.org/wiki/Pythagorean_theorem "Pythagorean theorem")
-. In general
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-does not equal
-![{\\displaystyle D\_{\\text{KL}}(Q\\parallel P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/33cd5593ce6b0527ee343372fcf0a745cb9cdb52)
-, and while this can be symmetrized (see
-[§ Symmetrised divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Symmetrised_divergence)
-), the asymmetry is an important part of the
-geometry.[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
-
-It generates a
-[topology](https://en.wikipedia.org/wiki/Topology#Topologies_on_sets "Topology")
-on the space of
-[probability distributions](https://en.wikipedia.org/wiki/Probability_distribution "Probability distribution")
-. More concretely, if
-![{\\displaystyle \\{P\_{1},P\_{2},\\ldots \\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1ff268a40cf47f496dfc3ff0425478093bc49e96)
-is a sequence of distributions such that
-
-![{\\displaystyle \\lim \_{n\\to \\infty }D\_{\\text{KL}}(P\_{n}\\parallel Q)=0{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f7527f531b3b491f7cc3eccc79da9be09247dbbe)
-
-then it is said that
-
-![{\\displaystyle P\_{n}\\xrightarrow {D} \\,Q{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e9758cb3479ccf538c74eb7e887c4568f987bae3)
-
-[Pinsker's inequality](https://en.wikipedia.org/wiki/Pinsker's_inequality "Pinsker's inequality")
-entails that
-
-![{\\displaystyle P\_{n}\\xrightarrow {D} P\\Rightarrow P\_{n}\\xrightarrow {TV} P{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a6001fee2dbafa3f33e0ed6c758e44d0f751a5d7)
-
-where the latter stands for the usual convergence in
-[total variation](https://en.wikipedia.org/wiki/Total_variation "Total variation")
-.
-
-### Fisher information metric
-
-Relative entropy is directly related to the
-[Fisher information metric](https://en.wikipedia.org/wiki/Fisher_information_metric "Fisher information metric")
-. This can be made explicit as follows. Assume that the probability
-distributions P and Q are both parameterized by some (possibly
-multi-dimensional) parameter
-![{\\displaystyle \\theta }](https://wikimedia.org/api/rest_v1/media/math/render/svg/6e5ab2664b422d53eb0c7df3b87e1360d75ad9af)
-. Consider then two close by values of
-![{\\displaystyle P=P(\\theta )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/83a5542468bf4a5ad9dee3d0f3ad02047a73da01)
-and
-![{\\displaystyle Q=P(\\theta \_{0})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/41b4e23cf75300ef3c256ed00c99f4c6ac348c20)
-so that the parameter
-![{\\displaystyle \\theta }](https://wikimedia.org/api/rest_v1/media/math/render/svg/6e5ab2664b422d53eb0c7df3b87e1360d75ad9af)
-differs by only a small amount from the parameter value
-![{\\displaystyle \\theta \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/18b67de6bf25dba7a24e66967ff6319858798734)
-. Specifically, up to first order one has (using the
-[Einstein summation convention](https://en.wikipedia.org/wiki/Einstein_summation_convention "Einstein summation convention")
-)
-![{\\displaystyle P(\\theta )=P(\\theta \_{0})+\\Delta \\theta \_{j}\\,P\_{j}(\\theta \_{0})+\\cdots }](https://wikimedia.org/api/rest_v1/media/math/render/svg/39a491014bbef8b352bee60fb833c91cd17c6437)
-
-with
-![{\\displaystyle \\Delta \\theta \_{j}=(\\theta -\\theta \_{0})\_{j}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f7bd79e9136dbee44c7e455332ac89706e6e8b49)
-a small change of
-![{\\displaystyle \\theta }](https://wikimedia.org/api/rest_v1/media/math/render/svg/6e5ab2664b422d53eb0c7df3b87e1360d75ad9af)
-in the j direction, and
-![{\\displaystyle P\_{j}\\left(\\theta \_{0}\\right)={\\frac {\\partial P}{\\partial \\theta \_{j}}}(\\theta \_{0})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c798c08f1c1be857631d461730e7314b794c1567)
-the corresponding rate of change in the probability distribution. Since relative
-entropy has an absolute minimum 0 for
-![{\\displaystyle P=Q}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2abc7e2c5a78e9e6cb7a2a907279953f9b4a3f52)
-, i.e.
-![{\\displaystyle \\theta =\\theta \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6092e68904d0c31801bee667fd2b3ebd40d6e5e4)
-, it changes only to _second_ order in the small parameters
-![{\\displaystyle \\Delta \\theta \_{j}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2687696576367a434e6dd95c336e73c1d4badcaf)
-. More formally, as for any minimum, the first derivatives of the divergence
-vanish
-
-![{\\displaystyle \\left.{\\frac {\\partial }{\\partial \\theta \_{j}}}\\right|\_{\\theta =\\theta \_{0}}D\_{\\text{KL}}(P(\\theta )\\parallel P(\\theta \_{0}))=0,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8dd547ff8248b6e69e3014b4458c82e92113cb8e)
-
-and by the
-[Taylor expansion](https://en.wikipedia.org/wiki/Taylor_series "Taylor series")
-one has up to second order
-
-![{\\displaystyle D\_{\\text{KL}}(P(\\theta )\\parallel P(\\theta \_{0}))={\\frac {1}{2}}\\,\\Delta \\theta \_{j}\\,\\Delta \\theta \_{k}\\,g\_{jk}(\\theta \_{0})+\\cdots }](https://wikimedia.org/api/rest_v1/media/math/render/svg/0ed16484f979f06d691f968cc77100f06e827349)
-
-where the
-[Hessian matrix](https://en.wikipedia.org/wiki/Hessian_matrix "Hessian matrix")
-of the divergence
-
-![{\\displaystyle g\_{jk}(\\theta \_{0})=\\left.{\\frac {\\partial ^{2}}{\\partial \\theta \_{j}\\,\\partial \\theta \_{k}}}\\right|\_{\\theta =\\theta \_{0}}D\_{\\text{KL}}(P(\\theta )\\parallel P(\\theta \_{0}))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fba580eb7b3c8496eebb609d2fca6f65e10a0cf6)
-
-must be
-[positive semi-definite](https://en.wikipedia.org/wiki/Positive-definite_matrix "Positive-definite matrix")
-. Letting
-![{\\displaystyle \\theta \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/18b67de6bf25dba7a24e66967ff6319858798734)
-vary (and dropping the subindex 0) the Hessian
-![{\\displaystyle g\_{jk}(\\theta )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2fcb737db9ee0c60db3b36b5f81a67083ceb0817)
-defines a (possibly degenerate)
-[Riemannian metric](https://en.wikipedia.org/wiki/Riemannian_metric "Riemannian metric")
-on the θ
-[parameter space](https://en.wikipedia.org/wiki/Parameter_space "Parameter space")
-, called the Fisher information metric.
-
-#### Fisher information metric theorem
-
-There is an associated
-theorem.[\[3\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback1959-3)
-When
-![{\\displaystyle p\_{(x,\\rho )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2e3ddcb4e03e8d98f0c05c23f7914a903ee47647)
-satisfies the following regularity conditions:
-
-![{\\displaystyle {\\frac {\\partial \\log(p)}{\\partial \\rho }},{\\frac {\\partial ^{2}\\log(p)}{\\partial \\rho ^{2}}},{\\frac {\\partial ^{3}\\log(p)}{\\partial \\rho ^{3}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a5f62261a92a9f9a1169723fd94341b01e412aed)
-exist,
-![{\\displaystyle {\\begin{aligned}\\left|{\\frac {\\partial p}{\\partial \\rho }}\\right|&<F(x):\\int \_{x=0}^{\\infty }F(x)\\,dx<\\infty ,\\\\\\left|{\\frac {\\partial ^{2}p}{\\partial \\rho ^{2}}}\\right|&<G(x):\\int \_{x=0}^{\\infty }G(x)\\,dx<\\infty \\\\\\left|{\\frac {\\partial ^{3}\\log(p)}{\\partial \\rho ^{3}}}\\right|&<H(x):\\int \_{x=0}^{\\infty }p(x,0)H(x)\\,dx<\\xi <\\infty \\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/33c02cd3c9aa88aed8394b4f0917fb34740afe7a)
-
-where ξ is independent of ρ
-![{\\displaystyle \\left.\\int \_{x=0}^{\\infty }{\\frac {\\partial p(x,\\rho )}{\\partial \\rho }}\\right|\_{\\rho =0}\\,dx=\\left.\\int \_{x=0}^{\\infty }{\\frac {\\partial ^{2}p(x,\\rho )}{\\partial \\rho ^{2}}}\\right|\_{\\rho =0}\\,dx=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/81bd4dadd20884ad83a9d1fce680183e2c2eaa1f)
-
-then:
-![{\\displaystyle {\\mathcal {D}}(p(x,0)\\parallel p(x,\\rho ))={\\frac {c\\rho ^{2}}{2}}+{\\mathcal {O}}\\left(\\rho ^{3}\\right){\\text{ as }}\\rho \\to 0{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fa70bfdc1ef68ff2c0949b31e719ac183fc74f24)
-
-### Variation of information
-
-Another information-theoretic metric is
-[variation of information](https://en.wikipedia.org/wiki/Variation_of_information "Variation of information")
-, which is roughly a symmetrization of
-[conditional entropy](https://en.wikipedia.org/wiki/Conditional_entropy "Conditional entropy")
-. It is a metric on the set of
-[partitions](https://en.wikipedia.org/wiki/Partition_of_a_set "Partition of a set")
-of a discrete
-[probability space](https://en.wikipedia.org/wiki/Probability_space "Probability space")
-.
-
-### MAUVE Metric
-
-MAUVE is a measure of the statistical gap between two text distributions, such
-as the difference between text generated by a model and human-written text. This
-measure is computed using Kullback–Leibler divergences between the two
-distributions in a quantized embedding space of a foundation model.
-
-Many of the other quantities of information theory can be interpreted as
-applications of relative entropy to specific cases.
-
-### Self-information
-
-The
-[self-information](https://en.wikipedia.org/wiki/Self-information "Self-information")
-, also known as the
-[information content](https://en.wikipedia.org/wiki/Information_content "Information content")
-of a signal, random variable, or
-[event](https://en.wikipedia.org/wiki/Event_%28probability_theory%29 "Event (probability theory)")
-is defined as the negative logarithm of the
-[probability](https://en.wikipedia.org/wiki/Probability "Probability") of the
-given outcome occurring.
-
-When applied to a
-[discrete random variable](https://en.wikipedia.org/wiki/Discrete_random_variable "Discrete random variable")
-, the self-information can be represented as\[_[citation
-needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed
-"Wikipedia:Citation needed")_\]
-
-![{\\displaystyle \\operatorname {\\operatorname {I} } (m)=D\_{\\text{KL}}\\left(\\delta \_{\\text{im}}\\parallel \\{p\_{i}\\}\\right),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/403686f36e08d2d510888265dc2aff716b8bca6b)
-
-is the relative entropy of the probability distribution
-![{\\displaystyle P(i)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d3f0c5cc0e8d31d705f7e08298daab4cb7da1662)
-from a
-[Kronecker delta](https://en.wikipedia.org/wiki/Kronecker_delta "Kronecker delta")
-representing certainty that
-![{\\displaystyle i=m}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4718f08ffcf5718164ddc37f4a382d04effe1014)
-— i.e. the number of extra bits that must be transmitted to identify i if only
-the probability distribution
-![{\\displaystyle P(i)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d3f0c5cc0e8d31d705f7e08298daab4cb7da1662)
-is available to the receiver, not the fact that
-![{\\displaystyle i=m}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4718f08ffcf5718164ddc37f4a382d04effe1014)
-.
-
-### Mutual information
-
-The
-[mutual information](https://en.wikipedia.org/wiki/Mutual_information "Mutual information")
-,
-
-![{\\displaystyle {\\begin{aligned}\\operatorname {I}
-(X;Y)&=D\_{\\text{KL}}(P\_{X,Y}\\parallel P\_{X}\\cdot
-P\_{Y})\\\\&=\\operatorname {E} \_{X}\[D\_{\\text{KL}}^{Y}(P\_{Y\\mid
-X}\\parallel P\_{Y})\]\\\\&=\\operatorname {E}
-\_{Y}\[D\_{\\text{KL}}^{X}(P\_{X\\mid Y}\\parallel
-P\_{X})\]\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5808992333117aa1618d86a9293c7bb0531b22d1)
-
-is the relative entropy of the
-[joint probability distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution "Joint probability distribution")
-![{\\displaystyle P\_{X,Y}(x,y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fb7b89359e2ab11faff329f878f6272ad4e39103)
-from the product
-![{\\displaystyle (P\_{X}\\cdot P\_{Y})(x,y)=P\_{X}(x)P\_{Y}(y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ff7faf65c801b47f9f7b3360b0428a676a828b3f)
-of the two
-[marginal probability distributions](https://en.wikipedia.org/wiki/Marginal_probability_distribution "Marginal probability distribution")
-— i.e. the expected number of extra bits that must be transmitted to identify X
-and Y if they are coded using only their marginal distributions instead of the
-joint distribution.
-
-### Shannon entropy
-
-The
-[Shannon entropy](https://en.wikipedia.org/wiki/Shannon_entropy "Shannon entropy")
-,
-
-![{\\displaystyle {\\begin{aligned}\\mathrm {H} (X)&=\\operatorname {E}
-\\left\[\\operatorname {I} \_{X}(x)\\right\]\\\\&=\\log
-N-D\_{\\text{KL}}{\\left(p\_{X}(x)\\parallel
-P\_{U}(X)\\right)}\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1404db97e6e3315749e73c5fd8237503a1fa58cf)
-
-is the number of bits which would have to be transmitted to identify X from N
-equally likely possibilities, _less_ the relative entropy of the uniform
-distribution on the
-[random variates](https://en.wikipedia.org/wiki/Random_variate "Random variate")
-of X,
-![{\\displaystyle P\_{U}(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d08085e4447acd3d0e19ca4ac57058ff75bd38be)
-, from the true distribution
-![{\\displaystyle P(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bf94167ebc67db1573e6be9e261cd0518e49e160)
-— i.e. _less_ the expected number of bits saved, which would have had to be sent
-if the value of X were coded according to the uniform distribution
-![{\\displaystyle P\_{U}(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d08085e4447acd3d0e19ca4ac57058ff75bd38be)
-rather than the true distribution
-![{\\displaystyle P(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bf94167ebc67db1573e6be9e261cd0518e49e160)
-. This definition of Shannon entropy forms the basis of
-[E.T. Jaynes](https://en.wikipedia.org/wiki/E.T._Jaynes "E.T. Jaynes") 's
-alternative generalization to continuous distributions, the
-[limiting density of discrete points](https://en.wikipedia.org/wiki/Limiting_density_of_discrete_points "Limiting density of discrete points")
-(as opposed to the usual
-[differential entropy](https://en.wikipedia.org/wiki/Differential_entropy "Differential entropy")
-), which defines the continuous entropy as
-![{\\displaystyle \\lim \_{N\\to \\infty }H\_{N}(X)=\\log N-\\int p(x)\\log {\\frac {p(x)}{m(x)}}\\,dx{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ef7fb5689851293e706d1d4d63d2bb55299f170b)
-which is equivalent to:
-![{\\displaystyle \\log(N)-D\_{\\text{KL}}(p(x)||m(x))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e9008c0b15e4f79b3ed0583f21b418b7f4f6f2ee)
-
-### Conditional entropy
-
-The
-[conditional entropy](https://en.wikipedia.org/wiki/Conditional_entropy "Conditional entropy")
-[\[34\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-CoverThomas-34),
-
-![{\\displaystyle {\\begin{aligned}\\mathrm {H} (X\\mid Y)&=\\log
-N-D\_{\\text{KL}}(P(X,Y)\\parallel P\_{U}(X)P(Y))\\\\\[5pt\]&=\\log
-N-D\_{\\text{KL}}(P(X,Y)\\parallel P(X)P(Y))-D\_{\\text{KL}}(P(X)\\parallel
-P\_{U}(X))\\\\\[5pt\]&=\\mathrm {H} (X)-\\operatorname {I}
-(X;Y)\\\\\[5pt\]&=\\log N-\\operatorname {E}
-\_{Y}\\left\[D\_{\\text{KL}}\\left(P\\left(X\\mid Y\\right)\\parallel
-P\_{U}(X)\\right)\\right\]\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/486a08374b73d457fbab6acab4f4014b045ee26a)
-
-is the number of bits which would have to be transmitted to identify X from N
-equally likely possibilities, _less_ the relative entropy of the true joint
-distribution
-![{\\displaystyle P(X,Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a6fd347c3c83e69250fe5a6c3648fed056ff8f68)
-from the product distribution
-![{\\displaystyle P\_{U}(X)P(Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6ee07039d976b9177d02c8e75b068f62b34292a0)
-from — i.e. _less_ the expected number of bits saved which would have had to be
-sent if the value of X were coded according to the uniform distribution
-![{\\displaystyle P\_{U}(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d08085e4447acd3d0e19ca4ac57058ff75bd38be)
-rather than the conditional distribution
-![{\\displaystyle P(X|Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/38c2f827fbc72053662e88bb04636eca0fc5cd0f)
-of X given Y.
-
-### Cross entropy
-
-When we have a set of possible events, coming from the distribution p, we can
-encode them (with a
-[lossless data compression](https://en.wikipedia.org/wiki/Lossless_compression "Lossless compression")
-) using
-[entropy encoding](https://en.wikipedia.org/wiki/Entropy_encoding "Entropy encoding")
-. This compresses the data by replacing each fixed-length input symbol with a
-corresponding unique, variable-length,
-[prefix-free code](https://en.wikipedia.org/wiki/Prefix-free_code "Prefix-free code")
-(e.g.: the events (A, B, C) with probabilities p = (1/2, 1/4, 1/4) can be
-encoded as the bits (0, 10, 11)). If we know the distribution p in advance, we
-can devise an encoding that would be optimal (e.g.: using
-[Huffman coding](https://en.wikipedia.org/wiki/Huffman_coding "Huffman coding")
-). Meaning the messages we encode will have the shortest length on average
-(assuming the encoded events are sampled from p), which will be equal to
-[Shannon's Entropy](https://en.wikipedia.org/wiki/Shannon's_entropy "Shannon's entropy")
-of p (denoted as
-![{\\displaystyle \\mathrm {H} (p)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/76d5ade34024658505d6df9f6114079845d005b5)
-). However, if we use a different probability distribution (q) when creating the
-entropy encoding scheme, then a larger number of
-[bits](https://en.wikipedia.org/wiki/Bit "Bit") will be used (on average) to
-identify an event from a set of possibilities. This new (larger) number is
-measured by the
-[cross entropy](https://en.wikipedia.org/wiki/Cross_entropy "Cross entropy")
-between p and q.
-
-The [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy "Cross entropy")
-between two
-[probability distributions](https://en.wikipedia.org/wiki/Probability_distribution "Probability distribution")
-(p and q) measures the average number of
-[bits](https://en.wikipedia.org/wiki/Bit "Bit") needed to identify an event from
-a set of possibilities, if a coding scheme is used based on a given probability
-distribution q, rather than the "true" distribution p. The cross entropy for two
-distributions p and q over the same
-[probability space](https://en.wikipedia.org/wiki/Probability_space "Probability space")
-is thus defined as follows.
-
-![{\\displaystyle \\mathrm {H} (p,q)=\\operatorname {E} \_{p}\[-\\log
-q\]=\\mathrm {H} (p)+D\_{\\text{KL}}(p\\parallel
-q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ce9707c4ceba15ce0d453ea474cba38592b1a4d1)
-
-For explicit derivation of this, see the
-[Motivation](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Motivation)
-section above.
-
-Under this scenario, relative entropies (kl-divergence) can be interpreted as
-the extra number of bits, on average, that are needed (beyond
-![{\\displaystyle \\mathrm {H} (p)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/76d5ade34024658505d6df9f6114079845d005b5)
-) for encoding the events because of using q for constructing the encoding
-scheme instead of p.
-
-In
-[Bayesian statistics](https://en.wikipedia.org/wiki/Bayesian_statistics "Bayesian statistics")
-, relative entropy can be used as a measure of the information gain in moving
-from a
-[prior distribution](https://en.wikipedia.org/wiki/Prior_distribution "Prior distribution")
-to a
-[posterior distribution](https://en.wikipedia.org/wiki/Posterior_distribution "Posterior distribution")
-:
-![{\\displaystyle p(x)\\to p(x\\mid I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cb998006aac30bccaa9826f5a593db16b833c8ca)
-. If some new fact
-![{\\displaystyle Y=y}](https://wikimedia.org/api/rest_v1/media/math/render/svg/678864c5e9a7ce08acfc22d0d7f726d2cade5b45)
-is discovered, it can be used to update the posterior distribution for X from
-![{\\displaystyle p(x\\mid I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ed634d96ca7387739783ba12b2a60f53420075b3)
-to a new posterior distribution
-![{\\displaystyle p(x\\mid y,I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dbb94d46086c0f668f208447d1094b3be4c57f4d)
-using
-[Bayes' theorem](https://en.wikipedia.org/wiki/Bayes'_theorem "Bayes' theorem")
-:
-
-![{\\displaystyle p(x\\mid y,I)={\\frac {p(y\\mid x,I)p(x\\mid I)}{p(y\\mid I)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a5e1963e6c1b97081f40d4e00a33a13bf23e019e)
-
-This distribution has a new
-[entropy](https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_information_theory "Entropy in thermodynamics and information theory")
-:
-
-![{\\displaystyle \\mathrm {H} {\\big (}p(x\\mid y,I){\\big )}=-\\sum \_{x}p(x\\mid y,I)\\log p(x\\mid y,I){\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/40ef14fbac8f28f971bbfdf3e20b6852865fae29)
-
-which may be less than or greater than the original entropy
-![{\\displaystyle \\mathrm {H} (p(x\\mid I))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/47e7c4d8c15ef8c0ece7e8a42867584c4f04ca3c)
-. However, from the standpoint of the new probability distribution one can
-estimate that to have used the original code based on
-![{\\displaystyle p(x\\mid I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ed634d96ca7387739783ba12b2a60f53420075b3)
-instead of a new code based on
-![{\\displaystyle p(x\\mid y,I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dbb94d46086c0f668f208447d1094b3be4c57f4d)
-would have added an expected number of bits:
-
-![{\\displaystyle D\_{\\text{KL}}{\\big (}p(x\\mid y,I)\\parallel p(x\\mid I){\\big )}=\\sum \_{x}p(x\\mid y,I)\\log {\\frac {p(x\\mid y,I)}{p(x\\mid I)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0d26a80847684e120153b8f0bf013212d2f24058)
-
-to the message length. This therefore represents the amount of useful
-information, or information gain, about X, that has been learned by discovering
-![{\\displaystyle Y=y}](https://wikimedia.org/api/rest_v1/media/math/render/svg/678864c5e9a7ce08acfc22d0d7f726d2cade5b45)
-.
-
-If a further piece of data,
-![{\\displaystyle Y\_{2}=y\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/052adff6c0f6624b25f30b42c942007137d47faf)
-, subsequently comes in, the probability distribution for x can be updated
-further, to give a new best guess
-![{\\displaystyle p(x\\mid y\_{1},y\_{2},I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/56d4b0464ccdda0a8de69b81efc90371fe6432fc)
-. If one reinvestigates the information gain for using
-![{\\displaystyle p(x\\mid y\_{1},I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6db097127a754553f560ee35afb78209b325174a)
-rather than
-![{\\displaystyle p(x\\mid I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ed634d96ca7387739783ba12b2a60f53420075b3)
-, it turns out that it may be either greater or less than previously estimated:
-
-![{\\displaystyle \\sum \_{x}p(x\\mid y\_{1},y\_{2},I)\\log {\\frac {p(x\\mid y\_{1},y\_{2},I)}{p(x\\mid I)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d5dac2e4e5ade5cf9586a78a96c727d6bae2edf8)
-may be ≤ or > than
-![{\\textstyle \\sum \_{x}p(x\\mid y\_{1},I)\\log {\\frac {p(x\\mid y\_{1},I)}{p(x\\mid I)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d10748d9317bd640b92ff47bf2da07e889e539be)
-
-and so the combined information gain does _not_ obey the triangle inequality:
-
-![{\\displaystyle D\_{\\text{KL}}{\\big (}p(x\\mid y\_{1},y\_{2},I)\\parallel p(x\\mid I){\\big )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c93e3853d8702d3203faa88ad7bb87e5039d415c)
-may be <, = or > than
-![{\\displaystyle D\_{\\text{KL}}{\\big (}p(x\\mid y\_{1},y\_{2},I)\\parallel p(x\\mid y\_{1},I){\\big )}+D\_{\\text{KL}}{\\big (}p(x\\mid y\_{1},I)\\parallel p(x\\mid I){\\big )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a971a78c5ace5b1f62f3e697a35b4017a9d407fb)
-
-All one can say is that on _average_, averaging using
-![{\\displaystyle p(y\_{2}\\mid y\_{1},x,I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d99961b15f4a711b0a733454ab8a7b5307a9131b)
-, the two sides will average out.
-
-### Bayesian experimental design
-
-A common goal in
-[Bayesian experimental design](https://en.wikipedia.org/wiki/Bayesian_experimental_design "Bayesian experimental design")
-is to maximise the expected relative entropy between the prior and the
-posterior.[\[35\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-35)
-When posteriors are approximated to be Gaussian distributions, a design
-maximising the expected relative entropy is called
-[Bayes d-optimal](https://en.wikipedia.org/wiki/D-optimal_design "D-optimal design")
-.
-
-Relative entropy
-![{\\textstyle D\_{\\text{KL}}{\\bigl (}p(x\\mid H\_{1})\\parallel p(x\\mid H\_{0}){\\bigr )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f0e2665a13b2a170c4cb5d28f14708037b7924ca)
-can also be interpreted as the expected **discrimination information** for
-![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
-over
-![{\\displaystyle H\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/43910602a221b7a4c373791f94793e3008622070)
-: the mean information per sample for discriminating in favor of a hypothesis
-![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
-against a hypothesis
-![{\\displaystyle H\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/43910602a221b7a4c373791f94793e3008622070)
-, when hypothesis
-![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
-is
-true.[\[36\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-36)
-Another name for this quantity, given to it by
-[I. J. Good](https://en.wikipedia.org/wiki/I._J._Good "I. J. Good") , is the
-expected weight of evidence for
-![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
-over
-![{\\displaystyle H\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/43910602a221b7a4c373791f94793e3008622070)
-to be expected from each sample.
-
-The expected weight of evidence for
-![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
-over
-![{\\displaystyle H\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/43910602a221b7a4c373791f94793e3008622070)
-is **not** the same as the information gain expected per sample about the
-probability distribution
-![{\\displaystyle p(H)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2290c0131714e85c56133f607eb41af87fdda3c1)
-of the hypotheses,
-
-![{\\displaystyle D\_{\\text{KL}}(p(x\\mid H\_{1})\\parallel p(x\\mid H\_{0}))\\neq IG=D\_{\\text{KL}}(p(H\\mid x)\\parallel p(H\\mid I)){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9d9922587515eaa137f6a34b189700306877a048)
-
-Either of the two quantities can be used as a
-[utility function](https://en.wikipedia.org/wiki/Utility_function "Utility function")
-in Bayesian experimental design, to choose an optimal next question to
-investigate: but they will in general lead to rather different experimental
-strategies.
-
-On the entropy scale of _information gain_ there is very little difference
-between near certainty and absolute certainty—coding according to a near
-certainty requires hardly any more bits than coding according to an absolute
-certainty. On the other hand, on the
-[logit](https://en.wikipedia.org/wiki/Logit "Logit") scale implied by weight of
-evidence, the difference between the two is enormous – infinite perhaps; this
-might reflect the difference between being almost sure (on a probabilistic
-level) that, say, the
-[Riemann hypothesis](https://en.wikipedia.org/wiki/Riemann_hypothesis "Riemann hypothesis")
-is correct, compared to being certain that it is correct because one has a
-[mathematical proof](https://en.wikipedia.org/wiki/Mathematical_proof "Mathematical proof")
-. These two different scales of
-[loss function](https://en.wikipedia.org/wiki/Loss_function "Loss function") for
-uncertainty are _both_ useful, according to how well each reflects the
-particular circumstances of the problem in question.
-
-### Principle of minimum discrimination information
-
-The idea of relative entropy as discrimination information led Kullback to
-propose the Principle of **Minimum Discrimination Information** (**MDI**): given
-new facts, a new distribution f should be chosen which is as hard to
-discriminate from the original distribution
-![{\\displaystyle f\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6423b30a4c5770c59b5ab92dcb4ce378755440ab)
-as possible; so that the new data produces as small an information gain
-![{\\displaystyle D\_{\\text{KL}}(f\\parallel f\_{0})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bb8c6740dba256c3e3fcd75f18b456890882fe98)
-as possible.
-
-For example, if one had a prior distribution
-![{\\displaystyle p(x,a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f99fe2c5da9eda4261abd3a3ba9e95a582258829)
-over x and a, and subsequently learnt the true distribution of a was
-![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
-, then the relative entropy between the new joint distribution for x and a,
-![{\\displaystyle q(x\\mid a)u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d46f0614827dab0040f75c8fe54904bb7f7d1ceb)
-, and the earlier prior distribution would be:
-
-![{\\displaystyle D\_{\\text{KL}}(q(x\\mid a)u(a)\\parallel p(x,a))=\\operatorname {E} \_{u(a)}\\left\\{D\_{\\text{KL}}(q(x\\mid a)\\parallel p(x\\mid a))\\right\\}+D\_{\\text{KL}}(u(a)\\parallel p(a)),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/458e2e6ba88a2aca56c0b96b0e6d3cbb416eae16)
-
-i.e. the sum of the relative entropy of
-![{\\displaystyle p(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c07e3f3507e18c9efcf41242ba875d747d4b878f)
-the prior distribution for a from the updated distribution
-![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
-, plus the expected value (using the probability distribution
-![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
-) of the relative entropy of the prior conditional distribution
-![{\\displaystyle p(x\\mid a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6322773741893591649ace67e0dc831a98805c7f)
-from the new conditional distribution
-![{\\displaystyle q(x\\mid a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b5984976e59f1aa29d0b7c7551c26b4e19cc8fed)
-. (Note that often the later expected value is called the _conditional relative
-entropy_ (or _conditional Kullback–Leibler divergence_) and denoted by
-![{\\displaystyle D\_{\\text{KL}}(q(x\\mid a)\\parallel p(x\\mid a))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/56f618aec5f161aa05c8a4b2d5de6b458b892aba)
-[\[3\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback1959-3)[\[34\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-CoverThomas-34))
-This is minimized if
-![{\\displaystyle q(x\\mid a)=p(x\\mid a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e8b3c003ea111edc7f55cdbc05c1a0bf2b2a75e0)
-over the whole support of
-![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
-; and we note that this result incorporates Bayes' theorem, if the new
-distribution
-![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
-is in fact a δ function representing certainty that a has one particular value.
-
-MDI can be seen as an extension of
-[Laplace](https://en.wikipedia.org/wiki/Laplace "Laplace") 's
-[Principle of Insufficient Reason](https://en.wikipedia.org/wiki/Principle_of_Insufficient_Reason "Principle of Insufficient Reason")
-, and the
-[Principle of Maximum Entropy](https://en.wikipedia.org/wiki/Principle_of_Maximum_Entropy "Principle of Maximum Entropy")
-of [E.T. Jaynes](https://en.wikipedia.org/wiki/E.T._Jaynes "E.T. Jaynes") . In
-particular, it is the natural extension of the principle of maximum entropy from
-discrete to continuous distributions, for which Shannon entropy ceases to be so
-useful (see _[differential
-entropy](https://en.wikipedia.org/wiki/Differential_entropy "Differential
-entropy")_), but the relative entropy continues to be just as relevant.
-
-In the engineering literature, MDI is sometimes called the **Principle of
-Minimum Cross-Entropy** (MCE) or **Minxent** for short. Minimising relative
-entropy from m to p with respect to m is equivalent to minimizing the
-cross-entropy of p and m, since
-
-![{\\displaystyle \\mathrm {H} (p,m)=\\mathrm {H} (p)+D\_{\\text{KL}}(p\\parallel m),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/20391c494fd8cb3a15b2bf92381d91ae947934c6)
-
-which is appropriate if one is trying to choose an adequate approximation to p.
-However, this is just as often _not_ the task one is trying to achieve. Instead,
-just as often it is m that is some fixed prior reference measure, and p that one
-is attempting to optimise by minimising
-![{\\displaystyle D\_{\\text{KL}}(p\\parallel m)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/29b125efdcb1fa150f3aa7c780e359d09b1eabed)
-subject to some constraint. This has led to some ambiguity in the literature,
-with some authors attempting to resolve the inconsistency by redefining
-cross-entropy to be
-![{\\displaystyle D\_{\\text{KL}}(p\\parallel m)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/29b125efdcb1fa150f3aa7c780e359d09b1eabed)
-, rather than
-![{\\displaystyle \\mathrm {H} (p,m)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/be29a0421e4b5a6d734025bc9222ca1aa100e293)
-\[_[citation needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed
-"Wikipedia:Citation needed")_\].
-
-[![](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/ArgonKLdivergence.png/250px-ArgonKLdivergence.png)](https://en.wikipedia.org/wiki/File:ArgonKLdivergence.png)
-
-Pressure versus volume plot of available work from a mole of argon gas relative to ambient, calculated as ![{\\displaystyle T\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2991f18ef81fec2f94e3a532df91a079f3f6697d) times the Kullback–Leibler divergence
-
-[Surprisals](https://en.wikipedia.org/wiki/Surprisal "Surprisal")
-[\[37\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-37)
-add where probabilities multiply. The surprisal for an event of probability p is
-defined as
-![{\\displaystyle s=-k\\ln p}](https://wikimedia.org/api/rest_v1/media/math/render/svg/da96ff26c096baca807e01eb60c2578113324686)
-. If k is
-![{\\displaystyle \\left\\{1,1/\\ln 2,1.38\\times 10^{-23}\\right\\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4e54eedf7d41bc0d5f15addb0a39ed0ec2a63d1a)
-then surprisal is in
-![{\\displaystyle \\{}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c1288432dfd9ce5f65857a9e6bbf0d4876b403d2)
-nats, bits, or
-![{\\displaystyle J/K\\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1d48054423445e38fab2d2eeb673148b758153bd)
-so that, for instance, there are N bits of surprisal for landing all "heads" on
-a toss of N coins.
-
-Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the
-_average surprisal_ S ([entropy](https://en.wikipedia.org/wiki/Entropy
-"Entropy")) for a given set of control parameters (like pressure P or volume V).
-This constrained
-[entropy maximization](https://en.wikipedia.org/wiki/Entropy_maximization "Entropy maximization")
-, both
-classically[\[38\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-38)
-and quantum
-mechanically,[\[39\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-39)
-minimizes
-[Gibbs](https://en.wikipedia.org/wiki/Josiah_Willard_Gibbs "Josiah Willard Gibbs")
-availability in entropy
-units[\[40\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-40)
-![{\\displaystyle A\\equiv -k\\ln Z}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d71f47d2e064d6dd4eb804d3fbc85414fa6d7f91)
-where Z is a constrained multiplicity or
-[partition function](https://en.wikipedia.org/wiki/Partition_function_%28mathematics%29 "Partition function (mathematics)")
-.
-
-When temperature T is fixed, free energy (![{\\displaystyle T\\times
-A}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6081e433612df99e17fdd5dc6e444c2eafe67bc4))
-is also minimized. Thus if
-![{\\displaystyle T,V}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a191b02cede513e88db4ae4995611bb2c722eb5b)
-and number of molecules N are constant, the
-[Helmholtz free energy](https://en.wikipedia.org/wiki/Helmholtz_free_energy "Helmholtz free energy")
-![{\\displaystyle F\\equiv U-TS}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a666284cd1282b2a96582ddbb8167627298963e1)
-(where U is energy and S is entropy) is minimized as a system "equilibrates." If
-T and P are held constant (say during processes in your body), the
-[Gibbs free energy](https://en.wikipedia.org/wiki/Gibbs_free_energy "Gibbs free energy")
-![{\\displaystyle G=U+PV-TS}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a5a60159352173c3a14b7c1331ecea327889fbb0)
-is minimized instead. The change in free energy under these conditions is a
-measure of available
-[work](https://en.wikipedia.org/wiki/Work_%28thermodynamics%29 "Work (thermodynamics)")
-that might be done in the process. Thus available work for an ideal gas at
-constant temperature
-![{\\displaystyle T\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2991f18ef81fec2f94e3a532df91a079f3f6697d)
-and pressure
-![{\\displaystyle P\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/07d20f5b8e09cda22c3552e1da5bf7c7709057d9)
-is
-![{\\displaystyle W=\\Delta G=NkT\_{o}\\Theta (V/V\_{o})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/50381560c4280ba174e4372e20e67f84a3e42227)
-where
-![{\\displaystyle V\_{o}=NkT\_{o}/P\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/96f665485e0792186953b11d48a462c38377174e)
-and
-![{\\displaystyle \\Theta (x)=x-1-\\ln x\\geq 0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1bb0a30e7c143bf0e0be43f2afc4988d7b598d94)
-(see also
-[Gibbs inequality](https://en.wikipedia.org/wiki/Gibbs_inequality "Gibbs inequality")
-).
-
-More
-generally[\[41\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-41)
-the [work available](https://en.wikipedia.org/wiki/Exergy "Exergy") relative to
-some ambient is obtained by multiplying ambient temperature
-![{\\displaystyle T\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2991f18ef81fec2f94e3a532df91a079f3f6697d)
-by relative entropy or _net surprisal_
-![{\\displaystyle \\Delta I\\geq 0,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/12b2fee10216ae36191e63f22abd073f0f1cdeeb)
-defined as the average value of
-![{\\displaystyle k\\ln(p/p\_{o})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/586545929842d36fc1185e8af64546dbd6fc770c)
-where
-![{\\displaystyle p\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2c058013bfdb87708b775d59bda857fd5e6d53b3)
-is the probability of a given state under ambient conditions. For instance, the
-work available in equilibrating a monatomic ideal gas to ambient values of
-![{\\displaystyle V\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4373adc7c6ba3ad0a230674368b6a3a2df6cf98d)
-and
-![{\\displaystyle T\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2991f18ef81fec2f94e3a532df91a079f3f6697d)
-is thus
-![{\\displaystyle W=T\_{o}\\Delta I}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b33ae1f27a5912e2d27ede43e55b9a9ec187a6f3)
-, where relative entropy
-
-![{\\displaystyle \\Delta I=Nk\\left\[\\Theta {\\left({\\frac
-{V}{V\_{o}}}\\right)}+{\\frac {3}{2}}\\Theta {\\left({\\frac
-{T}{T\_{o}}}\\right)}\\right\].}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0eb6f446447587fdb6044b12548247c26fddf955)
-
-The resulting contours of constant relative entropy, shown at right for a mole
-of Argon at
-[standard temperature and pressure](https://en.wikipedia.org/wiki/Standard_temperature_and_pressure "Standard temperature and pressure")
-, for example put limits on the conversion of hot to cold as in flame-powered
-air-conditioning or in the unpowered device to convert boiling-water to
-ice-water discussed
-here.[\[42\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-42)
-Thus relative entropy measures thermodynamic availability in bits.
-
-For
-[density matrices](https://en.wikipedia.org/wiki/Density_matrix "Density matrix")
-P and Q on a
-[Hilbert space](https://en.wikipedia.org/wiki/Hilbert_space "Hilbert space") ,
-the
-[quantum relative entropy](https://en.wikipedia.org/wiki/Quantum_relative_entropy "Quantum relative entropy")
-from Q to P is defined to be
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\operatorname {Tr} (P(\\log P-\\log Q)).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/85368a6e510d46a3201422e2510f91363d0b70cc)
-
-In
-[quantum information science](https://en.wikipedia.org/wiki/Quantum_information_science "Quantum information science")
-the minimum of
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
-over all separable states Q can also be used as a measure of
-[entanglement](https://en.wikipedia.org/wiki/Quantum_entanglement "Quantum entanglement")
-in the state P.
-
-Just as relative entropy of "actual from ambient" measures thermodynamic
-availability, relative entropy of "reality from a model" is also useful even if
-the only clues we have about reality are some experimental measurements. In the
-former case relative entropy describes _distance to equilibrium_ or (when
-multiplied by ambient temperature) the amount of _available work_, while in the
-latter case it tells you about surprises that reality has up its sleeve or, in
-other words, _how much the model has yet to learn_.
-
-Although this tool for evaluating models against systems that are accessible
-experimentally may be applied in any field, its application to selecting a
-[statistical model](https://en.wikipedia.org/wiki/Statistical_model "Statistical model")
-via
-[Akaike information criterion](https://en.wikipedia.org/wiki/Akaike_information_criterion "Akaike information criterion")
-are particularly well described in
-papers[\[43\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-43)
-and a
-book[\[44\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-44)
-by Burnham and Anderson. In a nutshell the relative entropy of reality from a
-model may be estimated, to within a constant additive term, by a function of the
-deviations observed between data and the model's predictions (like the
-[mean squared deviation](https://en.wikipedia.org/wiki/Mean_squared_deviation "Mean squared deviation")
-) . Estimates of such divergence for models that share the same additive term
-can in turn be used to select among models.
-
-When trying to fit parametrized models to data there are various estimators
-which attempt to minimize relative entropy, such as
-[maximum likelihood](https://en.wikipedia.org/wiki/Maximum_likelihood "Maximum likelihood")
-and
-[maximum spacing](https://en.wikipedia.org/wiki/Maximum_spacing_estimation "Maximum spacing estimation")
-estimators.\[_[citation
-needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed
-"Wikipedia:Citation needed")_\]
-
-[Kullback & Leibler (1951)](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#CITEREFKullbackLeibler1951)
-also considered the symmetrized
-function:[\[6\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullbackLeibler195180-6)
-
-![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)+D\_{\\text{KL}}(Q\\parallel P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/11b983ff758af97a9f2703d29a748f9b0f2f27a9)
-
-which they referred to as the "divergence", though today the "KL divergence"
-refers to the asymmetric function (see
-[§ Etymology](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Etymology)
-for the evolution of the term). This function is symmetric and nonnegative, and
-had already been defined and used by
-[Harold Jeffreys](https://en.wikipedia.org/wiki/Harold_Jeffreys "Harold Jeffreys")
-in
-1948;[\[7\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEJeffreys1948158-7)
-it is accordingly called the **Jeffreys divergence**.
-
-This quantity has sometimes been used for
-[feature selection](https://en.wikipedia.org/wiki/Feature_selection "Feature selection")
-in
-[classification](https://en.wikipedia.org/wiki/Statistical_classification "Statistical classification")
-problems, where P and Q are the conditional
-[Probability density functions](https://en.wikipedia.org/wiki/Probability_density_function "Probability density function")
-of a feature under two different classes. In the Banking and Finance industries,
-this quantity is referred to as **Population Stability Index** (**PSI**), and is
-used to assess distributional shifts in model features through time.
-
-An alternative is given via the
-![{\\displaystyle \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/b43d0ea3c9c025af1be9128e62a18fa74bedda2a)
-\-divergence,
-
-![{\\displaystyle D\_{\\lambda }(P\\parallel Q)=\\lambda D\_{\\text{KL}}(P\\parallel \\lambda P+(1-\\lambda )Q)+(1-\\lambda )D\_{\\text{KL}}(Q\\parallel \\lambda P+(1-\\lambda )Q){\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ae204c7b3367027e53d52d76c97e693e6e84be10)
-
-which can be interpreted as the expected information gain about X from
-discovering which probability distribution X is drawn from, P or Q, if they
-currently have probabilities
-![{\\displaystyle \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/b43d0ea3c9c025af1be9128e62a18fa74bedda2a)
-and
-![{\\displaystyle 1-\\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/da4535bdb18a0914505c66a316f858f601be0cf8)
-respectively.\[_[clarification
-needed](https://en.wikipedia.org/wiki/Wikipedia:Please_clarify "Wikipedia:Please
-clarify")_\] \[_[citation
-needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed
-"Wikipedia:Citation needed")_\]
-
-The value
-![{\\displaystyle \\lambda =0.5}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9355d8eeec28478037abac23ce379075d1cea5fb)
-gives the
-[Jensen–Shannon divergence](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence "Jensen–Shannon divergence")
-, defined by
-
-![{\\displaystyle D\_{\\text{JS}}={\\tfrac {1}{2}}D\_{\\text{KL}}(P\\parallel M)+{\\tfrac {1}{2}}D\_{\\text{KL}}(Q\\parallel M)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/566ad76576dda350c87c38c78a63fe8ba2b1b838)
-
-where M is the average of the two distributions,
-
-![{\\displaystyle M={\\tfrac {1}{2}}\\left(P+Q\\right){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a99d3183325024a989864a82d9c02d5213b681de)
-
-We can also interpret
-![{\\displaystyle D\_{\\text{JS}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6fc42cb8ebf661fea3c6d82f5a8261d5176fbd42)
-as the capacity of a noisy information channel with two inputs giving the output
-distributions P and Q. The Jensen–Shannon divergence, like all f\-divergences,
-is _locally_ proportional to the
-[Fisher information metric](https://en.wikipedia.org/wiki/Fisher_information_metric "Fisher information metric")
-. It is similar to the
-[Hellinger metric](https://en.wikipedia.org/wiki/Hellinger_metric "Hellinger metric")
-(in the sense that it induces the same
-[affine connection](https://en.wikipedia.org/wiki/Affine_connection "Affine connection")
-on a
-[statistical manifold](https://en.wikipedia.org/wiki/Statistical_manifold "Statistical manifold")
-).
-
-Furthermore, the Jensen–Shannon divergence can be generalized using abstract
-statistical M-mixtures relying on an abstract mean
-M.[\[45\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Nielsen2019-45)[\[46\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Nielsen2020-46)
-
-There are many other important measures of
-[probability distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
-. Some of these are particularly connected with relative entropy. For example:
-
-Other notable measures of distance include the
-[Hellinger distance](https://en.wikipedia.org/wiki/Hellinger_distance "Hellinger distance")
-, _histogram intersection_, _[Chi-squared
-statistic](https://en.wikipedia.org/wiki/Chi-squared_test "Chi-squared test")_,
-_quadratic form distance_, _[match
-distance](https://en.wikipedia.org/wiki/Matching_distance "Matching distance")_,
-_[Kolmogorov–Smirnov
-distance](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
-"Kolmogorov–Smirnov test")_, and _[earth mover's
-distance](https://en.wikipedia.org/wiki/Earth_mover's_distance "Earth mover's
-distance")_.[\[49\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-earth-49)
-
-Just as _absolute_ entropy serves as theoretical background for
-[data _compression_](https://en.wikipedia.org/wiki/Data_compression "Data compression")
-, _relative_ entropy serves as theoretical background for
-[data _differencing_](https://en.wikipedia.org/wiki/Data_differencing "Data differencing")
-– the absolute entropy of a set of data in this sense being the data required to
-reconstruct it (minimum compressed size), while the relative entropy of a target
-set of data, given a source set of data, is the data required to reconstruct the
-target _given_ the source (minimum size of a
-[patch](https://en.wikipedia.org/wiki/Patch_%28computing%29 "Patch (computing)")
-).
-
-*   [Akaike information criterion](https://en.wikipedia.org/wiki/Akaike_information_criterion "Akaike information criterion")
-*   [Bayesian information criterion](https://en.wikipedia.org/wiki/Bayesian_information_criterion "Bayesian information criterion")
-*   [Bregman divergence](https://en.wikipedia.org/wiki/Bregman_divergence "Bregman divergence")
-*   [Cross-entropy](https://en.wikipedia.org/wiki/Cross-entropy "Cross-entropy")
-*   [Deviance information criterion](https://en.wikipedia.org/wiki/Deviance_information_criterion "Deviance information criterion")
-*   [Entropic value at risk](https://en.wikipedia.org/wiki/Entropic_value_at_risk "Entropic value at risk")
-*   [Entropy power inequality](https://en.wikipedia.org/wiki/Entropy_power_inequality "Entropy power inequality")
-*   [Hellinger distance](https://en.wikipedia.org/wiki/Hellinger_distance "Hellinger distance")
-*   [Information gain in decision trees](https://en.wikipedia.org/wiki/Information_gain_in_decision_trees "Information gain in decision trees")
-*   [Information gain ratio](https://en.wikipedia.org/wiki/Information_gain_ratio "Information gain ratio")
-*   [Information theory and measure theory](https://en.wikipedia.org/wiki/Information_theory_and_measure_theory "Information theory and measure theory")
-*   [Jensen–Shannon divergence](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence "Jensen–Shannon divergence")
-*   [Quantum relative entropy](https://en.wikipedia.org/wiki/Quantum_relative_entropy "Quantum relative entropy")
-*   [Solomon Kullback](https://en.wikipedia.org/wiki/Solomon_Kullback "Solomon Kullback") and [Richard Leibler](https://en.wikipedia.org/wiki/Richard_Leibler "Richard Leibler")
-*   [Bhattacharyya distance](https://en.wikipedia.org/wiki/Bhattacharyya_distance "Bhattacharyya distance")
-
-* * *
-
-14.  Bishop, Christopher M. _Pattern recognition and machine learning_. p. 55\.
-     [OCLC](https://en.wikipedia.org/wiki/OCLC_%28identifier%29 "OCLC (identifier)")
-     [1334664824](https://search.worldcat.org/oclc/1334664824) .
-     
-
-22.  Soklakov, A. N. (2023). "Information Geometry of Risks and Returns". _Risk_.
-     **June**.
-     [SSRN](https://en.wikipedia.org/wiki/SSRN_%28identifier%29 "SSRN (identifier)")
-     [4134885](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4134885) .
-     
-23.  Henide, Karim (30 September 2024). "Flow Rider: Tradable Ecosystems' Relative
-     Entropy of Flows As a Determinant of Relative Value". _The Journal of
-     Investing_. **33** (6): 34–58\.
-     [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)")
-     :[10.3905/joi.2024.1.321](https://doi.org/10.3905%2Fjoi.2024.1.321).
-     
-24.  Sanov, I.N. (1957). "On the probability of large deviations of random
-     magnitudes". _Mat. Sbornik_. **42** (84): 11–44.
-     
-
-26.  Hobson, Arthur (1971). _Concepts in statistical mechanics_. New York: Gordon and
-     Breach.
-     [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)")
-     [978-0-677-03240-5](https://en.wikipedia.org/wiki/Special:BookSources/978-0-677-03240-5 "Special:BookSources/978-0-677-03240-5")
-     .
-     
-27.  Bonnici, V. (2020). "Kullback-Leibler divergence between quantum distributions,
-     and its upper-bound".
-     [arXiv](https://en.wikipedia.org/wiki/ArXiv_%28identifier%29 "ArXiv (identifier)")
-     :[2008.05932](https://arxiv.org/abs/2008.05932)
-     \[[cs.LG](https://arxiv.org/archive/cs.LG)\].
-     
-
-29.  Donsker, Monroe D.; Varadhan, SR Srinivasa (1983). "Asymptotic evaluation of
-     certain Markov process expectations for large time. IV". _Communications on Pure
-     and Applied Mathematics_. **36** (2): 183–212\.
-     [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)")
-     :[10.1002/cpa.3160360204](https://doi.org/10.1002%2Fcpa.3160360204).
-     
-
-31.  Belov, Dmitry I.; Armstrong, Ronald D. (2011-04-15). "Distributions of the
-     Kullback-Leibler divergence with applications". _British Journal of Mathematical
-     and Statistical Psychology_. **64** (2): 291–309\.
-     [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)")
-     :[10.1348/000711010x522227](https://doi.org/10.1348%2F000711010x522227).
-     [ISSN](https://en.wikipedia.org/wiki/ISSN_%28identifier%29 "ISSN (identifier)")
-     [0007-1102](https://search.worldcat.org/issn/0007-1102) .
-     [PMID](https://en.wikipedia.org/wiki/PMID_%28identifier%29 "PMID (identifier)")
-     [21492134](https://pubmed.ncbi.nlm.nih.gov/21492134) .
-     
-32.  Buchner, Johannes (2022-04-29). _An intuition for physicists: information gain
-     from experiments_.
-     [OCLC](https://en.wikipedia.org/wiki/OCLC_%28identifier%29 "OCLC (identifier)")
-     [1363563215](https://search.worldcat.org/oclc/1363563215) .
-     
-33.  Nielsen, Frank; Garcia, Vincent (2011). "Statistical exponential families: A
-     digest with flash cards".
-     [arXiv](https://en.wikipedia.org/wiki/ArXiv_%28identifier%29 "ArXiv (identifier)")
-     :[0911.4863](https://arxiv.org/abs/0911.4863)
-     \[[cs.LG](https://arxiv.org/archive/cs.LG)\].
-     
-34.  Cover, Thomas M.; Thomas, Joy A. (1991), _Elements of Information Theory_,
-     [John Wiley & Sons](https://en.wikipedia.org/wiki/John_Wiley_&_Sons "John Wiley & Sons")
-     , p. 22
-     
-
-44.  Burnham, Kenneth P. (December 2010). _Model selection and multimodel inference:
-     a practical information-theoretic approach_. Springer.
-     [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)")
-     [978-1-4419-2973-0](https://en.wikipedia.org/wiki/Special:BookSources/978-1-4419-2973-0 "Special:BookSources/978-1-4419-2973-0")
-     .
-     [OCLC](https://en.wikipedia.org/wiki/OCLC_%28identifier%29 "OCLC (identifier)")
-     [878132909](https://search.worldcat.org/oclc/878132909) .
-     
-
-47.  Bretagnolle, J.; Huber, C. (1978), "Estimation des densités : Risque minimax",
-     _Séminaire de Probabilités XII_, Lecture Notes in Mathematics (in French), vol.
-     649, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 342–363,
-     [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)")
-     :[10.1007/bfb0064610](https://doi.org/10.1007%2Fbfb0064610),
-     [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)")
-     [978-3-540-08761-8](https://en.wikipedia.org/wiki/Special:BookSources/978-3-540-08761-8 "Special:BookSources/978-3-540-08761-8")
-     ,
-     [S2CID](https://en.wikipedia.org/wiki/S2CID_%28identifier%29 "S2CID (identifier)")
-     [122597694](https://api.semanticscholar.org/CorpusID:122597694) Lemma 2.1
-     
-
-*   [Amari, Shun-ichi](https://en.wikipedia.org/wiki/Shun'ichi_Amari "Shun'ichi Amari") (2016). _Information Geometry and Its Applications_. Applied Mathematical Sciences. Vol. 194\. Springer Japan. pp. XIII, 374. [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)"):[10.1007/978-4-431-55978-8](https://doi.org/10.1007%2F978-4-431-55978-8). [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)") [978-4-431-55977-1](https://en.wikipedia.org/wiki/Special:BookSources/978-4-431-55977-1 "Special:BookSources/978-4-431-55977-1").
-*   Kullback, Solomon (1959), _Information Theory and Statistics_, [John Wiley & Sons](https://en.wikipedia.org/wiki/John_Wiley_&_Sons "John Wiley & Sons"). Republished by [Dover Publications](https://en.wikipedia.org/wiki/Dover_Publications "Dover Publications") in 1968; reprinted in 1978: [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)") [0-8446-5625-9](https://en.wikipedia.org/wiki/Special:BookSources/0-8446-5625-9 "Special:BookSources/0-8446-5625-9").
-*   Jeffreys, Harold (1948). _Theory of Probability_ (Second ed.). Oxford University Press.
-
-*   [Information Theoretical Estimators Toolbox](https://bitbucket.org/szzoli/ite/)
-*   [Ruby gem for calculating Kullback–Leibler divergence](https://github.com/evansenter/diverge)
-*   [Jon Shlens' tutorial on Kullback–Leibler divergence and likelihood theory](https://arxiv.org/abs/1404.2000)
-*   [Matlab code for calculating Kullback–Leibler divergence for discrete distributions](http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13089&objectType=file) [Archived](https://web.archive.org/web/20070929155435/http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13089&objectType=file) 2007-09-29 at the [Wayback Machine](https://en.wikipedia.org/wiki/Wayback_Machine "Wayback Machine")
-*   [Sergio Verdú](https://en.wikipedia.org/wiki/Sergio_Verd%C3%BA "Sergio Verdú"), [Relative Entropy](http://videolectures.net/nips09_verdu_re/), [NIPS](https://en.wikipedia.org/wiki/Conference_on_Neural_Information_Processing_Systems "Conference on Neural Information Processing Systems") 2009. One-hour video lecture.
-*   [A modern summary of info-theoretic divergence measures](https://arxiv.org/abs/math/0604246)
-```
-
-### `log://turn_3/get/log%3A/turn_2/search/**`
-scheme=log action=get attrs={"path":"log://turn_2/search/**","action":"get"}
-
-```
-MANIFEST get path="log://turn_2/search/**": 1 matched (1663 tokens)
-log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi (1663)
-```
-
-### `log://turn_3/set/known%3A/kl_math/forward_backward_uniform`
-scheme=log action=set attrs={"path":"known://kl_math/forward_backward_uniform","patch":"===================================================================\n--- known://kl_math/forward_backward_uniform\told\t\n+++ known://kl_math/forward_backward_uniform\tnew\t\n@@ -0,0 +1,20 @@\n+# Related\n+[main prompt](prompt://1)\n+[unknown resolved](unknown://kl_divergence/forward_backward_relationship)\n+[search results](log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi)\n+\n+# KL(P||U) and KL(U||P) for uniform U\n+\n+KL(P||U) = H(U) - H(P) = log(V) - H(P), where H(U)=log(150000)≈11.92\n+\n+For KL(P||U)=10, H(P)≈1.92 (concentrated P).\n+\n+KL(U||P) = -sum_{i} (1/V) log(P(i)) = log(V) - sum P(i) log(P(i)) = log(V) - H(P)\n+\n+From search sources:\n+- blester125: KL(P||U) reduces to H(U) - H(P) for uniform U.\n+- Wikipedia: Confirms KL definition, relative entropy.\n+\n+Exact relation: KL(P||U) + H(P) = log(V); KL(U||P) = log(V) - H(P)\n+\n+Thus KL(P||U) = KL(U||P). Both cannot be independently 10.0.\n\\n","merge":"<<<<<<< SEARCH\n=======\n# Related\n[main prompt](prompt://1)\n[unknown resolved](unknown://kl_divergence/forward_backward_relationship)\n[search results](log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi)\n\n# KL(P||U) and KL(U||P) for uniform U\n\nKL(P||U) = H(U) - H(P) = log(V) - H(P), where H(U)=log(150000)≈11.92\n\nFor KL(P||U)=10, H(P)≈1.92 (concentrated P).\n\nKL(U||P) = -sum_{i} (1/V) log(P(i)) = log(V) - sum P(i) log(P(i)) = log(V) - H(P)\n\nFrom search sources:\n- blester125: KL(P||U) reduces to H(U) - H(P) for uniform U.\n- Wikipedia: Confirms KL definition, relative entropy.\n\nExact relation: KL(P||U) + H(P) = log(V); KL(U||P) = log(V) - H(P)\n\nThus KL(P||U) = KL(U||P). Both cannot be independently 10.0.\n>>>>>>> REPLACE","beforeTokens":0,"afterTokens":371,"summary":"kl,math,uniform,forward,backward,relationship,entropy,h_u-log_v","action":"set"}
-
-```
-# Related
-[main prompt](prompt://1)
-[unknown resolved](unknown://kl_divergence/forward_backward_relationship)
-[search results](log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi)
-
-# KL(P||U) and KL(U||P) for uniform U
-
-KL(P||U) = H(U) - H(P) = log(V) - H(P), where H(U)=log(150000)≈11.92
-
-For KL(P||U)=10, H(P)≈1.92 (concentrated P).
-
-KL(U||P) = -sum_{i} (1/V) log(P(i)) = log(V) - sum P(i) log(P(i)) = log(V) - H(P)
-
-From search sources:
-- blester125: KL(P||U) reduces to H(U) - H(P) for uniform U.
-- Wikipedia: Confirms KL definition, relative entropy.
-
-Exact relation: KL(P||U) + H(P) = log(V); KL(U||P) = log(V) - H(P)
-
-Thus KL(P||U) = KL(U||P). Both cannot be independently 10.0.
-```
-
-### `log://turn_3/set/unknown%3A/kl_divergence/forward_backward_relationship`
-scheme=log action=set attrs={}
-
-```
-unknown://kl_divergence/forward_backward_relationship set to archived
-```
-
-### `log://turn_3/update/distilled_KL_math_relationship%3B_discovering_feasibility`
-scheme=log action=update attrs={"status":155,"action":"update"}
-
-```
-distilled KL math relationship; discovering feasibility
-```
-