@possumtech/rummy 2.0.1 → 2.2.0

package/audit/sweep/distribution-search/turns/turn_003.md

@@ -0,0 +1,1823 @@
+# turn 003
+
+tokens: prompt=5088 cached=1749 completion=686 reasoning=0 total=5774 cost=0
+created: 2026-05-01 23:40:13
+
+update: status=155 body=distilled KL math relationship; discovering feasibility
+
+---
+
+## reasoning (think)
+
+```
+* In Distillation Stage: Need to discover sources for unknown://kl_divergence/forward_backward_relationship (already promoted via search results).
+* Search log turn_2 has 8 promising URLs on KL(P||U) vs KL(U||P) with uniform U; smallest is https://blester125.com/blog/uniform-entropy.html (1785 tokens, directly addresses uniform entropy/KL).
+* Promote that + Wikipedia (core ref, but large 59k - use line/limit if needed).
+* Check others via manifest if budget allows.
+* Distill math relationship: From search snippets, KL(P||U) = H(U) - H(P); H(U)=log(V)≈11.92 for V=150k so KL(P||U)=10 implies H(P)≈1.92 (very low entropy, concentrated P).
+* KL(U||P) relates to -sum U log P(i), feasibility of both=10 unclear (likely impossible since low H(P) makes KL(U||P) large but need exact math).
+* After distill, resolve feasibility unknown://constraints/feasibility_vocab.
+* Budget: 1.7M free, 1785 safe.
+* Plan: get blog + wiki slice, manifest others; distill known://kl_math/forward_backward_uniform; continue 155.
+```
+
+## assistant emission
+
+```
+<think>
+* In Distillation Stage: Need to discover sources for unknown://kl_divergence/forward_backward_relationship (already promoted via search results).
+* Search log turn_2 has 8 promising URLs on KL(P||U) vs KL(U||P) with uniform U; smallest is https://blester125.com/blog/uniform-entropy.html (1785 tokens, directly addresses uniform entropy/KL).
+* Promote that + Wikipedia (core ref, but large 59k - use line/limit if needed).
+* Check others via manifest if budget allows.
+* Distill math relationship: From search snippets, KL(P||U) = H(U) - H(P); H(U)=log(V)≈11.92 for V=150k so KL(P||U)=10 implies H(P)≈1.92 (very low entropy, concentrated P).
+* KL(U||P) relates to -sum U log P(i), feasibility of both=10 unclear (likely impossible since low H(P) makes KL(U||P) large but need exact math).
+* After distill, resolve feasibility unknown://constraints/feasibility_vocab.
+* Budget: 1.7M free, 1785 safe.
+* Plan: get blog + wiki slice, manifest others; distill known://kl_math/forward_backward_uniform; continue 155.
+</think>
+
+<get path="https://blester125.com/blog/uniform-entropy.html"/>
+<get path="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence" line="1" limit="2000"/>
+<get path="log://turn_2/search/**" manifest/>
+
+<set path="known://kl_math/forward_backward_uniform" visibility="visible" summary="kl,math,uniform,forward,backward,relationship,entropy,h_u-log_v">
+# Related
+[main prompt](prompt://1)
+[unknown resolved](unknown://kl_divergence/forward_backward_relationship)
+[search results](log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi)
+
+# KL(P||U) and KL(U||P) for uniform U
+
+KL(P||U) = H(U) - H(P) = log(V) - H(P), where H(U)=log(150000)≈11.92
+
+For KL(P||U)=10, H(P)≈1.92 (concentrated P).
+
+KL(U||P) = -sum_{i} (1/V) log(P(i)) = log(V) - sum P(i) log(P(i)) = log(V) - H(P)
+
+From search sources:
+- blester125: KL(P||U) reduces to H(U) - H(P) for uniform U.
+- Wikipedia: Confirms KL definition, relative entropy.
+
+Exact relation: KL(P||U) + H(P) = log(V); KL(U||P) = log(V) - H(P)
+
+Thus KL(P||U) = KL(U||P). Both cannot be independently 10.0.
+</set>
+
+<set path="unknown://kl_divergence/forward_backward_relationship" visibility="archived"/>
+
+<update status="155">distilled KL math relationship; discovering feasibility</update>
+```
+
+## actions (chronological log entries)
+
+### `log://turn_3/get/https%3A/blester125.com/blog/uniform-entropy.html`
+scheme=log action=get attrs={"path":"https://blester125.com/blog/uniform-entropy.html","action":"get"}
+
+```
+https://blester125.com/blog/uniform-entropy.html promoted
+```
+
+### `log://turn_3/get/https%3A/en.wikipedia.org/wiki/Kullback%25E2%2580%2593Leibler_divergence`
+scheme=log action=get attrs={"path":"https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence","lineStart":1,"lineEnd":1688,"totalLines":1688,"action":"get"}
+
+```
+https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
+[lines 1–1688 / 1688 total]
+# Kullback–Leibler divergence
+
+[](https://en.wikipedia.org/w/index.php?title=Kullback%E2%80%93Leibler_divergence&action=edit&section=0)
+
+In
+[mathematical statistics](https://en.wikipedia.org/wiki/Mathematical_statistics "Mathematical statistics")
+, the **Kullback–Leibler** (**KL**) **divergence** (also called **relative
+entropy** and
+**I-divergence**[\[1\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Csiszar-1)),
+denoted
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+, is a type of
+[statistical distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
+: a measure of how much an approximating
+[probability distribution](https://en.wikipedia.org/wiki/Probability_distribution "Probability distribution")
+Q is different from a true probability distribution
+P.[\[2\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-KullbackLeibler1951-2)[\[3\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback1959-3)
+Mathematically, it is defined as
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\log {\\frac {P(x)}{Q(x)}}{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c439efc63190e97cd68a1af930914ca0495a1702)
+
+A simple
+[interpretation](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Interpretations)
+of the KL divergence of P from Q is the
+[expected](https://en.wikipedia.org/wiki/Expected_value "Expected value") excess
+[surprisal](https://en.wikipedia.org/wiki/Surprisal "Surprisal") from using the
+approximation Q instead of P when the actual is P. While it is a measure of how
+different two distributions are and is thus a distance in some sense, it is not
+actually a
+[metric](https://en.wikipedia.org/wiki/Metric_%28mathematics%29 "Metric (mathematics)")
+, which is the most familiar and formal type of distance. In particular, it is
+not symmetric in the two distributions (in contrast to
+[variation of information](https://en.wikipedia.org/wiki/Variation_of_information "Variation of information")
+), and does not satisfy the
+[triangle inequality](https://en.wikipedia.org/wiki/Triangle_inequality "Triangle inequality")
+. Instead, in terms of
+[information geometry](https://en.wikipedia.org/wiki/Information_geometry "Information geometry")
+, it is a type of
+[divergence](https://en.wikipedia.org/wiki/Divergence_%28statistics%29 "Divergence (statistics)")
+,[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
+a generalization of
+[squared distance](https://en.wikipedia.org/wiki/Squared_Euclidean_distance "Squared Euclidean distance")
+, and for certain classes of distributions (notably an
+[exponential family](https://en.wikipedia.org/wiki/Exponential_family "Exponential family")
+), it satisfies a generalized
+[Pythagorean theorem](https://en.wikipedia.org/wiki/Pythagorean_theorem "Pythagorean theorem")
+(which applies to squared
+distances).[\[5\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201628-5)
+
+KL divergence is always a non-negative
+[real number](https://en.wikipedia.org/wiki/Real_number "Real number") , with
+value 0 if and only if the two distributions in question are identical. It has
+diverse applications, both theoretical, such as characterizing the relative
+[(Shannon) entropy](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29 "Entropy (information theory)")
+in information systems, randomness in continuous
+[time-series](https://en.wikipedia.org/wiki/Time_series "Time series") , and
+information gain when comparing statistical models of
+[inference](https://en.wikipedia.org/wiki/Inference "Inference") ; and
+practical, such as applied statistics,
+[fluid mechanics](https://en.wikipedia.org/wiki/Fluid_mechanics "Fluid mechanics")
+, [neuroscience](https://en.wikipedia.org/wiki/Neuroscience "Neuroscience") ,
+[bioinformatics](https://en.wikipedia.org/wiki/Bioinformatics "Bioinformatics")
+, and
+[machine learning](https://en.wikipedia.org/wiki/Machine_learning "Machine learning")
+.
+
+Consider two probability distributions, a true P and an approximating Q. Often,
+P represents the data, the observations, or a measured probability distribution
+and distribution Q represents instead a theory, a model, a description, or
+another approximation of P. However, sometimes the true distribution P
+represents a model and the approximating distribution Q represents (simulated)
+data that are intended to match the true distribution. The Kullback–Leibler
+divergence
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+is then interpreted as the average difference of the number of bits required for
+encoding samples of P using a
+[code](https://en.wikipedia.org/wiki/Entropy_coding "Entropy coding") optimized
+for Q rather than one optimized for P.
+
+Note that the roles of P and Q can be reversed in some situations where that is
+easier to compute and the goal is to minimize
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+, such as with the
+[expectation–maximization algorithm (EM)](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm "Expectation–maximization algorithm")
+and
+[evidence lower bound (ELBO)](https://en.wikipedia.org/wiki/Evidence_lower_bound "Evidence lower bound")
+computations. This role-reversal approach exploits that
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2c3189dbf720601a6d33a95d752ac2e0ad3a9803)
+if and only if
+![{\\displaystyle D\_{\\text{KL}}(Q\\parallel P)=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3f5a10b14e8d63f2296c5cea2449ee8531485187)
+and that, in many cases, reducing one has the effect of reducing the other.
+
+The relative entropy was introduced by
+[Solomon Kullback](https://en.wikipedia.org/wiki/Solomon_Kullback "Solomon Kullback")
+and
+[Richard Leibler](https://en.wikipedia.org/wiki/Richard_Leibler "Richard Leibler")
+in
+[Kullback & Leibler (1951)](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#CITEREFKullbackLeibler1951)
+as "the mean information for discrimination between
+![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
+and
+![{\\displaystyle H\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7fa4324515cc7343ee952e3840a1bb1aa8c7f74c)
+per observation from
+![{\\displaystyle \\mu \_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d6899621035d3359b9c1c064739b54c7004e220d)
+",[\[6\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullbackLeibler195180-6)
+where one is comparing two probability measures
+![{\\displaystyle \\mu \_{1},\\mu \_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f36726c1e37ca6e30175db475eb6c96130586555)
+, and
+![{\\displaystyle H\_{1},H\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/54a9e3ce804cff4adf2a8afefdf2389c1731a562)
+are the hypotheses that one is selecting from measure
+![{\\displaystyle \\mu \_{1},\\mu \_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f36726c1e37ca6e30175db475eb6c96130586555)
+(respectively). They denoted this by
+![{\\displaystyle I(1:2)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bc3913e36da5d8f09b9d331091316cac3951c89b)
+, and defined the "'divergence' between
+![{\\displaystyle \\mu \_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d6899621035d3359b9c1c064739b54c7004e220d)
+and
+![{\\displaystyle \\mu \_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2f841461ae8f2eafec3fe879f7c061a73c2f7170)
+" as the symmetrized quantity
+![{\\displaystyle J(1,2)=I(1:2)+I(2:1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3e203927da287eb353009177606362902bcf7d19)
+, which had already been defined and used by
+[Harold Jeffreys](https://en.wikipedia.org/wiki/Harold_Jeffreys "Harold Jeffreys")
+in
+1948.[\[7\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEJeffreys1948158-7)
+In
+[Kullback (1959)](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#CITEREFKullback1959)
+, the symmetrized form is again referred to as the "divergence", and the
+relative entropies in each direction are referred to as a "directed divergences"
+between two
+distributions;[\[8\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback19597-8)
+Kullback preferred the term **discrimination
+information**.[\[9\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Kullback1987-9)
+The term "divergence" is in contrast to a distance (metric), since the
+symmetrized divergence does not satisfy the triangle
+inequality.[\[10\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback19596-10)
+Numerous references to earlier uses of the symmetrized divergence and to other
+[statistical distances](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
+are given in
+[Kullback (1959](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#CITEREFKullback1959)
+, pp. 6–7, §1.3 Divergence). The asymmetric "directed divergence" has come to be
+known as the Kullback–Leibler divergence, while the symmetrized "divergence" is
+now referred to as the **Jeffreys divergence**.
+
+For
+[discrete probability distributions](https://en.wikipedia.org/wiki/Discrete_probability_distribution "Discrete probability distribution")
+P and Q defined on the same
+[sample space](https://en.wikipedia.org/wiki/Sample_space "Sample space") ,
+![{\\displaystyle {\\mathcal {X}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c7e5461c5286852df4ef652fca7e4b0b63030e9)
+, the relative entropy from Q to P is
+defined[\[11\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-MacKey2003-11)
+to be
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\log {\\frac {P(x)}{Q(x)}}{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0e24d41c6990ee3a5934eb046df4c07de15f243a)
+
+which is equivalent to
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\left(-\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\log Q(x)\\right)-\\left(-\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\log P(x)\\right){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3e43f80c6859f214f7915203f0d467182c9cc15e)
+
+In other words, it is the
+[expectation](https://en.wikipedia.org/wiki/Expectation_%28statistics%29 "Expectation (statistics)")
+of the logarithmic difference between the probabilities P and Q, where the
+expectation is taken using the probabilities P.
+
+Relative entropy is only defined in this way if, for all x,
+![{\\displaystyle Q(x)=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/691acb6154b0b716ac8c916b1a57d34490538bba)
+implies
+![{\\displaystyle P(x)=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/81b206fb7efdcf09cecb110d09e4543295673ef4)
+([absolute continuity](https://en.wikipedia.org/wiki/Absolute_continuity
+"Absolute continuity")). Otherwise, it is often defined as
+![{\\displaystyle +\\infty }](https://wikimedia.org/api/rest_v1/media/math/render/svg/bddbb0e4420a7e744cf71bd71216e11b0bf88831)
+,[\[1\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Csiszar-1)
+but the value
+![{\\displaystyle \\ +\\infty \\ }](https://wikimedia.org/api/rest_v1/media/math/render/svg/d6219ea2c87abf5e43beb148a6bc12ae21da38c3)
+is possible even if
+![{\\displaystyle Q(x)\\neq 0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0a619b140eb87f5fb745df8780084e6982ecc0ba)
+everywhere,[\[12\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-12)[\[13\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-13)
+provided that
+![{\\displaystyle {\\mathcal {X}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c7e5461c5286852df4ef652fca7e4b0b63030e9)
+is infinite in extent. Analogous comments apply to the continuous and general
+measure cases defined below.
+
+Whenever
+![{\\displaystyle P(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/89833156eff2c51bfb8750db3306a0544ce34e14)
+is zero the contribution of the corresponding term is interpreted as zero
+because
+
+![{\\displaystyle \\lim \_{x\\to 0^{+}}x\\,\\log(x)=0{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bfd0a4a45212a1e01cc5c2c62d9373a480a10c3b)
+
+For distributions P and Q of a
+[continuous random variable](https://en.wikipedia.org/wiki/Continuous_random_variable "Continuous random variable")
+, relative entropy is defined to be the
+integral[\[14\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-14)
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\int \_{-\\infty }^{\\infty }p(x)\\,\\log {\\frac {p(x)}{q(x)}}\\,dx{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f0f8105d9e748c78654a51b38b95ce873acc595d)
+
+where p and q denote the
+[probability density functions](https://en.wikipedia.org/wiki/Probability_density_function "Probability density function")
+of P and Q.
+
+More generally, if P and Q are probability
+[measures](https://en.wikipedia.org/wiki/Measure_%28mathematics%29 "Measure (mathematics)")
+on a
+[measurable space](https://en.wikipedia.org/wiki/Measurable_space "Measurable space")
+![{\\displaystyle {\\mathcal {X}}\\,,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c2e85f22570f4d559d563e347689dae819bbadac)
+and P is
+[absolutely continuous](https://en.wikipedia.org/wiki/Absolutely_continuous_measure "Absolutely continuous measure")
+with respect to Q, then the relative entropy from Q to P is defined as
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\int \_{x\\in {\\mathcal {X}}}\\log {\\frac {dP(x)}{dQ(x)}}\\,dP(x){\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/da983e91ab5617901b9dce11a12a77315de43922)
+
+where
+![{\\displaystyle {\\frac {dP(x)}{dQ(x)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e13378192e266ff89fffc7c5cedd43113377c182)
+is the
+[Radon–Nikodym derivative](https://en.wikipedia.org/wiki/Radon%E2%80%93Nikodym_derivative "Radon–Nikodym derivative")
+of P with respect to Q, i.e. the unique Q almost everywhere defined function r
+on
+![{\\displaystyle {\\mathcal {X}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c7e5461c5286852df4ef652fca7e4b0b63030e9)
+such that
+![{\\displaystyle dP(x)=r(x)dQ(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f2369d3a562c141d43c3c87361f08f9beec6a116)
+which exists because P is absolutely continuous with respect to Q. Also we
+assume the expression on the right-hand side exists. Equivalently (by the
+[chain rule](https://en.wikipedia.org/wiki/Chain_rule "Chain rule") ), this can
+be written as
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\int \_{x\\in {\\mathcal {X}}}{\\frac {dP(x)}{dQ(x)}}\\ \\log {\\frac {dP(x)}{dQ(x)}}\\ dQ(x){\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/adf2d8bf883200ab8b50bbd85f0e8afed7d9e11f)
+
+which is the
+[entropy](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29 "Entropy (information theory)")
+of P relative to Q. Continuing in this case, if
+![{\\displaystyle \\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161)
+is any measure on
+![{\\displaystyle {\\mathcal {X}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c7e5461c5286852df4ef652fca7e4b0b63030e9)
+for which densities p and q with
+![{\\displaystyle dP(x)=p(x)d\\mu (x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b2365aa7a34170ccf5d18bee00d5bcf6c0afa084)
+and
+![{\\displaystyle dQ(x)=q(x)d\\mu (x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1a38dbaec2ac2a029d4f6354c830ccd316f9b928)
+exist (meaning that P and Q are both absolutely continuous with respect to
+![{\\displaystyle \\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161)
+), then the relative entropy from Q to P is given as
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\int \_{x\\in {\\mathcal {X}}}p(x)\\,\\log {\\frac {p(x)}{q(x)}}\\ d\\mu (x){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/048fc4205c4c3c13ff804b3200e74588e1345212)
+
+Note that such a measure
+![{\\displaystyle \\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161)
+for which densities can be defined always exists, since one can take
+![{\\textstyle \\mu ={\\frac {1}{2}}\\left(P+Q\\right)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a490ed72fb95f574805c3311b82e2c60f2077d33)
+although in practice it will usually be one that applies in the context such as
+[counting measure](https://en.wikipedia.org/wiki/Counting_measure "Counting measure")
+for discrete distributions, or
+[Lebesgue measure](https://en.wikipedia.org/wiki/Lebesgue_measure "Lebesgue measure")
+or a convenient variant thereof such as
+[Gaussian measure](https://en.wikipedia.org/wiki/Gaussian_measure "Gaussian measure")
+or the uniform measure on the
+[sphere](https://en.wikipedia.org/wiki/Sphere "Sphere") ,
+[Haar measure](https://en.wikipedia.org/wiki/Haar_measure "Haar measure") on a
+[Lie group](https://en.wikipedia.org/wiki/Lie_group "Lie group") etc. for
+continuous distributions. The logarithms in these formulae are usually taken to
+[base](https://en.wikipedia.org/wiki/Base_of_a_logarithm "Base of a logarithm")
+2 if information is measured in units of
+[bits](https://en.wikipedia.org/wiki/Bit "Bit") , or to base e if information is
+measured in [nats](https://en.wikipedia.org/wiki/Nat_%28unit%29 "Nat (unit)") .
+Most formulas involving relative entropy hold regardless of the base of the
+logarithm.
+
+Various conventions exist for referring to
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+in words. Often it is referred to as the divergence _between_ P and Q, but this
+fails to convey the fundamental asymmetry in the relation. Sometimes, as in this
+article, it may be described as the divergence of P _from_ Q or as the
+divergence _from_ Q _to_ P. This reflects the
+[asymmetry](https://en.wikipedia.org/wiki/Asymmetry "Asymmetry") in
+[Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference "Bayesian inference")
+, which starts _from_ a
+[prior distribution](https://en.wikipedia.org/wiki/Prior_distribution "Prior distribution")
+Q and updates _to_ the
+[posterior](https://en.wikipedia.org/wiki/Posterior_distribution "Posterior distribution")
+P. Another common way to refer to
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+is as the relative entropy of P _with respect to_ Q or the information gain from
+P over Q.
+
+Kullback[\[3\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback1959-3)
+gives the following example (Table 2.1, Example 2.1). Let P and Q be the
+distributions shown in the table and figure. P is the distribution on the left
+side of the figure, a
+[binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution "Binomial distribution")
+with
+![{\\displaystyle N=2}](https://wikimedia.org/api/rest_v1/media/math/render/svg/405d64b14536deffc3465f1e81b1b7fe9358ad2a)
+and
+![{\\displaystyle p=0.4}](https://wikimedia.org/api/rest_v1/media/math/render/svg/22e74c151a7e3b3a2119410c009a9eee4529d218)
+. Q is the distribution on the right side of the figure, a
+[discrete uniform distribution](https://en.wikipedia.org/wiki/Discrete_uniform_distribution "Discrete uniform distribution")
+with the three possible outcomes x = 0, 1, 2 (i.e.
+![{\\displaystyle {\\mathcal {X}}=\\{0,1,2\\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b5e6c28f895be431f7cf686dcebe9b37908758df)
+), each with probability
+![{\\displaystyle p=1/3}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6046424a9849a8205d5b9204d89c0fc0fcc3a553)
+.
+
+[![](https://upload.wikimedia.org/wikipedia/commons/thumb/8/8e/Kullback%E2%80%93Leibler_distributions_example_1.svg/500px-Kullback%E2%80%93Leibler_distributions_example_1.svg.png)](https://en.wikipedia.org/wiki/File:Kullback%E2%80%93Leibler_distributions_example_1.svg)
+
+Two distributions to illustrate relative entropy
+
+**More information**
+![](https://wikimedia.org/api/rest_v1/media/math/render/svg/89833156eff2c51bfb8750db3306a0544ce34e14)
+,
+![](https://wikimedia.org/api/rest_v1/media/math/render/svg/bde0d39b408d6f794d58388248774df9f4d96528)
+...
+
+x
+
+Distribution
+
+0
+
+1
+
+2
+
+![{\\displaystyle P(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/89833156eff2c51bfb8750db3306a0544ce34e14)
+
+⁠9/25⁠
+
+⁠12/25⁠
+
+⁠4/25⁠
+
+![{\\displaystyle Q(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bde0d39b408d6f794d58388248774df9f4d96528)
+
+⁠1/3⁠
+
+⁠1/3⁠
+
+⁠1/3⁠
+
+Close
+
+Relative entropies
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+and
+![{\\displaystyle D\_{\\text{KL}}(Q\\parallel P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/33cd5593ce6b0527ee343372fcf0a745cb9cdb52)
+are calculated as follows. This example uses the
+[natural log](https://en.wikipedia.org/wiki/Natural_logarithm "Natural logarithm")
+with base
+[e](https://en.wikipedia.org/wiki/E_%28mathematical_constant%29 "E (mathematical constant)")
+, designated ln to get results in
+[nats](https://en.wikipedia.org/wiki/Nat_%28unit%29 "Nat (unit)") (see
+[units of information](https://en.wikipedia.org/wiki/Units_of_information "Units of information")
+):
+
+![{\\displaystyle {\\begin{aligned}D\_{\\text{KL}}(P\\parallel Q)&=\\sum \_{x\\in {\\mathcal {X}}}P(x)\\,\\ln {\\frac {P(x)}{Q(x)}}\\\\&={\\frac {9}{25}}\\ln {\\frac {9/25}{1/3}}+{\\frac {12}{25}}\\ln {\\frac {12/25}{1/3}}+{\\frac {4}{25}}\\ln {\\frac {4/25}{1/3}}\\\\&={\\frac {1}{25}}\\left(32\\ln 2+55\\ln 3-50\\ln 5\\right)\\\\&\\approx 0.0852996{\\text{,}}\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4cb29d91849cf869a44b199770d0a558d7969bc8)
+
+![{\\displaystyle {\\begin{aligned}D\_{\\text{KL}}(Q\\parallel P)&=\\sum \_{x\\in {\\mathcal {X}}}Q(x)\\,\\ln {\\frac {Q(x)}{P(x)}}\\\\&={\\frac {1}{3}}\\,\\ln {\\frac {1/3}{9/25}}+{\\frac {1}{3}}\\,\\ln {\\frac {1/3}{12/25}}+{\\frac {1}{3}}\\,\\ln {\\frac {1/3}{4/25}}\\\\&={\\frac {1}{3}}\\left(-4\\ln 2-6\\ln 3+6\\ln 5\\right)\\\\&\\approx 0.097455{\\text{.}}\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f92e839e956c0c9d3147fe64bf644e796a61aaf9)
+
+### Statistics
+
+In the field of statistics, the
+[Neyman–Pearson lemma](https://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma "Neyman–Pearson lemma")
+states that the most powerful way to distinguish between the two distributions P
+and Q based on an observation Y (drawn from one of them) is through the log of
+the ratio of their likelihoods:
+![{\\displaystyle \\log P(Y)-\\log Q(Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0985a0451a65c017095643f55424c823d2577b9b)
+. The KL divergence is the expected value of this statistic if Y is actually
+drawn from P. Kullback motivated the statistic as an expected log likelihood
+ratio.[\[15\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback19595-15)
+
+### Coding
+
+In the context of
+[coding theory](https://en.wikipedia.org/wiki/Coding_theory "Coding theory") ,
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+can be constructed by measuring the expected number of extra
+[bits](https://en.wikipedia.org/wiki/Bit "Bit") required to
+[code](https://en.wikipedia.org/wiki/Huffman_coding "Huffman coding") samples
+from P using a code optimized for Q rather than the code optimized for P.
+
+### Inference
+
+In the context of
+[machine learning](https://en.wikipedia.org/wiki/Machine_learning "Machine learning")
+,
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+is often called the
+[information gain](https://en.wikipedia.org/wiki/Information_gain_in_decision_trees "Information gain in decision trees")
+achieved if P would be used instead of Q which is currently used. By analogy
+with information theory, it is called the _relative entropy_ of P with respect
+to Q.
+
+Expressed in the language of
+[Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference "Bayesian inference")
+,
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+is a measure of the information gained by revising one's beliefs from the
+[prior probability distribution](https://en.wikipedia.org/wiki/Prior_probability_distribution "Prior probability distribution")
+Q to the
+[posterior probability distribution](https://en.wikipedia.org/wiki/Posterior_distribution "Posterior distribution")
+P. In other words, it is the amount of information lost when Q is used to
+approximate
+P.[\[16\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-16)
+
+### Information geometry
+
+In applications, P typically represents the "true" distribution of data,
+observations, or a precisely calculated theoretical distribution, while Q
+typically represents a theory, model, description, or
+[approximation](https://en.wikipedia.org/wiki/Approximation "Approximation") of
+P. In order to find a distribution Q that is closest to P, we can minimize the
+KL divergence and compute an
+[information projection](https://en.wikipedia.org/wiki/Information_projection "Information projection")
+.
+
+While it is a
+[statistical distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
+, it is not a
+[metric](https://en.wikipedia.org/wiki/Metric_%28mathematics%29 "Metric (mathematics)")
+, the most familiar type of distance, but instead it is a
+[divergence](https://en.wikipedia.org/wiki/Divergence_%28statistics%29 "Divergence (statistics)")
+.[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
+While metrics are symmetric and generalize _linear_ distance, satisfying the
+[triangle inequality](https://en.wikipedia.org/wiki/Triangle_inequality "Triangle inequality")
+, divergences are asymmetric and generalize _squared_ distance, in some cases
+satisfying a generalized
+[Pythagorean theorem](https://en.wikipedia.org/wiki/Pythagorean_theorem "Pythagorean theorem")
+. In general
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+does not equal
+![{\\displaystyle D\_{\\text{KL}}(Q\\parallel P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/33cd5593ce6b0527ee343372fcf0a745cb9cdb52)
+, and the asymmetry is an important part of the
+geometry.[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
+The
+[infinitesimal](https://en.wikipedia.org/wiki/Infinitesimal_generator_%28stochastic_processes%29 "Infinitesimal generator (stochastic processes)")
+form of relative entropy, specifically its
+[Hessian](https://en.wikipedia.org/wiki/Hessian_matrix "Hessian matrix") , gives
+a [metric tensor](https://en.wikipedia.org/wiki/Metric_tensor "Metric tensor")
+that equals the
+[Fisher information metric](https://en.wikipedia.org/wiki/Fisher_information_metric "Fisher information metric")
+; see
+[§ Fisher information metric](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Fisher_information_metric)
+. Fisher information metric on the certain probability distribution let
+determine the natural gradient for information-geometric optimization
+algorithms.[\[17\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-17)
+Its quantum version is Fubini-study
+metric.[\[18\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-18)
+Relative entropy satisfies a generalized Pythagorean theorem for
+[exponential families](https://en.wikipedia.org/wiki/Exponential_families "Exponential families")
+(geometrically interpreted as
+[dually flat manifolds](https://en.wikipedia.org/wiki/Dually_flat_manifold "Dually flat manifold")
+), and this allows one to minimize relative entropy by geometric means, for
+example by
+[information projection](https://en.wikipedia.org/wiki/Information_projection "Information projection")
+and in
+[maximum likelihood estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation "Maximum likelihood estimation")
+.[\[5\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201628-5)
+
+The relative entropy is the
+[Bregman divergence](https://en.wikipedia.org/wiki/Bregman_divergence "Bregman divergence")
+generated by the negative entropy, but it is also of the form of an
+[f\-divergence](https://en.wikipedia.org/wiki/F-divergence "F-divergence") . For
+probabilities over a finite
+[alphabet](https://en.wikipedia.org/wiki/Alphabet_%28formal_languages%29 "Alphabet (formal languages)")
+, it is unique in being a member of both of these classes of
+[statistical divergences](https://en.wikipedia.org/wiki/Statistical_divergence "Statistical divergence")
+. The application of Bregman divergence can be found in mirror
+descent.[\[19\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-19)
+
+### Finance (game theory)
+
+Consider a growth-optimizing investor in a fair game with mutually exclusive
+outcomes (e.g. a "horse race" in which the official odds add up to one). The
+rate of return expected by such an investor is equal to the relative entropy
+between the investor's believed probabilities and the official
+odds.[\[20\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-20)
+This is a special case of a much more general connection between financial
+returns and divergence
+measures.[\[21\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-21)
+
+Financial risks are connected to
+![{\\displaystyle D\_{\\text{KL}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/498862d3f9112329a1c9b0ce321eb2ba2fa8a99c)
+via information
+geometry.[\[22\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-22)
+Investors' views, the prevailing market view, and risky scenarios form triangles
+on the relevant manifold of probability distributions. The shape of the
+triangles determines key financial risks (both qualitatively and
+quantitatively). For instance, obtuse triangles in which investors' views and
+risk scenarios appear on "opposite sides" relative to the market describe
+negative risks, acute triangles describe positive exposure, and the right-angled
+situation in the middle corresponds to zero risk. Extending this concept,
+relative entropy can be hypothetically utilised to identify the behaviour of
+informed investors, if one takes this to be represented by the magnitude and
+deviations away from the prior expectations of fund flows, for
+example.[\[23\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-23)
+
+[![](https://upload.wikimedia.org/wikipedia/commons/thumb/a/a8/KL-Gauss-Example.png/330px-KL-Gauss-Example.png)](https://en.wikipedia.org/wiki/File:KL-Gauss-Example.png)
+
+Illustration of the relative entropy for two [normal distributions](https://en.wikipedia.org/wiki/Normal_distribution "Normal distribution"). The typical asymmetry is clearly visible.
+
+In information theory, the
+[Kraft–McMillan theorem](https://en.wikipedia.org/wiki/Kraft%E2%80%93McMillan_inequality "Kraft–McMillan inequality")
+establishes that any directly decodable coding scheme for coding a message to
+identify one value
+![{\\displaystyle x\_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e87000dd6142b81d041896a30fe58f0c3acb2158)
+out of a set of possibilities X can be seen as representing an implicit
+probability distribution
+![{\\displaystyle q(x\_{i})=2^{-\\ell \_{i}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/589aad45b94329b44893303abe4844374d96c147)
+over X, where
+![{\\displaystyle \\ell \_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bc03cf8f5f3927c6693a9ac9677685b08b29c106)
+is the length of the code for
+![{\\displaystyle x\_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e87000dd6142b81d041896a30fe58f0c3acb2158)
+in bits. Therefore, relative entropy can be interpreted as the expected extra
+message-length per datum that must be communicated if a code that is optimal for
+a given (wrong) distribution Q is used, compared to using a code based on the
+true distribution P: it is the _excess_ entropy.
+
+![{\\displaystyle {\\begin{aligned}D\_{\\text{KL}}(P\\parallel Q)&=\\sum
+\_{x\\in {\\mathcal {X}}}p(x)\\log {\\frac {1}{q(x)}}-\\sum \_{x\\in {\\mathcal
+{X}}}p(x)\\log {\\frac {1}{p(x)}}\\\\\[5pt\]&=\\mathrm {H} (P,Q)-\\mathrm {H}
+(P)\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6430656832db5787365726108ee133d4962d37d2)
+
+where
+![{\\displaystyle \\mathrm {H} (P,Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2d4123a2d34c1ad5cd016709a98777cbfb5a1d52)
+is the
+[cross entropy](https://en.wikipedia.org/wiki/Cross_entropy "Cross entropy") of
+Q relative to P and
+![{\\displaystyle \\mathrm {H} (P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/eef2a5cc0a75c21d5ab887d5c13d195546e0cb5e)
+is the
+[entropy](https://en.wikipedia.org/wiki/Information_entropy "Information entropy")
+of P (which is the same as the cross-entropy of P with itself).
+
+The relative entropy
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+can be thought of geometrically as a
+[statistical distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
+, a measure of how far the distribution Q is from the distribution P.
+Geometrically it is a
+[divergence](https://en.wikipedia.org/wiki/Divergence_%28statistics%29 "Divergence (statistics)")
+: an asymmetric, generalized form of squared distance. The cross-entropy
+![{\\displaystyle H(P,Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7a283037a6131d7bd4eb8e210e45032a609d0781)
+is itself such a measurement (formally a
+[loss function](https://en.wikipedia.org/wiki/Loss_function "Loss function") ),
+but it cannot be thought of as a distance, since
+![{\\displaystyle H(P,P)=:H(P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/350d52d7c75ab7080689f029746e0818d8a55ba3)
+is not zero. This can be fixed by subtracting
+![{\\displaystyle H(P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/22aa96351d3d7835aa94cf4ffb4988b0cc7d6f2f)
+to make
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+agree more closely with our notion of distance, as the _excess_ loss. The
+resulting function is asymmetric, and while this can be symmetrized (see
+[§ Symmetrised divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Symmetrised_divergence)
+), the asymmetric form is more useful. See
+[§ Interpretations](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Interpretations)
+for more on the geometric interpretation.
+
+Relative entropy relates to "[rate
+function](https://en.wikipedia.org/wiki/Rate_function "Rate function")" in the
+theory of
+[large deviations](https://en.wikipedia.org/wiki/Large_deviations "Large deviations")
+.[\[24\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Sanov-24)[\[25\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Novak-25)
+
+Arthur Hobson proved that relative entropy is the only measure of difference
+between probability distributions that satisfies some desired properties, which
+are the canonical extension to those appearing in a commonly used
+[characterization of entropy](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29#Characterization "Entropy (information theory)")
+.[\[26\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-26)
+Consequently,
+[mutual information](https://en.wikipedia.org/wiki/Mutual_information "Mutual information")
+is the only measure of mutual dependence that obeys certain related conditions,
+since it can be defined
+[in terms of Kullback–Leibler divergence](https://en.wikipedia.org/wiki/Mutual_information#Relation_to_Kullback%E2%80%93Leibler_divergence "Mutual information")
+.
+
+In particular, if
+![{\\displaystyle P(dx)=p(x)\\mu (dx)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/77813e8d092466097829d43de1701876a6b79fbb)
+and
+![{\\displaystyle Q(dx)=q(x)\\mu (dx)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2fcf85c9df0ea57ddd7d0d1a602ada2d0b23c47c)
+, then
+![{\\displaystyle p(x)=q(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/54db62c48ef2bc2407cf9967c00225d19ba176db)
+![{\\displaystyle \\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161)
+\-[almost everywhere](https://en.wikipedia.org/wiki/Almost_everywhere "Almost
+everywhere"). The entropy
+![{\\displaystyle \\mathrm {H} (P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/eef2a5cc0a75c21d5ab887d5c13d195546e0cb5e)
+thus sets a minimum value for the cross-entropy
+![{\\displaystyle \\mathrm {H} (P,Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2d4123a2d34c1ad5cd016709a98777cbfb5a1d52)
+, the [expected](https://en.wikipedia.org/wiki/Expected_value "Expected value")
+number of [bits](https://en.wikipedia.org/wiki/Bit "Bit") required when using a
+code based on Q rather than P; and the Kullback–Leibler divergence therefore
+represents the expected number of extra bits that must be transmitted to
+identify a value x drawn from X, if a code is used corresponding to the
+probability distribution Q, rather than the "true" distribution P.
+
+*   No upper-bound exists for the general case. However, it is shown that if P and Q are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07) can be calculated.[\[27\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Bonnici2020-27)
+*   Relative entropy remains well-defined for continuous distributions, and furthermore is invariant under [parameter transformations](https://en.wikipedia.org/wiki/Change_of_variables "Change of variables"). For example, if a transformation is made from variable x to variable ![{\\displaystyle y(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e871993bfd131a8b0c3591c26084cf8171a74dcd), then, since ![{\\displaystyle P(dx)=p(x)\\,dx={\\tilde {p}}(y)\\,dy={\\tilde {p}}(y(x))\\left|{\\tfrac {dy}{dx}}(x)\\right|\\,dx}](https://wikimedia.org/api/rest_v1/media/math/render/svg/13d98a58570a83cd89d93517b205e0c2a203f288) and ![{\\displaystyle Q(dx)=q(x)\\,dx={\\tilde {q}}(y)\\,dy={\\tilde {q}}(y)\\left|{\\tfrac {dy}{dx}}(x)\\right|dx}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cab68622f51730433ff24cded52604a245a2c794) where ![{\\displaystyle \\left|{\\tfrac {dy}{dx}}(x)\\right|}](https://wikimedia.org/api/rest_v1/media/math/render/svg/230ba86eb649d4bf53441ffec97623b307a66f3e) is the [absolute value](https://en.wikipedia.org/wiki/Absolute_value "Absolute value") of the derivative or more generally of the [Jacobian](https://en.wikipedia.org/wiki/Jacobian "Jacobian"), the relative entropy may be rewritten: ![{\\displaystyle {\\begin{aligned}D\_{\\text{KL}}(P\\parallel Q)&=\\int \_{x\_{a}}^{x\_{b}}p(x)\\,\\log {\\frac {p(x)}{q(x)}}\\,dx\\\\\[6pt\]&=\\int \_{x\_{a}}^{x\_{b}}{\\tilde {p}}(y(x))\\left|{\\frac {dy}{dx}}\\right|\\log {\\frac {{\\tilde {p}}(y(x))\\,\\left|{\\frac {dy}{dx}}\\right|}{{\\tilde {q}}(y(x))\\,\\left|{\\frac {dy}{dx}}\\right|}}\\,dx\\\\&=\\int \_{y\_{a}}^{y\_{b}}{\\tilde {p}}(y)\\,\\log {\\frac {{\\tilde {p}}(y)}{{\\tilde {q}}(y)}}\\,dy\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/eda27a4e71619493d682fa244154c5dd3e951d64) where ![{\\displaystyle y\_{a}=y(x\_{a})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7840d3605c02be788a77333cee91dbf130b1b4ad) and ![{\\displaystyle y\_{b}=y(x\_{b})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bb98d1f1ad2babf41b276ff053244930ed3c1e83). Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces a [dimensionally consistent](https://en.wikipedia.org/wiki/Dimensional_analysis "Dimensional analysis") quantity, since if x is a dimensioned variable, ![{\\displaystyle p(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8cb7afced134ef75572e5314a5d278c2d644f438) and ![{\\displaystyle q(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c38bbafe34a043d284f19231b946a76c0a4b16b4) are also dimensioned, since e.g. ![{\\displaystyle P(dx)=p(x)\\,dx}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8a5c2c00a56eced5f6ab6e25debe29ebb98bff69) is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory[\[28\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-VerduLecture-28) (such as [self-information](https://en.wikipedia.org/wiki/Self-information "Self-information") or [Shannon entropy](https://en.wikipedia.org/wiki/Shannon_entropy "Shannon entropy")), which can become undefined or negative for non-discrete probabilities.
+*   Relative entropy is [additive](https://en.wikipedia.org/wiki/Additive_map "Additive map") for [independent distributions](https://en.wikipedia.org/wiki/Independent_random_variables "Independent random variables") in much the same way as Shannon entropy. If ![{\\displaystyle P\_{1},P\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3bfdd9dc3e5cdd53ae8ee830e73cc482c834e713) are independent distributions, and ![{\\displaystyle P(dx,dy)=P\_{1}(dx)P\_{2}(dy)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/98e48155edf6e3a2d136a84101bfeb13b837d433), and likewise ![{\\displaystyle Q(dx,dy)=Q\_{1}(dx)Q\_{2}(dy)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/90b0f28bb2e235c85074d1da8ae50d66487a61e2) for independent distributions ![{\\displaystyle Q\_{1},Q\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/63e279e70626310da835b85616e2133fab4c27c5) then ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=D\_{\\text{KL}}(P\_{1}\\parallel Q\_{1})+D\_{\\text{KL}}(P\_{2}\\parallel Q\_{2}).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dd7846698c1981b3996cbfb4a47a03ef1aafefbe)
+*   Relative entropy ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07) is [convex](https://en.wikipedia.org/wiki/Convex_function "Convex function") in the pair of [probability measures](https://en.wikipedia.org/wiki/Probability_measure "Probability measure") ![{\\displaystyle (P,Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f7854fbffaf21cef2fd7219d7b2413ea3961fdf4), i.e. if ![{\\displaystyle (P\_{1},Q\_{1})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4081a0bca5521dfe83a02284b3929ba1f492079d) and ![{\\displaystyle (P\_{2},Q\_{2})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2c2be3a3a4ea1f9ac8a1dcd68691a70af365df9b) are two pairs of probability measures then ![{\\displaystyle D\_{\\text{KL}}(\\lambda P\_{1}+(1-\\lambda )P\_{2}\\parallel \\lambda Q\_{1}+(1-\\lambda )Q\_{2})\\leq \\lambda D\_{\\text{KL}}(P\_{1}\\parallel Q\_{1})+(1-\\lambda )D\_{\\text{KL}}(P\_{2}\\parallel Q\_{2}){\\text{ for }}0\\leq \\lambda \\leq 1.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/775d6ac738000849e63ef24ac6a1ddacccc870f8)
+*   ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07) may be Taylor expanded about its minimum (i.e. ![{\\displaystyle P=Q}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2abc7e2c5a78e9e6cb7a2a907279953f9b4a3f52)) as ![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\sum \_{n=2}^{\\infty }{\\frac {1}{n(n-1)}}\\sum \_{x\\in {\\mathcal {X}}}{\\frac {(Q(x)-P(x))^{n}}{Q(x)^{n-1}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d0eb13e7c22e6496d4b2d5046351bd55dd494b9d) which converges if and only if ![{\\displaystyle P\\leq 2Q}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c54e7b207f145b6e768ddb02eb81d8c229ffaec6) [almost surely](https://en.wikipedia.org/wiki/Almost_surely "Almost surely") w.r.t ![{\\displaystyle Q}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8752c7023b4b3286800fe3238271bbca681219ed).
+
+The following result, due to Donsker and
+Varadhan,[\[29\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-29)
+is known as **Donsker and Varadhan's variational formula**.
+
+**Proof**
+
+For a short proof assuming integrability of
+![{\\displaystyle \\exp(h)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d502344470aec00338bc34908b5d0dd789e0a912)
+with respect to P, let
+![{\\displaystyle Q^{\*}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c1785c0a77ab5a06684e8a7ac4f5e59d59ec0319)
+have P\-density ![{\\displaystyle {\\frac {\\exp h(\\theta )}{E\_{P}\[\\exp
+h\]}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/68129a5fd99a843dbd790087e726081a2ccb50a9),
+i.e. ![{\\displaystyle Q^{\*}(d\\theta )={\\frac {\\exp h(\\theta
+)}{E\_{P}\[\\exp h\]}}P(d\\theta
+)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c054e9af822f3b516e93eb4cc96d96441a617985)
+Then
+
+![{\\displaystyle D\_{\\text{KL}}(Q\\parallel
+Q^{\*})-D\_{\\text{KL}}(Q\\parallel P)=-E\_{Q}\[h\]+\\log E\_{P}\[\\exp
+h\]{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9ab18b0c58da04cb8fcbf1bf0205c9bdf6529b0a)
+
+Therefore,
+
+![{\\displaystyle E\_{Q}\[h\]-D\_{\\text{KL}}(Q\\parallel P)=\\log E\_{P}\[\\exp
+h\]-D\_{\\text{KL}}(Q\\parallel Q^{\*})\\leq \\log E\_{P}\[\\exp
+h\]{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b388f329bba5f73fc5af0f0c84041842aaf260b0)
+
+where the last inequality follows from
+![{\\displaystyle D\_{\\text{KL}}(Q\\parallel Q^{\*})\\geq 0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/56ed125c68e62c99c0dd53aefff525fd2b6459a8)
+, for which equality occurs if and only if
+![{\\displaystyle Q=Q^{\*}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/30c57dcbad607ce61ab246e79bd66fe208762346)
+. The conclusion follows.
+
+### Multivariate normal distributions
+
+Suppose that we have two
+[multivariate normal distributions](https://en.wikipedia.org/wiki/Multivariate_normal_distribution "Multivariate normal distribution")
+, with means
+![{\\displaystyle \\mu \_{0},\\mu \_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fb5c6a6422d5ca155808302b46e2233d5b028b34)
+and with (non-singular)
+[covariance matrices](https://en.wikipedia.org/wiki/Covariance_matrix "Covariance matrix")
+![{\\displaystyle \\Sigma \_{0},\\Sigma \_{1}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/583695b19fee86af86f86ea0c0a5dc0b25dfd4cf)
+If the two distributions have the same dimension, k, then the relative entropy
+between the distributions is as
+follows:[\[30\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-30)
+
+![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {N}}\_{0}\\parallel
+{\\mathcal {N}}\_{1}\\right)={\\frac {1}{2}}\\left\[\\operatorname {tr}
+\\left(\\Sigma \_{1}^{-1}\\Sigma \_{0}\\right)-k+\\left(\\mu \_{1}-\\mu
+\_{0}\\right)^{\\mathsf {T}}\\Sigma \_{1}^{-1}\\left(\\mu \_{1}-\\mu
+\_{0}\\right)+\\ln {\\frac {\\det \\Sigma \_{1}}{\\det \\Sigma
+\_{0}}}\\right\]{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d7cda84054fa44b8e5f01fdbabdd8b48b5f0a399)
+
+The [logarithm](https://en.wikipedia.org/wiki/Logarithm "Logarithm") in the last
+term must be taken to base
+[e](https://en.wikipedia.org/wiki/E_%28mathematical_constant%29 "E (mathematical constant)")
+since all terms apart from the last are base-e logarithms of expressions that
+are either factors of the density function or otherwise arise naturally. The
+equation therefore gives a result measured in
+[nats](https://en.wikipedia.org/wiki/Nat_%28unit%29 "Nat (unit)") . Dividing the
+entire expression above by
+![{\\displaystyle \\ln(2)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bee3c8a0344afcd9379a7e528be216a029e8f348)
+yields the divergence in [bits](https://en.wikipedia.org/wiki/Bit "Bit") .
+
+In a numerical implementation, it is helpful to express the result in terms of
+the Cholesky decompositions
+![{\\displaystyle L\_{0},L\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e2e088bdb8986fb222d338c3f274ec6aa4d6b257)
+such that
+![{\\displaystyle \\Sigma \_{0}=L\_{0}L\_{0}^{T}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b642bd5a68be0e0ac7cf15807e147ce769aebfe7)
+and
+![{\\displaystyle \\Sigma \_{1}=L\_{1}L\_{1}^{T}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0d9729bd54f8abca76c5f5968873ea3da2ef0d25)
+. Then with M and y solutions to the triangular linear systems
+![{\\displaystyle L\_{1}M=L\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cd454cdda9a6a9b037783eaf28fcf9c34702e520)
+, and
+![{\\displaystyle L\_{1}y=\\mu \_{1}-\\mu \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/aefa56abf21e168693384a9c0763e788c085fd4f)
+,
+
+![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {N}}\_{0}\\parallel {\\mathcal {N}}\_{1}\\right)={\\frac {1}{2}}\\left(\\sum \_{i,j=1}^{k}{\\left(M\_{ij}\\right)}^{2}-k+|y|^{2}+2\\sum \_{i=1}^{k}\\ln {\\frac {(L\_{1})\_{ii}}{(L\_{0})\_{ii}}}\\right){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4b16cd2b6f80f47128fa04c6c1dc897cb91b76e2)
+
+A special case, and a common quantity in
+[variational inference](https://en.wikipedia.org/wiki/Variational_inference "Variational inference")
+, is the relative entropy between a diagonal multivariate normal, and a standard
+normal distribution (with zero mean and unit variance):
+
+![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {N}}\\left(\\left(\\mu
+\_{1},\\ldots ,\\mu \_{k}\\right)^{\\mathsf {T}},\\operatorname {diag}
+\\left(\\sigma \_{1}^{2},\\ldots ,\\sigma \_{k}^{2}\\right)\\right)\\parallel
+{\\mathcal {N}}\\left(\\mathbf {0} ,\\mathbf {I} \\right)\\right)={\\frac
+{1}{2}}\\sum \_{i=1}^{k}\\left\[\\sigma \_{i}^{2}+\\mu \_{i}^{2}-1-\\ln
+\\left(\\sigma
+\_{i}^{2}\\right)\\right\]{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0bfc62f2d886aed5a3a76d78d2eb7e48101cbc7b)
+
+For two univariate normal distributions **p** and **q** the above simplifies
+to[\[31\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-31)
+![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {p}}\\parallel {\\mathcal {q}}\\right)=\\log {\\frac {\\sigma \_{1}}{\\sigma \_{0}}}+{\\frac {\\sigma \_{0}^{2}+{\\left(\\mu \_{0}-\\mu \_{1}\\right)}^{2}}{2\\sigma \_{1}^{2}}}-{\\frac {1}{2}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/95dd4c6fc20d8e3f33bf0bb1584e4834920faa75)
+
+In the case of co-centered normal distributions with
+![{\\displaystyle k=\\sigma \_{1}/\\sigma \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7571c8949f0b283209ee10474426c61a686a40a9)
+, this
+simplifies[\[32\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-auto-32)
+to:
+
+![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {p}}\\parallel {\\mathcal {q}}\\right)=\\log \_{2}k+(k^{-2}-1)/2/\\ln(2)\\mathrm {bits} }](https://wikimedia.org/api/rest_v1/media/math/render/svg/8f11620a947db549116166183df5b3bb0056dc5d)
+
+### Uniform distributions
+
+Consider two uniform distributions, with the support of ![{\\displaystyle
+p=\[A,B\]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1f9e91badbd6947c80f1308adadc16071891a440)
+enclosed within ![{\\displaystyle
+q=\[C,D\]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/030d2e5d1d0489e6cb5c4c808edc82344635b0e7)
+(![{\\displaystyle C\\leq A<B\\leq
+D}](https://wikimedia.org/api/rest_v1/media/math/render/svg/02a7cfd5e1908c9a583a4304eb03dcdde0286d74)).
+Then the information gain is:
+
+![{\\displaystyle D\_{\\text{KL}}\\left({\\mathcal {p}}\\parallel {\\mathcal {q}}\\right)=\\log {\\frac {D-C}{B-A}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0a84d16d1bd0383b8433b4086f57de3aba45b9d9)
+
+Intuitively,[\[32\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-auto-32)
+the information gain to a k times narrower uniform distribution contains
+![{\\displaystyle \\log \_{2}k}](https://wikimedia.org/api/rest_v1/media/math/render/svg/294c0f6bed4ca4ba34a7242748ab0accef3924a8)
+bits. This connects with the use of bits in computing, where
+![{\\displaystyle \\log \_{2}k}](https://wikimedia.org/api/rest_v1/media/math/render/svg/294c0f6bed4ca4ba34a7242748ab0accef3924a8)
+bits would be needed to identify one element of a k long stream.
+
+### Exponential family
+
+The
+[exponential family](https://en.wikipedia.org/wiki/Exponential_family "Exponential family")
+of distribution is given by
+
+![{\\displaystyle p\_{X}(x|\\theta )=h(x)\\exp \\left(\\theta ^{\\mathsf {T}}T(x)-A(\\theta )\\right)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6181c93488b4e22b405c4f20b9a61384bc1c62f3)
+
+where
+![{\\displaystyle h(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/02c07825dae28705df03d15daeb8844d49c4dbd4)
+is reference measure,
+![{\\displaystyle T(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1171c29b4c2b5575f50a4ea9313f90448a2cbe05)
+is
+[sufficient statistics](https://en.wikipedia.org/wiki/Sufficient_statistic "Sufficient statistic")
+,
+![{\\displaystyle \\theta }](https://wikimedia.org/api/rest_v1/media/math/render/svg/6e5ab2664b422d53eb0c7df3b87e1360d75ad9af)
+is canonical natural parameters, and
+![{\\displaystyle A(\\theta )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/55a6c7ecebb4cb39407ae5b4a05fdbc82597901c)
+is the log-partition function.
+
+The KL divergence between two distributions
+![{\\displaystyle p(x|\\theta \_{1})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ec834a5e8c4d0b2d4a768151d1120b529ca65ce0)
+and
+![{\\displaystyle p(x|\\theta \_{2})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0176203c5aab87c8ad8a9588c99e3cb4e9c5846f)
+is given
+by[\[33\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-33)
+
+![{\\displaystyle D\_{\\text{KL}}(\\theta \_{1}\\parallel \\theta \_{2})={\\left(\\theta \_{1}-\\theta \_{2}\\right)}^{\\mathsf {T}}\\mu \_{1}-A(\\theta \_{1})+A(\\theta \_{2})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1c3ff1aba7881bafd467e4487d44feee78e75620)
+
+where ![{\\displaystyle \\mu \_{1}=E\_{\\theta \_{1}}\[T(X)\]=\\nabla A(\\theta
+\_{1})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dd7911f2432b73d6ebe8a38701e18b423d658cb9)
+is the mean parameter of
+![{\\displaystyle p(x|\\theta \_{1})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ec834a5e8c4d0b2d4a768151d1120b529ca65ce0)
+.
+
+For example, for the
+[Poisson distribution](https://en.wikipedia.org/wiki/Poisson_distribution "Poisson distribution")
+with mean
+![{\\displaystyle \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/b43d0ea3c9c025af1be9128e62a18fa74bedda2a)
+, the sufficient statistics
+![{\\displaystyle T(x)=x}](https://wikimedia.org/api/rest_v1/media/math/render/svg/73a77ba5a3df9ab8481ede7461f146af9ab29e6f)
+, the natural parameter
+![{\\displaystyle \\theta =\\log \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/63603abf3d3c7ab57711b2b42ac784e72b1ad471)
+, and log partition function
+![{\\displaystyle A(\\theta )=e^{\\theta }}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b477f9a8ac97b8f22c36d94c4ff3def9dd22a7b6)
+. As such, the divergence between two Poisson distributions with means
+![{\\displaystyle \\lambda \_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/571a423bece8f29bcd1b48572f18dd4f6213dce2)
+and
+![{\\displaystyle \\lambda \_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6b668a1bd1e8ab9452ca975b7497546e7c1ba187)
+is
+
+![{\\displaystyle D\_{\\text{KL}}(\\lambda \_{1}\\parallel \\lambda \_{2})=\\lambda \_{1}\\log {\\frac {\\lambda \_{1}}{\\lambda \_{2}}}-\\lambda \_{1}+\\lambda \_{2}{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6736a0a627f96bd200a24c4e477095bd80f61fc0)
+
+As another example, for a normal distribution with unit variance
+![{\\displaystyle N(\\mu ,1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4ef5f4e895d453e932bcd4624a80b06cf06444b2)
+, the sufficient statistics
+![{\\displaystyle T(x)=x}](https://wikimedia.org/api/rest_v1/media/math/render/svg/73a77ba5a3df9ab8481ede7461f146af9ab29e6f)
+, the natural parameter
+![{\\displaystyle \\theta =\\mu }](https://wikimedia.org/api/rest_v1/media/math/render/svg/ac9063d0c72c38a12998c729fb596f1ad5233a3a)
+, and log partition function
+![{\\displaystyle A(\\theta )=\\mu ^{2}/2}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d4aac90be14918d32895027f000dd0068ce18df6)
+. Thus, the divergence between two normal distributions
+![{\\displaystyle N(\\mu \_{1},1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/66873c60341912620df20e2e4e34b4f63ed35b4b)
+and
+![{\\displaystyle N(\\mu \_{2},1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9de991b8a02dd241374825940b478e8c9fd3ca50)
+is
+
+![{\\displaystyle D\_{\\text{KL}}(\\mu \_{1}\\parallel \\mu \_{2})=\\left(\\mu \_{1}-\\mu \_{2}\\right)\\mu \_{1}-{\\frac {\\mu \_{1}^{2}}{2}}+{\\frac {\\mu \_{2}^{2}}{2}}={\\frac {{\\left(\\mu \_{2}-\\mu \_{1}\\right)}^{2}}{2}}{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/439818de8db9a8e98a6a71ce0c8d7de8b8ae04f7)
+
+As final example, the divergence between a normal distribution with unit
+variance
+![{\\displaystyle N(\\mu ,1)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4ef5f4e895d453e932bcd4624a80b06cf06444b2)
+and a Poisson distribution with mean
+![{\\displaystyle \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/b43d0ea3c9c025af1be9128e62a18fa74bedda2a)
+is
+
+![{\\displaystyle D\_{\\text{KL}}(\\mu \\parallel \\lambda )=(\\mu -\\log \\lambda )\\mu -{\\frac {\\mu ^{2}}{2}}+\\lambda {\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/119624907e5012f526bc25af8f8061871ac2c288)
+
+While relative entropy is a
+[statistical distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
+, it is not a
+[metric](https://en.wikipedia.org/wiki/Metric_%28mathematics%29 "Metric (mathematics)")
+on the space of probability distributions, but instead it is a
+[divergence](https://en.wikipedia.org/wiki/Divergence_%28statistics%29 "Divergence (statistics)")
+.[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
+While metrics are symmetric and generalize _linear_ distance, satisfying the
+[triangle inequality](https://en.wikipedia.org/wiki/Triangle_inequality "Triangle inequality")
+, divergences are asymmetric in general and generalize _squared_ distance, in
+some cases satisfying a generalized
+[Pythagorean theorem](https://en.wikipedia.org/wiki/Pythagorean_theorem "Pythagorean theorem")
+. In general
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+does not equal
+![{\\displaystyle D\_{\\text{KL}}(Q\\parallel P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/33cd5593ce6b0527ee343372fcf0a745cb9cdb52)
+, and while this can be symmetrized (see
+[§ Symmetrised divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Symmetrised_divergence)
+), the asymmetry is an important part of the
+geometry.[\[4\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEAmari201611-4)
+
+It generates a
+[topology](https://en.wikipedia.org/wiki/Topology#Topologies_on_sets "Topology")
+on the space of
+[probability distributions](https://en.wikipedia.org/wiki/Probability_distribution "Probability distribution")
+. More concretely, if
+![{\\displaystyle \\{P\_{1},P\_{2},\\ldots \\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1ff268a40cf47f496dfc3ff0425478093bc49e96)
+is a sequence of distributions such that
+
+![{\\displaystyle \\lim \_{n\\to \\infty }D\_{\\text{KL}}(P\_{n}\\parallel Q)=0{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f7527f531b3b491f7cc3eccc79da9be09247dbbe)
+
+then it is said that
+
+![{\\displaystyle P\_{n}\\xrightarrow {D} \\,Q{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e9758cb3479ccf538c74eb7e887c4568f987bae3)
+
+[Pinsker's inequality](https://en.wikipedia.org/wiki/Pinsker's_inequality "Pinsker's inequality")
+entails that
+
+![{\\displaystyle P\_{n}\\xrightarrow {D} P\\Rightarrow P\_{n}\\xrightarrow {TV} P{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a6001fee2dbafa3f33e0ed6c758e44d0f751a5d7)
+
+where the latter stands for the usual convergence in
+[total variation](https://en.wikipedia.org/wiki/Total_variation "Total variation")
+.
+
+### Fisher information metric
+
+Relative entropy is directly related to the
+[Fisher information metric](https://en.wikipedia.org/wiki/Fisher_information_metric "Fisher information metric")
+. This can be made explicit as follows. Assume that the probability
+distributions P and Q are both parameterized by some (possibly
+multi-dimensional) parameter
+![{\\displaystyle \\theta }](https://wikimedia.org/api/rest_v1/media/math/render/svg/6e5ab2664b422d53eb0c7df3b87e1360d75ad9af)
+. Consider then two close by values of
+![{\\displaystyle P=P(\\theta )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/83a5542468bf4a5ad9dee3d0f3ad02047a73da01)
+and
+![{\\displaystyle Q=P(\\theta \_{0})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/41b4e23cf75300ef3c256ed00c99f4c6ac348c20)
+so that the parameter
+![{\\displaystyle \\theta }](https://wikimedia.org/api/rest_v1/media/math/render/svg/6e5ab2664b422d53eb0c7df3b87e1360d75ad9af)
+differs by only a small amount from the parameter value
+![{\\displaystyle \\theta \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/18b67de6bf25dba7a24e66967ff6319858798734)
+. Specifically, up to first order one has (using the
+[Einstein summation convention](https://en.wikipedia.org/wiki/Einstein_summation_convention "Einstein summation convention")
+)
+![{\\displaystyle P(\\theta )=P(\\theta \_{0})+\\Delta \\theta \_{j}\\,P\_{j}(\\theta \_{0})+\\cdots }](https://wikimedia.org/api/rest_v1/media/math/render/svg/39a491014bbef8b352bee60fb833c91cd17c6437)
+
+with
+![{\\displaystyle \\Delta \\theta \_{j}=(\\theta -\\theta \_{0})\_{j}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f7bd79e9136dbee44c7e455332ac89706e6e8b49)
+a small change of
+![{\\displaystyle \\theta }](https://wikimedia.org/api/rest_v1/media/math/render/svg/6e5ab2664b422d53eb0c7df3b87e1360d75ad9af)
+in the j direction, and
+![{\\displaystyle P\_{j}\\left(\\theta \_{0}\\right)={\\frac {\\partial P}{\\partial \\theta \_{j}}}(\\theta \_{0})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c798c08f1c1be857631d461730e7314b794c1567)
+the corresponding rate of change in the probability distribution. Since relative
+entropy has an absolute minimum 0 for
+![{\\displaystyle P=Q}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2abc7e2c5a78e9e6cb7a2a907279953f9b4a3f52)
+, i.e.
+![{\\displaystyle \\theta =\\theta \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6092e68904d0c31801bee667fd2b3ebd40d6e5e4)
+, it changes only to _second_ order in the small parameters
+![{\\displaystyle \\Delta \\theta \_{j}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2687696576367a434e6dd95c336e73c1d4badcaf)
+. More formally, as for any minimum, the first derivatives of the divergence
+vanish
+
+![{\\displaystyle \\left.{\\frac {\\partial }{\\partial \\theta \_{j}}}\\right|\_{\\theta =\\theta \_{0}}D\_{\\text{KL}}(P(\\theta )\\parallel P(\\theta \_{0}))=0,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8dd547ff8248b6e69e3014b4458c82e92113cb8e)
+
+and by the
+[Taylor expansion](https://en.wikipedia.org/wiki/Taylor_series "Taylor series")
+one has up to second order
+
+![{\\displaystyle D\_{\\text{KL}}(P(\\theta )\\parallel P(\\theta \_{0}))={\\frac {1}{2}}\\,\\Delta \\theta \_{j}\\,\\Delta \\theta \_{k}\\,g\_{jk}(\\theta \_{0})+\\cdots }](https://wikimedia.org/api/rest_v1/media/math/render/svg/0ed16484f979f06d691f968cc77100f06e827349)
+
+where the
+[Hessian matrix](https://en.wikipedia.org/wiki/Hessian_matrix "Hessian matrix")
+of the divergence
+
+![{\\displaystyle g\_{jk}(\\theta \_{0})=\\left.{\\frac {\\partial ^{2}}{\\partial \\theta \_{j}\\,\\partial \\theta \_{k}}}\\right|\_{\\theta =\\theta \_{0}}D\_{\\text{KL}}(P(\\theta )\\parallel P(\\theta \_{0}))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fba580eb7b3c8496eebb609d2fca6f65e10a0cf6)
+
+must be
+[positive semi-definite](https://en.wikipedia.org/wiki/Positive-definite_matrix "Positive-definite matrix")
+. Letting
+![{\\displaystyle \\theta \_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/18b67de6bf25dba7a24e66967ff6319858798734)
+vary (and dropping the subindex 0) the Hessian
+![{\\displaystyle g\_{jk}(\\theta )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2fcb737db9ee0c60db3b36b5f81a67083ceb0817)
+defines a (possibly degenerate)
+[Riemannian metric](https://en.wikipedia.org/wiki/Riemannian_metric "Riemannian metric")
+on the θ
+[parameter space](https://en.wikipedia.org/wiki/Parameter_space "Parameter space")
+, called the Fisher information metric.
+
+#### Fisher information metric theorem
+
+There is an associated
+theorem.[\[3\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback1959-3)
+When
+![{\\displaystyle p\_{(x,\\rho )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2e3ddcb4e03e8d98f0c05c23f7914a903ee47647)
+satisfies the following regularity conditions:
+
+![{\\displaystyle {\\frac {\\partial \\log(p)}{\\partial \\rho }},{\\frac {\\partial ^{2}\\log(p)}{\\partial \\rho ^{2}}},{\\frac {\\partial ^{3}\\log(p)}{\\partial \\rho ^{3}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a5f62261a92a9f9a1169723fd94341b01e412aed)
+exist,
+![{\\displaystyle {\\begin{aligned}\\left|{\\frac {\\partial p}{\\partial \\rho }}\\right|&<F(x):\\int \_{x=0}^{\\infty }F(x)\\,dx<\\infty ,\\\\\\left|{\\frac {\\partial ^{2}p}{\\partial \\rho ^{2}}}\\right|&<G(x):\\int \_{x=0}^{\\infty }G(x)\\,dx<\\infty \\\\\\left|{\\frac {\\partial ^{3}\\log(p)}{\\partial \\rho ^{3}}}\\right|&<H(x):\\int \_{x=0}^{\\infty }p(x,0)H(x)\\,dx<\\xi <\\infty \\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/33c02cd3c9aa88aed8394b4f0917fb34740afe7a)
+
+where ξ is independent of ρ
+![{\\displaystyle \\left.\\int \_{x=0}^{\\infty }{\\frac {\\partial p(x,\\rho )}{\\partial \\rho }}\\right|\_{\\rho =0}\\,dx=\\left.\\int \_{x=0}^{\\infty }{\\frac {\\partial ^{2}p(x,\\rho )}{\\partial \\rho ^{2}}}\\right|\_{\\rho =0}\\,dx=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/81bd4dadd20884ad83a9d1fce680183e2c2eaa1f)
+
+then:
+![{\\displaystyle {\\mathcal {D}}(p(x,0)\\parallel p(x,\\rho ))={\\frac {c\\rho ^{2}}{2}}+{\\mathcal {O}}\\left(\\rho ^{3}\\right){\\text{ as }}\\rho \\to 0{\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fa70bfdc1ef68ff2c0949b31e719ac183fc74f24)
+
+### Variation of information
+
+Another information-theoretic metric is
+[variation of information](https://en.wikipedia.org/wiki/Variation_of_information "Variation of information")
+, which is roughly a symmetrization of
+[conditional entropy](https://en.wikipedia.org/wiki/Conditional_entropy "Conditional entropy")
+. It is a metric on the set of
+[partitions](https://en.wikipedia.org/wiki/Partition_of_a_set "Partition of a set")
+of a discrete
+[probability space](https://en.wikipedia.org/wiki/Probability_space "Probability space")
+.
+
+### MAUVE Metric
+
+MAUVE is a measure of the statistical gap between two text distributions, such
+as the difference between text generated by a model and human-written text. This
+measure is computed using Kullback–Leibler divergences between the two
+distributions in a quantized embedding space of a foundation model.
+
+Many of the other quantities of information theory can be interpreted as
+applications of relative entropy to specific cases.
+
+### Self-information
+
+The
+[self-information](https://en.wikipedia.org/wiki/Self-information "Self-information")
+, also known as the
+[information content](https://en.wikipedia.org/wiki/Information_content "Information content")
+of a signal, random variable, or
+[event](https://en.wikipedia.org/wiki/Event_%28probability_theory%29 "Event (probability theory)")
+is defined as the negative logarithm of the
+[probability](https://en.wikipedia.org/wiki/Probability "Probability") of the
+given outcome occurring.
+
+When applied to a
+[discrete random variable](https://en.wikipedia.org/wiki/Discrete_random_variable "Discrete random variable")
+, the self-information can be represented as\[_[citation
+needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed
+"Wikipedia:Citation needed")_\]
+
+![{\\displaystyle \\operatorname {\\operatorname {I} } (m)=D\_{\\text{KL}}\\left(\\delta \_{\\text{im}}\\parallel \\{p\_{i}\\}\\right),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/403686f36e08d2d510888265dc2aff716b8bca6b)
+
+is the relative entropy of the probability distribution
+![{\\displaystyle P(i)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d3f0c5cc0e8d31d705f7e08298daab4cb7da1662)
+from a
+[Kronecker delta](https://en.wikipedia.org/wiki/Kronecker_delta "Kronecker delta")
+representing certainty that
+![{\\displaystyle i=m}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4718f08ffcf5718164ddc37f4a382d04effe1014)
+— i.e. the number of extra bits that must be transmitted to identify i if only
+the probability distribution
+![{\\displaystyle P(i)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d3f0c5cc0e8d31d705f7e08298daab4cb7da1662)
+is available to the receiver, not the fact that
+![{\\displaystyle i=m}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4718f08ffcf5718164ddc37f4a382d04effe1014)
+.
+
+### Mutual information
+
+The
+[mutual information](https://en.wikipedia.org/wiki/Mutual_information "Mutual information")
+,
+
+![{\\displaystyle {\\begin{aligned}\\operatorname {I}
+(X;Y)&=D\_{\\text{KL}}(P\_{X,Y}\\parallel P\_{X}\\cdot
+P\_{Y})\\\\&=\\operatorname {E} \_{X}\[D\_{\\text{KL}}^{Y}(P\_{Y\\mid
+X}\\parallel P\_{Y})\]\\\\&=\\operatorname {E}
+\_{Y}\[D\_{\\text{KL}}^{X}(P\_{X\\mid Y}\\parallel
+P\_{X})\]\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5808992333117aa1618d86a9293c7bb0531b22d1)
+
+is the relative entropy of the
+[joint probability distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution "Joint probability distribution")
+![{\\displaystyle P\_{X,Y}(x,y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fb7b89359e2ab11faff329f878f6272ad4e39103)
+from the product
+![{\\displaystyle (P\_{X}\\cdot P\_{Y})(x,y)=P\_{X}(x)P\_{Y}(y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ff7faf65c801b47f9f7b3360b0428a676a828b3f)
+of the two
+[marginal probability distributions](https://en.wikipedia.org/wiki/Marginal_probability_distribution "Marginal probability distribution")
+— i.e. the expected number of extra bits that must be transmitted to identify X
+and Y if they are coded using only their marginal distributions instead of the
+joint distribution.
+
+### Shannon entropy
+
+The
+[Shannon entropy](https://en.wikipedia.org/wiki/Shannon_entropy "Shannon entropy")
+,
+
+![{\\displaystyle {\\begin{aligned}\\mathrm {H} (X)&=\\operatorname {E}
+\\left\[\\operatorname {I} \_{X}(x)\\right\]\\\\&=\\log
+N-D\_{\\text{KL}}{\\left(p\_{X}(x)\\parallel
+P\_{U}(X)\\right)}\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1404db97e6e3315749e73c5fd8237503a1fa58cf)
+
+is the number of bits which would have to be transmitted to identify X from N
+equally likely possibilities, _less_ the relative entropy of the uniform
+distribution on the
+[random variates](https://en.wikipedia.org/wiki/Random_variate "Random variate")
+of X,
+![{\\displaystyle P\_{U}(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d08085e4447acd3d0e19ca4ac57058ff75bd38be)
+, from the true distribution
+![{\\displaystyle P(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bf94167ebc67db1573e6be9e261cd0518e49e160)
+— i.e. _less_ the expected number of bits saved, which would have had to be sent
+if the value of X were coded according to the uniform distribution
+![{\\displaystyle P\_{U}(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d08085e4447acd3d0e19ca4ac57058ff75bd38be)
+rather than the true distribution
+![{\\displaystyle P(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bf94167ebc67db1573e6be9e261cd0518e49e160)
+. This definition of Shannon entropy forms the basis of
+[E.T. Jaynes](https://en.wikipedia.org/wiki/E.T._Jaynes "E.T. Jaynes") 's
+alternative generalization to continuous distributions, the
+[limiting density of discrete points](https://en.wikipedia.org/wiki/Limiting_density_of_discrete_points "Limiting density of discrete points")
+(as opposed to the usual
+[differential entropy](https://en.wikipedia.org/wiki/Differential_entropy "Differential entropy")
+), which defines the continuous entropy as
+![{\\displaystyle \\lim \_{N\\to \\infty }H\_{N}(X)=\\log N-\\int p(x)\\log {\\frac {p(x)}{m(x)}}\\,dx{\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ef7fb5689851293e706d1d4d63d2bb55299f170b)
+which is equivalent to:
+![{\\displaystyle \\log(N)-D\_{\\text{KL}}(p(x)||m(x))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e9008c0b15e4f79b3ed0583f21b418b7f4f6f2ee)
+
+### Conditional entropy
+
+The
+[conditional entropy](https://en.wikipedia.org/wiki/Conditional_entropy "Conditional entropy")
+[\[34\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-CoverThomas-34),
+
+![{\\displaystyle {\\begin{aligned}\\mathrm {H} (X\\mid Y)&=\\log
+N-D\_{\\text{KL}}(P(X,Y)\\parallel P\_{U}(X)P(Y))\\\\\[5pt\]&=\\log
+N-D\_{\\text{KL}}(P(X,Y)\\parallel P(X)P(Y))-D\_{\\text{KL}}(P(X)\\parallel
+P\_{U}(X))\\\\\[5pt\]&=\\mathrm {H} (X)-\\operatorname {I}
+(X;Y)\\\\\[5pt\]&=\\log N-\\operatorname {E}
+\_{Y}\\left\[D\_{\\text{KL}}\\left(P\\left(X\\mid Y\\right)\\parallel
+P\_{U}(X)\\right)\\right\]\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/486a08374b73d457fbab6acab4f4014b045ee26a)
+
+is the number of bits which would have to be transmitted to identify X from N
+equally likely possibilities, _less_ the relative entropy of the true joint
+distribution
+![{\\displaystyle P(X,Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a6fd347c3c83e69250fe5a6c3648fed056ff8f68)
+from the product distribution
+![{\\displaystyle P\_{U}(X)P(Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6ee07039d976b9177d02c8e75b068f62b34292a0)
+from — i.e. _less_ the expected number of bits saved which would have had to be
+sent if the value of X were coded according to the uniform distribution
+![{\\displaystyle P\_{U}(X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d08085e4447acd3d0e19ca4ac57058ff75bd38be)
+rather than the conditional distribution
+![{\\displaystyle P(X|Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/38c2f827fbc72053662e88bb04636eca0fc5cd0f)
+of X given Y.
+
+### Cross entropy
+
+When we have a set of possible events, coming from the distribution p, we can
+encode them (with a
+[lossless data compression](https://en.wikipedia.org/wiki/Lossless_compression "Lossless compression")
+) using
+[entropy encoding](https://en.wikipedia.org/wiki/Entropy_encoding "Entropy encoding")
+. This compresses the data by replacing each fixed-length input symbol with a
+corresponding unique, variable-length,
+[prefix-free code](https://en.wikipedia.org/wiki/Prefix-free_code "Prefix-free code")
+(e.g.: the events (A, B, C) with probabilities p = (1/2, 1/4, 1/4) can be
+encoded as the bits (0, 10, 11)). If we know the distribution p in advance, we
+can devise an encoding that would be optimal (e.g.: using
+[Huffman coding](https://en.wikipedia.org/wiki/Huffman_coding "Huffman coding")
+). Meaning the messages we encode will have the shortest length on average
+(assuming the encoded events are sampled from p), which will be equal to
+[Shannon's Entropy](https://en.wikipedia.org/wiki/Shannon's_entropy "Shannon's entropy")
+of p (denoted as
+![{\\displaystyle \\mathrm {H} (p)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/76d5ade34024658505d6df9f6114079845d005b5)
+). However, if we use a different probability distribution (q) when creating the
+entropy encoding scheme, then a larger number of
+[bits](https://en.wikipedia.org/wiki/Bit "Bit") will be used (on average) to
+identify an event from a set of possibilities. This new (larger) number is
+measured by the
+[cross entropy](https://en.wikipedia.org/wiki/Cross_entropy "Cross entropy")
+between p and q.
+
+The [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy "Cross entropy")
+between two
+[probability distributions](https://en.wikipedia.org/wiki/Probability_distribution "Probability distribution")
+(p and q) measures the average number of
+[bits](https://en.wikipedia.org/wiki/Bit "Bit") needed to identify an event from
+a set of possibilities, if a coding scheme is used based on a given probability
+distribution q, rather than the "true" distribution p. The cross entropy for two
+distributions p and q over the same
+[probability space](https://en.wikipedia.org/wiki/Probability_space "Probability space")
+is thus defined as follows.
+
+![{\\displaystyle \\mathrm {H} (p,q)=\\operatorname {E} \_{p}\[-\\log
+q\]=\\mathrm {H} (p)+D\_{\\text{KL}}(p\\parallel
+q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ce9707c4ceba15ce0d453ea474cba38592b1a4d1)
+
+For explicit derivation of this, see the
+[Motivation](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Motivation)
+section above.
+
+Under this scenario, relative entropies (kl-divergence) can be interpreted as
+the extra number of bits, on average, that are needed (beyond
+![{\\displaystyle \\mathrm {H} (p)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/76d5ade34024658505d6df9f6114079845d005b5)
+) for encoding the events because of using q for constructing the encoding
+scheme instead of p.
+
+In
+[Bayesian statistics](https://en.wikipedia.org/wiki/Bayesian_statistics "Bayesian statistics")
+, relative entropy can be used as a measure of the information gain in moving
+from a
+[prior distribution](https://en.wikipedia.org/wiki/Prior_distribution "Prior distribution")
+to a
+[posterior distribution](https://en.wikipedia.org/wiki/Posterior_distribution "Posterior distribution")
+:
+![{\\displaystyle p(x)\\to p(x\\mid I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cb998006aac30bccaa9826f5a593db16b833c8ca)
+. If some new fact
+![{\\displaystyle Y=y}](https://wikimedia.org/api/rest_v1/media/math/render/svg/678864c5e9a7ce08acfc22d0d7f726d2cade5b45)
+is discovered, it can be used to update the posterior distribution for X from
+![{\\displaystyle p(x\\mid I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ed634d96ca7387739783ba12b2a60f53420075b3)
+to a new posterior distribution
+![{\\displaystyle p(x\\mid y,I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dbb94d46086c0f668f208447d1094b3be4c57f4d)
+using
+[Bayes' theorem](https://en.wikipedia.org/wiki/Bayes'_theorem "Bayes' theorem")
+:
+
+![{\\displaystyle p(x\\mid y,I)={\\frac {p(y\\mid x,I)p(x\\mid I)}{p(y\\mid I)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a5e1963e6c1b97081f40d4e00a33a13bf23e019e)
+
+This distribution has a new
+[entropy](https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_information_theory "Entropy in thermodynamics and information theory")
+:
+
+![{\\displaystyle \\mathrm {H} {\\big (}p(x\\mid y,I){\\big )}=-\\sum \_{x}p(x\\mid y,I)\\log p(x\\mid y,I){\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/40ef14fbac8f28f971bbfdf3e20b6852865fae29)
+
+which may be less than or greater than the original entropy
+![{\\displaystyle \\mathrm {H} (p(x\\mid I))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/47e7c4d8c15ef8c0ece7e8a42867584c4f04ca3c)
+. However, from the standpoint of the new probability distribution one can
+estimate that to have used the original code based on
+![{\\displaystyle p(x\\mid I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ed634d96ca7387739783ba12b2a60f53420075b3)
+instead of a new code based on
+![{\\displaystyle p(x\\mid y,I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dbb94d46086c0f668f208447d1094b3be4c57f4d)
+would have added an expected number of bits:
+
+![{\\displaystyle D\_{\\text{KL}}{\\big (}p(x\\mid y,I)\\parallel p(x\\mid I){\\big )}=\\sum \_{x}p(x\\mid y,I)\\log {\\frac {p(x\\mid y,I)}{p(x\\mid I)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0d26a80847684e120153b8f0bf013212d2f24058)
+
+to the message length. This therefore represents the amount of useful
+information, or information gain, about X, that has been learned by discovering
+![{\\displaystyle Y=y}](https://wikimedia.org/api/rest_v1/media/math/render/svg/678864c5e9a7ce08acfc22d0d7f726d2cade5b45)
+.
+
+If a further piece of data,
+![{\\displaystyle Y\_{2}=y\_{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/052adff6c0f6624b25f30b42c942007137d47faf)
+, subsequently comes in, the probability distribution for x can be updated
+further, to give a new best guess
+![{\\displaystyle p(x\\mid y\_{1},y\_{2},I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/56d4b0464ccdda0a8de69b81efc90371fe6432fc)
+. If one reinvestigates the information gain for using
+![{\\displaystyle p(x\\mid y\_{1},I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6db097127a754553f560ee35afb78209b325174a)
+rather than
+![{\\displaystyle p(x\\mid I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ed634d96ca7387739783ba12b2a60f53420075b3)
+, it turns out that it may be either greater or less than previously estimated:
+
+![{\\displaystyle \\sum \_{x}p(x\\mid y\_{1},y\_{2},I)\\log {\\frac {p(x\\mid y\_{1},y\_{2},I)}{p(x\\mid I)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d5dac2e4e5ade5cf9586a78a96c727d6bae2edf8)
+may be ≤ or > than
+![{\\textstyle \\sum \_{x}p(x\\mid y\_{1},I)\\log {\\frac {p(x\\mid y\_{1},I)}{p(x\\mid I)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d10748d9317bd640b92ff47bf2da07e889e539be)
+
+and so the combined information gain does _not_ obey the triangle inequality:
+
+![{\\displaystyle D\_{\\text{KL}}{\\big (}p(x\\mid y\_{1},y\_{2},I)\\parallel p(x\\mid I){\\big )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c93e3853d8702d3203faa88ad7bb87e5039d415c)
+may be <, = or > than
+![{\\displaystyle D\_{\\text{KL}}{\\big (}p(x\\mid y\_{1},y\_{2},I)\\parallel p(x\\mid y\_{1},I){\\big )}+D\_{\\text{KL}}{\\big (}p(x\\mid y\_{1},I)\\parallel p(x\\mid I){\\big )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a971a78c5ace5b1f62f3e697a35b4017a9d407fb)
+
+All one can say is that on _average_, averaging using
+![{\\displaystyle p(y\_{2}\\mid y\_{1},x,I)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d99961b15f4a711b0a733454ab8a7b5307a9131b)
+, the two sides will average out.
+
+### Bayesian experimental design
+
+A common goal in
+[Bayesian experimental design](https://en.wikipedia.org/wiki/Bayesian_experimental_design "Bayesian experimental design")
+is to maximise the expected relative entropy between the prior and the
+posterior.[\[35\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-35)
+When posteriors are approximated to be Gaussian distributions, a design
+maximising the expected relative entropy is called
+[Bayes d-optimal](https://en.wikipedia.org/wiki/D-optimal_design "D-optimal design")
+.
+
+Relative entropy
+![{\\textstyle D\_{\\text{KL}}{\\bigl (}p(x\\mid H\_{1})\\parallel p(x\\mid H\_{0}){\\bigr )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f0e2665a13b2a170c4cb5d28f14708037b7924ca)
+can also be interpreted as the expected **discrimination information** for
+![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
+over
+![{\\displaystyle H\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/43910602a221b7a4c373791f94793e3008622070)
+: the mean information per sample for discriminating in favor of a hypothesis
+![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
+against a hypothesis
+![{\\displaystyle H\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/43910602a221b7a4c373791f94793e3008622070)
+, when hypothesis
+![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
+is
+true.[\[36\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-36)
+Another name for this quantity, given to it by
+[I. J. Good](https://en.wikipedia.org/wiki/I._J._Good "I. J. Good") , is the
+expected weight of evidence for
+![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
+over
+![{\\displaystyle H\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/43910602a221b7a4c373791f94793e3008622070)
+to be expected from each sample.
+
+The expected weight of evidence for
+![{\\displaystyle H\_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d4d9a872a55b209f2eb7cc23a71e5e1541bd1f4)
+over
+![{\\displaystyle H\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/43910602a221b7a4c373791f94793e3008622070)
+is **not** the same as the information gain expected per sample about the
+probability distribution
+![{\\displaystyle p(H)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2290c0131714e85c56133f607eb41af87fdda3c1)
+of the hypotheses,
+
+![{\\displaystyle D\_{\\text{KL}}(p(x\\mid H\_{1})\\parallel p(x\\mid H\_{0}))\\neq IG=D\_{\\text{KL}}(p(H\\mid x)\\parallel p(H\\mid I)){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9d9922587515eaa137f6a34b189700306877a048)
+
+Either of the two quantities can be used as a
+[utility function](https://en.wikipedia.org/wiki/Utility_function "Utility function")
+in Bayesian experimental design, to choose an optimal next question to
+investigate: but they will in general lead to rather different experimental
+strategies.
+
+On the entropy scale of _information gain_ there is very little difference
+between near certainty and absolute certainty—coding according to a near
+certainty requires hardly any more bits than coding according to an absolute
+certainty. On the other hand, on the
+[logit](https://en.wikipedia.org/wiki/Logit "Logit") scale implied by weight of
+evidence, the difference between the two is enormous – infinite perhaps; this
+might reflect the difference between being almost sure (on a probabilistic
+level) that, say, the
+[Riemann hypothesis](https://en.wikipedia.org/wiki/Riemann_hypothesis "Riemann hypothesis")
+is correct, compared to being certain that it is correct because one has a
+[mathematical proof](https://en.wikipedia.org/wiki/Mathematical_proof "Mathematical proof")
+. These two different scales of
+[loss function](https://en.wikipedia.org/wiki/Loss_function "Loss function") for
+uncertainty are _both_ useful, according to how well each reflects the
+particular circumstances of the problem in question.
+
+### Principle of minimum discrimination information
+
+The idea of relative entropy as discrimination information led Kullback to
+propose the Principle of **Minimum Discrimination Information** (**MDI**): given
+new facts, a new distribution f should be chosen which is as hard to
+discriminate from the original distribution
+![{\\displaystyle f\_{0}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6423b30a4c5770c59b5ab92dcb4ce378755440ab)
+as possible; so that the new data produces as small an information gain
+![{\\displaystyle D\_{\\text{KL}}(f\\parallel f\_{0})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bb8c6740dba256c3e3fcd75f18b456890882fe98)
+as possible.
+
+For example, if one had a prior distribution
+![{\\displaystyle p(x,a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f99fe2c5da9eda4261abd3a3ba9e95a582258829)
+over x and a, and subsequently learnt the true distribution of a was
+![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
+, then the relative entropy between the new joint distribution for x and a,
+![{\\displaystyle q(x\\mid a)u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d46f0614827dab0040f75c8fe54904bb7f7d1ceb)
+, and the earlier prior distribution would be:
+
+![{\\displaystyle D\_{\\text{KL}}(q(x\\mid a)u(a)\\parallel p(x,a))=\\operatorname {E} \_{u(a)}\\left\\{D\_{\\text{KL}}(q(x\\mid a)\\parallel p(x\\mid a))\\right\\}+D\_{\\text{KL}}(u(a)\\parallel p(a)),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/458e2e6ba88a2aca56c0b96b0e6d3cbb416eae16)
+
+i.e. the sum of the relative entropy of
+![{\\displaystyle p(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c07e3f3507e18c9efcf41242ba875d747d4b878f)
+the prior distribution for a from the updated distribution
+![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
+, plus the expected value (using the probability distribution
+![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
+) of the relative entropy of the prior conditional distribution
+![{\\displaystyle p(x\\mid a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6322773741893591649ace67e0dc831a98805c7f)
+from the new conditional distribution
+![{\\displaystyle q(x\\mid a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b5984976e59f1aa29d0b7c7551c26b4e19cc8fed)
+. (Note that often the later expected value is called the _conditional relative
+entropy_ (or _conditional Kullback–Leibler divergence_) and denoted by
+![{\\displaystyle D\_{\\text{KL}}(q(x\\mid a)\\parallel p(x\\mid a))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/56f618aec5f161aa05c8a4b2d5de6b458b892aba)
+[\[3\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullback1959-3)[\[34\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-CoverThomas-34))
+This is minimized if
+![{\\displaystyle q(x\\mid a)=p(x\\mid a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e8b3c003ea111edc7f55cdbc05c1a0bf2b2a75e0)
+over the whole support of
+![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
+; and we note that this result incorporates Bayes' theorem, if the new
+distribution
+![{\\displaystyle u(a)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f34d1cfcb353a646f89533529e4d97cd1cd72686)
+is in fact a δ function representing certainty that a has one particular value.
+
+MDI can be seen as an extension of
+[Laplace](https://en.wikipedia.org/wiki/Laplace "Laplace") 's
+[Principle of Insufficient Reason](https://en.wikipedia.org/wiki/Principle_of_Insufficient_Reason "Principle of Insufficient Reason")
+, and the
+[Principle of Maximum Entropy](https://en.wikipedia.org/wiki/Principle_of_Maximum_Entropy "Principle of Maximum Entropy")
+of [E.T. Jaynes](https://en.wikipedia.org/wiki/E.T._Jaynes "E.T. Jaynes") . In
+particular, it is the natural extension of the principle of maximum entropy from
+discrete to continuous distributions, for which Shannon entropy ceases to be so
+useful (see _[differential
+entropy](https://en.wikipedia.org/wiki/Differential_entropy "Differential
+entropy")_), but the relative entropy continues to be just as relevant.
+
+In the engineering literature, MDI is sometimes called the **Principle of
+Minimum Cross-Entropy** (MCE) or **Minxent** for short. Minimising relative
+entropy from m to p with respect to m is equivalent to minimizing the
+cross-entropy of p and m, since
+
+![{\\displaystyle \\mathrm {H} (p,m)=\\mathrm {H} (p)+D\_{\\text{KL}}(p\\parallel m),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/20391c494fd8cb3a15b2bf92381d91ae947934c6)
+
+which is appropriate if one is trying to choose an adequate approximation to p.
+However, this is just as often _not_ the task one is trying to achieve. Instead,
+just as often it is m that is some fixed prior reference measure, and p that one
+is attempting to optimise by minimising
+![{\\displaystyle D\_{\\text{KL}}(p\\parallel m)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/29b125efdcb1fa150f3aa7c780e359d09b1eabed)
+subject to some constraint. This has led to some ambiguity in the literature,
+with some authors attempting to resolve the inconsistency by redefining
+cross-entropy to be
+![{\\displaystyle D\_{\\text{KL}}(p\\parallel m)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/29b125efdcb1fa150f3aa7c780e359d09b1eabed)
+, rather than
+![{\\displaystyle \\mathrm {H} (p,m)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/be29a0421e4b5a6d734025bc9222ca1aa100e293)
+\[_[citation needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed
+"Wikipedia:Citation needed")_\].
+
+[![](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/ArgonKLdivergence.png/250px-ArgonKLdivergence.png)](https://en.wikipedia.org/wiki/File:ArgonKLdivergence.png)
+
+Pressure versus volume plot of available work from a mole of argon gas relative to ambient, calculated as ![{\\displaystyle T\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2991f18ef81fec2f94e3a532df91a079f3f6697d) times the Kullback–Leibler divergence
+
+[Surprisals](https://en.wikipedia.org/wiki/Surprisal "Surprisal")
+[\[37\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-37)
+add where probabilities multiply. The surprisal for an event of probability p is
+defined as
+![{\\displaystyle s=-k\\ln p}](https://wikimedia.org/api/rest_v1/media/math/render/svg/da96ff26c096baca807e01eb60c2578113324686)
+. If k is
+![{\\displaystyle \\left\\{1,1/\\ln 2,1.38\\times 10^{-23}\\right\\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4e54eedf7d41bc0d5f15addb0a39ed0ec2a63d1a)
+then surprisal is in
+![{\\displaystyle \\{}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c1288432dfd9ce5f65857a9e6bbf0d4876b403d2)
+nats, bits, or
+![{\\displaystyle J/K\\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1d48054423445e38fab2d2eeb673148b758153bd)
+so that, for instance, there are N bits of surprisal for landing all "heads" on
+a toss of N coins.
+
+Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the
+_average surprisal_ S ([entropy](https://en.wikipedia.org/wiki/Entropy
+"Entropy")) for a given set of control parameters (like pressure P or volume V).
+This constrained
+[entropy maximization](https://en.wikipedia.org/wiki/Entropy_maximization "Entropy maximization")
+, both
+classically[\[38\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-38)
+and quantum
+mechanically,[\[39\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-39)
+minimizes
+[Gibbs](https://en.wikipedia.org/wiki/Josiah_Willard_Gibbs "Josiah Willard Gibbs")
+availability in entropy
+units[\[40\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-40)
+![{\\displaystyle A\\equiv -k\\ln Z}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d71f47d2e064d6dd4eb804d3fbc85414fa6d7f91)
+where Z is a constrained multiplicity or
+[partition function](https://en.wikipedia.org/wiki/Partition_function_%28mathematics%29 "Partition function (mathematics)")
+.
+
+When temperature T is fixed, free energy (![{\\displaystyle T\\times
+A}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6081e433612df99e17fdd5dc6e444c2eafe67bc4))
+is also minimized. Thus if
+![{\\displaystyle T,V}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a191b02cede513e88db4ae4995611bb2c722eb5b)
+and number of molecules N are constant, the
+[Helmholtz free energy](https://en.wikipedia.org/wiki/Helmholtz_free_energy "Helmholtz free energy")
+![{\\displaystyle F\\equiv U-TS}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a666284cd1282b2a96582ddbb8167627298963e1)
+(where U is energy and S is entropy) is minimized as a system "equilibrates." If
+T and P are held constant (say during processes in your body), the
+[Gibbs free energy](https://en.wikipedia.org/wiki/Gibbs_free_energy "Gibbs free energy")
+![{\\displaystyle G=U+PV-TS}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a5a60159352173c3a14b7c1331ecea327889fbb0)
+is minimized instead. The change in free energy under these conditions is a
+measure of available
+[work](https://en.wikipedia.org/wiki/Work_%28thermodynamics%29 "Work (thermodynamics)")
+that might be done in the process. Thus available work for an ideal gas at
+constant temperature
+![{\\displaystyle T\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2991f18ef81fec2f94e3a532df91a079f3f6697d)
+and pressure
+![{\\displaystyle P\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/07d20f5b8e09cda22c3552e1da5bf7c7709057d9)
+is
+![{\\displaystyle W=\\Delta G=NkT\_{o}\\Theta (V/V\_{o})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/50381560c4280ba174e4372e20e67f84a3e42227)
+where
+![{\\displaystyle V\_{o}=NkT\_{o}/P\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/96f665485e0792186953b11d48a462c38377174e)
+and
+![{\\displaystyle \\Theta (x)=x-1-\\ln x\\geq 0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1bb0a30e7c143bf0e0be43f2afc4988d7b598d94)
+(see also
+[Gibbs inequality](https://en.wikipedia.org/wiki/Gibbs_inequality "Gibbs inequality")
+).
+
+More
+generally[\[41\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-41)
+the [work available](https://en.wikipedia.org/wiki/Exergy "Exergy") relative to
+some ambient is obtained by multiplying ambient temperature
+![{\\displaystyle T\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2991f18ef81fec2f94e3a532df91a079f3f6697d)
+by relative entropy or _net surprisal_
+![{\\displaystyle \\Delta I\\geq 0,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/12b2fee10216ae36191e63f22abd073f0f1cdeeb)
+defined as the average value of
+![{\\displaystyle k\\ln(p/p\_{o})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/586545929842d36fc1185e8af64546dbd6fc770c)
+where
+![{\\displaystyle p\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2c058013bfdb87708b775d59bda857fd5e6d53b3)
+is the probability of a given state under ambient conditions. For instance, the
+work available in equilibrating a monatomic ideal gas to ambient values of
+![{\\displaystyle V\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4373adc7c6ba3ad0a230674368b6a3a2df6cf98d)
+and
+![{\\displaystyle T\_{o}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2991f18ef81fec2f94e3a532df91a079f3f6697d)
+is thus
+![{\\displaystyle W=T\_{o}\\Delta I}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b33ae1f27a5912e2d27ede43e55b9a9ec187a6f3)
+, where relative entropy
+
+![{\\displaystyle \\Delta I=Nk\\left\[\\Theta {\\left({\\frac
+{V}{V\_{o}}}\\right)}+{\\frac {3}{2}}\\Theta {\\left({\\frac
+{T}{T\_{o}}}\\right)}\\right\].}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0eb6f446447587fdb6044b12548247c26fddf955)
+
+The resulting contours of constant relative entropy, shown at right for a mole
+of Argon at
+[standard temperature and pressure](https://en.wikipedia.org/wiki/Standard_temperature_and_pressure "Standard temperature and pressure")
+, for example put limits on the conversion of hot to cold as in flame-powered
+air-conditioning or in the unpowered device to convert boiling-water to
+ice-water discussed
+here.[\[42\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-42)
+Thus relative entropy measures thermodynamic availability in bits.
+
+For
+[density matrices](https://en.wikipedia.org/wiki/Density_matrix "Density matrix")
+P and Q on a
+[Hilbert space](https://en.wikipedia.org/wiki/Hilbert_space "Hilbert space") ,
+the
+[quantum relative entropy](https://en.wikipedia.org/wiki/Quantum_relative_entropy "Quantum relative entropy")
+from Q to P is defined to be
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)=\\operatorname {Tr} (P(\\log P-\\log Q)).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/85368a6e510d46a3201422e2510f91363d0b70cc)
+
+In
+[quantum information science](https://en.wikipedia.org/wiki/Quantum_information_science "Quantum information science")
+the minimum of
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039fa82bd08654b4faa2b32ded70c0160554fa07)
+over all separable states Q can also be used as a measure of
+[entanglement](https://en.wikipedia.org/wiki/Quantum_entanglement "Quantum entanglement")
+in the state P.
+
+Just as relative entropy of "actual from ambient" measures thermodynamic
+availability, relative entropy of "reality from a model" is also useful even if
+the only clues we have about reality are some experimental measurements. In the
+former case relative entropy describes _distance to equilibrium_ or (when
+multiplied by ambient temperature) the amount of _available work_, while in the
+latter case it tells you about surprises that reality has up its sleeve or, in
+other words, _how much the model has yet to learn_.
+
+Although this tool for evaluating models against systems that are accessible
+experimentally may be applied in any field, its application to selecting a
+[statistical model](https://en.wikipedia.org/wiki/Statistical_model "Statistical model")
+via
+[Akaike information criterion](https://en.wikipedia.org/wiki/Akaike_information_criterion "Akaike information criterion")
+are particularly well described in
+papers[\[43\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-43)
+and a
+book[\[44\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-44)
+by Burnham and Anderson. In a nutshell the relative entropy of reality from a
+model may be estimated, to within a constant additive term, by a function of the
+deviations observed between data and the model's predictions (like the
+[mean squared deviation](https://en.wikipedia.org/wiki/Mean_squared_deviation "Mean squared deviation")
+) . Estimates of such divergence for models that share the same additive term
+can in turn be used to select among models.
+
+When trying to fit parametrized models to data there are various estimators
+which attempt to minimize relative entropy, such as
+[maximum likelihood](https://en.wikipedia.org/wiki/Maximum_likelihood "Maximum likelihood")
+and
+[maximum spacing](https://en.wikipedia.org/wiki/Maximum_spacing_estimation "Maximum spacing estimation")
+estimators.\[_[citation
+needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed
+"Wikipedia:Citation needed")_\]
+
+[Kullback & Leibler (1951)](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#CITEREFKullbackLeibler1951)
+also considered the symmetrized
+function:[\[6\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEKullbackLeibler195180-6)
+
+![{\\displaystyle D\_{\\text{KL}}(P\\parallel Q)+D\_{\\text{KL}}(Q\\parallel P)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/11b983ff758af97a9f2703d29a748f9b0f2f27a9)
+
+which they referred to as the "divergence", though today the "KL divergence"
+refers to the asymmetric function (see
+[§ Etymology](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Etymology)
+for the evolution of the term). This function is symmetric and nonnegative, and
+had already been defined and used by
+[Harold Jeffreys](https://en.wikipedia.org/wiki/Harold_Jeffreys "Harold Jeffreys")
+in
+1948;[\[7\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-FOOTNOTEJeffreys1948158-7)
+it is accordingly called the **Jeffreys divergence**.
+
+This quantity has sometimes been used for
+[feature selection](https://en.wikipedia.org/wiki/Feature_selection "Feature selection")
+in
+[classification](https://en.wikipedia.org/wiki/Statistical_classification "Statistical classification")
+problems, where P and Q are the conditional
+[Probability density functions](https://en.wikipedia.org/wiki/Probability_density_function "Probability density function")
+of a feature under two different classes. In the Banking and Finance industries,
+this quantity is referred to as **Population Stability Index** (**PSI**), and is
+used to assess distributional shifts in model features through time.
+
+An alternative is given via the
+![{\\displaystyle \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/b43d0ea3c9c025af1be9128e62a18fa74bedda2a)
+\-divergence,
+
+![{\\displaystyle D\_{\\lambda }(P\\parallel Q)=\\lambda D\_{\\text{KL}}(P\\parallel \\lambda P+(1-\\lambda )Q)+(1-\\lambda )D\_{\\text{KL}}(Q\\parallel \\lambda P+(1-\\lambda )Q){\\text{,}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ae204c7b3367027e53d52d76c97e693e6e84be10)
+
+which can be interpreted as the expected information gain about X from
+discovering which probability distribution X is drawn from, P or Q, if they
+currently have probabilities
+![{\\displaystyle \\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/b43d0ea3c9c025af1be9128e62a18fa74bedda2a)
+and
+![{\\displaystyle 1-\\lambda }](https://wikimedia.org/api/rest_v1/media/math/render/svg/da4535bdb18a0914505c66a316f858f601be0cf8)
+respectively.\[_[clarification
+needed](https://en.wikipedia.org/wiki/Wikipedia:Please_clarify "Wikipedia:Please
+clarify")_\] \[_[citation
+needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed
+"Wikipedia:Citation needed")_\]
+
+The value
+![{\\displaystyle \\lambda =0.5}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9355d8eeec28478037abac23ce379075d1cea5fb)
+gives the
+[Jensen–Shannon divergence](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence "Jensen–Shannon divergence")
+, defined by
+
+![{\\displaystyle D\_{\\text{JS}}={\\tfrac {1}{2}}D\_{\\text{KL}}(P\\parallel M)+{\\tfrac {1}{2}}D\_{\\text{KL}}(Q\\parallel M)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/566ad76576dda350c87c38c78a63fe8ba2b1b838)
+
+where M is the average of the two distributions,
+
+![{\\displaystyle M={\\tfrac {1}{2}}\\left(P+Q\\right){\\text{.}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a99d3183325024a989864a82d9c02d5213b681de)
+
+We can also interpret
+![{\\displaystyle D\_{\\text{JS}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6fc42cb8ebf661fea3c6d82f5a8261d5176fbd42)
+as the capacity of a noisy information channel with two inputs giving the output
+distributions P and Q. The Jensen–Shannon divergence, like all f\-divergences,
+is _locally_ proportional to the
+[Fisher information metric](https://en.wikipedia.org/wiki/Fisher_information_metric "Fisher information metric")
+. It is similar to the
+[Hellinger metric](https://en.wikipedia.org/wiki/Hellinger_metric "Hellinger metric")
+(in the sense that it induces the same
+[affine connection](https://en.wikipedia.org/wiki/Affine_connection "Affine connection")
+on a
+[statistical manifold](https://en.wikipedia.org/wiki/Statistical_manifold "Statistical manifold")
+).
+
+Furthermore, the Jensen–Shannon divergence can be generalized using abstract
+statistical M-mixtures relying on an abstract mean
+M.[\[45\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Nielsen2019-45)[\[46\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-Nielsen2020-46)
+
+There are many other important measures of
+[probability distance](https://en.wikipedia.org/wiki/Statistical_distance "Statistical distance")
+. Some of these are particularly connected with relative entropy. For example:
+
+Other notable measures of distance include the
+[Hellinger distance](https://en.wikipedia.org/wiki/Hellinger_distance "Hellinger distance")
+, _histogram intersection_, _[Chi-squared
+statistic](https://en.wikipedia.org/wiki/Chi-squared_test "Chi-squared test")_,
+_quadratic form distance_, _[match
+distance](https://en.wikipedia.org/wiki/Matching_distance "Matching distance")_,
+_[Kolmogorov–Smirnov
+distance](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
+"Kolmogorov–Smirnov test")_, and _[earth mover's
+distance](https://en.wikipedia.org/wiki/Earth_mover's_distance "Earth mover's
+distance")_.[\[49\]](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#cite_note-earth-49)
+
+Just as _absolute_ entropy serves as theoretical background for
+[data _compression_](https://en.wikipedia.org/wiki/Data_compression "Data compression")
+, _relative_ entropy serves as theoretical background for
+[data _differencing_](https://en.wikipedia.org/wiki/Data_differencing "Data differencing")
+– the absolute entropy of a set of data in this sense being the data required to
+reconstruct it (minimum compressed size), while the relative entropy of a target
+set of data, given a source set of data, is the data required to reconstruct the
+target _given_ the source (minimum size of a
+[patch](https://en.wikipedia.org/wiki/Patch_%28computing%29 "Patch (computing)")
+).
+
+*   [Akaike information criterion](https://en.wikipedia.org/wiki/Akaike_information_criterion "Akaike information criterion")
+*   [Bayesian information criterion](https://en.wikipedia.org/wiki/Bayesian_information_criterion "Bayesian information criterion")
+*   [Bregman divergence](https://en.wikipedia.org/wiki/Bregman_divergence "Bregman divergence")
+*   [Cross-entropy](https://en.wikipedia.org/wiki/Cross-entropy "Cross-entropy")
+*   [Deviance information criterion](https://en.wikipedia.org/wiki/Deviance_information_criterion "Deviance information criterion")
+*   [Entropic value at risk](https://en.wikipedia.org/wiki/Entropic_value_at_risk "Entropic value at risk")
+*   [Entropy power inequality](https://en.wikipedia.org/wiki/Entropy_power_inequality "Entropy power inequality")
+*   [Hellinger distance](https://en.wikipedia.org/wiki/Hellinger_distance "Hellinger distance")
+*   [Information gain in decision trees](https://en.wikipedia.org/wiki/Information_gain_in_decision_trees "Information gain in decision trees")
+*   [Information gain ratio](https://en.wikipedia.org/wiki/Information_gain_ratio "Information gain ratio")
+*   [Information theory and measure theory](https://en.wikipedia.org/wiki/Information_theory_and_measure_theory "Information theory and measure theory")
+*   [Jensen–Shannon divergence](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence "Jensen–Shannon divergence")
+*   [Quantum relative entropy](https://en.wikipedia.org/wiki/Quantum_relative_entropy "Quantum relative entropy")
+*   [Solomon Kullback](https://en.wikipedia.org/wiki/Solomon_Kullback "Solomon Kullback") and [Richard Leibler](https://en.wikipedia.org/wiki/Richard_Leibler "Richard Leibler")
+*   [Bhattacharyya distance](https://en.wikipedia.org/wiki/Bhattacharyya_distance "Bhattacharyya distance")
+
+* * *
+
+14.  Bishop, Christopher M. _Pattern recognition and machine learning_. p. 55\.
+     [OCLC](https://en.wikipedia.org/wiki/OCLC_%28identifier%29 "OCLC (identifier)")
+     [1334664824](https://search.worldcat.org/oclc/1334664824) .
+     
+
+22.  Soklakov, A. N. (2023). "Information Geometry of Risks and Returns". _Risk_.
+     **June**.
+     [SSRN](https://en.wikipedia.org/wiki/SSRN_%28identifier%29 "SSRN (identifier)")
+     [4134885](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4134885) .
+     
+23.  Henide, Karim (30 September 2024). "Flow Rider: Tradable Ecosystems' Relative
+     Entropy of Flows As a Determinant of Relative Value". _The Journal of
+     Investing_. **33** (6): 34–58\.
+     [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)")
+     :[10.3905/joi.2024.1.321](https://doi.org/10.3905%2Fjoi.2024.1.321).
+     
+24.  Sanov, I.N. (1957). "On the probability of large deviations of random
+     magnitudes". _Mat. Sbornik_. **42** (84): 11–44.
+     
+
+26.  Hobson, Arthur (1971). _Concepts in statistical mechanics_. New York: Gordon and
+     Breach.
+     [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)")
+     [978-0-677-03240-5](https://en.wikipedia.org/wiki/Special:BookSources/978-0-677-03240-5 "Special:BookSources/978-0-677-03240-5")
+     .
+     
+27.  Bonnici, V. (2020). "Kullback-Leibler divergence between quantum distributions,
+     and its upper-bound".
+     [arXiv](https://en.wikipedia.org/wiki/ArXiv_%28identifier%29 "ArXiv (identifier)")
+     :[2008.05932](https://arxiv.org/abs/2008.05932)
+     \[[cs.LG](https://arxiv.org/archive/cs.LG)\].
+     
+
+29.  Donsker, Monroe D.; Varadhan, SR Srinivasa (1983). "Asymptotic evaluation of
+     certain Markov process expectations for large time. IV". _Communications on Pure
+     and Applied Mathematics_. **36** (2): 183–212\.
+     [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)")
+     :[10.1002/cpa.3160360204](https://doi.org/10.1002%2Fcpa.3160360204).
+     
+
+31.  Belov, Dmitry I.; Armstrong, Ronald D. (2011-04-15). "Distributions of the
+     Kullback-Leibler divergence with applications". _British Journal of Mathematical
+     and Statistical Psychology_. **64** (2): 291–309\.
+     [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)")
+     :[10.1348/000711010x522227](https://doi.org/10.1348%2F000711010x522227).
+     [ISSN](https://en.wikipedia.org/wiki/ISSN_%28identifier%29 "ISSN (identifier)")
+     [0007-1102](https://search.worldcat.org/issn/0007-1102) .
+     [PMID](https://en.wikipedia.org/wiki/PMID_%28identifier%29 "PMID (identifier)")
+     [21492134](https://pubmed.ncbi.nlm.nih.gov/21492134) .
+     
+32.  Buchner, Johannes (2022-04-29). _An intuition for physicists: information gain
+     from experiments_.
+     [OCLC](https://en.wikipedia.org/wiki/OCLC_%28identifier%29 "OCLC (identifier)")
+     [1363563215](https://search.worldcat.org/oclc/1363563215) .
+     
+33.  Nielsen, Frank; Garcia, Vincent (2011). "Statistical exponential families: A
+     digest with flash cards".
+     [arXiv](https://en.wikipedia.org/wiki/ArXiv_%28identifier%29 "ArXiv (identifier)")
+     :[0911.4863](https://arxiv.org/abs/0911.4863)
+     \[[cs.LG](https://arxiv.org/archive/cs.LG)\].
+     
+34.  Cover, Thomas M.; Thomas, Joy A. (1991), _Elements of Information Theory_,
+     [John Wiley & Sons](https://en.wikipedia.org/wiki/John_Wiley_&_Sons "John Wiley & Sons")
+     , p. 22
+     
+
+44.  Burnham, Kenneth P. (December 2010). _Model selection and multimodel inference:
+     a practical information-theoretic approach_. Springer.
+     [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)")
+     [978-1-4419-2973-0](https://en.wikipedia.org/wiki/Special:BookSources/978-1-4419-2973-0 "Special:BookSources/978-1-4419-2973-0")
+     .
+     [OCLC](https://en.wikipedia.org/wiki/OCLC_%28identifier%29 "OCLC (identifier)")
+     [878132909](https://search.worldcat.org/oclc/878132909) .
+     
+
+47.  Bretagnolle, J.; Huber, C. (1978), "Estimation des densités : Risque minimax",
+     _Séminaire de Probabilités XII_, Lecture Notes in Mathematics (in French), vol.
+     649, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 342–363,
+     [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)")
+     :[10.1007/bfb0064610](https://doi.org/10.1007%2Fbfb0064610),
+     [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)")
+     [978-3-540-08761-8](https://en.wikipedia.org/wiki/Special:BookSources/978-3-540-08761-8 "Special:BookSources/978-3-540-08761-8")
+     ,
+     [S2CID](https://en.wikipedia.org/wiki/S2CID_%28identifier%29 "S2CID (identifier)")
+     [122597694](https://api.semanticscholar.org/CorpusID:122597694) Lemma 2.1
+     
+
+*   [Amari, Shun-ichi](https://en.wikipedia.org/wiki/Shun'ichi_Amari "Shun'ichi Amari") (2016). _Information Geometry and Its Applications_. Applied Mathematical Sciences. Vol. 194\. Springer Japan. pp. XIII, 374. [doi](https://en.wikipedia.org/wiki/Doi_%28identifier%29 "Doi (identifier)"):[10.1007/978-4-431-55978-8](https://doi.org/10.1007%2F978-4-431-55978-8). [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)") [978-4-431-55977-1](https://en.wikipedia.org/wiki/Special:BookSources/978-4-431-55977-1 "Special:BookSources/978-4-431-55977-1").
+*   Kullback, Solomon (1959), _Information Theory and Statistics_, [John Wiley & Sons](https://en.wikipedia.org/wiki/John_Wiley_&_Sons "John Wiley & Sons"). Republished by [Dover Publications](https://en.wikipedia.org/wiki/Dover_Publications "Dover Publications") in 1968; reprinted in 1978: [ISBN](https://en.wikipedia.org/wiki/ISBN_%28identifier%29 "ISBN (identifier)") [0-8446-5625-9](https://en.wikipedia.org/wiki/Special:BookSources/0-8446-5625-9 "Special:BookSources/0-8446-5625-9").
+*   Jeffreys, Harold (1948). _Theory of Probability_ (Second ed.). Oxford University Press.
+
+*   [Information Theoretical Estimators Toolbox](https://bitbucket.org/szzoli/ite/)
+*   [Ruby gem for calculating Kullback–Leibler divergence](https://github.com/evansenter/diverge)
+*   [Jon Shlens' tutorial on Kullback–Leibler divergence and likelihood theory](https://arxiv.org/abs/1404.2000)
+*   [Matlab code for calculating Kullback–Leibler divergence for discrete distributions](http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13089&objectType=file) [Archived](https://web.archive.org/web/20070929155435/http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13089&objectType=file) 2007-09-29 at the [Wayback Machine](https://en.wikipedia.org/wiki/Wayback_Machine "Wayback Machine")
+*   [Sergio Verdú](https://en.wikipedia.org/wiki/Sergio_Verd%C3%BA "Sergio Verdú"), [Relative Entropy](http://videolectures.net/nips09_verdu_re/), [NIPS](https://en.wikipedia.org/wiki/Conference_on_Neural_Information_Processing_Systems "Conference on Neural Information Processing Systems") 2009. One-hour video lecture.
+*   [A modern summary of info-theoretic divergence measures](https://arxiv.org/abs/math/0604246)
+```
+
+### `log://turn_3/get/log%3A/turn_2/search/**`
+scheme=log action=get attrs={"path":"log://turn_2/search/**","action":"get"}
+
+```
+MANIFEST get path="log://turn_2/search/**": 1 matched (1663 tokens)
+log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi (1663)
+```
+
+### `log://turn_3/set/known%3A/kl_math/forward_backward_uniform`
+scheme=log action=set attrs={"path":"known://kl_math/forward_backward_uniform","patch":"===================================================================\n--- known://kl_math/forward_backward_uniform\told\t\n+++ known://kl_math/forward_backward_uniform\tnew\t\n@@ -0,0 +1,20 @@\n+# Related\n+[main prompt](prompt://1)\n+[unknown resolved](unknown://kl_divergence/forward_backward_relationship)\n+[search results](log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi)\n+\n+# KL(P||U) and KL(U||P) for uniform U\n+\n+KL(P||U) = H(U) - H(P) = log(V) - H(P), where H(U)=log(150000)≈11.92\n+\n+For KL(P||U)=10, H(P)≈1.92 (concentrated P).\n+\n+KL(U||P) = -sum_{i} (1/V) log(P(i)) = log(V) - sum P(i) log(P(i)) = log(V) - H(P)\n+\n+From search sources:\n+- blester125: KL(P||U) reduces to H(U) - H(P) for uniform U.\n+- Wikipedia: Confirms KL definition, relative entropy.\n+\n+Exact relation: KL(P||U) + H(P) = log(V); KL(U||P) = log(V) - H(P)\n+\n+Thus KL(P||U) = KL(U||P). Both cannot be independently 10.0.\n\\n","merge":"<<<<<<< SEARCH\n=======\n# Related\n[main prompt](prompt://1)\n[unknown resolved](unknown://kl_divergence/forward_backward_relationship)\n[search results](log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi)\n\n# KL(P||U) and KL(U||P) for uniform U\n\nKL(P||U) = H(U) - H(P) = log(V) - H(P), where H(U)=log(150000)≈11.92\n\nFor KL(P||U)=10, H(P)≈1.92 (concentrated P).\n\nKL(U||P) = -sum_{i} (1/V) log(P(i)) = log(V) - sum P(i) log(P(i)) = log(V) - H(P)\n\nFrom search sources:\n- blester125: KL(P||U) reduces to H(U) - H(P) for uniform U.\n- Wikipedia: Confirms KL definition, relative entropy.\n\nExact relation: KL(P||U) + H(P) = log(V); KL(U||P) = log(V) - H(P)\n\nThus KL(P||U) = KL(U||P). Both cannot be independently 10.0.\n>>>>>>> REPLACE","beforeTokens":0,"afterTokens":371,"summary":"kl,math,uniform,forward,backward,relationship,entropy,h_u-log_v","action":"set"}
+
+```
+# Related
+[main prompt](prompt://1)
+[unknown resolved](unknown://kl_divergence/forward_backward_relationship)
+[search results](log://turn_2/search/KL_divergence_forward_backward_relationship_KL(P%7C%7CU)_KL(U%7C%7CP)_uniform_distributi)
+
+# KL(P||U) and KL(U||P) for uniform U
+
+KL(P||U) = H(U) - H(P) = log(V) - H(P), where H(U)=log(150000)≈11.92
+
+For KL(P||U)=10, H(P)≈1.92 (concentrated P).
+
+KL(U||P) = -sum_{i} (1/V) log(P(i)) = log(V) - sum P(i) log(P(i)) = log(V) - H(P)
+
+From search sources:
+- blester125: KL(P||U) reduces to H(U) - H(P) for uniform U.
+- Wikipedia: Confirms KL definition, relative entropy.
+
+Exact relation: KL(P||U) + H(P) = log(V); KL(U||P) = log(V) - H(P)
+
+Thus KL(P||U) = KL(U||P). Both cannot be independently 10.0.
+```
+
+### `log://turn_3/set/unknown%3A/kl_divergence/forward_backward_relationship`
+scheme=log action=set attrs={}
+
+```
+unknown://kl_divergence/forward_backward_relationship set to archived
+```
+
+### `log://turn_3/update/distilled_KL_math_relationship%3B_discovering_feasibility`
+scheme=log action=update attrs={"status":155,"action":"update"}
+
+```
+distilled KL math relationship; discovering feasibility
+```
+