math-skill 1.0.0 → 2.1.0

package/skills/information-theory/SKILL.md

@@ -1,242 +1,140 @@
----
-name: information-theory
-description: |
-  科研模式触发：熵计算、信道容量、编码理论、模型选择等信息论数学形式化问题。
-  生活模式触发：评估信息价值、减少冗余、判断该获取哪些信息等日常信息决策问题。
-  English (Research mode): Trigger for entropy computation, channel capacity, coding theory, model selection.
-  English (Life mode): Trigger for assessing information value, reducing redundancy, evaluating what information to seek.
----
-
-# 📡 信息论思想
-
-> "信息是不确定性的减少——知道更多意味着怀疑更少。"
-> "Information is reduction of uncertainty — knowing more means doubting less."
->
-> —— 信息论、编码理论、统计推断
-> —— Information Theory, Coding Theory, Statistical Inference
-
-## 核心原则 / Core Principle
-
-三个核心洞见：
-1. **不确定性是可量化的**——熵 H(X) 给出不确定性的精确度量，而非主观感受
-   **Uncertainty is quantifiable** — entropy H(X) gives a precise measure of uncertainty, not a subjective feeling
-2. **压缩与通信存在基本极限**——信源编码定理与信道编码定理揭示了信息处理的数学边界，不可逾越
-   **Compression and communication have fundamental limits** — source coding and channel coding theorems reveal mathematical boundaries of information processing, unbreakable
-3. **信息增益连接不确定性与决策**——观察 Y 对 X 的信息增益 I(X;Y) = H(X) − H(X|Y) 指导在不确定性下应优先获取何种信息
-   **Information gain connects uncertainty and decision-making** — I(X;Y) = H(X) − H(X|Y) guides which information to prioritize under uncertainty
-
-> **数学形式化 / Mathematical Formalization**（科研模式参考）
->
-> Shannon 熵 H(X) = −Σ p(x) log p(x) 量化随机变量的平均"惊奇度"——概率越低的事件发生时带来的惊奇越大，熵是所有事件惊奇度的期望。
->
-> 信息增益 I(X;Y) = H(X) − H(X|Y) 量化观察 Y 之后对 X 不确定性的减少——这正是"信息"的数学定义。
->
-> 信源编码定理：最优压缩的平均编码长度 ≥ H(X) bits/symbol，低于熵限则必然丢失信息。
->
-> 信道编码定理：可靠通信的速率上限为信道容量 C = max I(X;Y)，当传输速率 R < C 时存在编码方案使误码率趋零，R > C 时可靠通信不可能。
->
-> 详细数学依据见 `original-texts.md`
-
-## 不适用场景 / When NOT to Use
-
-- **问题无概率结构**（如纯粹符号推理、逻辑演绎）——熵与信息增益需要概率分布，无概率则无信息论 `[科研]`
-  **No probabilistic structure** (e.g., pure symbolic reasoning, logical deduction) — entropy and information gain require probability distributions `[科研]`
-- **纯确定性场景无不确定性**（如已知精确答案的数学问题）——熵为零时信息论退化为平凡结论 `[科研]`
-  **Purely deterministic scenarios with no uncertainty** (e.g., math problems with exact known answers) — when entropy is zero, information theory degenerates to trivial conclusions `[科研]`
-- **定性判断无需量化**（如美学评价、情感判断）——信息论量化的是概率意义上的不确定性，非语义歧义 `[通用]`
-  **Qualitative judgments need no quantification** (e.g., aesthetic evaluation, emotional judgment) — information theory quantifies uncertainty in the probabilistic sense, not semantic ambiguity `[通用]`
-- **被噪音信息淹没而忽略关键信息**——在信息过载的环境中迷失于无关细节 `[生活]`
-  **Overwhelmed by noise information while ignoring key information** — lost in irrelevant details in an information-overloaded environment `[生活]`
-- **追求更多信息而不评估其价值**——数量不等于质量，未经筛选的信息可能是噪音 `[生活]`
-  **Seeking more information without evaluating its value** — quantity ≠ quality; unfiltered information may be noise `[生活]`
-- **信息过载导致决策瘫痪**——信息量超过处理能力时反而降低决策质量 `[生活]`
-  **Decision paralysis from information overload** — when information volume exceeds processing capacity, decision quality actually declines `[生活]`
-
-## 何时使用 / When to Use
-
-### 科研触发条件 / Research Trigger Conditions
-
-- 需要度量不确定性的大小（熵 H(X) 量化随机变量的"混乱程度"）
-  Need to measure the magnitude of uncertainty (entropy H(X) quantifies the "disorder" of a random variable)
-- 需要比较不同信息源的价值（互信息 I(X;Y) 衡量哪个观察 Y 最能减少关于 X 的不确定性）
-  Need to compare the value of different information sources (mutual information I(X;Y) measures which observation Y best reduces uncertainty about X)
-- 需要最优数据压缩（信源编码定理保证最优压缩极限为 H(X) bits/symbol）
-  Need optimal data compression (source coding theorem guarantees the optimal compression limit is H(X) bits/symbol)
-- 需要在噪声环境下可靠通信（信道编码定理保证速率低于容量 C 时可靠传输可行）
-  Need reliable communication under noise (channel coding theorem guarantees reliable transmission when rate is below capacity C)
-- 需要特征选择或模型选择（互信息筛选特征，AIC/BIC/MDL 作为信息准则选择模型）
-  Need feature selection or model selection (mutual information filters features, AIC/BIC/MDL as information criteria for model selection)
-- 需要贝叶斯模型比较（KL 散度 D(P||Q) 衡量分布间信息距离，贝叶斯因子量化模型间证据比）
-  Need Bayesian model comparison (KL divergence D(P||Q) measures information distance between distributions, Bayes factor quantifies evidence ratio between models)
-
-### 生活触发条件 / Life Trigger Conditions
-
-- 需要评估某条信息是否值得获取——不是所有信息都有同等价值
-  Need to assess whether a piece of information is worth acquiring — not all information has equal value
-- 需要在信息过载中筛选关键信息——去除冗余、聚焦核心
-  Need to filter key information from overload — remove redundancy, focus on the core
-- 需要判断哪个解释更合理——最简洁且最有解释力的解释优于复杂但增量解释力很小的解释
-  Need to judge which explanation is more reasonable — the simplest yet most explanatory beats complex ones with marginal explanatory gain
-- 需要决定是先收集更多信息还是果断行动——信息不足时先收集关键信息，信息足够时果断行动
-  Need to decide whether to gather more information or act decisively — collect key info when insufficient, act decisively when sufficient
-- 需要提高沟通效率——识别信息传递的瓶颈，用最精炼的方式传达关键内容
-  Need to improve communication efficiency — identify bottlenecks, convey key content in the most concise way
-
-## 方法流程 / Method
-
-### 第一步：识别信息源与不确定度 / Step 1: Identify Source and Uncertainty
-
-**科研模式 / Research Mode:**
-- **随机变量 X** 是什么？——定义信息源，明确要研究的不确定性对象
-- **概率分布 p(x)**——离散分布用概率表，连续分布用密度函数
-- **计算 H(X) = −Σ p(x) log p(x)**——量化当前不确定性水平
-- **识别要减少的不确定性**——明确"知道什么之后不确定性会降低？"
-
-**生活模式 / Life Mode:**
-- 你面对的不确定性是什么？你最想搞清楚哪件事？——明确"知道什么之后不确定性会降低"是关键
-  What uncertainty are you facing? What do you most want to figure out? — clarifying "what knowledge would reduce uncertainty" is the key
-
-**共通要点 / Common Point:**
-- 核心都是明确不确定性对象——科研用数学定义 X 和 p(x)，生活用问题描述
-  The core is clarifying the object of uncertainty — research uses mathematical definitions of X and p(x), life uses problem description
-
-### 第二步：量化信息增益 / Step 2: Quantify Information Gain
-
-**科研模式 / Research Mode:**
-- **计算条件熵 H(X|Y)**——观察 Y 之后 X 的剩余不确定性
-- **计算互信息 I(X;Y) = H(X) − H(X|Y)**——Y 对 X 提供的信息量
-- **识别最优观察**——哪个 Y 使 I(X;Y) 最大？该观察最值得获取
-- **链式规则**——H(X₁,...,Xₙ) = H(X₁) + H(X₂|X₁) + ... + H(Xₙ|X₁,...,Xₙ₋₁)，逐个变量拆解联合不确定性
-
-**生活模式 / Life Mode:**
-- 这条信息能减少多少不确定性？——不是所有信息都有同等价值，有的信息一针见血，有的信息只是噪音
-  How much uncertainty can this information reduce? — not all information has equal value; some is incisive, some is just noise
-
-**共通要点 / Common Point:**
-- 核心都是评估信息的价值——科研用 I(X;Y) 精确量化，生活用直觉判断哪条信息最能减少不确定性
-  The core is evaluating information value — research uses I(X;Y) for precise quantification, life uses intuitive judgment of which info best reduces uncertainty
-
-### 第三步：选择编码策略 / Step 3: Choose Coding Strategy
-
-**科研模式 / Research Mode:**
-- **信源编码（压缩）**：Huffman 编码（贪心最优前缀码，平均长度接近 H(X)）、算术编码（更接近熵限）、通用编码（LZ77/LZ78/LZW，无需已知分布）
-- **信道编码（纠错）**：Hamming 码（最小距离 3，纠正 1 位错）、Reed-Solomon 码（突发纠错）、LDPC/Turbo 码（逼近 Shannon 极限）
-- **编码选择原则**：压缩需求用信源编码→逼近 H(X)；噪声防护用信道编码→逼近 Shannon 极限 (R→C)
-
-**生活模式 / Life Mode:**
-- 如何用最精炼的方式传递关键信息？——去除冗余、聚焦核心、避免信息过载
-  How to convey key information in the most concise way? — remove redundancy, focus on the core, avoid information overload
-
-**共通要点 / Common Point:**
-- 核心都是压缩与传递的效率——科研用编码定理逼近数学极限，生活用精炼表达逼近沟通极限
-  The core is efficiency of compression and transmission — research uses coding theorems to approach mathematical limits, life uses concise expression to approach communication limits
-
-### 第四步：评估信道容量 / Step 4: Evaluate Channel Capacity
-
-**科研模式 / Research Mode:**
-- **计算信道容量 C = max_{p(x)} I(X;Y)**——在所有输入分布 p(x) 上最大化互信息
-- **比较传输速率 R 与容量 C**：R < C → 可靠通信可行；R > C → 必定出错
-- **噪声模型**：BSC（二元对称信道，翻转概率 p）、BEC（二元擦除信道，擦除概率 ε）、AWGN（加性白高斯噪声信道）
-- **容量公式示例**：BSC 容量 C = 1 − H(p)；AWGN 容量 C = ½ log(1 + S/N)
-
-**生活模式 / Life Mode:**
-- 信息传递的瓶颈在哪？——不是所有信息都能有效传达，识别沟通的限制条件
-  Where is the bottleneck in information transmission? — not all information can be effectively conveyed; identify communication constraints
-
-**共通要点 / Common Point:**
-- 核心都是识别信息传递的限制——科研用信道容量 C 量化数学极限，生活用沟通瓶颈识别实际限制
-  The core is identifying limits of information transmission — research uses channel capacity C to quantify mathematical limits, life uses communication bottlenecks to identify practical limits
-
-### 第五步：应用信息准则 / Step 5: Apply Information Criteria
-
-**科研模式 / Research Mode:**
-- **AIC（赤池信息准则）**：AIC = −2lnL + 2k——偏重拟合，适合预测目标
-- **BIC（贝叶斯信息准则）**：BIC = −2lnL + k·ln(n)——偏重简约，适合解释目标
-- **KL 散度 D(P||Q) = Σ p(x) log(p(x)/q(x))**——衡量用 Q 替代 P 的信息损失，注意不对称性 D(P||Q) ≠ D(Q||P)
-- **MDL 原则（最小描述长度）**：选择使"数据描述长度 + 模型描述长度"最小的模型——信息论版本的奥卡姆剃刀
-
-**生活模式 / Life Mode:**
-- 如何判断哪个解释更好？——最简洁且最解释力的解释优于复杂但增量解释力很小的解释（奥卡姆剃刀的信息论版本）
-  How to judge which explanation is better? — the simplest yet most explanatory explanation beats complex ones with marginal explanatory gain (the information-theoretic version of Occam's razor)
-
-**共通要点 / Common Point:**
-- 核心都是简洁性与解释力的权衡——科研用 AIC/BIC/KL/MDL 精确量化，生活用"简洁且有解释力"直觉判断
-  The core is balancing simplicity and explanatory power — research uses AIC/BIC/KL/MDL for precise quantification, life uses "simple yet explanatory" intuitive judgment
-
-### 第六步：做出信息最优决策 / Step 6: Make Information-Optimal Decision
-
-**科研模式 / Research Mode:**
-- **贝叶斯实验设计**：选择使期望信息增益 max E[I(θ;Y)] 最大的实验——优先获取最能减少不确定性的数据
-- **最小化 KL 散度**：决策输出分布 Q 应尽量接近目标分布 P，即 min D(P||Q)
-- **最大熵原则**：在已知约束下选择使 H(X) 最大的分布——最少假设，最保守推断
-- **信息瓶颈**：min I(X;T) − βI(T;Y)——在压缩 X 为 T 时保留关于 Y 的最大相关信息
-
-**生活模式 / Life Mode:**
-- 在当前信息水平下做最优决策——信息不足时先收集关键信息，信息足够时果断行动
-  Make the optimal decision at the current information level — collect key info when insufficient, act decisively when sufficient
-
-**共通要点 / Common Point:**
-- 核心都是信息与行动的平衡——科研用信息增益最大化设计实验，生活用"信息不足先收集、信息足够果断行动"策略
-  The core is balancing information and action — research uses information gain maximization to design experiments, life uses "collect when insufficient, act when sufficient" strategy
-
-## 常见错误 / Common Errors
-
-| 错误 / Error | 数学批评 / Mathematical Critique | 正确做法 / Correct Approach | 标签 / Tag |
-|---|---|---|---|
-| 把信息等同于比特而非概率减少 | 熵 H(X) 是概率分布的函数，不是比特的物理量；信息增益 I(X;Y) 是不确定性的减少量 | 理解信息为不确定性的减少：I(X;Y) = H(X) − H(X|Y)，比特只是量化单位 | `[科研]` |
-| 忽视信道容量极限 | R > C 时无论何种编码都不能实现可靠通信；信道容量是数学极限而非工程限制 | 计算信道容量 C = max I(X;Y)，确保 R < C | `[科研]` |
-| 混淆熵与方差 | 熵 H(X) 度量分布的"扩散度"（概率意义上），方差 Var(X) 度量分布的"展开度"（距离意义上）；两者不等价 | 熵量化概率结构的不确定性，方差量化数值偏差；连续分布熵可为负 | `[科研]` |
-| 过度压缩低于熵限 | 信源编码定理保证最优压缩 ≥ H(X) bits/symbol；低于此限必然丢失信息 | 最优压缩极限为 H(X)，接受此极限并据此设计编码 | `[科研]` |
-| 忽视 KL 散度不对称性 | D(P||Q) ≠ D(Q||P)；D(P||Q) 衡量"用 Q 近似 P 的信息损失"，D(Q||P) 衡量"用 P 近似 Q 的信息损失" | 明确 KL 散度的方向性：D(P||Q) 用于编码（用 Q 编码 P 的额外代价） | `[科研]` |
-| 把相关性等同于信息 | 相关 ρ(X,Y) 仅衡量线性关联；互信息 I(X;Y) 衡量所有依赖关系（含非线性）；I(X;Y)=0 ⇔ X,Y 独立，但 ρ=0 ≠ X,Y 独立 | 用互信息 I(X;Y) 代替相关系数 ρ 评估变量间依赖关系 | `[科研]` |
-| 把定性判断强行量化 | 信息论量化的是概率意义上的不确定性，非语义歧义或主观感受 | 区分概率不确定性与语义歧义，不要对定性问题强行套用熵公式 | `[通用]` |
-| 被噪音信息淹没而忽略关键信息 | 信息过载时，大量低价值信息掩盖了少量高价值信息 | 先评估每条信息的不确定性减少能力，优先关注高信息增益的信息 | `[生活]` |
-| 追求更多信息而不评估其价值 | 数量 ≠ 质量；未经筛选的信息可能是噪音而非信息 | 每获取一条信息前先问"这条信息能减少多少不确定性？" | `[生活]` |
-| 信息过载导致决策瘫痪 | 信息量超过处理能力时，决策质量反而下降 | 设定信息收集的止损点——信息足够时果断行动，不再追求完美信息 | `[生活]` |
-
-## 操作规程 / Operating Procedure
-
-当本 skill 被触发时，首先判断模式：
-
-- **科研模式**：问题涉及熵计算、信道容量、编码理论、模型选择等数学形式化
-- **生活模式**：问题涉及评估信息价值、减少冗余、判断该获取哪些信息等日常决策
-
----
-
-### 科研模式输出格式 / Research Mode Output Format
-
-1. **[信息源]:[描述]** H(X) = [值]——定义随机变量 X，计算其熵，量化当前不确定性
-2. **[信息增益]:[描述]** I(X;Y) = [值]——计算互信息，识别最有价值的观察 Y
-3. **[编码策略]:[选择]**——信源编码（压缩）或信道编码（纠错），说明逼近哪种极限
-4. **[信道容量]:[描述]** C = [值]——计算信道容量，比较传输速率 R 与 C
-5. **[信息准则]:[AIC/BIC/KL/MDL]**——说明选择的信息准则及理由
-6. **[最优决策]:[说明]**——基于信息增益最大化或 KL 散度最小化的决策建议
-
-**科研模式输出必须包含以上 6 项，不得只输出分析性文字而不给出结论。**
-
----
-
-### 生活模式输出格式 / Life Mode Output Format
-
-1. **[核心不确定性]:[描述]** — 你最想搞清楚什么
-   **[Core Uncertainty]:[description]** — What you most want to figure out
-2. **[信息价值排序]:[列表]** — 哪些信息最能减少不确定性，价值高低排序
-   **[Information Value Ranking]:[list]** — Which information best reduces uncertainty, ranked by value
-3. **[信息瓶颈]:[识别]** — 信息传递或获取的限制在哪
-   **[Information Bottleneck]:[identification]** — Where are the limits on information transmission or acquisition
-4. **[精炼传递]:[建议]** — 如何用最精炼的方式聚焦关键信息
-   **[Concise Transmission]:[suggestion]** — How to focus key information in the most concise way
-5. **[最优解释]:[判断]** — 哪个解释最简洁且最有力
-   **[Optimal Explanation]:[judgment]** — Which explanation is simplest yet most powerful
-6. **[行动建议]:[步骤]** — 在当前信息水平下最务实的行动方案
-   **[Action Advice]:[steps]** — The most pragmatic action plan at the current information level
-
-**生活模式输出必须包含以上 6 项，不得只输出分析性文字而不给出结论。**
-
-## 与其他 skill 的关系 / Relations to Other Skills
-
-- **概率与统计**(熵与信息增益补充概率推理)——熵 H(X) 度量概率分布的不确定性，信息增益 I(X;Y) 是贝叶斯更新的信息论表达
-- **优化思想**(信道容量最大化是优化问题)——C = max_{p(x)} I(X;Y) 是在输入分布上的最优化问题
-- **变换思想**(编码是信息空间的变换)——编码将信息从原始空间映射到压缩/纠错空间，是信息空间的变换
-- **建模思想**(信息准则指导模型选择)——AIC/BIC/MDL 从信息论角度量化模型的拟合与复杂度权衡
-- **算法思想**(压缩算法是计算实现)——Huffman/LZ/LDPC 等算法是信息论极限的逼近实现
+---
+name: information-theory
+description: |
+  触发：问题需量化不确定性、以"信息=不确定性的减少"度量信息价值；或需计算熵/互信息/KL 散度/信道容量，为压缩/KV 压缩/量化/路由设计信息准则时调用。
+  English: Trigger when a problem needs to quantify uncertainty, measure information value as "information = reduction of uncertainty"; or compute entropy / mutual information / KL divergence / channel capacity, or design information criteria for compression / KV-cache compression / quantization / routing.
+---
+
+> **语言路由**：若用户消息为英文，请读取并遵循同目录下的 `SKILL.en.md`，按其操作规程以英文输出；中文消息则继续使用本文件。
+
+# 📡 信息论思想 / Information Theory
+
+> "信息是不确定性的减少——知道更多意味着怀疑更少。"
+> "Information is reduction of uncertainty — knowing more means doubting less."
+>
+> —— 信息论、编码理论、统计推断 / Information Theory, Coding Theory, Statistical Inference
+
+## 核心原则 / Core Principle
+
+**信息是不确定性的减少——不确定性可由熵精确量化，压缩与通信存在由熵限与信道容量刻画的不可逾越极限，信息增益则指导在不确定性下应优先获取何种观察。**
+
+**Information is reduction of uncertainty — uncertainty is precisely quantified by entropy; compression and communication have unbreakable limits set by the entropy bound and channel capacity; information gain guides which observation to prioritize under uncertainty.**
+
+> **数学形式化 / Mathematical Formalization**
+>
+> Shannon 熵 $H(X) = -\sum_x p(x)\log p(x)$ 量化随机变量的平均"惊奇度"——概率越低的事件发生时惊奇越大，熵是惊奇度的期望。
+>
+> 互信息 $I(X;Y) = H(X) - H(X|Y) = \sum_{x,y} p(x,y)\log\frac{p(x,y)}{p(x)p(y)}$，即观察 $Y$ 后对 $X$ 不确定性的减少——这正是"信息"的数学定义。
+>
+> KL 散度 $D_{KL}(P\|Q) = \sum_x p(x)\log\frac{p(x)}{q(x)}$ 衡量用 $Q$ 替代 $P$ 的信息损失，**不对称** $D(P\|Q)\neq D(Q\|P)$。
+>
+> 信源编码定理：最优压缩的平均编码长度 $\ge H(X)$ bits/symbol，低于熵限则必然丢失信息。
+>
+> 信道编码定理：可靠通信的速率上限为信道容量 $C = \max_{p(x)} I(X;Y)$；$R<C$ 时存在编码使误码率趋零，$R>C$ 时可靠通信不可能。
+>
+> 率失真函数 $R(D) = \min_{p(\hat{x}|x):\,\mathbb{E}[d(x,\hat{x})]\le D} I(X;\hat{X})$ 给定失真 $D$ 下的最小信息率——有损压缩的极限。
+>
+> 详细数学依据见 `original-texts.md`。
+
+## GPU 友好性 / GPU-Friendliness（横切检查）
+
+信息论量用于**压缩/剪枝/量化/KV 压缩/路由设计**时，须过 `../../references/gpu-friendly-math.md` 八维门。核心判据：**局部、可融合的熵/KL 估计 = 友好；全局精确信息估计 = 不友好**。
+
+- **量化校准（per-block 熵/Hessian）**：局部、可批量 GEMM 化、低精度可行——友好（维度 1/2/5）。
+- **KV-Cache 压缩（信息瓶颈/块摘要）**：低秩块摘要压缩显存——友好（维度 4）；Plücker 式块摘要见 `../../references/books/algebraic-geometry-rising-sea.md`。
+- **互信息剪枝/特征选择**：用局部或低秩近似估计 $I(X;Y)$ 则可改造；若需全分布精确估计则不友好。
+- **信息增益路由**：全局精确 $I(X;Y)$ 需 $O(n^2)$ 显存、高精度、不可融合——反模式；改用 softmax/热带门控等连续可微近似（维度 3/6/8）。
+- **反模式**：在完整 token 分布上精确计算熵/互信息——显存爆炸、需 fp64、串行依赖，"美但不可算"。
+
+八维最低判定（正式术语）：**张量化**看熵/KL/MI 是否可按 token/block/batch 估计；**GEMM 可映射**看压缩、投影、校准是否落线性代数；**复杂度**避免全分布精确估计；**显存与 KV-Cache**量化 KV/激活/码本占用；**低精度稳定**检查 log/softmax/KL 动态范围；**并行与通信**看分块统计是否可跨设备归约；**稀疏结构**看码本/路由是否块结构化；**算子融合**看统计、mask、量化能否融合。
+
+> 配合 `../../references/books/matrix-analysis.md`（低秩压缩）、`../../references/books/abstract-algebra.md`（编码/有限域）。
+
+## 不适用场景 / When NOT to Use
+
+- **问题无概率结构**（如纯粹符号推理、逻辑演绎）——熵与信息增益需要概率分布，无概率则无信息论。
+- **纯确定性场景无不确定性**（如已知精确答案的数学问题）——熵为零时信息论退化为平凡结论。
+- **定性判断无需量化**（如美学评价、情感判断）——信息论量化的是概率意义上的不确定性，非语义歧义。
+
+## 何时使用 / When to Use
+
+- 需要度量不确定性的大小（熵 $H(X)$ 量化随机变量的"混乱程度"）。
+- 需要比较不同信息源的价值（互信息 $I(X;Y)$ 衡量哪个观察 $Y$ 最能减少关于 $X$ 的不确定性）。
+- 需要最优数据压缩（信源编码定理保证最优压缩极限为 $H(X)$ bits/symbol）。
+- 需要在噪声环境下可靠通信（信道编码定理保证 $R<C$ 时可靠传输可行）。
+- 需要特征选择或模型选择（互信息筛选特征，AIC/BIC/MDL 作为信息准则选择模型）。
+- 需要贝叶斯模型比较（KL 散度 $D(P\|Q)$ 衡量分布间信息距离，贝叶斯因子量化模型间证据比）。
+- **为压缩 / KV-Cache 压缩 / 量化 / 路由设计信息准则**，并评估其 GPU 可行性。
+
+## 方法流程 / Method
+
+### 第一步：识别信息源与不确定度 / Identify Source and Uncertainty
+- **随机变量 $X$** 是什么？——定义信息源，明确要研究的不确定性对象。
+- **概率分布 $p(x)$**——离散分布用概率表，连续分布用密度函数。
+- **计算 $H(X) = -\sum p(x)\log p(x)$**——量化当前不确定性水平。
+- **识别要减少的不确定性**——明确"知道什么之后不确定性会降低？"
+
+### 第二步：量化信息增益 / Quantify Information Gain
+- **计算条件熵 $H(X|Y)$**——观察 $Y$ 之后 $X$ 的剩余不确定性。
+- **计算互信息 $I(X;Y) = H(X) - H(X|Y)$**——$Y$ 对 $X$ 提供的信息量。
+- **识别最优观察**——哪个 $Y$ 使 $I(X;Y)$ 最大？该观察最值得获取。
+- **链式规则**——$H(X_1,\dots,X_n) = H(X_1) + H(X_2|X_1) + \dots + H(X_n|X_1,\dots,X_{n-1})$，逐变量拆解联合不确定性。
+
+### 第三步：选择编码策略 / Choose Coding Strategy
+- **信源编码（压缩）**：Huffman 编码（贪心最优前缀码，平均长度接近 $H(X)$）、算术编码（更接近熵限）、通用编码（LZ77/LZ78/LZW，无需已知分布）。
+- **信道编码（纠错）**：Hamming 码（最小距离 3，纠正 1 位错）、Reed-Solomon 码（突发纠错）、LDPC/Turbo 码（逼近 Shannon 极限）。
+- **编码选择原则**：压缩需求用信源编码→逼近 $H(X)$；噪声防护用信道编码→逼近 Shannon 极限 ($R\to C$)。
+
+### 第四步：评估信道容量 / Evaluate Channel Capacity
+- **计算信道容量 $C = \max_{p(x)} I(X;Y)$**——在所有输入分布上最大化互信息。
+- **比较传输速率 $R$ 与容量 $C$**：$R<C$ → 可靠通信可行；$R>C$ → 必定出错。
+- **噪声模型**：BSC（二元对称信道，翻转概率 $p$）、BEC（二元擦除信道，擦除概率 $\varepsilon$）、AWGN（加性白高斯噪声信道）。
+- **容量公式示例**：BSC 容量 $C = 1 - H(p)$；AWGN 容量 $C = \frac{1}{2}\log(1 + S/N)$。
+
+### 第五步：应用信息准则 / Apply Information Criteria
+- **AIC（赤池信息准则）**：$\text{AIC} = -2\ln L + 2k$——偏重拟合，适合预测目标。
+- **BIC（贝叶斯信息准则）**：$\text{BIC} = -2\ln L + k\ln n$——偏重简约，适合解释目标。
+- **KL 散度 $D_{KL}(P\|Q) = \sum p(x)\log\frac{p(x)}{q(x)}$**——衡量用 $Q$ 替代 $P$ 的信息损失，注意不对称性。
+- **MDL 原则（最小描述长度）**：选择使"数据描述长度 + 模型描述长度"最小的模型——信息论版本的奥卡姆剃刀。
+
+### 第六步：做出信息最优决策 / Make Information-Optimal Decision
+- **贝叶斯实验设计**：选择使期望信息增益 $\max\,\mathbb{E}[I(\theta;Y)]$ 最大的实验——优先获取最能减少不确定性的数据。
+- **最小化 KL 散度**：决策输出分布 $Q$ 应尽量接近目标分布 $P$，即 $\min D(P\|Q)$。
+- **最大熵原则**：在已知约束下选择使 $H(X)$ 最大的分布——最少假设，最保守推断。
+- **信息瓶颈**：$\min I(X;T) - \beta I(T;Y)$——在压缩 $X$ 为 $T$ 时保留关于 $Y$ 的最大相关信息。
+
+## 常见错误 / Common Errors
+
+| 错误 / Error | 批评 / Critique | 正确做法 / Correct Approach |
+|---|---|---|
+| 把信息等同于比特而非概率减少 | 熵 $H(X)$ 是概率分布的函数，比特只是量化单位 | 理解信息为不确定性的减少 $I(X;Y)=H(X)-H(X\|Y)$ |
+| 忽视信道容量极限 | $R>C$ 时无论何种编码都不能可靠通信 | 计算容量 $C=\max I(X;Y)$，确保 $R<C$ |
+| 混淆熵与方差 | 熵度量概率结构的"扩散度"，方差度量数值"展开度"，两者不等价 | 概率不确定性用熵，数值偏差用方差；连续熵可为负 |
+| 过度压缩低于熵限 | 最优压缩 $\ge H(X)$ bits/symbol，低于此限必然丢信息 | 接受熵限并据此设计编码 |
+| 忽视 KL 散度不对称性 | $D(P\|Q)\ne D(Q\|P)$，方向不同含义不同 | 明确方向：$D(P\|Q)$ 用于"用 $Q$ 编码 $P$ 的额外代价" |
+| 把相关性等同于信息 | 相关 $\rho$ 仅衡量线性关联；$I(X;Y)=0 \Leftrightarrow$ 独立，但 $\rho=0 \not\Rightarrow$ 独立 | 用互信息 $I(X;Y)$ 评估依赖关系 |
+| 把定性判断强行量化 | 信息论量化概率不确定性，非语义歧义或主观感受 | 区分概率不确定性与语义歧义，不对定性问题套熵公式 |
+| 全局精确熵/互信息不可算 | 在完整分布上精确估计 $I(X;Y)$ 需 $O(n^2)$ 显存与高精度，GPU 不可行 | 用局部/采样/低秩估计，过 GPU 八维门 |
+
+## 操作规程 / Operating Procedure
+
+当本 skill 被触发时，输出必须包含：
+
+1. **[信息源]:[描述]** $H(X)=$ [值]——定义随机变量 $X$，计算其熵，量化当前不确定性。
+2. **[信息增益]:[描述]** $I(X;Y)=$ [值]——计算互信息，识别最有价值的观察 $Y$。
+3. **[编码策略]:[选择]**——信源编码（压缩）或信道编码（纠错），说明逼近哪种极限。
+4. **[信道容量]:[描述]** $C=$ [值]——计算信道容量，比较传输速率 $R$ 与 $C$。
+5. **[信息准则]:[AIC/BIC/KL/MDL]**——说明选择的信息准则及理由。
+6. **[最优决策]:[说明]**——基于信息增益最大化或 KL 散度最小化的决策建议。
+7. **[GPU 可行性]**（若用于压缩/KV 压缩/量化/路由）——熵/KL/互信息估计是局部可融合还是全局精确？过八维门，标注友好/可改造/不友好 + 改造建议。
+
+**输出不得只给分析而无结论。**
+
+## 与其他 skill 的关系 / Relations to Other Skills
+
+- **概率与统计**：熵与信息增益补充概率推理——$H(X)$ 度量分布不确定性，$I(X;Y)$ 是贝叶斯更新的信息论表达。
+- **优化思想**：信道容量最大化是优化问题——$C=\max_{p(x)} I(X;Y)$ 是输入分布上的最优化。
+- **变换思想**：编码是信息空间的变换——信源编码变换为高效表示，信道编码变换为抗干扰表示。
+- **建模思想**：信息准则指导模型选择——AIC/BIC/MDL 从信息论角度量化拟合与复杂度权衡。
+- **算法思想**：压缩算法是计算实现——Huffman/LZ/LDPC 等算法是信息论极限的逼近实现。
+- **现代数学激活**：`../../references/books/matrix-analysis.md`（低秩压缩）、`../../references/books/abstract-algebra.md`（编码/有限域）、`../../references/books/algebraic-geometry-rising-sea.md`（Plücker KV 压缩）。

package/skills/information-theory/original-texts.en.md

@@ -0,0 +1,127 @@
+# Mathematical Sources and Classic Texts
+
+## Shannon's Information Theory (1948)
+
+> H(X) = -Sum p(x) log p(x) -- Entropy, the average uncertainty of an information source
+> I(X;Y) = H(X) - H(X|Y) -- Mutual information, the information contribution of observing Y about X
+> C = max_{p(x)} I(X;Y) -- Channel capacity, the upper bound on the rate of reliable communication
+>
+> Source coding theorem: optimal compression >= H(X) bits/symbol
+> Channel coding theorem: when R < C, there exists a coding scheme that drives the error rate to zero
+
+**Meaning**: Information can be precisely quantified; there exist insurmountable mathematical limits on compression and communication, not merely limits imposed by the current state of engineering. Entropy measures "average surprise" -- low-probability events carry more information when they occur. Mutual information measures "reduction in uncertainty" -- how much observing Y reduces our uncertainty about X.
+
+**Mathematical background**: Claude Shannon founded information theory in "A Mathematical Theory of Communication" (1948). Core contributions: (1) Defined entropy H(X) = -Sum p(x) log p(x), where the choice of logarithm base determines the unit (base 2 -> bits, base e -> nats); (2) The source coding theorem proves that the average length L of an optimal prefix code satisfies L >= H(X), with equality if and only if all probabilities are powers of 2; (3) The channel coding theorem proves that when R < C there exist coding sequences such that P(error) -> 0, while for R > C, P(error) > 0 is unavoidable. Shannon's proofs use random coding arguments -- constructing a random codebook and then showing that the error rate of a "typical" code approaches zero, thereby establishing the existence of at least one such code.
+
+## Huffman Coding (1952)
+
+> Greedy construction of optimal prefix codes: repeatedly merge the two symbols with the smallest probabilities, building the tree bottom-up.
+> Average code length L(Huffman) <= H(X) + 1; when probabilities are powers of 2, L = H(X).
+
+**Meaning**: An optimal prefix code (unambiguously decodable) can be constructed exactly via a greedy algorithm, with average length approaching the entropy bound.
+
+**Mathematical background**: David Huffman published this algorithm in 1952 as an MIT student, proving its optimality. Huffman coding is a variable-length prefix code -- more frequent symbols receive shorter codewords, and the prefix condition guarantees that no codeword is a prefix of another, enabling instantaneous decoding. When symbol probabilities are not all powers of 2, the average length of Huffman coding is strictly greater than H(X); arithmetic coding encodes an entire message as an interval and can approach the entropy bound more closely.
+
+## Hamming Codes (1950)
+
+> [7,4] Hamming code: 4 information bits + 3 parity bits = 7-bit codeword, minimum Hamming distance d_min = 3
+> Can correct 1-bit errors or detect 2-bit errors
+> General [2^r - 1, 2^r - 1 - r] Hamming code: r parity bits, minimum distance 3
+
+**Meaning**: The first systematic construction of error-correcting codes -- a small amount of redundancy can protect information from corruption by noise.
+
+**Mathematical background**: Richard Hamming, working at Bell Labs, designed the first systematic error-correcting code after hardware errors in early computers caused his programs to crash. Hamming distance d(x,y) = the number of positions at which two binary strings differ; a minimum distance d_min >= 2t+1 allows correction of t-bit errors. Hamming codes are perfect codes -- the sphere packing exactly fills the entire space. Subsequent developments: Reed-Solomon codes (1960) for burst error correction and CD/DVD; LDPC codes (Gallager 1962) and Turbo codes (1993) approach the Shannon limit.
+
+## Kolmogorov Complexity (1965)
+
+> K(x) = the length of the shortest program that outputs x (on a universal Turing machine)
+> Kolmogorov complexity is an algorithmic measure of information -- complementary to Shannon entropy
+>
+> Uncomputability: K(x) is not computable in general (Chaitin's incompleteness theorem)
+
+**Meaning**: The shortest program description length is the intrinsic information content of an object -- a more fundamental measure of information than Shannon entropy, independent of any probability distribution.
+
+**Mathematical background**: Andrey Kolmogorov proposed algorithmic information theory in 1965, defining K(x) as the length |p| of the shortest program p that outputs x on a universal Turing machine U: K_U(x) = min{|p| : U(p) = x}. Key properties: (1) Uncomputability -- no algorithm can compute K(x) in general; (2) Invariance theorem -- the values of K_U(x) on different universal Turing machines differ by at most a constant c (depending on the choice of machine); (3) Relation to entropy -- for i.i.d. sequences, E[K(x)] is approximately nH(X) (the algorithmic analogue of Shannon entropy); (4) Chaitin's incompleteness theorem -- a formal system cannot prove K(x) > c for most x. Kolmogorov complexity is the theoretical foundation of the MDL principle.
+
+## KL Divergence (Kullback-Leibler, 1951)
+
+> D(P||Q) = Sum p(x) log(p(x)/q(x)) = E_P[log(p(X)/q(X))]
+>
+> D(P||Q) >= 0, with equality if and only if P = Q
+> D(P||Q) is not equal to D(Q||P) -- it is asymmetric
+> D(P||Q) is not a distance (asymmetric, does not satisfy the triangle inequality)
+
+**Meaning**: KL divergence measures "the number of extra bits needed to encode data from distribution P using distribution Q" -- a one-directional measure of information loss.
+
+**Mathematical background**: Solomon Kullback and Richard Leibler proposed relative entropy in 1951, defined as D(P||Q) = Sum p(x) log(p(x)/q(x)). Core properties: (1) Non-negativity D(P||Q) >= 0 (proved via Gibbs' inequality); (2) Asymmetry D(P||Q) is not equal to D(Q||P) -- the interpretation of D(P||Q) is "the expected extra length of encoding data from P using Q," and the direction cannot be interchanged; (3) Relation to mutual information: I(X;Y) = D(p(x,y)||p(x)p(y)) -- mutual information is the KL divergence between the joint distribution and the product of the marginals; (4) Relation to entropy: H(P) + D(P||Q) = Sum p(x)(-log q(x)) -- cross-entropy equals entropy plus KL divergence. The cross-entropy loss function is central to deep learning: minimizing H(P) + D(P||Q) = minimizing Sum p(x)(-log q(x)).
+
+## Rate-Distortion Theory (Shannon, 1959)
+
+> R(D) = min_{p(z|x): E[d(x,z)] <= D} I(X;Z)
+>
+> The minimum rate R(D) required under an allowed distortion D
+> R(0) = H(X) (lossless compression), R(D_max) = 0 (maximum allowable distortion)
+
+**Meaning**: Lossy compression has a fundamental limit given by the rate-distortion function R(D) -- lower rates require accepting greater distortion.
+
+**Mathematical background**: Shannon extended the source coding theorem to lossy compression in 1959. The rate-distortion function R(D) is defined as the minimum of mutual information I(X;Z) subject to the constraint that expected distortion E[d(x,z)] <= D. The distortion measure d(x,z) can be Hamming distortion (binary), squared error (continuous), etc. R(D) is monotonically decreasing and convex -- greater tolerance for distortion yields lower required rate. The inverse function D(R) gives the minimum achievable distortion at a given rate. Practical applications: JPEG/MPEG compression, speech coding, and related technologies follow this theoretical framework.
+
+## Fisher Information (1925)
+
+> I(theta) = E[(d log f(X;theta)/d theta)^2] = -E[d^2 log f(X;theta)/d theta^2]
+>
+> Cramer-Rao lower bound: Var(theta_hat) >= 1/I(theta), for any unbiased estimator theta_hat
+> Fisher information measures the information content of data about parameter theta
+
+**Meaning**: Fisher information is the information-theoretic measure in statistical inference -- it connects information theory to the theoretical limits of parameter estimation.
+
+**Mathematical background**: R.A. Fisher proposed the concept of information content I(theta) in 1925, measuring the discriminative power of a single observation with respect to parameter theta. Cramer (1946) and Rao (1945) independently proved the Cramer-Rao lower bound Var(theta_hat) >= 1/(nI(theta)). Key connections: (1) Fisher information and KL divergence -- I(theta) = lim_{theta' -> theta} 2D(f(x;theta)||f(x;theta'))/(theta - theta')^2, the local second-order approximation of KL divergence; (2) Fisher information and Bayesian information -- the Jeffreys prior p(theta) proportional to |I(theta)|^(1/2) makes the volume uniform under the Fisher information metric in parameter space; (3) Large-sample properties of maximum likelihood estimation -- theta_hat_MLE is asymptotically normal N(theta, 1/(nI(theta))), achieving the Cramer-Rao lower bound. Fisher information is the bridge between information theory and statistical inference.
+
+## MDL Principle (Rissanen, 1978)
+
+> MDL(M, D) = L(D|M) + L(M)
+>
+> Choose the model that minimizes "description length of data given the model + description length of the model itself"
+> L(D|M) = -log P(D|M) (negative log-likelihood of the data)
+> L(M) = model encoding length (complexity penalty)
+
+**Meaning**: MDL is the information-theoretic version of Occam's razor -- a good model is both accurate (short data description) and parsimonious (short model description).
+
+**Mathematical background**: Jorma Rissanen proposed the Minimum Description Length principle in 1978, reducing model selection to a coding problem. Relationships between MDL and other criteria: (1) MDL and BIC -- two-part MDL (L(D|M) + L(M)) is asymptotically equivalent to BIC (-2 ln L + k ln(n)) for large samples; (2) MDL and Kolmogorov complexity -- ideal MDL uses Kolmogorov complexity K(M) to measure model complexity, but K(M) is uncomputable; practical MDL approximates it with parameter encoding length. Normalized MDL (1996) uses mixture coding to handle parameters, avoiding the arbitrary-precision problem. The central insight of MDL: model selection is fundamentally data compression -- the best model is the one that compresses the data best.
+
+## Mutual Information and Channel Capacity
+
+> I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) = H(X) + H(Y) - H(X,Y)
+>
+> Symmetry of mutual information: I(X;Y) = I(Y;X)
+> Non-negativity of mutual information: I(X;Y) >= 0, with equality if and only if X and Y are independent
+>
+> Channel capacity C = max_{p(x)} I(X;Y)
+> Typical set A_epsilon^n: {x^n : |-log p(x^n)/n - H(X)| < epsilon}
+
+**Meaning**: Mutual information is a precise measure of the dependence between two random variables -- it is zero if and only if they are independent, and positive if statistical association exists. Channel capacity is the maximum of mutual information over all input distributions, defining the theoretical limit of reliable communication.
+
+**Mathematical background**: Mutual information I(X;Y) = Sum p(x,y) log(p(x,y)/(p(x)p(y))) = D(p(x,y)||p(x)p(y)) -- the KL divergence between the joint distribution and the product of the marginals. The typical set is the key tool in Shannon's proofs of coding theorems: for an i.i.d. source X^n, typical sequences x^n satisfy |-1/n log p(x^n) - H(X)| < epsilon; the probability of the typical set approaches 1 (as n -> infinity), yet the size of the typical set is approximately 2^{nH(X)}, far smaller than the entire space 2^n (when H(X) < 1). This explains why compression to nH(X) bits suffices to cover nearly all valid sequences.
+
+## Joint Entropy, Conditional Entropy, and Chain Rules
+
+> H(X,Y) = H(X) + H(Y|X) -- Chain rule for joint entropy
+> H(Y|X) = Sum p(x) H(Y|X=x) -- Conditional entropy
+> I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) -- Mutual information
+> I(X;Y|Z) = H(X|Z) - H(X|Y,Z) -- Conditional mutual information
+> Chain expansion of I(X1;X2;...;Xn): I(X;Y,Z) = I(X;Y) + I(X;Z|Y)
+
+**Meaning**: The chain rule for entropy decomposes joint uncertainty into the contributions of individual variables -- enabling an understanding of the information structure of multivariate systems.
+
+**Mathematical background**: Joint entropy H(X,Y) = -Sum p(x,y) log p(x,y) measures the total uncertainty of a pair of random variables. The chain rule H(X,Y) = H(X) + H(Y|X) states: once X is known, the additional uncertainty of Y is H(Y|X). Generalized to n variables: H(X1,...,Xn) = Sum_i H(Xi|X1,...,Xi-1). Conditional mutual information I(X;Y|Z) measures "the additional information that Y provides about X, given that Z is already known" -- a key tool in information bottleneck methods and causal inference. The chain expansion of multivariate mutual information: I(X;Y,Z) = I(X;Y) + I(X;Z|Y), showing that the joint information of Y and Z about X equals the direct information from Y plus the conditional information from Z.
+
+## Everyday Insights from Information Theory and Coding Theory
+
+> "The core of information theory is not about computing precise bit counts, but about cultivating the habit of mind that uncertainty is quantifiable, information has value, and fundamental limits cannot be transcended."
+
+- Do not ignore uncertainty -- entropy H(X) quantifies it
+- Information has direction and magnitude -- mutual information I(X;Y) measures it
+- Compression has limits -- the source coding theorem defines them
+- Communication has limits -- the channel coding theorem defines them
+- Model selection has information-theoretic criteria -- AIC/BIC/MDL guide it
+- There is an information-theoretic distance between probability distributions -- KL divergence measures it