math-skill 3.0.1 → 3.1.1

package/design-patterns/compression/low-rank-kv-cache.md

@@ -2,16 +2,16 @@
 > **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
-当 LLM 推理的 KV-Cache 显存占用成为瓶颈时使用：长上下文推理（$L > 8K$）、多轮对话累积、边缘部署、投机解码/束搜索。核心诉求：**以可控信息损失将 KV-Cache 从 $O(Ld)$ 压缩到 $O(kd)$，$k \ll L$**。
+当 LLM 推理的 KV-Cache 显存占用成为瓶颈时使用：长上下文推理（$L > 8K$）、多轮对话累积、边缘部署、投机解码/束搜索。核心诉求：**以可控信息损失将 KV-Cache 从 $O(Ld)$ 压缩到低秩因子存储 $O(Lk + kd)$，$k \ll d$**。
 
 ## 数学思想来源
-- 透镜：lenses/spectral.md（谱分量识别与截断）、lenses/variational.md（压缩率与重构误差的帕累托取舍）、lenses/duality.md（核范数-谱范数对偶）
-- 知识：knowledge-base/matrix-analysis/low-rank-approximation.md（Eckart-Young 最优逼近、随机化 SVD）、knowledge-base/matrix-analysis/projection.md（正交投影到主子空间）、knowledge-base/matrix-analysis/matrix-perturbation.md（Weyl 扰动界）
+- 透镜：../../lenses/spectral.md（谱分量识别与截断）、../../lenses/variational.md（压缩率与重构误差的帕累托取舍）、../../lenses/duality.md（核范数-谱范数对偶）
+- 知识：../../knowledge-base/matrix-analysis/low-rank-approximation.md（Eckart-Young 最优逼近、随机化 SVD）、../../knowledge-base/matrix-analysis/projection.md（正交投影到主子空间）、../../knowledge-base/matrix-analysis/matrix-perturbation.md（Weyl 扰动界）
 
 ## 需要的数学知识
 - **Eckart-Young-Mirsky 定理**：$A_k = U_k \Sigma_k V_k^H$，$\|A - A_k\|_F = \sqrt{\sum_{i>k} \sigma_i^2}$，截断 SVD = 最优秩-$k$ 逼近
 - **随机化 SVD**：$Y = A\Omega$（$\Omega$ 随机高斯），$Y = QR$，$B = Q^H A$，对 $B$ 做 SVD；复杂度 $O(Ldk)$，核心全是 matmul
-- **Weyl 扰动界**：$\|A - A_k\|_2 = \sigma_{k+1}$，压缩后 attention score 最大偏差 $\leq \sigma_{k+1}$
+- **Eckart-Young 谱范数误差**：$\|A - A_k\|_2 = \sigma_{k+1}$（仅界定矩阵本身的最优秩-$k$ 压缩误差；Weyl 只用于分析扰动后的奇异值变化）。将其推广到 attention 输出误差需要额外条件：$\|\text{Attn}(Q,K,V) - \text{Attn}(Q,K_k,V_k)\|$ 的完整界需要 (a) $\|Q\|$ 有界、(b) softmax 的 Lipschitz 常数（依赖温度和 score 范围）、(c) K 和 V **两者**的压缩误差。粗略地：误差 $\lesssim C \cdot (\|Q\| \cdot \|K - K_k\| \cdot \|V\| + \|Q\| \cdot \|K_k\| \cdot \|V - V_k\|) / \tau$，其中 $C$ 取决于 softmax Lipschitz 常数，$\tau$ 为温度参数。直接用 $\sigma_{k+1}$ 界 attention score 偏差**仅在固定 Q 且忽略 V 压缩误差的简化假设下成立**。
 - **有效秩**：$r_{\text{eff}}(K) = \|K\|_F^2 / \|K\|_2^2$，指导 $k$ 的自适应选取
 
 ## AI 模块形式
@@ -21,11 +21,31 @@
 
 方法1 - 离线周期压缩（最实用）：
   Omega = randn(d, k+p)                    // 随机投影，p=5 过采样
-  Q_k = qr(K @ Omega)[0]                   // L×(k+p) 正交基底（GEMM + QR）
-  B_k = Q_k^T @ K                          // (k+p)×d 小矩阵 GEMM
+  Q_base = qr(K @ Omega)[0]                // L×(k+p) 过采样正交基底（GEMM + QR）
+  B_k = Q_base^T @ K                       // (k+p)×d 小矩阵 GEMM
   U_r, S_r, Vt_r = svd(B_k)               // 小矩阵 SVD
-  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d 压缩 Key
-  // Attention: softmax(Q @ K_comp^T / √d) @ V_comp，序列维度 L → k
+  Q_final = Q_base @ U_r[:, :k]            // L×k 最终左因子
+  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d 压缩 Key（低秩因子）
+  // V_comp 定义：对 V 做类似的截断 SVD，V_comp = Σ_k^{(V)} · Vt_k^{(V)}（k×d 压缩 Value）
+  // 或者，若 K 和 V 共享左因子 Q_final，则 V_comp = Q_final^T @ V（投影到同一低秩子空间）
+  //
+  // ⚠ 关键区分——低秩因子不能直接替代原始序列参与 softmax attention：
+  //   K ≈ Q_final @ K_comp（L×d 重构），softmax 是非线性操作，
+  //   softmax(Q @ K_comp^T / √d) @ V_comp ≠ softmax(Q @ K^T / √d) @ V
+  //   因此 k 个"压缩 token"的解释仅对线性注意力成立，对 softmax 注意力不成立。
+  //
+  // 模式 A - 标准 softmax attention（省显存；可省内积维度但不省序列长度）：
+  //   不必物化 K_recon。若 K ≈ Q_final @ K_comp，则
+  //   logits = (Q @ K_comp^T) @ Q_final^T / √d // Q 为 m×d 时复杂度 O(mdk + mLk)
+  //   attn = softmax(logits)                   // softmax 仍在 L 个位置上归一化
+  //   若 V ≈ Q_final @ V_comp，则 output = (attn @ Q_final) @ V_comp
+  //   // 优势：存储从 O(Ld) 降至 O(Lk + kd)，QK/AV 的内维从 d 降到 k
+  //   // 限制：注意力矩阵仍是 m×L，不能把 k 个因子当作 k 个 token
+  //
+  // 模式 B - 线性注意力（核特征映射 φ；需压缩 φ(K), V 的可加统计量）：
+  //   用核特征映射 φ 替代 softmax，Attn = φ(Q) @ (φ(K)^T @ V) / (φ(Q) @ φ(K)^T @ 1)
+  //   只有当压缩对象是 φ(K)^T V、φ(K)^T 1 等可加统计量，或直接在 φ(K) 空间做低秩/聚合时，
+  //   历史状态才能从 L 个 token 变成 k 个统计因子。直接对 K_comp 做 φ 一般不等价。
 
 方法2 - 流式增量压缩（低延迟）：
   维护基底 (U_basis ∈ R^{k×d})，新 token 到来：
@@ -37,7 +57,7 @@
 ```
 
 ## 可实现结构
-- **周期压缩层**：每 M=64 步触发随机化 SVD，$L \times d \to k \times d$
+- **周期压缩层**：每 M=64 步触发随机化 SVD，$L \times d \to (L \times k) + (k \times d)$
 - **双缓冲**：压缩基底 + 近期 $w$ 个原始 token，兼顾精度与压缩率
 - **共享基底**：多头共享 Key 列空间基底，各头只存系数
 - **量化基底**：压缩后进一步 INT8/FP8 量化，双重压缩
@@ -45,13 +65,19 @@
 ## GPU 可行性
 - 张量化/GEMM：随机化 SVD = 3 次 GEMM + 1 次小 SVD，完美映射 Tensor Core
 - 复杂度：$O(Ldk)$ 远优于 $O(Ld^2)$ 完整 SVD；$k \sim 256$ 时开销可忽略
-- 显存：KV-Cache 从 $O(Ld)$ 到 $O(kd)$，$k/L \sim 1/16$ 可达 16x 压缩
+- 显存：Key-Cache 以低秩因子形式存储（$Q_{\text{final}} \in \mathbb{R}^{L \times k}$ + $B_k \in \mathbb{R}^{k \times d}$），总参数 $Lk + kd$，压缩比 $Ld/(Lk+kd) \approx d/k$（当 $L \gg k$）。注意 $Q_{\text{final}}$ 仍有 $L$ 维，序列长度未缩短；softmax attention 可用因子化 GEMM 计算长度为 $L$ 的 logits，不必物化完整 $L \times d$ 矩阵。V-Cache 需独立做类似压缩。端到端压缩比取决于 K/V 两者的存储格式与秩选取
 - 低精度：SVD 建议 fp32（小矩阵可接受）；压缩后 KV 可回 bf16 存储
 - 并行：多层/多头压缩完全独立；增量更新 $O(kd)$ 极低延迟
-- 算子融合：$K\Omega$ + QR 可部分融合；attention 的 QK^T 维度已缩小
+- 算子融合：$K\Omega$ + QR 可部分融合；softmax 路径可把 QK/AV 的内维从 $d$ 降到 $k$，但 softmax 归一化长度仍为 $L$；线性注意力只有在压缩可加统计量时才真正把历史状态降到 $k$
+
+**量化评估示例**（标准 transformer, d=128, n=2048, rank k=64）：
+- D3: SVD 计算 O(n·d·k) ≈ 2048·128·64 ≈ 16.8M FLOPs（一次性）；softmax 推理每个 query 的因子化 QK/AV 约 O(dk + n·k)，仍需长度 n 的 softmax；线性注意力若压缩统计量才可消去 n
+- D4: KV-Cache 从 O(n·d) = 2048·128·2B ≈ 512KB；因子格式 O(n·k + k·d) ≈ 2048·64·2B + 64·128·2B ≈ 278KB（压缩比 ~1.8x）。最终左因子+系数格式（$Q_{\text{final}} \in \mathbb{R}^{n \times k}$ 正交基 + $B_k \in \mathbb{R}^{k \times d}$）总参数 $nk + kd$，压缩比 $nd/(nk+kd) = d/k \cdot 1/(1 + d/n) \approx d/k = 2x$，V-Cache 需独立压缩
+- D5: 截断 SVD 在 bf16 下 σ_k 附近奇异值误差放大 ~κ(A)，需注意
+- D8: SVD → matmul 可融合；在线更新用 incremental SVD 避免全量重算
 
 ## 论文表述方式
-"基于 Eckart-Young-Mirsky 定理，采用随机化 SVD 将 KV-Cache 投影到最优秩-$k$ 子空间，以 $O(Ldk)$ 复杂度实现 $L/k$ 倍显存压缩，Weyl 扰动界保证 attention score 偏差不超过 $\sigma_{k+1}$。"
+"基于 Eckart-Young-Mirsky 定理，采用随机化 SVD 将 KV-Cache 投影到秩-$k$ 子空间，以 $O(Ldk)$ 复杂度将存储从 $O(Ld)$ 压缩至 $O(Lk + kd)$（基底+系数格式）。对标准 softmax attention，低秩因子不能解释为 $k$ 个压缩 token；softmax 仍在长度 $L$ 上归一化，但可用因子化 GEMM 计算 logits 与 value 聚合，避免物化完整 $L \times d$ 重构并将 QK/AV 的内维从 $d$ 降到 $k$。仅在线性注意力中，且压缩的是 $\phi(K)^T V$、$\phi(K)^T\mathbf{1}$ 等可加统计量时，历史状态才可真正从 $L$ 降至 $k$ 个统计因子。Eckart-Young 的谱范数误差给出 K/V 矩阵本身的最优秩-$k$ 压缩误差 $\sigma_{k+1}$；attention 输出的端到端误差界进一步依赖 query 范数、softmax Lipschitz 常数和温度参数。实际显存压缩比取决于存储格式及 V-Cache 的独立压缩策略。"
 
 ## 风险
 - **秩选取不当**：$k$ 过小导致 $\sigma_{k+1}$ 不可忽略，长距离 recall 下降；需监控奇异值衰减曲线

package/design-patterns/compression/spectral-token-pruning.en.md

@@ -5,40 +5,42 @@
 Use when pruning must be based on the structural importance of tokens (rather than raw attention scores alone): KV-Cache eviction, long-document summarization, inference acceleration (reducing $O(L^2)$), multimodal vision token compression. Core objective: **quantify the structural importance of each token via spectral methods, achieving pruning with minimal information loss**.
 
 ## Mathematical Foundations
-- Lenses: lenses/spectral.md (identifying dominant spectral components, discarding redundant ones), lenses/algorithmic.md (complexity classification and approximation algorithms), lenses/perturbation.md (pruning = sparse perturbation, Weyl bound for spectral drift estimation)
-- Knowledge: knowledge-base/matrix-analysis/spectral-decomposition.md (spectral radius, eigenvector centrality), knowledge-base/matrix-analysis/matrix-perturbation.md (Geršgorin discs, perturbation bounds), knowledge-base/matrix-analysis/positive-semidefinite.md (Gram matrix PSD structure)
+- Lenses: ../../lenses/spectral.en.md (identifying dominant spectral components, discarding redundant ones), ../../lenses/algorithmic.en.md (complexity classification and approximation algorithms), ../../lenses/perturbation.en.md (pruning = sparse perturbation, Cauchy interlacing theorem / Bauer-Fike pseudospectral analysis for spectral drift bounds)
+- Knowledge: ../../knowledge-base/matrix-analysis/spectral-decomposition.en.md (spectral radius, eigenvector centrality), ../../knowledge-base/matrix-analysis/matrix-perturbation.en.md (Geršgorin discs, perturbation bounds), ../../knowledge-base/matrix-analysis/positive-semidefinite.en.md (Gram matrix PSD structure)
 
 ## Required Mathematical Background
-- **Eigenvector Centrality**: the principal eigenvector of the attention matrix $A$ satisfies $Ax = \lambda_1 x$; component $x_i$ quantifies the global influence of token $i$ (Perron--Frobenius guarantees non-negativity)
+- **Eigenvector Centrality**: For an unmasked, positive, irreducible row-stochastic attention matrix $A$, the right principal eigenvector $Ax = \lambda_1 x$ degenerates to the all-ones vector (since $A \mathbf{1} = \mathbf{1}$), making it useless for distinguishing token importance; the left principal eigenvector $x^T A = \lambda_1 x^T$ (equivalently $A^T x = \lambda_1 x$) can be used as a stationary-distribution / PageRank-like score. For causal or heavily masked attention, the chain is often reducible and the stationary distribution may collapse toward early tokens; use a K/V similarity graph, a symmetrized graph, or teleportation $A_\alpha=\alpha A+(1-\alpha)\mathbf{1}\pi^T$ before interpreting PageRank.
 - **Spectral Gap**: $\Delta = \lambda_1 - \lambda_2$ governs the rate of information diffusion; large $\Delta \Rightarrow$ a few tokens dominate $\Rightarrow$ safe to prune
 - **Fiedler Vector**: the second-smallest eigenvector of the Laplacian $L_{\text{sym}}$ yields the optimal bipartition; small magnitude = partition boundary = important
-- **Weyl Perturbation Bound**: pruning = removing rows/columns = rank-1 perturbation $E$, $|\lambda_i(A) - \lambda_i(A_{\text{pruned}})| \leq \|E\|_2$
+- **Spectral Perturbation Analysis**: Pruning changes the matrix dimension and cannot directly apply the Weyl theorem. For a principal submatrix of a fixed Hermitian matrix (e.g., a symmetrized matrix $S=(A+A^T)/2$ without re-normalization), Cauchy interlacing bounds eigenvalue interlacing. If the Laplacian / attention graph is re-normalized after pruning, the matrix itself has changed, so this bound no longer applies directly. For non-symmetric row-stochastic matrices, spectral radius perturbation can be bounded via Bauer--Fike or pseudospectral analysis, though the bounds are less tight than in the Hermitian case.
 
 ## AI Module Specification
 ```
 Module: SpectralTokenPruner
 Input: K ∈ R^{L×d}    Parameters: retention ratio ρ ∈ (0,1]
 
-Method 1 - Spectral centrality pruning (power iteration):
-  A = softmax(K @ K^T / √d)                  // L×L similarity graph
+Method 1 - Spectral centrality pruning (power iteration, left eigenvector):
+  A = softmax(K @ K^T / √d)                  // L×L row-stochastic similarity graph; handle causal/masked cases separately
+  // ⚠ A is row-stochastic: right principal eigenvector = all-ones (degenerate); must use left eigenvector
+  // Left principal eigenvector = right eigenvector of A^T; PageRank interpretation is stable only for irreducible/teleported chains
   v = ones(L) / √L
-  for t in range(5): v = A @ v; v = v / ‖v‖  // power iteration O(L²·T)
-  indices = topk(v, ceil(ρ * L))              // retain tokens with highest centrality
+  for t in range(5): v = A^T @ v; v = v / ‖v‖  // power iteration on A^T (not A), O(L²·T)
+  indices = topk(v, ceil(ρ * L))              // retain tokens with highest left-eigenvector centrality
 
 Method 2 - Geršgorin cheap pruning (zero iterations):
-  A = softmax(K @ K^T / √d)
-  gersh_score = |diag(A)| + sum(|A|, dim=1)    // disc upper bound, O(L²) elementwise
-  indices = topk(gersh_score, ceil(ρ * L))
+  S = K @ K^T / √d                         // L×L raw similarity matrix (pre-softmax, so row sums vary)
+  gersh_score = sum(|S|, dim=1) - |diag(S)|  // Geršgorin disc radius R_i = sum_{j!=i}|S_{ij}|, measures connectivity
+  indices = topk(gersh_score, ceil(ρ * L))    // O(L²) elementwise, no power iteration needed
 
 Method 3 - Differentiable spectral pruning (end-to-end):
-  v = power_iteration(A, T=5)
+  v = power_iteration(A^T, T=5)               // left principal eigenvector (power iteration on A^T)
   gate = sigmoid(v @ W_gate / τ)               // soft gating, τ annealing
   K_gated = gate * K                            // per-token scaling
   L_sparse = ‖gate‖_1 / L                       // sparsity regularization
 ```
 
 ## Implementable Architectures
-- **Power iteration centrality**: 5-step matvec to estimate the principal eigenvector, $O(L^2 \cdot 5)$, suitable for moderate-length sequences
+- **Power iteration centrality**: 5-step matvec to estimate the principal eigenvector of $A^T$ (i.e., the left principal eigenvector / stationary distribution of $A$), $O(L^2 \cdot 5)$, suitable for moderate-length sequences
 - **Sampling approximation**: for $L > 4096$, sample $m$ anchor points and construct an $m \times m$ submatrix for spectral analysis
 - **Multi-head fusion**: average the attention graphs across different heads before performing spectral analysis
 - **Progressive pruning**: incrementally increase the pruning ratio across layers (light pruning in shallow layers, heavy pruning in deep layers)
@@ -52,7 +54,7 @@ Method 3 - Differentiable spectral pruning (end-to-end):
 - Operator fusion: $KK^T$ + row-sum + topk can be fused into a single kernel
 
 ## Paper-Worthy Formulation
-"We cast token pruning as spectral sparsification of a directed graph: leveraging the Perron--Frobenius principal eigenvector of the attention matrix to quantify global centrality, with the Weyl perturbation bound guaranteeing that post-pruning spectral drift does not exceed the $\ell_2$ norm of the removed tokens, while Geršgorin discs provide an $O(L^2)$ inexpensive alternative."
+"We cast token pruning as spectral sparsification of a directed graph: on unmasked irreducible row-stochastic graphs, left-eigenvector / PageRank-like centrality can quantify global token importance; for causal or heavily masked attention, use a K/V similarity graph, a symmetrized graph, or teleported PageRank to avoid stationary-mass collapse toward early tokens. Cauchy interlacing applies to principal submatrices of fixed Hermitian matrices; after re-normalization or for non-symmetric graphs, use weaker but applicable diagnostics such as pseudospectral analysis, Bauer--Fike, or Geršgorin discs."
 
 ## Risks
 - **$L \times L$ matrix memory bottleneck**: for long sequences the similarity matrix itself may exceed available memory, necessitating sampling or chunking

package/design-patterns/compression/spectral-token-pruning.md

@@ -5,40 +5,42 @@
 当需要基于 token 的结构性重要性（而非单纯 attention score）进行剪枝时使用：KV-Cache 驱逐、长文档摘要、推理加速（降低 $O(L^2)$）、多模态视觉 token 压缩。核心诉求：**用谱方法量化每个 token 的结构性重要性，实现信息损失最小的剪枝**。
 
 ## 数学思想来源
-- 透镜：lenses/spectral.md（识别主导谱分量、丢弃冗余分量）、lenses/algorithmic.md（复杂度分类与近似算法）、lenses/perturbation.md（剪枝 = 稀疏扰动，Weyl 界估谱漂移）
-- 知识：knowledge-base/matrix-analysis/spectral-decomposition.md（谱半径、特征向量中心性）、knowledge-base/matrix-analysis/matrix-perturbation.md（Geršgorin 圆盘、扰动界）、knowledge-base/matrix-analysis/positive-semidefinite.md（Gram 矩阵 PSD 结构）
+- 透镜：../../lenses/spectral.md（识别主导谱分量、丢弃冗余分量）、../../lenses/algorithmic.md（复杂度分类与近似算法）、../../lenses/perturbation.md（剪枝 = 稀疏扰动，Cauchy 交错定理 / Bauer-Fike 伪谱分析界定谱漂移）
+- 知识：../../knowledge-base/matrix-analysis/spectral-decomposition.md（谱半径、特征向量中心性）、../../knowledge-base/matrix-analysis/matrix-perturbation.md（Geršgorin 圆盘、扰动界）、../../knowledge-base/matrix-analysis/positive-semidefinite.md（Gram 矩阵 PSD 结构）
 
 ## 需要的数学知识
-- **特征向量中心性**：attention 矩阵 $A$ 的主特征向量 $Ax = \lambda_1 x$，分量 $x_i$ 量化 token $i$ 的全局影响力（Perron-Frobenius 保证非负）
+- **特征向量中心性**：对未 mask、正且不可约的行随机（行 softmax）attention 矩阵 $A$，右主特征向量 $Ax = \lambda_1 x$ 退化为全 1 向量（因 $A \mathbf{1} = \mathbf{1}$），无法区分 token 重要性；可使用左主特征向量 $x^T A = \lambda_1 x^T$（等价于 $A^T x = \lambda_1 x$）作为平稳分布/PageRank 式重要性。对 causal 或强 mask attention，链通常非不可约，平稳分布可能偏向早期 token；应改用 K/V 相似度图、对称化图，或加入 teleportation $A_\alpha=\alpha A+(1-\alpha)\mathbf{1}\pi^T$ 后再解释 PageRank。
 - **谱间隙**：$\Delta = \lambda_1 - \lambda_2$ 决定信息扩散速度，$\Delta$ 大 $\Rightarrow$ 少数 token 主导 $\Rightarrow$ 安全剪枝
 - **Fiedler 向量**：Laplacian $L_{\text{sym}}$ 的第二小特征向量给出最优二分割，幅值小 = 分割边界 = 重要
-- **Weyl 扰动界**：剪枝 = 删除行/列 = 秩-1 扰动 $E$，$|\lambda_i(A) - \lambda_i(A_{\text{pruned}})| \leq \|E\|_2$
+- **谱扰动分析**：剪枝改变矩阵维度，不能直接套用 Weyl 定理。对固定 Hermitian 矩阵的主子矩阵（如不重新归一化的对称化矩阵 $S=(A+A^T)/2$），可用 Cauchy 交错定理界定特征值交错；若剪枝后重新归一化 Laplacian/attention graph，矩阵本身已改变，不能直接套该界。对非对称行随机矩阵，谱半径扰动可用 Bauer-Fike 或伪谱分析，但界不如 Hermitian 情形紧凑。
 
 ## AI 模块形式
 ```
 模块：SpectralTokenPruner
 输入：K ∈ R^{L×d}    参数：保留比例 ρ ∈ (0,1]
 
-方法1 - 谱中心性剪枝（幂迭代）：
-  A = softmax(K @ K^T / √d)                  // L×L 相似度图
+方法1 - 谱中心性剪枝（幂迭代，左特征向量）：
+  A = softmax(K @ K^T / √d)                  // L×L 行随机相似度图；causal/mask 场景需另行处理
+  // ⚠ A 为行随机矩阵，右主特征向量 = 全 1 向量（退化），必须用左主特征向量
+  // 左主特征向量 = A^T 的右主特征向量；仅在链不可约/加 teleportation 后可稳定解释为 PageRank
   v = ones(L) / √L
-  for t in range(5): v = A @ v; v = v / ‖v‖  // 幂迭代 O(L²·T)
-  indices = topk(v, ceil(ρ * L))              // 保留中心性最高的 token
+  for t in range(5): v = A^T @ v; v = v / ‖v‖  // 幂迭代 A^T（非 A），O(L²·T)
+  indices = topk(v, ceil(ρ * L))              // 保留左特征向量中心性最高的 token
 
 方法2 - Geršgorin 廉价剪枝（零迭代）：
-  A = softmax(K @ K^T / √d)
-  gersh_score = |diag(A)| + sum(|A|, dim=1)    // 圆盘上界，O(L²) elementwise
-  indices = topk(gersh_score, ceil(ρ * L))
+  S = K @ K^T / √d                         // L×L 原始相似度矩阵（非 softmax，行使行和有区分度）
+  gersh_score = sum(|S|, dim=1) - |diag(S)|  // Geršgorin 圆盘半径 R_i = Σ_{j≠i}|S_{ij}|，衡量连通度
+  indices = topk(gersh_score, ceil(ρ * L))    // O(L²) elementwise，无需幂迭代
 
 方法3 - 可微谱剪枝（端到端）：
-  v = power_iteration(A, T=5)
+  v = power_iteration(A^T, T=5)               // 左主特征向量（A^T 的幂迭代）
   gate = sigmoid(v @ W_gate / τ)               // 软门控，τ 退火
   K_gated = gate * K                            // 逐 token 缩放
   L_sparse = ‖gate‖_1 / L                       // 稀疏正则
 ```
 
 ## 可实现结构
-- **幂迭代中心性**：5 步 matvec 估计主特征向量，$O(L^2 \cdot 5)$，适合中等序列
+- **幂迭代中心性**：5 步 matvec 估计 $A^T$ 的主特征向量（即 $A$ 的左主特征向量 / 平稳分布），$O(L^2 \cdot 5)$，适合中等序列
 - **采样近似**：$L > 4096$ 时采样 $m$ 个锚点，构造 $m \times m$ 子矩阵做谱分析
 - **多头融合**：不同头的 attention 图取平均后再做谱分析
 - **渐进式剪枝**：逐层递增剪枝比例（浅层少剪、深层多剪）
@@ -52,7 +54,7 @@
 - 算子融合：$KK^T$ + row-sum + topk 可融合为单一 kernel
 
 ## 论文表述方式
-"将 token 剪枝建模为有向图的谱稀疏化：利用 attention 矩阵的 Perron-Frobenius 主特征向量量化全局中心性，Weyl 扰动界保证剪枝后谱漂移不超过被移除 token 的 $\ell_2$ 范数，Geršgorin 圆盘提供 $O(L^2)$ 廉价替代。"
+"将 token 剪枝建模为有向图的谱稀疏化：在未 mask 且不可约的行随机图上，利用左主特征向量/PageRank 式中心性量化全局重要性；对 causal 或强 mask attention，则改用 K/V 相似度图、对称化图或 teleportation 后的 PageRank，避免平稳分布退化到早期 token。对固定 Hermitian 子矩阵可用 Cauchy 交错分析谱漂移；重新归一化或非对称图则需改用伪谱、Bauer-Fike 或 Geršgorin 圆盘等更弱但适用的扰动诊断。"
 
 ## 风险
 - **$L \times L$ 矩阵显存瓶颈**：长序列下相似度矩阵本身可能超出显存，必须采样或分块

package/design-patterns/compression/topology-preserving-compression.en.md

@@ -2,11 +2,11 @@
 > **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Target Problem
-Use when compressing representations while homologically preserving the intrinsic topological structure of the data (connected components, loops, cavities): latent-space compression (a toroidal manifold must not collapse into a line segment), knowledge distillation (student--teacher homological equivalence), 3D mesh simplification (genus invariance), KV-Cache semantic preservation (cluster structure must not collapse). Core objective: **compress dimensions or counts while ensuring that changes in the persistence diagram of persistent homology remain controlled**.
+Use when compressing representations while homologically preserving the intrinsic topological structure of the data (connected components, loops, cavities): latent-space compression (a toroidal manifold must not collapse into a line segment), knowledge distillation (student--teacher homological equivalence), 3D mesh simplification (genus invariance), KV-Cache semantic preservation (cluster structure must not collapse). Core objective: **compress dimensions or counts while constraining and measuring changes in the persistence diagram of persistent homology**.
 
 ## Mathematical Foundations
-- Lenses: lenses/topological.md (topological invariants -- connected components and hole counts are invariant under continuous deformation), lenses/spectral.md (Gauss--Bonnet linking curvature and Euler characteristic), lenses/variational.md (constrained variational of compression ratio vs. topological fidelity)
-- Knowledge: knowledge-base/topology/persistent-homology.md (persistent homology, Vietoris--Rips filtration, bottleneck distance), knowledge-base/topology/euler-characteristic.md (Euler characteristic for rapid topological diagnostics), knowledge-base/matrix-analysis/matrix-perturbation.md (Davis--Kahan subspace perturbation bound)
+- Lenses: ../../lenses/topological.en.md (topological invariants -- connected components and hole counts are invariant under continuous deformation), ../../lenses/spectral.en.md (Gauss--Bonnet linking curvature and Euler characteristic), ../../lenses/variational.en.md (constrained variational of compression ratio vs. topological fidelity)
+- Knowledge: ../../knowledge-base/topology/persistent-homology.en.md (persistent homology, Vietoris--Rips filtration, bottleneck distance), ../../knowledge-base/topology/euler-characteristic.en.md (Euler characteristic for rapid topological diagnostics), ../../knowledge-base/matrix-analysis/matrix-perturbation.en.md (Davis--Kahan subspace perturbation bound)
 
 ## Required Mathematical Background
 - **Stability Theorem of Persistent Homology**: $d_B(D(X), D(Y)) \leq d_{GH}(X, Y)$; the Hausdorff distance bounds the change in persistence diagrams
@@ -46,15 +46,15 @@ Method 3 - Topology monitoring + adaptive compression ratio (at inference time):
 - **Topological diagnostics dashboard**: real-time plotting of $\beta_0(\epsilon), \beta_1(\epsilon)$ curves during training
 
 ## GPU Feasibility
-- Tensorization / GEMM: the distance matrix `cdist` is GEMM-dominated; $\chi(\epsilon)$ involves threshold counting + cumulative sum
-- Complexity: full persistent homology $O(N^3)$ is infeasible; Euler curve $O(N^2 |\epsilon|)$ is feasible; Landmark $O(m^3)$
-- Memory: the $N \times N$ distance matrix requires chunking or landmark reduction for $N > 8K$
-- Low precision: distance computation is stable in bf16 (positive-number addition); $\chi$ involves integer arithmetic with no precision concerns
-- Parallelism: each threshold in the $\epsilon$ sweep is independently parallel; landmark selection can be batch-parallelized
-- Operator fusion: cdist + threshold + count can be fused to avoid materializing the large distance matrix
+- **D1[v]/D2[v]**: the distance matrix `cdist` is GEMM-dominated; $\chi(\epsilon)$ involves threshold counting + cumulative sum
+- **D3[~]**: full persistent homology $O(N^3)$ is infeasible; Euler curve $O(N^2 |\epsilon|)$ is feasible; Landmark $O(m^3)$
+- **D4[~]**: the $N \times N$ distance matrix requires chunking or landmark reduction for $N > 8K$
+- **D5[v]**: distance computation is stable in bf16 (positive-number addition); $\chi$ involves integer arithmetic with no precision concerns
+- **D6[v]**: each threshold in the $\epsilon$ sweep is independently parallel; landmark selection can be batch-parallelized
+- **D8[v]**: cdist + threshold + count can be fused to avoid materializing the large distance matrix
 
 ## Paper-Worthy Formulation
-"Grounded in the Bottleneck stability theorem of persistent homology, we achieve $O(N^2)$-complexity topology-preserving regularization via Euler characteristic curve matching, ensuring that post-compression deviations in Betti numbers over persistence intervals are controlled by the Hausdorff distance."
+"Grounded in the Bottleneck stability theorem of persistent homology, we build a computable topology-preserving regularizer via Euler-characteristic curves or landmark approximations. Bottleneck distance can bound persistence-diagram changes when the filtration-function perturbation is controlled; Euler curves are incomplete proxies and do not by themselves guarantee per-degree Betti-number deviations, so persistence / Betti-curve deviations should be measured."
 
 ## Risks
 - **Persistent homology computational bottleneck**: exact boundary matrix reduction is highly serial ($O(N^3)$), necessitating reliance on Euler curve or landmark approximations

package/design-patterns/compression/topology-preserving-compression.md

@@ -2,11 +2,11 @@
 > **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
-当压缩表示时需要同构保持数据的本质拓扑结构（连通性、环、空腔）时使用：隐空间压缩（环形流形不能压成线段）、知识蒸馏（学生-教师同调等价）、3D 网格简化（genus 不变）、KV-Cache 语义保持（聚类结构不坍缩）。核心诉求：**压缩维度或数量，保证持续同调的 persistence diagram 变化可控**。
+当压缩表示时需要同构保持数据的本质拓扑结构（连通性、环、空腔）时使用：隐空间压缩（环形流形不能压成线段）、知识蒸馏（学生-教师同调等价）、3D 网格简化（genus 不变）、KV-Cache 语义保持（聚类结构不坍缩）。核心诉求：**压缩维度或数量，同时约束并实测持续同调的 persistence diagram 变化**。
 
 ## 数学思想来源
-- 透镜：lenses/topological.md（拓扑不变量——连通性、洞数在连续变形下不变）、lenses/spectral.md（Gauss-Bonnet 连接曲率与欧拉示性数）、lenses/variational.md（压缩率 vs. 拓扑保真度的约束优化）
-- 知识：knowledge-base/topology/persistent-homology.md（持续同调、Vietoris-Rips 滤流、Bottleneck 距离）、knowledge-base/topology/euler-characteristic.md（欧拉示性数快速拓扑诊断）、knowledge-base/matrix-analysis/matrix-perturbation.md（Davis-Kahan 子空间扰动界）
+- 透镜：../../lenses/topological.md（拓扑不变量——连通性、洞数在连续变形下不变）、../../lenses/spectral.md（Gauss-Bonnet 连接曲率与欧拉示性数）、../../lenses/variational.md（压缩率 vs. 拓扑保真度的约束优化）
+- 知识：../../knowledge-base/topology/persistent-homology.md（持续同调、Vietoris-Rips 滤流、Bottleneck 距离）、../../knowledge-base/topology/euler-characteristic.md（欧拉示性数快速拓扑诊断）、../../knowledge-base/matrix-analysis/matrix-perturbation.md（Davis-Kahan 子空间扰动界）
 
 ## 需要的数学知识
 - **持续同调稳定性定理**：$d_B(D(X), D(Y)) \leq d_{GH}(X, Y)$，Hausdorff 距离界定 persistence diagram 变化
@@ -46,15 +46,15 @@
 - **拓扑诊断仪表板**：训练中实时绘制 $\beta_0(\epsilon), \beta_1(\epsilon)$ 曲线
 
 ## GPU 可行性
-- 张量化/GEMM：距离矩阵 `cdist` 核心为 GEMM；$\chi(\epsilon)$ 为阈值计数 + 累积和
-- 复杂度：完整持续同调 $O(N^3)$ 不可行；Euler curve $O(N^2 |\epsilon|)$ 可行；Landmark $O(m^3)$
-- 显存：$N \times N$ 距离矩阵在 $N > 8K$ 时需分块或 Landmark 降维
-- 低精度：距离计算在 bf16 下稳定（正数加法）；$\chi$ 为整数运算无精度问题
-- 并行：$\epsilon$ 扫描的每个阈值独立并行；Landmark 选择可批并行
-- 算子融合：cdist + threshold + count 可融合避免物化大距离矩阵
+- **D1[v]/D2[v]**：距离矩阵 `cdist` 核心为 GEMM；$\chi(\epsilon)$ 为阈值计数 + 累积和
+- **D3[~]**：完整持续同调 $O(N^3)$ 不可行；Euler curve $O(N^2 |\epsilon|)$ 可行；Landmark $O(m^3)$
+- **D4[~]**：$N \times N$ 距离矩阵在 $N > 8K$ 时需分块或 Landmark 降维
+- **D5[v]**：距离计算在 bf16 下稳定（正数加法）；$\chi$ 为整数运算无精度问题
+- **D6[v]**：$\epsilon$ 扫描的每个阈值独立并行；Landmark 选择可批并行
+- **D8[v]**：cdist + threshold + count 可融合避免物化大距离矩阵
 
 ## 论文表述方式
-"以持续同调的 Bottleneck 稳定性定理为理论基础，通过 Euler characteristic curve 匹配实现 $O(N^2)$ 复杂度的拓扑保真正则化，保证压缩后 Betti 数在持续区间上的偏差受 Hausdorff 距离控制。"
+"以持续同调的 Bottleneck 稳定性定理为理论基础，通过 Euler characteristic curve 或 landmark 近似构造可计算的拓扑正则。Bottleneck 距离可在滤流函数扰动受控时界定 persistence diagram 的变化；Euler curve 只是不完整代理，不能单独保证每阶 Betti 数偏差受控，需报告 persistence/Betti 曲线的实测偏差。"
 
 ## 风险
 - **持续同调计算瓶颈**：精确边界矩阵约化高度串行（$O(N^3)$），必须依赖 Euler curve 或 Landmark 近似

package/design-patterns/loss/constraint-penalty.en.md

@@ -5,8 +5,8 @@
 When the design involves hard constraints (e.g., probability simplex, orthogonality, capacity limits, load balancing) but end-to-end training is required. Typical scenarios: (1) MoE routing probabilities must lie on the $K$-simplex with load balancing; (2) Expert activation count is constrained (top-k); (3) Subspace projection matrices must satisfy orthogonality $W^T W = I$; (4) Feature norms are bounded $\|z\| \leq R$. Core objective: **transform mathematical constraints into differentiable penalty terms integrated into gradient-based variational**.
 
 ## Mathematical Inspiration
-- Lenses: lenses/variational.md (constrained variational, Lagrangian duality, KKT conditions), lenses/geometric.md (manifold projection)
-- Knowledge: knowledge-base/optimization/lagrangian-duality.md (augmented Lagrangian method, penalty function method), knowledge-base/matrix-analysis/projection.md (projection operators, constraint sets)
+- Lenses: ../../lenses/variational.en.md (constrained variational, Lagrangian duality, KKT conditions), ../../lenses/geometric.en.md (manifold projection)
+- Knowledge: ../../knowledge-base/optimization/lagrangian-duality.en.md (augmented Lagrangian method, penalty function method), ../../knowledge-base/matrix-analysis/projection.en.md (projection operators, constraint sets)
 
 ## Required Mathematical Knowledge
 - **Penalty Function Method**: $\min f(x)$ s.t. $g(x)=0 \to \min f(x) + \rho/2 \cdot \|g(x)\|^2$ -- $\rho$ is gradually increased (exterior point method), driving constraint violation $\|g(x)\| \to 0$
@@ -29,6 +29,10 @@ Method 2 - Augmented Lagrangian Method (ALM):
   // lambda is a learnable parameter (nn.Parameter), updated via gradient ascent:
   lambda.data += rho * g(x).detach()   // dual ascent step
   // Converges faster than pure penalty, avoids rho -> infinity
+  For inequality constraints h(x) <= 0:
+  L_ALM = Sum_j 1/(2*rho) * [max(0, lambda_j + rho*h_j(x))^2 - lambda_j^2]
+  // lambda update: lambda_j <- max(0, lambda_j + rho*h_j(x))
+  // Strictly feasible constraints (lambda_j + rho*h_j(x) < 0) are not penalized
 
 Method 3 - Softmax Projection onto Simplex (load balancing special case):
   p = softmax(logits / tau)           // project onto Delta^{K-1}
@@ -43,7 +47,7 @@ Method 4 - Orthogonal Constraint Projection:
 
 ## Implementable Architectures
 - **Loss Wrapper**: ConstraintLoss(base_loss, constraints, rho_schedule) -- forward computes base_loss + Sum constraint.penalty()
-- **Dual Variable Management**: nn.Parameter stores lambda; negative learning rate set in the optimizer to achieve gradient ascent
+- **Dual Variable Management**: equality multipliers can use nn.Parameter with a negative learning rate for gradient ascent; inequality multipliers must satisfy $\lambda \geq 0$, so use explicit projected updates `lambda <- max(0, lambda + rho*h(x))` or clamp after optimizer steps
 - **Warm-up Strategy**: Optimize only base_loss for the first $N$ steps, then progressively activate constraint penalties
 - **Constraint Monitoring**: Record $\|g(x)\|$ at each step for visualization and adaptive rho adjustment
 
@@ -58,7 +62,7 @@ Method 4 - Orthogonal Constraint Projection:
 - **Operator Fusion**: Constraint computation + weighted summation + merging with base_loss can be fused into a single kernel
 
 ## Paper Phrasing
-"We employ the augmented Lagrangian method to convert hard constraints $g(x)=0$ into differentiable penalty terms $\lambda^T g(x) + \rho/2 \|g(x)\|^2$. Through alternating dual ascent updates of $\lambda$, constraint satisfaction is guaranteed to converge at rate $O(1/\rho)$ without requiring $\rho \to \infty$, reducing constraint violations by 3x compared to pure penalty methods in experiments."
+"We employ the augmented Lagrangian method to convert hard constraints $g(x)=0$ into differentiable penalty terms $\lambda^T g(x) + \rho/2 \|g(x)\|^2$, using alternating dual ascent updates of $\lambda$ rather than relying only on $\rho \to \infty$. Under convexity, constraint qualifications such as LICQ/MFCQ, and sufficiently accurate inner solves, ALM can converge to KKT solutions; in non-convex training, report the measured constraint-violation curve instead of claiming a universal $O(1/\rho)$ rate."
 
 ## Risks
 - Excessively fast rho growth leads to ill-conditioned variational landscapes (deteriorating condition numbers), causing gradient vanishing or explosion

package/design-patterns/loss/constraint-penalty.md

@@ -8,9 +8,9 @@
 核心诉求：**将数学约束转化为可微惩罚项，融入梯度优化流程**。
 
 ## 数学思想来源
-- 透镜：lenses/variational.md（约束优化、拉格朗日对偶、KKT 条件）、lenses/geometric.md（流形投影）
-- 知识：knowledge-base/optimization/lagrangian-duality.md（增广拉格朗日法、惩罚函数法）、
-  knowledge-base/matrix-analysis/projection.md（投影算子、约束集）
+- 透镜：../../lenses/variational.md（约束优化、拉格朗日对偶、KKT 条件）、../../lenses/geometric.md（流形投影）
+- 知识：../../knowledge-base/optimization/lagrangian-duality.md（增广拉格朗日法、惩罚函数法）、
+  ../../knowledge-base/matrix-analysis/projection.md（投影算子、约束集）
 
 ## 需要的数学知识
 - **罚函数法**：min f(x) s.t. g(x)=0 → min f(x) + ρ/2 · ‖g(x)‖²
@@ -37,6 +37,10 @@
   // λ 为可学习参数（nn.Parameter），通过梯度上升更新：
   λ.data += ρ * g(x).detach()   // 对偶上升步
   // 比纯罚函数收敛快，避免 ρ → ∞
+  对不等式约束 h(x) ≤ 0：
+  L_ALM = Σ_j 1/(2ρ) · [max(0, λ_j + ρ·h_j(x))² - λ_j²]
+  // λ 更新：λ_j ← max(0, λ_j + ρ·h_j(x))
+  // 严格可行（λ_j + ρ·h_j(x) < 0）的约束不被惩罚
 
 方法3 - Softmax 投影到单纯形（负载均衡特例）：
   p = softmax(logits / τ)           // 投影到 Δ^{K-1}
@@ -52,7 +56,7 @@
 ## 可实现结构
 - **Loss Wrapper**：ConstraintLoss(base_loss, constraints, ρ_schedule)
   forward 时计算 base_loss + Σ constraint.penalty()
-- **对偶变量管理**：nn.Parameter 存储 λ，optimizer 中为 λ 设置负学习率实现梯度上升
+- **对偶变量管理**：等式乘子 λ 可用 nn.Parameter + 负学习率做梯度上升；不等式乘子必须保持 λ ≥ 0，建议用显式投影更新 `λ ← max(0, λ + ρ·h(x))` 或在 optimizer step 后 clamp
 - **warm-up 策略**：前 N 步只优化 base_loss，之后逐步激活约束惩罚
 - **约束监控**：每 step 记录 ‖g(x)‖ 用于可视化和自适应 ρ 调整
 
@@ -68,8 +72,7 @@
 
 ## 论文表述方式
 "采用增广拉格朗日法将硬约束 g(x)=0 转化为可微惩罚项 λ^Tg(x) + ρ/2‖g(x)‖²，
-通过对偶变量 λ 的交替上升更新，在无需 ρ→∞ 的条件下保证约束满足度以 O(1/ρ) 收敛，
-相比纯罚函数法在实验中减少 3× 的约束违反。"
+并通过对偶变量 λ 的交替上升更新避免单纯依赖 ρ→∞。在凸性、约束规格（如 LICQ/MFCQ）和内层子问题求解精度满足条件时，ALM 可收敛到 KKT 解；非凸训练中应报告实际约束违反曲线，而不能宣称通用 O(1/ρ) 速率。"
 
 ## 风险
 - ρ 增大过快导致优化 landscape 病态（条件数恶化），梯度消失或爆炸

package/design-patterns/loss/contrastive-loss.en.md

@@ -5,8 +5,8 @@
 When the model needs to learn "what is similar to what and what is different from what." Typical scenarios: (1) Different augmented views of the same input should be pulled closer (positive pairs), while different inputs should be pushed apart (negative pairs); (2) Shared representations should capture cross-task commonalities, while Private representations should distinguish task-specific features; (3) In the expert embedding space, similar inputs should be routed to the same expert. Core objective: **learn relative relationships rather than absolute values**.
 
 ## Mathematical Inspiration
-- Lenses: lenses/geometric.md (metric spaces and distance functions), lenses/probabilistic.md (mutual information maximization)
-- Knowledge: knowledge-base/probability/entropy.md (conditional distributions and likelihood), knowledge-base/differential-geometry/manifold.md (geodesics and curvature)
+- Lenses: ../../lenses/geometric.en.md (metric spaces and distance functions), ../../lenses/probabilistic.en.md (mutual information maximization)
+- Knowledge: ../../knowledge-base/probability/entropy.en.md (conditional distributions and likelihood), ../../knowledge-base/differential-geometry/manifold.en.md (geodesics and curvature)
 
 ## Required Mathematical Knowledge
 - **InfoNCE Loss**: $L = -\log[\exp(\text{sim}(q,k^+)/\tau) / \sum_j \exp(\text{sim}(q,k_j)/\tau)]$ -- essentially a lower-bound estimate of mutual information in contrastive learning; $\tau$ is the temperature parameter controlling distribution sharpness
@@ -14,6 +14,32 @@ When the model needs to learn "what is similar to what and what is different fro
 - **NT-Xent (Normalized Temperature-scaled Cross Entropy)**: Softmax contrastive loss on the unit sphere $S^{d-1}$, with normalization eliminating scale effects
 - **Debiased Contrastive Learning**: Corrects for false negatives in negative samples, using prior $\tau^+$ to estimate the true negative sample distribution
 
+## Alignment-Uniformity Framework
+
+Representation quality in contrastive learning decomposes into two independent objectives (Wang & Isola, 2020):
+
+- **Alignment**: Positive pairs should have similar representations
+  $L_{\text{align}} = \mathbb{E}_{(x, x^+)}[\|f(x) - f(x^+)\|^2]$
+- **Uniformity**: Representations should be uniformly distributed on the unit hypersphere $S^{d-1}$
+  $L_{\text{uniform}} = \log \mathbb{E}_{x, x'}[\exp(-2\|f(x) - f(x')\|^2)]$
+
+**InfoNCE and alignment-uniformity**: As $N \to \infty$ with appropriate temperature $\tau$, the InfoNCE loss asymptotically decomposes into alignment + uniformity. For finite $N$, InfoNCE provides a lower bound on $I(X;Z)$, with tightness increasing in $N$.
+
+**Conditions for uniformity to hold**:
+- Sufficiently large negative sample count $N$ (theory requires $N \to \infty$; in practice $N \geq 1024$ is typically sufficient)
+- Temperature $\tau$ not too large ($\tau \to \infty$ causes loss to degenerate to a constant, eliminating the uniformity-driving force)
+- Representation dimension $d$ sufficient to support the intrinsic dimension of the data
+
+**Conditions for uniformity to fail**:
+- Insufficient negatives $\rightarrow$ weak uniformity pressure; representations may cluster on a local region of the sphere
+- Representation collapse: all inputs map to the same (or few) points, trivially minimizing alignment but completely destroying uniformity
+- Temperature $\tau$ too large $\rightarrow$ softmax degenerates to uniform distribution, gradients vanish, no uniformity guarantee
+- Severe positive/negative imbalance within the batch without queue compensation
+
+**What can be guaranteed at most**: Under ideal conditions (sufficiently large $N$, appropriate $\tau$, no collapse), minimizing contrastive loss is equivalent to jointly maximizing positive-pair alignment and representation uniformity.
+
+**What cannot be guaranteed**: Optimality of learned representations for downstream tasks (uniformity $\neq$ task relevance); semantic-level alignment (only geometric-level positive-pair proximity is guaranteed).
+
 ## AI Module Form
 ```
 Module: ContrastiveLoss
@@ -40,17 +66,17 @@ Hard Negative Mining:
 - **Multi-Granularity Contrast**: Apply contrastive objectives simultaneously at token-level, sequence-level, and expert-level
 
 ## GPU Feasibility
-- **Tensorization**: Similarity computation is $z_a @ z_n^T$ -- standard GEMM $(B \times d) @ (d \times N) = B \times N$
-- **GEMM-mappability**: Core computation is 1-2 matrix multiplications, perfectly mapped to cuBLAS
-- **Complexity**: $O(B \cdot N \cdot d)$ computation + $O(B \cdot N)$ storage for the logits matrix; approximately 64MB when B=256, N=65536
-- **Memory & KV-Cache**: Negative sample queue occupies $N \cdot d \cdot 4$ bytes, approximately 65536 * 256 * 4 = 64MB, fixed overhead
-- **Low Precision Stability**: Cosine similarity + softmax under fp16 requires caution for exp overflow; use log-sum-exp trick
-- **Parallelism & Communication**: On multiple GPUs, use all-gather to collect negatives from other GPUs to enlarge $N$ (MoCo v3 strategy)
-- **Sparse Structure**: After hard negative mining, only $k \ll N$ negatives are retained, effectively sparsifying the logits
-- **Operator Fusion**: L2-norm -> matmul -> scale -> log-softmax -> nll_loss can be fused
+- **D1[v]**: Similarity computation is $z_a @ z_n^T$ -- standard GEMM $(B \times d) @ (d \times N) = B \times N$
+- **D2[v]**: Core computation is 1-2 matrix multiplications, perfectly mapped to cuBLAS
+- **D3[v]**: $O(B \cdot N \cdot d)$ computation + $O(B \cdot N)$ storage for the logits matrix; approximately 64MB when B=256, N=65536
+- **D4[v]**: Negative sample queue occupies $N \cdot d \cdot 4$ bytes, approximately 65536 * 256 * 4 = 64MB, fixed overhead
+- **D5[v]**: Cosine similarity + softmax under fp16 requires caution for exp overflow; use log-sum-exp trick
+- **D6[v]**: On multiple GPUs, use all-gather to collect negatives from other GPUs to enlarge $N$ (MoCo v3 strategy)
+- **D7[v]**: After hard negative mining, only $k \ll N$ negatives are retained, effectively sparsifying the logits
+- **D8[v]**: L2-norm -> matmul -> scale -> log-softmax -> nll_loss can be fused
 
 ## Paper Phrasing
-"We employ temperature-scaled InfoNCE contrastive loss with a momentum encoder maintaining a negative sample queue of N=65536, maximizing the lower bound of positive-pair mutual information on the unit sphere. Theoretical analysis shows this lower bound converges at rate $O(1/\sqrt{N})$."
+"We employ temperature-scaled InfoNCE contrastive loss with a momentum encoder maintaining a negative-sample queue, optimizing proxies for positive-pair alignment and representation uniformity on the unit sphere. Mutual-information lower bounds and sampling-error rates depend on the negative-sample distribution, independence assumptions, and queue staleness; report ablations over queue size, temperature, negative-mining strategy, and downstream metrics."
 
 ## Risks
 - Too small $\tau$ causes training instability (excessively large gradients); too large $\tau$ makes all samples indistinguishable (degenerates to uniform distribution)

package/design-patterns/loss/contrastive-loss.md

@@ -8,9 +8,9 @@
 (3) 专家嵌入空间中相似输入应路由到同一专家。核心诉求：**学习相对关系而非绝对值**。
 
 ## 数学思想来源
-- 透镜：lenses/geometric.md（度量空间与距离函数）、lenses/probabilistic.md（互信息最大化）
-- 知识：knowledge-base/probability/entropy.md（条件分布与似然）、
-  knowledge-base/differential-geometry/manifold.md（测地线与曲率）
+- 透镜：../../lenses/geometric.md（度量空间与距离函数）、../../lenses/probabilistic.md（互信息最大化）
+- 知识：../../knowledge-base/probability/entropy.md（条件分布与似然）、
+  ../../knowledge-base/differential-geometry/manifold.md（测地线与曲率）
 
 ## 需要的数学知识
 - **InfoNCE 损失**：L = -log[exp(sim(q,k⁺)/τ) / Σ_j exp(sim(q,k_j)/τ)]
@@ -22,6 +22,32 @@
 - **去偏对比学习 (Debiased Contrastive)**：
   修正负样本中的假阴性问题，使用先验 τ⁺ 估计真实负样本分布
 
+## 对齐-均匀性理论 (Alignment-Uniformity Framework)
+
+对比学习的表示质量可分解为两个独立目标（Wang & Isola, 2020）：
+
+- **对齐 (Alignment)**：正对的表示应接近
+  $L_{\text{align}} = \mathbb{E}_{(x, x^+)}[\|f(x) - f(x^+)\|^2]$
+- **均匀性 (Uniformity)**：表示应在单位超球面 $S^{d-1}$ 上均匀分布
+  $L_{\text{uniform}} = \log \mathbb{E}_{x, x'}[\exp(-2\|f(x) - f(x')\|^2)]$
+
+**InfoNCE 与对齐-均匀性的关系**：当 $N \to \infty$ 且温度 $\tau$ 适当时，InfoNCE 损失渐近分解为 alignment + uniformity 之和。有限 $N$ 下，InfoNCE 给出 $I(X;Z)$ 的下界，下界紧度随 $N$ 增大。
+
+**均匀性成立的条件**：
+- 负样本数 $N$ 充分大（理论要求 $N \to \infty$，实践中 $N \geq 1024$ 通常足够）
+- 温度 $\tau$ 不过大（$\tau \to \infty$ 时 loss 退化为常数，丧失均匀性驱动力）
+- 表示维度 $d$ 足够支撑数据的本征维度
+
+**均匀性不成立的条件**：
+- 负样本不足 → 均匀性驱动力弱，表示可能聚集在球面局部
+- 表示坍塌 (representation collapse)：所有输入映射到同一/少数点，trivially 最小化 alignment 但完全丧失 uniformity
+- 温度 $\tau$ 过大 → softmax 趋于均匀分布，梯度消失，无均匀性保证
+- batch 内正负对比例严重失衡且未通过队列补偿
+
+**最多能保证什么**：在理想条件下（$N$ 充分大、$\tau$ 合适、无坍塌），对比损失最小化等价于同时最大化正对对齐度和表示均匀性。
+
+**不能保证什么**：不能保证学到的表示对下游任务最优（均匀性 ≠ 任务相关性）；不能保证语义层级的对齐（仅保证几何层面的正对接近）。
+
 ## AI 模块形式
 ```
 模块：ContrastiveLoss
@@ -48,18 +74,18 @@ Hard Negative Mining:
 - **多粒度对比**：同时在 token-level、sequence-level、expert-level 施加对比
 
 ## GPU 可行性
-- **张量化**：sim 计算为 z_a @ z_n^T → 标准 GEMM (B×d) @ (d×N) = B×N
-- **GEMM 可映射**：核心就是 1-2 次矩阵乘法，完美映射 cuBLAS
-- **复杂度**：O(B·N·d) 计算 + O(B·N) 存储 logits 矩阵，B=256,N=65536 时约 64MB
-- **显存与 KV-Cache**：负样本队列占 N·d·4 bytes ≈ 65536·256·4 = 64MB，固定开销
-- **低精度稳定**：cosine 相似度 + softmax 在 fp16 下需注意 exp 溢出，用 log-sum-exp trick
-- **并行与通信**：多 GPU 时用 all-gather 收集其他 GPU 的负样本扩大 N（MoCo v3 策略）
-- **稀疏结构**：hard negative mining 后只保留 k<<N 个负样本，有效稀疏化 logits
-- **算子融合**：L2-norm → matmul → scale → log-softmax → nll_loss 可融合
+- **D1[v]**：sim 计算为 z_a @ z_n^T → 标准 GEMM (B×d) @ (d×N) = B×N
+- **D2[v]**：核心就是 1-2 次矩阵乘法，完美映射 cuBLAS
+- **D3[v]**：O(B·N·d) 计算 + O(B·N) 存储 logits 矩阵，B=256,N=65536 时约 64MB
+- **D4[v]**：负样本队列占 N·d·4 bytes ≈ 65536·256·4 = 64MB，固定开销
+- **D5[v]**：cosine 相似度 + softmax 在 fp16 下需注意 exp 溢出，用 log-sum-exp trick
+- **D6[v]**：多 GPU 时用 all-gather 收集其他 GPU 的负样本扩大 N（MoCo v3 策略）
+- **D7[v]**：hard negative mining 后只保留 k<<N 个负样本，有效稀疏化 logits
+- **D8[v]**：L2-norm → matmul → scale → log-softmax → nll_loss 可融合
 
 ## 论文表述方式
 "采用温度缩放的 InfoNCE 对比损失，通过动量编码器维护 N=65536 的负样本队列，
-在单位球面上最大化正对互信息的下界，理论分析表明该下界以 O(1/√N) 收敛。"
+在单位球面上优化正对齐与表示均匀性代理。互信息下界和采样误差界依赖负样本分布、独立性与队列陈旧度假设，论文中应报告队列大小、温度、负样本策略和下游指标的消融。"
 
 ## 风险
 - τ 过小导致训练不稳定（梯度过大），τ 过大导致所有样本不分难易（退化均匀分布）