math-skill 3.0.0 → 3.1.0

package/design-patterns/attention/spectral-attention.md

@@ -1,8 +1,8 @@
 # 谱注意力 / Spectral Attention
-> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
-当输入信号具有**频域/谱结构**（周期性、循环对称性、图结构）时，在谱域而非空域计算注意力可大幅降低复杂度并利用信号的内在结构。典型场景：时间序列预测（周期性信号）、图神经网络（图 Laplacian 谱分解）、位置编码（RoPE/ALiBi 的频率解释）、长序列注意力的 $O(n \log n)$ 加速。
+当输入信号具有**频域/谱结构**（周期性、循环对称性、图结构）时，在谱域而非空域计算注意力可大幅降低复杂度并利用信号的内在结构。典型场景：时间序列预测（周期性信号）、图神经网络（图 Laplacian 谱分解）、位置编码（RoPE/ALiBi 的频率解释）、长序列中**位置依赖型**注意力的 $O(n \log n)$ 加速（需循环/Toeplitz 结构假设，不适用于通用 content-dependent softmax 注意力）。
 
 ## 数学思想来源
 - 透镜：[duality（对偶透镜 — 频域变换）, symmetry（对称透镜 — 循环/平移不变性）]
@@ -11,12 +11,14 @@
 ## 需要的数学知识
 - **离散傅里叶变换 DFT / FFT**：$O(n \log n)$ 频域变换，循环卷积定理
 - **图 Laplacian 谱分解**：$L = U \Lambda U^T$，$U$ 为图 Fourier 基
-- **循环群 $\mathbb{Z}_n$ 的不可约表示**：DFT 矩阵即循环群的表示矩阵（参见 `references/books/abstract-algebra.md` Ch.4, Ch.11）
+- **循环群 $\mathbb{Z}_n$ 的不可约表示**：DFT 矩阵即循环群的表示矩阵（参见 `../../references/books/abstract-algebra.md` Ch.4, Ch.11）
 
 ## AI 模块形式
 
 **核心思路**：将注意力从空域的 $Q K^T$ 转换到谱域的对角/稀疏运算：
 
+> **关键前提条件**：上述「谱域对角化 = 注意力」的等价性**仅在注意力矩阵为循环矩阵（circulant）或 Toeplitz 矩阵时成立**，即注意力权重仅依赖于 token 之间的相对位置 $a_{ij} = f(i-j)$，而与 token 内容无关。此时循环卷积定理保证 FFT 可将卷积核对角化。对于标准的 content-dependent softmax 注意力 $\text{softmax}(QK^T/\sqrt{d})$，注意力矩阵由 query/key 内容决定，一般**不是**循环矩阵或 Toeplitz 矩阵，因此 FFT 逐元素乘法**不能**等价替代 $QK^T$ 计算。方案 A/C 本质上是位置依赖的卷积式注意力近似，而非通用 softmax 注意力的精确替代。
+
 **方案 A：FFT 加速注意力（时间序列）**：
 ```python
 # 将循环卷积写成谱域逐元素乘
@@ -54,18 +56,30 @@ attn = ifft(scores_freq)
 - **频率感知位置编码**：RoPE 的本质即循环群 $\mathbb{Z}$ 的酉表示（参见抽象代数 Ch.4），可推广到其他群
 
 ## GPU 可行性
-- **维度 1 张量化 ✅**：FFT 和矩阵乘均为标准张量运算
-- **维度 2 GEMM 可映射 ✅**：谱投影 $U^T Q$ 为标准 GEMM；FFT 虽非 GEMM 但有高度优化的 cuFFT 实现
-- **维度 3 复杂度 ✅**：FFT 注意力 $O(n \log n \cdot d)$，远优于 $O(n^2 d)$
-- **维度 4 显存 ✅**：频域表示不增加额外维度，谱投影可降低到 $k \ll n$ 维
-- **维度 5 低精度 ⚠️**：复数 FFT 在 fp16 下精度损失，需 fp32 或使用实数 FFT (RFFT)
-- **维度 6 并行 ✅**：FFT 可跨 batch/head 并行，cuFFT 支持多流
-- **维度 7 稀疏 ✅**：谱域中高频分量可截断（结构化稀疏），只保留 top-k 频率
-- **维度 8 算子融合 ⚠️**：FFT 与 attention 融合需自定义 kernel，标准库无现成融合
+- **D1[v]**：FFT 和矩阵乘均为标准张量运算
+- **D2[v]**：谱投影 $U^T Q$ 为标准 GEMM；FFT 虽非 GEMM 但有高度优化的 cuFFT 实现
+- **D3[~]**：FFT 注意力 $O(n \log n \cdot d)$，远优于 $O(n^2 d)$——**但仅当注意力具有平移不变/位置依赖结构时成立**（如卷积式注意力）。通用 content-dependent softmax 注意力仍为 $O(n^2 d)$；linear attention 近似可达 $O(n d^2)$，但其机制与 FFT 谱方法不同。
+- **D4[v]**：频域表示不增加额外维度，谱投影可降低到 $k \ll n$ 维
+- **D5[~]**：复数 FFT 在 fp16 下精度损失，需 fp32 或使用实数 FFT (RFFT)
+- **D6[v]**：FFT 可跨 batch/head 并行，cuFFT 支持多流
+- **D7[v]**：谱域中高频分量可截断（结构化稀疏），只保留 top-k 频率
+- **D8[~]**：FFT 与 attention 融合需自定义 kernel，标准库无现成融合
+
+**量化评估示例**（标准 transformer, d=128, n=2048, h=16）：
+- D3: FFT 路径 FLOPs ≈ 2·n·log₂(n)·d ≈ 2·2048·11·128 ≈ 5.8M vs 标准 attention 2·n²·d ≈ 1.1G（仅当 attention 为卷积核时）
+- D4: 无需物化 n×n attention matrix；FFT 中间结果 O(n·d) ≈ 1MB
+- D5: FFT 在 bf16 下 twiddle factor 误差 ~10⁻³，可接受
+- D8: FFT + pointwise 可融合为单个 CUDA kernel
 
 ## 论文表述方式
-"我们提出谱域注意力机制，通过将注意力计算转换到 Fourier/Laplacian 谱域，利用循环卷积定理将序列注意力复杂度从 $O(n^2)$ 降至 $O(n \log n)$，同时通过频率自适应权重保留对不同尺度依赖关系的建模能力。"
+"我们提出谱域注意力机制，通过将注意力计算转换到 Fourier/Laplacian 谱域，利用循环卷积定理将**具有平移不变结构的**序列注意力复杂度从 $O(n^2)$ 降至 $O(n \log n)$，同时通过频率自适应权重保留对不同尺度依赖关系的建模能力。注意：此加速**要求注意力模式具有位置依赖（而非内容依赖）结构**；对于通用 content-dependent softmax 注意力，该谱域等价性不成立。"
+
+## 适用条件
+- **FFT 谱注意力的核心限制**：方案 A/C 的 FFT 加速**要求注意力矩阵为循环矩阵或 Toeplitz 矩阵**，即注意力权重仅依赖于相对位置 $a_{ij} = f(i-j)$，而与 token 的具体内容（query/key 向量）无关。
+- **不满足条件的情形**：大多数 NLP 任务中的标准 softmax 注意力 $\text{softmax}(QK^T/\sqrt{d})$ 是 content-dependent 的——注意力权重由 query 和 key 的内容共同决定，产生的注意力矩阵一般不具有循环/Toeplitz 结构。在此情形下，FFT 逐元素乘法**不能**替代 $QK^T$ 运算，$O(n \log n)$ 复杂度优势也不成立。
+- **满足条件的情形**：位置编码（如 RoPE/ALiBi 的频率分量）、时间序列的周期性卷积核、具有平移不变先验的固定模式注意力。方案 B（图谱注意力）不受此限制，因为它在图 Laplacian 特征基上做投影而非利用循环卷积定理。
 
 ## 风险
+- **[x] Content-dependent 注意力不兼容**：FFT 谱方法的核心假设（循环/Toeplitz 结构）与 NLP 中主流的 content-dependent softmax 注意力**根本矛盾**。将 FFT 方法不加区分地应用于通用注意力机制是数学上的错误——两者计算的是不同的量。使用时必须明确声明所做的位置依赖假设。
 - **非周期性假设违反**：FFT 隐式假设周期性边界条件，对自然语言等非周期信号产生谱泄漏，需加窗函数或零填充。
 - **图 Laplacian 预计算成本**：大规模图的特征分解 $O(n^3)$ 不可行，需近似（Nyström/Lanczos），且动态图需重算。

package/design-patterns/compression/leverage-score-selection.en.md

@@ -1,16 +1,16 @@
 # Leverage Score Selection
-> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Target Problem
-Use when selecting the most representative rows / columns / tokens from a large-scale matrix while guaranteeing the precision of downstream linear algebra operations: KV-Cache token selection, data coreset construction, Nystrom landmark sampling, distributed gradient compression. Core objective: **sample based on statistical leverage scores derived from subspace projections, with probabilistic guarantees of approximating full-computation accuracy**.
+Use when selecting representative rows / columns / tokens from a large-scale matrix while seeking verifiable error bounds for downstream linear-algebra tasks: KV-Cache token selection, data coreset construction, Nystrom landmark sampling, distributed gradient compression. Core objective: **sample using statistical leverage scores derived from subspace projections, approximating full computation with high probability under matrix-sampling assumptions**.
 
 ## Mathematical Foundations
-- Lenses: lenses/spectral.md (leverage scores = projection energy of row vectors onto the principal subspace), lenses/probabilistic.md (probabilistic sampling and concentration inequality guarantees), lenses/algorithmic.md (complexity--accuracy trade-offs of randomized algorithms)
-- Knowledge: knowledge-base/matrix-analysis/low-rank-approximation.md (randomized SVD, nuclear norm), knowledge-base/matrix-analysis/projection.md (diagonal entries of the projection matrix = leverage scores), knowledge-base/probability/concentration-inequality.md (Bernstein matrix concentration bound)
+- Lenses: ../../lenses/spectral.en.md (leverage scores = projection energy of row vectors onto the principal subspace), ../../lenses/probabilistic.en.md (probabilistic sampling and concentration inequality guarantees), ../../lenses/algorithmic.en.md (complexity--accuracy trade-offs of randomized algorithms)
+- Knowledge: ../../knowledge-base/matrix-analysis/low-rank-approximation.en.md (randomized SVD, nuclear norm), ../../knowledge-base/matrix-analysis/projection.en.md (diagonal entries of the projection matrix = leverage scores), ../../knowledge-base/probability/concentration-inequality.en.md (Bernstein matrix concentration bound)
 
 ## Required Mathematical Background
 - **Statistical Leverage Scores**: $\ell_i = \|(V_k V_k^T)_i\|^2 = (V_k V_k^T)_{ii}$, the projection energy of the $i$-th row onto the rank-$k$ subspace; $\sum_i \ell_i = k$
-- **Leverage Score Sampling Guarantee**: sampling $s = O(k \log k / \epsilon^2)$ rows with probabilities $p_i = \ell_i / k$ yields a $(1+\epsilon)$ approximation to full least squares (Drineas--Mahoney)
+- **Leverage Score Sampling Bound**: in least-squares / subspace-embedding settings with independent reweighted sampling, sampling $s = O(k \log k / \epsilon^2)$ rows with probabilities $p_i = \ell_i / k$ can yield a $(1+\epsilon)$ approximation (Drineas--Mahoney-style results)
 - **Bernstein Matrix Bound**: after sampling, $\|\hat{A}^T \hat{A} - A^T A\|_2 \leq \epsilon \|A\|_F^2$ with probability $\geq 1-\delta$
 - **Fast Approximation**: $\tilde{\ell}_i = \|(A\Omega)_i\|^2$ ($\Omega$ random Gaussian), avoids full SVD, $O(Ndk)$
 - **DPP Extension**: Determinantal Point Process $P(S) \propto \det(L_S)$ adds a diversity guarantee on top of leverage scores
@@ -48,20 +48,20 @@ Method 4 - DPP greedy diverse selection:
 
 ## Implementable Architectures
 - **Random projection leverage layer**: 1 GEMM + 1 QR yields approximate leverage scores in $O(Ndk)$
-- **KV-Cache eviction policy**: evict by leverage score ranking, with stronger theoretical guarantees than attention-score-based eviction
-- **Coreset constructor**: leverage score sampling + importance reweighting, ensuring empirical risk approximates the full-data risk
+- **KV-Cache eviction policy**: evict by leverage score ranking; the theory is mainly for matrix subspace approximation, so transfer to attention quality requires separate validation
+- **Coreset constructor**: leverage score sampling + importance reweighting, controlling empirical-risk approximation under the relevant sampling assumptions
 - **DPP greedy extension**: leverage scores + exclusion penalty, balancing importance and diversity
 
 ## GPU Feasibility
-- Tensorization / GEMM: $A\Omega$ is a GEMM; QR via cuSOLVER; leverage scores = elementwise row-wise sum of squares
-- Complexity: $O(Ndk)$ is far superior to $O(Nd^2)$ full SVD; top-s selection in $O(N \log s)$
-- Memory: $\Omega$ is only $d \times k$ (KB-scale); $Q$ is the same size as $A$ but can be computed in batches
-- Low precision: QR recommended in fp32 (small matrix with $k+p$ columns, negligible overhead); leverage scores can be cast back to bf16
-- Parallelism: the $A\Omega$ GEMM is highly parallelizable; top-s can use parallel radix sort
-- Operator fusion: $A\Omega$ + QR + row-norm² can be fused to avoid materializing intermediate matrices
+- **D1[v]/D2[v]**: $A\Omega$ is a GEMM; QR via cuSOLVER; leverage scores = elementwise row-wise sum of squares
+- **D3[v]**: $O(Ndk)$ is far superior to $O(Nd^2)$ full SVD; top-s selection in $O(N \log s)$
+- **D4[v]**: $\Omega$ is only $d \times k$ (KB-scale); $Q$ is the same size as $A$ but can be computed in batches
+- **D5[~]**: QR recommended in fp32 (small matrix with $k+p$ columns, negligible overhead); leverage scores can be cast back to bf16
+- **D6[v]**: the $A\Omega$ GEMM is highly parallelizable; top-s can use parallel radix sort
+- **D8[v]**: $A\Omega$ + QR + row-norm² can be fused to avoid materializing intermediate matrices
 
 ## Paper-Worthy Formulation
-"We adopt statistical leverage scores as the importance metric for token selection: approximating rank-$k$ subspace leverage scores via random projection in $O(Ndk)$, with the Drineas--Mahoney theory guaranteeing that $O(k \log k / \epsilon^2)$ samples suffice for a $(1+\epsilon)$ approximation of the full subspace."
+"We adopt statistical leverage scores as the importance metric for token / row selection: approximate rank-$k$ subspace leverage scores via random projection in $O(Ndk)$, and use Drineas--Mahoney-style bounds to choose sample counts when the target is subspace / least-squares approximation under independent reweighted sampling and effective-rank diagnostics. For KV-cache eviction, attention/output error and task metrics must still be reported."
 
 ## Risks
 - **Rank parameter $k$ selection**: leverage scores depend on the rank-$k$ subspace; an incorrect $k$ leads to biased sampling -- the effective rank must be diagnosed first

package/design-patterns/compression/leverage-score-selection.md

@@ -1,16 +1,16 @@
 # Leverage Score Selection（杠杆分数选择）
-> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
-当需要从大规模矩阵中选取最有代表性的行/列/token，且需保证下游线性代数运算精度时使用：KV-Cache token 选择、数据 coreset 构建、Nyström landmark 采样、分布式梯度压缩。核心诉求：**基于子空间投影的统计杠杆分数做采样，以概率保证逼近全量计算精度**。
+当需要从大规模矩阵中选取最有代表性的行/列/token，并希望在下游线性代数任务中获得可检验误差界时使用：KV-Cache token 选择、数据 coreset 构建、Nyström landmark 采样、分布式梯度压缩。核心诉求：**基于子空间投影的统计杠杆分数做采样，在矩阵采样理论假设下以高概率逼近全量计算**。
 
 ## 数学思想来源
-- 透镜：lenses/spectral.md（杠杆分数 = 行向量在主子空间上的投影能量）、lenses/probabilistic.md（概率采样与浓度不等式保证）、lenses/algorithmic.md（随机化算法的复杂度与精度权衡）
-- 知识：knowledge-base/matrix-analysis/low-rank-approximation.md（随机化 SVD、核范数）、knowledge-base/matrix-analysis/projection.md（投影矩阵对角元 = 杠杆分数）、knowledge-base/probability/concentration-inequality.md（Bernstein 矩阵浓度界）
+- 透镜：../../lenses/spectral.md（杠杆分数 = 行向量在主子空间上的投影能量）、../../lenses/probabilistic.md（概率采样与浓度不等式保证）、../../lenses/algorithmic.md（随机化算法的复杂度与精度权衡）
+- 知识：../../knowledge-base/matrix-analysis/low-rank-approximation.md（随机化 SVD、核范数）、../../knowledge-base/matrix-analysis/projection.md（投影矩阵对角元 = 杠杆分数）、../../knowledge-base/probability/concentration-inequality.md（Bernstein 矩阵浓度界）
 
 ## 需要的数学知识
 - **统计杠杆分数**：$\ell_i = \|(V_k V_k^T)_i\|^2 = (V_k V_k^T)_{ii}$，第 $i$ 行在 rank-$k$ 子空间的投影能量；$\sum_i \ell_i = k$
-- **杠杆分数采样保证**：以 $p_i = \ell_i / k$ 采样 $s = O(k \log k / \epsilon^2)$ 行，$(1+\epsilon)$ 近似全量最小二乘（Drineas-Mahoney）
+- **杠杆分数采样界**：在最小二乘/子空间嵌入设定和独立重加权采样条件下，以 $p_i = \ell_i / k$ 采样 $s = O(k \log k / \epsilon^2)$ 行可得到 $(1+\epsilon)$ 近似（Drineas-Mahoney 类结果）
 - **Bernstein 矩阵界**：采样后 $\|\hat{A}^T \hat{A} - A^T A\|_2 \leq \epsilon \|A\|_F^2$，概率 $\geq 1-\delta$
 - **快速近似**：$\tilde{\ell}_i = \|(A\Omega)_i\|^2$（$\Omega$ 随机高斯），避免完整 SVD，$O(Ndk)$
 - **DPP 扩展**：行列式点过程 $P(S) \propto \det(L_S)$ 在杠杆分数基础上增加多样性保证
@@ -48,20 +48,20 @@
 
 ## 可实现结构
 - **随机投影杠杆层**：1 次 GEMM + 1 次 QR 即得近似杠杆分数，$O(Ndk)$
-- **KV-Cache 驱逐策略**：按杠杆分数排序驱逐，比 attention score 驱逐更有理论保证
-- **Coreset 构建器**：杠杆分数采样 + 重要性重加权，保证经验风险逼近全量
+- **KV-Cache 驱逐策略**：按杠杆分数排序驱逐；理论界主要来自矩阵子空间近似，迁移到 attention 质量需单独验证
+- **Coreset 构建器**：杠杆分数采样 + 重要性重加权，在相应采样假设下控制经验风险近似误差
 - **DPP 贪心扩展**：杠杆分数 + 互斥惩罚，兼顾重要性与多样性
 
 ## GPU 可行性
-- 张量化/GEMM：$A\Omega$ 为 GEMM；QR 有 cuSOLVER；杠杆分数 = elementwise 逐行平方和
-- 复杂度：$O(Ndk)$ 远优于 $O(Nd^2)$ 完整 SVD；top-s 选择 $O(N \log s)$
-- 显存：$\Omega$ 仅 $d \times k$（KB 级）；$Q$ 与 $A$ 同尺寸但可分批计算
-- 低精度：QR 建议 fp32（小矩阵 $k+p$ 列，开销可忽略）；杠杆分数可回 bf16
-- 并行：$A\Omega$ 的 GEMM 高度并行；top-s 可用 radix sort 并行
-- 算子融合：$A\Omega$ + QR + row-norm² 可融合避免物化中间矩阵
+- **D1[v]/D2[v]**：$A\Omega$ 为 GEMM；QR 有 cuSOLVER；杠杆分数 = elementwise 逐行平方和
+- **D3[v]**：$O(Ndk)$ 远优于 $O(Nd^2)$ 完整 SVD；top-s 选择 $O(N \log s)$
+- **D4[v]**：$\Omega$ 仅 $d \times k$（KB 级）；$Q$ 与 $A$ 同尺寸但可分批计算
+- **D5[~]**：QR 建议 fp32（小矩阵 $k+p$ 列，开销可忽略）；杠杆分数可回 bf16
+- **D6[v]**：$A\Omega$ 的 GEMM 高度并行；top-s 可用 radix sort 并行
+- **D8[v]**：$A\Omega$ + QR + row-norm² 可融合避免物化中间矩阵
 
 ## 论文表述方式
-"采用统计杠杆分数作为 token 选择的重要性度量：通过随机投影在 $O(Ndk)$ 内近似 rank-$k$ 子空间杠杆分数，Drineas-Mahoney 理论保证 $O(k \log k / \epsilon^2)$ 次采样即可 $(1+\epsilon)$ 近似全量子空间。"
+"采用统计杠杆分数作为 token/行选择的重要性度量：通过随机投影在 $O(Ndk)$ 内近似 rank-$k$ 子空间杠杆分数，并在独立重加权采样、有效秩诊断和目标为子空间/最小二乘近似的前提下，使用 Drineas-Mahoney 类界选择样本数。对 KV-cache eviction，仍需报告 attention/output 误差与任务指标。"
 
 ## 风险
 - **秩参数 $k$ 选择**：杠杆分数依赖 rank-$k$ 子空间，$k$ 选错导致采样偏差；需先诊断有效秩

package/design-patterns/compression/low-rank-kv-cache.en.md

@@ -1,17 +1,17 @@
 # Low-Rank KV-Cache
-> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Target Problem
-Use when KV-Cache memory consumption becomes the bottleneck during LLM inference: long-context inference ($L > 8K$), multi-turn conversation accumulation, edge deployment, speculative decoding / beam search. Core objective: **compress the KV-Cache from $O(Ld)$ to $O(kd)$ with $k \ll L$, under controlled information loss**.
+Use when KV-Cache memory consumption becomes the bottleneck during LLM inference: long-context inference ($L > 8K$), multi-turn conversation accumulation, edge deployment, speculative decoding / beam search. Core objective: **compress the KV-Cache from $O(Ld)$ to low-rank factor storage $O(Lk + kd)$ with $k \ll d$, under controlled information loss**.
 
 ## Mathematical Foundations
-- Lenses: lenses/spectral.md (spectral component identification and truncation), lenses/variational.md (Pareto trade-off between compression ratio and reconstruction error), lenses/duality.md (nuclear norm--spectral norm duality)
-- Knowledge: knowledge-base/matrix-analysis/low-rank-approximation.md (Eckart--Young optimal approximation, randomized SVD), knowledge-base/matrix-analysis/projection.md (orthogonal projection onto principal subspace), knowledge-base/matrix-analysis/matrix-perturbation.md (Weyl perturbation bound)
+- Lenses: ../../lenses/spectral.en.md (spectral component identification and truncation), ../../lenses/variational.en.md (Pareto trade-off between compression ratio and reconstruction error), ../../lenses/duality.en.md (nuclear norm--spectral norm duality)
+- Knowledge: ../../knowledge-base/matrix-analysis/low-rank-approximation.en.md (Eckart--Young optimal approximation, randomized SVD), ../../knowledge-base/matrix-analysis/projection.en.md (orthogonal projection onto principal subspace), ../../knowledge-base/matrix-analysis/matrix-perturbation.en.md (Weyl perturbation bound)
 
 ## Required Mathematical Background
 - **Eckart--Young--Mirsky Theorem**: $A_k = U_k \Sigma_k V_k^H$, $\|A - A_k\|_F = \sqrt{\sum_{i>k} \sigma_i^2}$; truncated SVD = optimal rank-$k$ approximation
 - **Randomized SVD**: $Y = A\Omega$ ($\Omega$ random Gaussian), $Y = QR$, $B = Q^H A$, perform SVD on $B$; complexity $O(Ldk)$, dominated by matrix multiplications
-- **Weyl Perturbation Bound**: $\|A - A_k\|_2 = \sigma_{k+1}$; maximum deviation of attention scores after compression $\leq \sigma_{k+1}$
+- **Eckart--Young spectral-norm error**: $\|A - A_k\|_2 = \sigma_{k+1}$ (bounds only the optimal rank-$k$ compression error of the matrix itself; Weyl is for singular-value changes under perturbation). Extending this to attention output error requires additional conditions: bounding $\|\text{Attn}(Q,K,V) - \text{Attn}(Q,K_k,V_k)\|$ requires (a) bounded $\|Q\|$, (b) the softmax Lipschitz constant (which depends on temperature and score range), and (c) compression errors for **both** K and V. Roughly: error $\lesssim C \cdot (\|Q\| \cdot \|K - K_k\| \cdot \|V\| + \|Q\| \cdot \|K_k\| \cdot \|V - V_k\|) / \tau$, where $C$ depends on the softmax Lipschitz constant and $\tau$ is the temperature parameter. Directly using $\sigma_{k+1}$ to bound attention score deviation **holds only under the simplifying assumptions of fixed Q and ignoring V compression error**.
 - **Effective Rank**: $r_{\text{eff}}(K) = \|K\|_F^2 / \|K\|_2^2$, guides adaptive selection of $k$
 
 ## AI Module Specification
@@ -21,11 +21,32 @@ Input: K ∈ R^{L×d}, V ∈ R^{L×d}    Parameters: target rank k << L, update
 
 Method 1 - Offline periodic compression (most practical):
   Omega = randn(d, k+p)                    // random projection, p=5 oversampling
-  Q_k = qr(K @ Omega)[0]                   // L×(k+p) orthonormal basis (GEMM + QR)
-  B_k = Q_k^T @ K                          // (k+p)×d small matrix GEMM
+  Q_base = qr(K @ Omega)[0]                // L×(k+p) oversampled orthonormal basis (GEMM + QR)
+  B_k = Q_base^T @ K                       // (k+p)×d small matrix GEMM
   U_r, S_r, Vt_r = svd(B_k)               // small matrix SVD
-  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d compressed Key
-  // Attention: softmax(Q @ K_comp^T / √d) @ V_comp, sequence dimension L → k
+  Q_final = Q_base @ U_r[:, :k]            // L×k final left factor
+  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d compressed Key (low-rank factor)
+  // V_comp definition: apply analogous truncated SVD to V, V_comp = Σ_k^{(V)} · Vt_k^{(V)} (k×d compressed Value)
+  // Alternatively, if K and V share the left factor Q_final, then V_comp = Q_final^T @ V (project onto the same low-rank subspace)
+  //
+  // ⚠ Critical distinction -- low-rank factors CANNOT directly replace the original sequence in softmax attention:
+  //   K ≈ Q_final @ K_comp (L×d reconstruction); softmax is a nonlinear operation, so
+  //   softmax(Q @ K_comp^T / √d) @ V_comp ≠ softmax(Q @ K^T / √d) @ V
+  //   The "k compressed tokens" interpretation is only valid for linear attention, not softmax attention.
+  //
+  // Mode A - Standard softmax attention (saves memory; may reduce inner dimension, not sequence length):
+  //   K_recon need not be materialized. If K ≈ Q_final @ K_comp, then
+  //   logits = (Q @ K_comp^T) @ Q_final^T / √d // for Q in R^{m×d}: O(mdk + mLk)
+  //   attn = softmax(logits)                   // softmax is still normalized over L positions
+  //   If V ≈ Q_final @ V_comp, output = (attn @ Q_final) @ V_comp
+  //   // Advantage: storage reduced from O(Ld) to O(Lk + kd); QK/AV inner dimension goes from d to k
+  //   // Limitation: the attention matrix is still m×L; the k factors are not k tokens
+  //
+  // Mode B - Linear attention (kernel feature map φ; compress additive statistics of φ(K), V):
+  //   Replace softmax with kernel feature map φ: Attn = φ(Q) @ (φ(K)^T @ V) / (φ(Q) @ φ(K)^T @ 1)
+  //   History can shrink from L tokens to k statistical factors only if the compressed objects are
+  //   additive statistics such as φ(K)^T V and φ(K)^T 1, or if low-rank/aggregation is performed
+  //   directly in φ(K) space. Applying φ directly to K_comp is not generally equivalent.
 
 Method 2 - Streaming incremental compression (low latency):
   Maintain basis (U_basis ∈ R^{k×d}), on new token arrival:
@@ -37,7 +58,7 @@ Method 3 - Layer-wise adaptive: allocate per-layer, per-head k based on effectiv
 ```
 
 ## Implementable Architectures
-- **Periodic compression layer**: trigger randomized SVD every $M=64$ steps, $L \times d \to k \times d$
+- **Periodic compression layer**: trigger randomized SVD every $M=64$ steps, $L \times d \to (L \times k) + (k \times d)$
 - **Double buffering**: compressed basis + recent $w$ raw tokens, balancing accuracy and compression ratio
 - **Shared basis**: multi-head sharing of the Key column-space basis, each head stores only coefficients
 - **Quantized basis**: further INT8/FP8 quantization after compression, achieving dual compression
@@ -45,13 +66,19 @@ Method 3 - Layer-wise adaptive: allocate per-layer, per-head k based on effectiv
 ## GPU Feasibility
 - Tensorization / GEMM: randomized SVD = 3 GEMMs + 1 small SVD, maps perfectly onto Tensor Cores
 - Complexity: $O(Ldk)$ is far superior to $O(Ld^2)$ full SVD; overhead negligible for $k \sim 256$
-- Memory: KV-Cache reduced from $O(Ld)$ to $O(kd)$; $k/L \sim 1/16$ yields 16x compression
+- Memory: Key-Cache stored in low-rank factor form ($Q_{\text{final}} \in \mathbb{R}^{L \times k}$ + $B_k \in \mathbb{R}^{k \times d}$), total parameters $Lk + kd$, compression ratio $Ld/(Lk+kd) \approx d/k$ (when $L \gg k$). Note $Q_{\text{final}}$ still has $L$ dimension, so sequence length is NOT reduced; softmax attention can compute length-$L$ logits with factorized GEMMs without materializing a full $L \times d$ reconstruction. V-Cache requires independent compression. End-to-end compression ratio depends on K/V storage format and rank selection
 - Low precision: SVD recommended in fp32 (acceptable for small matrices); compressed KV can be stored back in bf16
 - Parallelism: compression across layers / heads is fully independent; incremental update $O(kd)$ with very low latency
-- Operator fusion: $K\Omega$ + QR can be partially fused; the QK^T dimension in attention is already reduced
+- Operator fusion: $K\Omega$ + QR can be partially fused; in the softmax path, QK/AV inner dimension can drop from $d$ to $k$, but softmax is still normalized over length $L$; linear attention only shrinks history to $k$ when compressing additive statistics
+
+**Quantitative assessment example** (standard transformer, d=128, n=2048, rank k=64):
+- D3: SVD computation O(n·d·k) ≈ 2048·128·64 ≈ 16.8M FLOPs (one-time); softmax inference per query with factors is about O(dk + n·k), with a length-n softmax still required; linear attention removes n only when compressed statistics are used
+- D4: KV-Cache from O(n·d) = 2048·128·2B ≈ 512KB; factor format O(n·k + k·d) ≈ 2048·64·2B + 64·128·2B ≈ 278KB (compression ratio ~1.8x); final-left-factor + coefficient format ($Q_{\text{final}} \in \mathbb{R}^{n \times k}$ columns + $B_k \in \mathbb{R}^{k \times d}$) total parameters $nk + kd$, compression ratio $nd/(nk+kd) = d/k \cdot 1/(1+d/n) \approx d/k = 2x$, V-Cache requires independent compression
+- D5: Truncated SVD under bf16 amplifies singular value errors near σ_k by ~κ(A), requires caution
+- D8: SVD → matmul can be fused; online updates use incremental SVD to avoid full recomputation
 
 ## Paper-Worthy Formulation
-"Building on the Eckart--Young--Mirsky theorem, we employ randomized SVD to project the KV-Cache onto the optimal rank-$k$ subspace, achieving $L/k$-fold memory compression at $O(Ldk)$ complexity, with the Weyl perturbation bound guaranteeing that attention score deviations do not exceed $\sigma_{k+1}$."
+"Building on the Eckart--Young--Mirsky theorem, we employ randomized SVD to project the KV-Cache onto a rank-$k$ subspace, compressing storage from $O(Ld)$ to $O(Lk + kd)$ (basis+coefficient format) at $O(Ldk)$ complexity. For standard softmax attention, low-rank factors cannot be interpreted as $k$ compressed tokens; softmax is still normalized over length $L$, but factorized GEMMs can compute logits and value aggregation without materializing a full $L \times d$ reconstruction, reducing the QK/AV inner dimension from $d$ to $k$. Only in linear attention, and only when compressing additive statistics such as $\phi(K)^T V$ and $\phi(K)^T\mathbf{1}$, can the historical state truly shrink from $L$ tokens to $k$ statistical factors. The Eckart--Young spectral-norm error gives $\sigma_{k+1}$ as the optimal rank-$k$ compression error for the K/V matrices themselves; the end-to-end attention output error bound further depends on query norms, the softmax Lipschitz constant, and the temperature parameter. The actual memory compression ratio depends on storage format and the independent V-Cache compression strategy."
 
 ## Risks
 - **Improper rank selection**: $k$ too small causes $\sigma_{k+1}$ to be non-negligible, degrading long-range recall; singular value decay curves must be monitored

package/design-patterns/compression/low-rank-kv-cache.md

@@ -1,17 +1,17 @@
 # Low-Rank KV-Cache（低秩 KV 缓存压缩）
-> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
-当 LLM 推理的 KV-Cache 显存占用成为瓶颈时使用：长上下文推理（$L > 8K$）、多轮对话累积、边缘部署、投机解码/束搜索。核心诉求：**以可控信息损失将 KV-Cache 从 $O(Ld)$ 压缩到 $O(kd)$，$k \ll L$**。
+当 LLM 推理的 KV-Cache 显存占用成为瓶颈时使用：长上下文推理（$L > 8K$）、多轮对话累积、边缘部署、投机解码/束搜索。核心诉求：**以可控信息损失将 KV-Cache 从 $O(Ld)$ 压缩到低秩因子存储 $O(Lk + kd)$，$k \ll d$**。
 
 ## 数学思想来源
-- 透镜：lenses/spectral.md（谱分量识别与截断）、lenses/variational.md（压缩率与重构误差的帕累托取舍）、lenses/duality.md（核范数-谱范数对偶）
-- 知识：knowledge-base/matrix-analysis/low-rank-approximation.md（Eckart-Young 最优逼近、随机化 SVD）、knowledge-base/matrix-analysis/projection.md（正交投影到主子空间）、knowledge-base/matrix-analysis/matrix-perturbation.md（Weyl 扰动界）
+- 透镜：../../lenses/spectral.md（谱分量识别与截断）、../../lenses/variational.md（压缩率与重构误差的帕累托取舍）、../../lenses/duality.md（核范数-谱范数对偶）
+- 知识：../../knowledge-base/matrix-analysis/low-rank-approximation.md（Eckart-Young 最优逼近、随机化 SVD）、../../knowledge-base/matrix-analysis/projection.md（正交投影到主子空间）、../../knowledge-base/matrix-analysis/matrix-perturbation.md（Weyl 扰动界）
 
 ## 需要的数学知识
 - **Eckart-Young-Mirsky 定理**：$A_k = U_k \Sigma_k V_k^H$，$\|A - A_k\|_F = \sqrt{\sum_{i>k} \sigma_i^2}$，截断 SVD = 最优秩-$k$ 逼近
 - **随机化 SVD**：$Y = A\Omega$（$\Omega$ 随机高斯），$Y = QR$，$B = Q^H A$，对 $B$ 做 SVD；复杂度 $O(Ldk)$，核心全是 matmul
-- **Weyl 扰动界**：$\|A - A_k\|_2 = \sigma_{k+1}$，压缩后 attention score 最大偏差 $\leq \sigma_{k+1}$
+- **Eckart-Young 谱范数误差**：$\|A - A_k\|_2 = \sigma_{k+1}$（仅界定矩阵本身的最优秩-$k$ 压缩误差；Weyl 只用于分析扰动后的奇异值变化）。将其推广到 attention 输出误差需要额外条件：$\|\text{Attn}(Q,K,V) - \text{Attn}(Q,K_k,V_k)\|$ 的完整界需要 (a) $\|Q\|$ 有界、(b) softmax 的 Lipschitz 常数（依赖温度和 score 范围）、(c) K 和 V **两者**的压缩误差。粗略地：误差 $\lesssim C \cdot (\|Q\| \cdot \|K - K_k\| \cdot \|V\| + \|Q\| \cdot \|K_k\| \cdot \|V - V_k\|) / \tau$，其中 $C$ 取决于 softmax Lipschitz 常数，$\tau$ 为温度参数。直接用 $\sigma_{k+1}$ 界 attention score 偏差**仅在固定 Q 且忽略 V 压缩误差的简化假设下成立**。
 - **有效秩**：$r_{\text{eff}}(K) = \|K\|_F^2 / \|K\|_2^2$，指导 $k$ 的自适应选取
 
 ## AI 模块形式
@@ -21,11 +21,31 @@
 
 方法1 - 离线周期压缩（最实用）：
   Omega = randn(d, k+p)                    // 随机投影，p=5 过采样
-  Q_k = qr(K @ Omega)[0]                   // L×(k+p) 正交基底（GEMM + QR）
-  B_k = Q_k^T @ K                          // (k+p)×d 小矩阵 GEMM
+  Q_base = qr(K @ Omega)[0]                // L×(k+p) 过采样正交基底（GEMM + QR）
+  B_k = Q_base^T @ K                       // (k+p)×d 小矩阵 GEMM
   U_r, S_r, Vt_r = svd(B_k)               // 小矩阵 SVD
-  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d 压缩 Key
-  // Attention: softmax(Q @ K_comp^T / √d) @ V_comp，序列维度 L → k
+  Q_final = Q_base @ U_r[:, :k]            // L×k 最终左因子
+  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d 压缩 Key（低秩因子）
+  // V_comp 定义：对 V 做类似的截断 SVD，V_comp = Σ_k^{(V)} · Vt_k^{(V)}（k×d 压缩 Value）
+  // 或者，若 K 和 V 共享左因子 Q_final，则 V_comp = Q_final^T @ V（投影到同一低秩子空间）
+  //
+  // ⚠ 关键区分——低秩因子不能直接替代原始序列参与 softmax attention：
+  //   K ≈ Q_final @ K_comp（L×d 重构），softmax 是非线性操作，
+  //   softmax(Q @ K_comp^T / √d) @ V_comp ≠ softmax(Q @ K^T / √d) @ V
+  //   因此 k 个"压缩 token"的解释仅对线性注意力成立，对 softmax 注意力不成立。
+  //
+  // 模式 A - 标准 softmax attention（省显存；可省内积维度但不省序列长度）：
+  //   不必物化 K_recon。若 K ≈ Q_final @ K_comp，则
+  //   logits = (Q @ K_comp^T) @ Q_final^T / √d // Q 为 m×d 时复杂度 O(mdk + mLk)
+  //   attn = softmax(logits)                   // softmax 仍在 L 个位置上归一化
+  //   若 V ≈ Q_final @ V_comp，则 output = (attn @ Q_final) @ V_comp
+  //   // 优势：存储从 O(Ld) 降至 O(Lk + kd)，QK/AV 的内维从 d 降到 k
+  //   // 限制：注意力矩阵仍是 m×L，不能把 k 个因子当作 k 个 token
+  //
+  // 模式 B - 线性注意力（核特征映射 φ；需压缩 φ(K), V 的可加统计量）：
+  //   用核特征映射 φ 替代 softmax，Attn = φ(Q) @ (φ(K)^T @ V) / (φ(Q) @ φ(K)^T @ 1)
+  //   只有当压缩对象是 φ(K)^T V、φ(K)^T 1 等可加统计量，或直接在 φ(K) 空间做低秩/聚合时，
+  //   历史状态才能从 L 个 token 变成 k 个统计因子。直接对 K_comp 做 φ 一般不等价。
 
 方法2 - 流式增量压缩（低延迟）：
   维护基底 (U_basis ∈ R^{k×d})，新 token 到来：
@@ -37,7 +57,7 @@
 ```
 
 ## 可实现结构
-- **周期压缩层**：每 M=64 步触发随机化 SVD，$L \times d \to k \times d$
+- **周期压缩层**：每 M=64 步触发随机化 SVD，$L \times d \to (L \times k) + (k \times d)$
 - **双缓冲**：压缩基底 + 近期 $w$ 个原始 token，兼顾精度与压缩率
 - **共享基底**：多头共享 Key 列空间基底，各头只存系数
 - **量化基底**：压缩后进一步 INT8/FP8 量化，双重压缩
@@ -45,13 +65,19 @@
 ## GPU 可行性
 - 张量化/GEMM：随机化 SVD = 3 次 GEMM + 1 次小 SVD，完美映射 Tensor Core
 - 复杂度：$O(Ldk)$ 远优于 $O(Ld^2)$ 完整 SVD；$k \sim 256$ 时开销可忽略
-- 显存：KV-Cache 从 $O(Ld)$ 到 $O(kd)$，$k/L \sim 1/16$ 可达 16x 压缩
+- 显存：Key-Cache 以低秩因子形式存储（$Q_{\text{final}} \in \mathbb{R}^{L \times k}$ + $B_k \in \mathbb{R}^{k \times d}$），总参数 $Lk + kd$，压缩比 $Ld/(Lk+kd) \approx d/k$（当 $L \gg k$）。注意 $Q_{\text{final}}$ 仍有 $L$ 维，序列长度未缩短；softmax attention 可用因子化 GEMM 计算长度为 $L$ 的 logits，不必物化完整 $L \times d$ 矩阵。V-Cache 需独立做类似压缩。端到端压缩比取决于 K/V 两者的存储格式与秩选取
 - 低精度：SVD 建议 fp32（小矩阵可接受）；压缩后 KV 可回 bf16 存储
 - 并行：多层/多头压缩完全独立；增量更新 $O(kd)$ 极低延迟
-- 算子融合：$K\Omega$ + QR 可部分融合；attention 的 QK^T 维度已缩小
+- 算子融合：$K\Omega$ + QR 可部分融合；softmax 路径可把 QK/AV 的内维从 $d$ 降到 $k$，但 softmax 归一化长度仍为 $L$；线性注意力只有在压缩可加统计量时才真正把历史状态降到 $k$
+
+**量化评估示例**（标准 transformer, d=128, n=2048, rank k=64）：
+- D3: SVD 计算 O(n·d·k) ≈ 2048·128·64 ≈ 16.8M FLOPs（一次性）；softmax 推理每个 query 的因子化 QK/AV 约 O(dk + n·k)，仍需长度 n 的 softmax；线性注意力若压缩统计量才可消去 n
+- D4: KV-Cache 从 O(n·d) = 2048·128·2B ≈ 512KB；因子格式 O(n·k + k·d) ≈ 2048·64·2B + 64·128·2B ≈ 278KB（压缩比 ~1.8x）。最终左因子+系数格式（$Q_{\text{final}} \in \mathbb{R}^{n \times k}$ 正交基 + $B_k \in \mathbb{R}^{k \times d}$）总参数 $nk + kd$，压缩比 $nd/(nk+kd) = d/k \cdot 1/(1 + d/n) \approx d/k = 2x$，V-Cache 需独立压缩
+- D5: 截断 SVD 在 bf16 下 σ_k 附近奇异值误差放大 ~κ(A)，需注意
+- D8: SVD → matmul 可融合；在线更新用 incremental SVD 避免全量重算
 
 ## 论文表述方式
-"基于 Eckart-Young-Mirsky 定理，采用随机化 SVD 将 KV-Cache 投影到最优秩-$k$ 子空间，以 $O(Ldk)$ 复杂度实现 $L/k$ 倍显存压缩，Weyl 扰动界保证 attention score 偏差不超过 $\sigma_{k+1}$。"
+"基于 Eckart-Young-Mirsky 定理，采用随机化 SVD 将 KV-Cache 投影到秩-$k$ 子空间，以 $O(Ldk)$ 复杂度将存储从 $O(Ld)$ 压缩至 $O(Lk + kd)$（基底+系数格式）。对标准 softmax attention，低秩因子不能解释为 $k$ 个压缩 token；softmax 仍在长度 $L$ 上归一化，但可用因子化 GEMM 计算 logits 与 value 聚合，避免物化完整 $L \times d$ 重构并将 QK/AV 的内维从 $d$ 降到 $k$。仅在线性注意力中，且压缩的是 $\phi(K)^T V$、$\phi(K)^T\mathbf{1}$ 等可加统计量时，历史状态才可真正从 $L$ 降至 $k$ 个统计因子。Eckart-Young 的谱范数误差给出 K/V 矩阵本身的最优秩-$k$ 压缩误差 $\sigma_{k+1}$；attention 输出的端到端误差界进一步依赖 query 范数、softmax Lipschitz 常数和温度参数。实际显存压缩比取决于存储格式及 V-Cache 的独立压缩策略。"
 
 ## 风险
 - **秩选取不当**：$k$ 过小导致 $\sigma_{k+1}$ 不可忽略，长距离 recall 下降；需监控奇异值衰减曲线

package/design-patterns/compression/spectral-token-pruning.en.md

@@ -1,44 +1,46 @@
 # Spectral Token Pruning
-> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as ✅ verified / ⚠️ retrofittable (needs validation) / ❌ infeasible. Unmarked claims are theoretically possible but require engineering validation.
+> **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Target Problem
 Use when pruning must be based on the structural importance of tokens (rather than raw attention scores alone): KV-Cache eviction, long-document summarization, inference acceleration (reducing $O(L^2)$), multimodal vision token compression. Core objective: **quantify the structural importance of each token via spectral methods, achieving pruning with minimal information loss**.
 
 ## Mathematical Foundations
-- Lenses: lenses/spectral.md (identifying dominant spectral components, discarding redundant ones), lenses/algorithmic.md (complexity classification and approximation algorithms), lenses/perturbation.md (pruning = sparse perturbation, Weyl bound for spectral drift estimation)
-- Knowledge: knowledge-base/matrix-analysis/spectral-decomposition.md (spectral radius, eigenvector centrality), knowledge-base/matrix-analysis/matrix-perturbation.md (Geršgorin discs, perturbation bounds), knowledge-base/matrix-analysis/positive-semidefinite.md (Gram matrix PSD structure)
+- Lenses: ../../lenses/spectral.en.md (identifying dominant spectral components, discarding redundant ones), ../../lenses/algorithmic.en.md (complexity classification and approximation algorithms), ../../lenses/perturbation.en.md (pruning = sparse perturbation, Cauchy interlacing theorem / Bauer-Fike pseudospectral analysis for spectral drift bounds)
+- Knowledge: ../../knowledge-base/matrix-analysis/spectral-decomposition.en.md (spectral radius, eigenvector centrality), ../../knowledge-base/matrix-analysis/matrix-perturbation.en.md (Geršgorin discs, perturbation bounds), ../../knowledge-base/matrix-analysis/positive-semidefinite.en.md (Gram matrix PSD structure)
 
 ## Required Mathematical Background
-- **Eigenvector Centrality**: the principal eigenvector of the attention matrix $A$ satisfies $Ax = \lambda_1 x$; component $x_i$ quantifies the global influence of token $i$ (Perron--Frobenius guarantees non-negativity)
+- **Eigenvector Centrality**: For an unmasked, positive, irreducible row-stochastic attention matrix $A$, the right principal eigenvector $Ax = \lambda_1 x$ degenerates to the all-ones vector (since $A \mathbf{1} = \mathbf{1}$), making it useless for distinguishing token importance; the left principal eigenvector $x^T A = \lambda_1 x^T$ (equivalently $A^T x = \lambda_1 x$) can be used as a stationary-distribution / PageRank-like score. For causal or heavily masked attention, the chain is often reducible and the stationary distribution may collapse toward early tokens; use a K/V similarity graph, a symmetrized graph, or teleportation $A_\alpha=\alpha A+(1-\alpha)\mathbf{1}\pi^T$ before interpreting PageRank.
 - **Spectral Gap**: $\Delta = \lambda_1 - \lambda_2$ governs the rate of information diffusion; large $\Delta \Rightarrow$ a few tokens dominate $\Rightarrow$ safe to prune
 - **Fiedler Vector**: the second-smallest eigenvector of the Laplacian $L_{\text{sym}}$ yields the optimal bipartition; small magnitude = partition boundary = important
-- **Weyl Perturbation Bound**: pruning = removing rows/columns = rank-1 perturbation $E$, $|\lambda_i(A) - \lambda_i(A_{\text{pruned}})| \leq \|E\|_2$
+- **Spectral Perturbation Analysis**: Pruning changes the matrix dimension and cannot directly apply the Weyl theorem. For a principal submatrix of a fixed Hermitian matrix (e.g., a symmetrized matrix $S=(A+A^T)/2$ without re-normalization), Cauchy interlacing bounds eigenvalue interlacing. If the Laplacian / attention graph is re-normalized after pruning, the matrix itself has changed, so this bound no longer applies directly. For non-symmetric row-stochastic matrices, spectral radius perturbation can be bounded via Bauer--Fike or pseudospectral analysis, though the bounds are less tight than in the Hermitian case.
 
 ## AI Module Specification
 ```
 Module: SpectralTokenPruner
 Input: K ∈ R^{L×d}    Parameters: retention ratio ρ ∈ (0,1]
 
-Method 1 - Spectral centrality pruning (power iteration):
-  A = softmax(K @ K^T / √d)                  // L×L similarity graph
+Method 1 - Spectral centrality pruning (power iteration, left eigenvector):
+  A = softmax(K @ K^T / √d)                  // L×L row-stochastic similarity graph; handle causal/masked cases separately
+  // ⚠ A is row-stochastic: right principal eigenvector = all-ones (degenerate); must use left eigenvector
+  // Left principal eigenvector = right eigenvector of A^T; PageRank interpretation is stable only for irreducible/teleported chains
   v = ones(L) / √L
-  for t in range(5): v = A @ v; v = v / ‖v‖  // power iteration O(L²·T)
-  indices = topk(v, ceil(ρ * L))              // retain tokens with highest centrality
+  for t in range(5): v = A^T @ v; v = v / ‖v‖  // power iteration on A^T (not A), O(L²·T)
+  indices = topk(v, ceil(ρ * L))              // retain tokens with highest left-eigenvector centrality
 
 Method 2 - Geršgorin cheap pruning (zero iterations):
-  A = softmax(K @ K^T / √d)
-  gersh_score = |diag(A)| + sum(|A|, dim=1)    // disc upper bound, O(L²) elementwise
-  indices = topk(gersh_score, ceil(ρ * L))
+  S = K @ K^T / √d                         // L×L raw similarity matrix (pre-softmax, so row sums vary)
+  gersh_score = sum(|S|, dim=1) - |diag(S)|  // Geršgorin disc radius R_i = sum_{j!=i}|S_{ij}|, measures connectivity
+  indices = topk(gersh_score, ceil(ρ * L))    // O(L²) elementwise, no power iteration needed
 
 Method 3 - Differentiable spectral pruning (end-to-end):
-  v = power_iteration(A, T=5)
+  v = power_iteration(A^T, T=5)               // left principal eigenvector (power iteration on A^T)
   gate = sigmoid(v @ W_gate / τ)               // soft gating, τ annealing
   K_gated = gate * K                            // per-token scaling
   L_sparse = ‖gate‖_1 / L                       // sparsity regularization
 ```
 
 ## Implementable Architectures
-- **Power iteration centrality**: 5-step matvec to estimate the principal eigenvector, $O(L^2 \cdot 5)$, suitable for moderate-length sequences
+- **Power iteration centrality**: 5-step matvec to estimate the principal eigenvector of $A^T$ (i.e., the left principal eigenvector / stationary distribution of $A$), $O(L^2 \cdot 5)$, suitable for moderate-length sequences
 - **Sampling approximation**: for $L > 4096$, sample $m$ anchor points and construct an $m \times m$ submatrix for spectral analysis
 - **Multi-head fusion**: average the attention graphs across different heads before performing spectral analysis
 - **Progressive pruning**: incrementally increase the pruning ratio across layers (light pruning in shallow layers, heavy pruning in deep layers)
@@ -52,7 +54,7 @@ Method 3 - Differentiable spectral pruning (end-to-end):
 - Operator fusion: $KK^T$ + row-sum + topk can be fused into a single kernel
 
 ## Paper-Worthy Formulation
-"We cast token pruning as spectral sparsification of a directed graph: leveraging the Perron--Frobenius principal eigenvector of the attention matrix to quantify global centrality, with the Weyl perturbation bound guaranteeing that post-pruning spectral drift does not exceed the $\ell_2$ norm of the removed tokens, while Geršgorin discs provide an $O(L^2)$ inexpensive alternative."
+"We cast token pruning as spectral sparsification of a directed graph: on unmasked irreducible row-stochastic graphs, left-eigenvector / PageRank-like centrality can quantify global token importance; for causal or heavily masked attention, use a K/V similarity graph, a symmetrized graph, or teleported PageRank to avoid stationary-mass collapse toward early tokens. Cauchy interlacing applies to principal submatrices of fixed Hermitian matrices; after re-normalization or for non-symmetric graphs, use weaker but applicable diagnostics such as pseudospectral analysis, Bauer--Fike, or Geršgorin discs."
 
 ## Risks
 - **$L \times L$ matrix memory bottleneck**: for long sequences the similarity matrix itself may exceed available memory, necessitating sampling or chunking

package/design-patterns/compression/spectral-token-pruning.md

@@ -1,44 +1,46 @@
 # Spectral Token Pruning（谱 Token 剪枝）
-> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「✅ 已验证 / ⚠️ 可改造需验证 / ❌ 不可行」。未标注的视为理论可行，需工程验证。
+> **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
 当需要基于 token 的结构性重要性（而非单纯 attention score）进行剪枝时使用：KV-Cache 驱逐、长文档摘要、推理加速（降低 $O(L^2)$）、多模态视觉 token 压缩。核心诉求：**用谱方法量化每个 token 的结构性重要性，实现信息损失最小的剪枝**。
 
 ## 数学思想来源
-- 透镜：lenses/spectral.md（识别主导谱分量、丢弃冗余分量）、lenses/algorithmic.md（复杂度分类与近似算法）、lenses/perturbation.md（剪枝 = 稀疏扰动，Weyl 界估谱漂移）
-- 知识：knowledge-base/matrix-analysis/spectral-decomposition.md（谱半径、特征向量中心性）、knowledge-base/matrix-analysis/matrix-perturbation.md（Geršgorin 圆盘、扰动界）、knowledge-base/matrix-analysis/positive-semidefinite.md（Gram 矩阵 PSD 结构）
+- 透镜：../../lenses/spectral.md（识别主导谱分量、丢弃冗余分量）、../../lenses/algorithmic.md（复杂度分类与近似算法）、../../lenses/perturbation.md（剪枝 = 稀疏扰动，Cauchy 交错定理 / Bauer-Fike 伪谱分析界定谱漂移）
+- 知识：../../knowledge-base/matrix-analysis/spectral-decomposition.md（谱半径、特征向量中心性）、../../knowledge-base/matrix-analysis/matrix-perturbation.md（Geršgorin 圆盘、扰动界）、../../knowledge-base/matrix-analysis/positive-semidefinite.md（Gram 矩阵 PSD 结构）
 
 ## 需要的数学知识
-- **特征向量中心性**：attention 矩阵 $A$ 的主特征向量 $Ax = \lambda_1 x$，分量 $x_i$ 量化 token $i$ 的全局影响力（Perron-Frobenius 保证非负）
+- **特征向量中心性**：对未 mask、正且不可约的行随机（行 softmax）attention 矩阵 $A$，右主特征向量 $Ax = \lambda_1 x$ 退化为全 1 向量（因 $A \mathbf{1} = \mathbf{1}$），无法区分 token 重要性；可使用左主特征向量 $x^T A = \lambda_1 x^T$（等价于 $A^T x = \lambda_1 x$）作为平稳分布/PageRank 式重要性。对 causal 或强 mask attention，链通常非不可约，平稳分布可能偏向早期 token；应改用 K/V 相似度图、对称化图，或加入 teleportation $A_\alpha=\alpha A+(1-\alpha)\mathbf{1}\pi^T$ 后再解释 PageRank。
 - **谱间隙**：$\Delta = \lambda_1 - \lambda_2$ 决定信息扩散速度，$\Delta$ 大 $\Rightarrow$ 少数 token 主导 $\Rightarrow$ 安全剪枝
 - **Fiedler 向量**：Laplacian $L_{\text{sym}}$ 的第二小特征向量给出最优二分割，幅值小 = 分割边界 = 重要
-- **Weyl 扰动界**：剪枝 = 删除行/列 = 秩-1 扰动 $E$，$|\lambda_i(A) - \lambda_i(A_{\text{pruned}})| \leq \|E\|_2$
+- **谱扰动分析**：剪枝改变矩阵维度，不能直接套用 Weyl 定理。对固定 Hermitian 矩阵的主子矩阵（如不重新归一化的对称化矩阵 $S=(A+A^T)/2$），可用 Cauchy 交错定理界定特征值交错；若剪枝后重新归一化 Laplacian/attention graph，矩阵本身已改变，不能直接套该界。对非对称行随机矩阵，谱半径扰动可用 Bauer-Fike 或伪谱分析，但界不如 Hermitian 情形紧凑。
 
 ## AI 模块形式
 ```
 模块：SpectralTokenPruner
 输入：K ∈ R^{L×d}    参数：保留比例 ρ ∈ (0,1]
 
-方法1 - 谱中心性剪枝（幂迭代）：
-  A = softmax(K @ K^T / √d)                  // L×L 相似度图
+方法1 - 谱中心性剪枝（幂迭代，左特征向量）：
+  A = softmax(K @ K^T / √d)                  // L×L 行随机相似度图；causal/mask 场景需另行处理
+  // ⚠ A 为行随机矩阵，右主特征向量 = 全 1 向量（退化），必须用左主特征向量
+  // 左主特征向量 = A^T 的右主特征向量；仅在链不可约/加 teleportation 后可稳定解释为 PageRank
   v = ones(L) / √L
-  for t in range(5): v = A @ v; v = v / ‖v‖  // 幂迭代 O(L²·T)
-  indices = topk(v, ceil(ρ * L))              // 保留中心性最高的 token
+  for t in range(5): v = A^T @ v; v = v / ‖v‖  // 幂迭代 A^T（非 A），O(L²·T)
+  indices = topk(v, ceil(ρ * L))              // 保留左特征向量中心性最高的 token
 
 方法2 - Geršgorin 廉价剪枝（零迭代）：
-  A = softmax(K @ K^T / √d)
-  gersh_score = |diag(A)| + sum(|A|, dim=1)    // 圆盘上界，O(L²) elementwise
-  indices = topk(gersh_score, ceil(ρ * L))
+  S = K @ K^T / √d                         // L×L 原始相似度矩阵（非 softmax，行使行和有区分度）
+  gersh_score = sum(|S|, dim=1) - |diag(S)|  // Geršgorin 圆盘半径 R_i = Σ_{j≠i}|S_{ij}|，衡量连通度
+  indices = topk(gersh_score, ceil(ρ * L))    // O(L²) elementwise，无需幂迭代
 
 方法3 - 可微谱剪枝（端到端）：
-  v = power_iteration(A, T=5)
+  v = power_iteration(A^T, T=5)               // 左主特征向量（A^T 的幂迭代）
   gate = sigmoid(v @ W_gate / τ)               // 软门控，τ 退火
   K_gated = gate * K                            // 逐 token 缩放
   L_sparse = ‖gate‖_1 / L                       // 稀疏正则
 ```
 
 ## 可实现结构
-- **幂迭代中心性**：5 步 matvec 估计主特征向量，$O(L^2 \cdot 5)$，适合中等序列
+- **幂迭代中心性**：5 步 matvec 估计 $A^T$ 的主特征向量（即 $A$ 的左主特征向量 / 平稳分布），$O(L^2 \cdot 5)$，适合中等序列
 - **采样近似**：$L > 4096$ 时采样 $m$ 个锚点，构造 $m \times m$ 子矩阵做谱分析
 - **多头融合**：不同头的 attention 图取平均后再做谱分析
 - **渐进式剪枝**：逐层递增剪枝比例（浅层少剪、深层多剪）
@@ -52,7 +54,7 @@
 - 算子融合：$KK^T$ + row-sum + topk 可融合为单一 kernel
 
 ## 论文表述方式
-"将 token 剪枝建模为有向图的谱稀疏化：利用 attention 矩阵的 Perron-Frobenius 主特征向量量化全局中心性，Weyl 扰动界保证剪枝后谱漂移不超过被移除 token 的 $\ell_2$ 范数，Geršgorin 圆盘提供 $O(L^2)$ 廉价替代。"
+"将 token 剪枝建模为有向图的谱稀疏化：在未 mask 且不可约的行随机图上，利用左主特征向量/PageRank 式中心性量化全局重要性；对 causal 或强 mask attention，则改用 K/V 相似度图、对称化图或 teleportation 后的 PageRank，避免平稳分布退化到早期 token。对固定 Hermitian 子矩阵可用 Cauchy 交错分析谱漂移；重新归一化或非对称图则需改用伪谱、Bauer-Fike 或 Geršgorin 圆盘等更弱但适用的扰动诊断。"
 
 ## 风险
 - **$L \times L$ 矩阵显存瓶颈**：长序列下相似度矩阵本身可能超出显存，必须采样或分块