math-skill 3.0.1 → 3.1.1

package/design-patterns/attention/information-bottleneck-attention.en.md

@@ -6,7 +6,7 @@ When the attention mechanism needs to **selectively transmit useful information
 
 ## Mathematical Inspiration
 - Lenses: [categorical (information-theoretic framework unifying attention design), variational (constrained variational and Lagrangian duality)]
-- Knowledge: [`../../knowledge-base/probability/information-bottleneck.md` (IB objective and variational lower bound), `../../knowledge-base/probability/kl-divergence.md` (implementation of KL regularization), `../../knowledge-base/probability/entropy.md` (mutual information estimation)]
+- Knowledge: [`../../knowledge-base/probability/information-bottleneck.en.md` (IB objective and variational lower bound), `../../knowledge-base/probability/kl-divergence.en.md` (implementation of KL regularization), `../../knowledge-base/probability/entropy.en.md` (mutual information estimation)]
 
 ## Required Mathematical Knowledge
 - **Information Bottleneck Objective**: $\min I(X;Z) - \beta I(Z;Y)$, balancing compression and prediction
@@ -19,27 +19,47 @@ When the attention mechanism needs to **selectively transmit useful information
 
 **Scheme A: KL-Regularized Attention (Simplest IB Attention)**:
 ```python
-# KL divergence between attention distribution q(z|x) and uniform prior as IB regularizer
+# ⚠ Note the semantic direction of KL:
+# KL(attn || uniform) = log(n) - H(attn)
+# Minimizing +beta * KL(attn || uniform) = maximizing H(attn) → pushes toward uniform (max entropy / max compression)
+# This does NOT induce sparsity by itself! Sparsity emerges from the tension with task_loss.
+# For explicit sparsity induction, use an entropy penalty +beta * H(attn) (minimize entropy).
+
 scores = (Q @ K.T) / sqrt(d)
 attn = softmax(scores)
 uniform_prior = ones_like(attn) / n
-kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
-loss = task_loss + beta * kl_reg
+
+# Correct IB compression term: KL(q(z|x) || p(z)), where p(z) is a fixed prior
+# Minimizing this term → compresses information (pushes toward prior); tension with task_loss yields useful attention
+kl_compression = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+# = log(n) - H(attn)
+
+# IB objective: min task_loss + beta * I(X;Z), where I(X;Z) ≈ KL(attn || prior)
+loss_ib = task_loss + beta * kl_compression
+# Note: this term alone pushes toward uniform; sparsity arises from the counter-pull of task_loss
+
+# Alternative: explicit sparsity induction (not IB compression, but entropy penalty)
+entropy = -(attn * log(attn + eps)).sum(dim=-1).mean()
+loss_sparse = task_loss + beta * entropy  # minimize entropy → concentrated attention → implicit Top-K
+
 output = attn @ V
 ```
 
 **Scheme B: Variational Information Bottleneck Attention (VIB-Attention)**:
 ```python
-# Introduce stochastic bottleneck Z ~ q(Z|attention) to limit information throughput
+# Introduce a continuous stochastic bottleneck Z ~ q(Z|context) after value aggregation
 scores = (Q @ K.T) / sqrt(d)
 attn = softmax(scores)
-mu_z, log_var_z = attn, linear(attn)  # mean and learnable variance
-z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # reparameterization + softmax
-kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs prior
-output = z @ V
+context = attn @ V
+mu_z, log_var_z = linear_mu(context), linear_logvar(context)
+z = mu_z + exp(0.5 * log_var_z) * randn_like(mu_z)  # Gaussian reparameterization
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL[q(z|context)||N(0,I)]
+output = linear_out(z)
 loss = task_loss + beta * kl
 ```
 
+If the bottleneck is applied directly on the attention simplex (e.g., noisy logits followed by softmax), the random variable is logistic-normal / Concrete-like. The closed-form Gaussian KL above no longer applies; use Monte Carlo KL, a Concrete KL approximation, or place the Gaussian KL on pre-softmax logits.
+
 **Scheme C: Mutual Information Maximization Attention (DIM Style)**:
 ```python
 # Directly maximize I(Z;Y) via InfoNCE, with KL constraining I(X;Z)
@@ -50,7 +70,7 @@ loss = -info_nce + beta * kl_bottleneck
 ```
 
 ## Implementable Architectures
-- **IB-Sparse Attention**: KL regularization serves as a soft constraint on attention sparsity -- larger KL relative to uniform prior leads to more concentrated attention, yielding implicit Top-K selection
+- **IB-Sparse Attention**: The IB compression term KL(attn || uniform) by itself pushes toward uniform (maximum entropy), but its tension with task_loss forces the attention to balance between "compressing all information" and "selectively transmitting useful information," implicitly producing non-uniform attention. For **explicit** sparsity induction (Top-K selection), use an entropy penalty `+beta * H(attn)` (minimize attention entropy → concentration), rather than relying on the IB compression term alone
 - **IB Interpretation of Dropout**: Dropout is a form of stochastic information bottleneck -- randomly blocking information channels forces the model to learn robust representations. The dropout rate corresponds to the $\beta$ parameter
 - **Multi-Head Information Allocation**: Different heads learn different information bottlenecks (different $\beta$); some heads transmit global information, others only local information
 
@@ -61,11 +81,11 @@ loss = -info_nce + beta * kl_bottleneck
 - **D4**: VIB requires additional $\mu_z$ and $\log\sigma_z$, approximately doubling attention weight memory
 - **D5**: log/exp in KL computations are stable under bf16 (standard log-softmax tricks)
 - **D6**: Regularization can be computed in parallel with forward propagation, introducing no serial dependencies
-- **D7**: KL regularization implicitly induces attention sparsity, amenable to block-sparse acceleration
+- **D7[~]**: KL regularization by itself does not induce sparsity (pushes toward uniform distribution); for sparsity, use an entropy penalty +beta * H(attn), which then enables block-sparse acceleration
 - **D8**: KL can be fused into the softmax kernel (FusedSoftmaxKL)
 
 ## Paper Phrasing
-"We propose information bottleneck attention, which models attention as an information bottleneck variational problem. By maximizing the mutual information between output and target while minimizing redundant information transmission from the input, it achieves theoretically optimal information selection and demonstrates enhanced sparsity and interpretability in experiments."
+"We propose information bottleneck attention, which models attention as an information bottleneck optimization problem. By maximizing the mutual information between output and target while constraining redundant information transmission from the input, the IB compression term by itself pushes toward a uniform distribution (maximum entropy), and its tension with the task loss produces non-uniform attention; explicit sparsity induction requires an additional entropy penalty term."
 
 ## Risks
 - **High Variance of Mutual Information Estimators**: The MINE/NWJ/InfoNCE estimators in Scheme C exhibit high variance in high-dimensional spaces, potentially causing training instability. It is recommended to first validate the basic effect of IB attention with Scheme A (KL regularization) before attempting the full IB objective.

package/design-patterns/attention/information-bottleneck-attention.md

@@ -19,27 +19,47 @@
 
 **方案 A：KL 正则化注意力（最简 IB 注意力）**：
 ```python
-# 注意力分布 q(z|x) 与均匀先验的 KL 散度作为 IB 正则
+# ⚠ 注意 KL 方向的语义：
+# KL(attn || uniform) = log(n) - H(attn)
+# 最小化 +beta * KL(attn || uniform) = 最大化 H(attn) → 推向均匀分布（最大熵/最大压缩）
+# 这本身不诱导稀疏性！稀疏性来自压缩项与 task_loss 的张力。
+# 若需要显式诱导稀疏注意力，应使用熵惩罚 +beta * H(attn)（最小化熵）。
+
 scores = (Q @ K.T) / sqrt(d)
 attn = softmax(scores)
 uniform_prior = ones_like(attn) / n
-kl_reg = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
-loss = task_loss + beta * kl_reg
+
+# 正确的 IB 压缩项：KL(q(z|x) || p(z))，p(z) 为固定先验
+# 最小化此项 → 压缩信息（推向先验），与 task_loss 的张力产生有用注意力
+kl_compression = (attn * (log(attn + eps) - log(uniform_prior))).sum(dim=-1).mean()
+# = log(n) - H(attn)
+
+# IB 目标：min task_loss + beta * I(X;Z)，其中 I(X;Z) ≈ KL(attn || prior)
+loss_ib = task_loss + beta * kl_compression
+# 注意：此项单独作用会推向均匀；稀疏性来自 task_loss 的反向拉力
+
+# 替代方案：显式稀疏性诱导（非 IB 压缩，而是熵惩罚）
+entropy = -(attn * log(attn + eps)).sum(dim=-1).mean()
+loss_sparse = task_loss + beta * entropy  # 最小化熵 → 注意力集中 → 隐式 Top-K
+
 output = attn @ V
 ```
 
 **方案 B：变分信息瓶颈注意力（VIB-Attention）**：
 ```python
-# 引入随机瓶颈 Z ~ q(Z|attention)，限制信息通过量
+# 引入随机连续瓶颈 Z ~ q(Z|context)，限制 value 聚合后的信息通过量
 scores = (Q @ K.T) / sqrt(d)
 attn = softmax(scores)
-mu_z, log_var_z = attn, linear(attn)  # 均值与可学习方差
-z = softmax(mu_z + exp(0.5 * log_var_z) * randn_like(mu_z))  # 重参数化 + softmax
-kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL vs 先验
-output = z @ V
+context = attn @ V
+mu_z, log_var_z = linear_mu(context), linear_logvar(context)
+z = mu_z + exp(0.5 * log_var_z) * randn_like(mu_z)  # 高斯重参数化
+kl = 0.5 * (mu_z^2 + exp(log_var_z) - log_var_z - 1).sum(-1).mean()  # KL[q(z|context)||N(0,I)]
+output = linear_out(z)
 loss = task_loss + beta * kl
 ```
 
+若瓶颈直接作用在 attention simplex 上（如对 logits 加噪后 softmax），变量分布是 logistic-normal / Concrete 类，不能再使用普通高斯到标准正态的闭式 KL；应使用 Monte Carlo KL、Concrete KL 近似，或把 KL 放在 softmax 前的 logits 高斯变量上。
+
 **方案 C：互信息最大化注意力（DIM 风格）**：
 ```python
 # 用 InfoNCE 直接最大化 I(Z;Y)，配合 KL 约束 I(X;Z)
@@ -50,7 +70,7 @@ loss = -info_nce + beta * kl_bottleneck
 ```
 
 ## 可实现结构
-- **IB-Sparse Attention**：将 KL 正则化为注意力稀疏性的软约束——KL 对均匀先越大 → 注意力越集中 → 隐式 Top-K 选择
+- **IB-Sparse Attention**：IB 压缩项 KL(attn || uniform) 本身推向均匀分布（最大熵），但与 task_loss 的张力迫使注意力在"压缩所有信息"与"选择性传递有用信息"之间取舍，从而隐式产生非均匀注意力。若要**显式**诱导稀疏（Top-K 选择），应使用熵惩罚 `+beta * H(attn)`（最小化注意力熵 → 集中化），而非依赖 IB 压缩项
 - **Dropout 的 IB 解释**：Dropout 是一种随机信息瓶颈——随机阻断部分信息通道，迫使模型学习鲁棒表示。Dropout rate 对应 $\beta$ 参数
 - **Multi-Head 信息分配**：不同 head 学习不同的信息瓶颈（不同 $\beta$），有的 head 传递全局信息，有的只传递局部信息
 
@@ -61,12 +81,12 @@ loss = -info_nce + beta * kl_bottleneck
 - **D4[~]**：VIB 需额外 $\mu_z$ 和 $\log\sigma_z$，约 2x 注意力权重显存
 - **D5[v]**：KL 中 log/exp 在 bf16 下稳定（标准 log-softmax 技巧）
 - **D6[v]**：正则项可与前向传播并行，不引入串行依赖
-- **D7[v]**：KL 正则隐式诱导注意力稀疏性，可利用 block-sparse 加速
+- **D7[~]**：KL 正则本身不诱导稀疏性（推向均匀分布）；若需稀疏，应改用熵惩罚 +beta * H(attn)，此时可 block-sparse 加速
 - **D8[v]**：KL 可融入 softmax kernel（FusedSoftmaxKL）
 
 ## 论文表述方式
-"我们提出信息瓶颈注意力机制，通过将注意力建模为信息瓶颈优化问题，在最大化输出与目标的互信息的同时最小化输入信息的冗余传递，实现理论上最优的信息选择，并在实验中展现出更强的稀疏性和可解释性。"
+"我们提出信息瓶颈注意力机制，通过将注意力建模为信息瓶颈优化问题，在最大化输出与目标的互信息的同时约束输入信息的冗余传递。IB 压缩项本身推向均匀分布（最大熵），与任务损失的张力产生非均匀注意力；若要显式诱导稀疏性，需额外使用熵惩罚项。"
 
 ## 风险
 - **互信息估计的方差问题**：方案 C 中的 MINE/NWJ/InfoNCE 估计器在高维空间中方差大，可能导致训练不稳定。建议先用方案 A（KL 正则）验证 IB 注意力的基本效果，再尝试完整 IB 目标。
-- **$\beta$ 调参困难**：$\beta$ 控制压缩-预测权衡，不同任务最优 $\beta$ 差异大。$\beta$ 过小退化为标准注意力，$\beta$ 过大导致欠拟合。建议自适应 $\beta$ 调度或信息平面监控。
+- **$\beta$ 调参困难**：$\beta$ 控制压缩-预测权衡，不同任务最优 $\beta$ 差异大。$\beta$ 过小退化为标准注意力，$\beta$ 过大导致欠拟合。建议自适应 $\beta$ 调度或信息平面监控。

package/design-patterns/attention/projection-attention.en.md

@@ -6,7 +6,7 @@ When the token/key space dimensionality is too high, the standard attention $Q K
 
 ## Mathematical Inspiration
 - Lenses: [projection, spectral, probabilistic]
-- Knowledge: [`../../knowledge-base/probability/concentration-inequality.md` (high-dimensional concentration inequalities provide theoretical explanation for attention collapse), `../../knowledge-base/probability/entropy.md` (entropy of attention distributions as a quality metric)]
+- Knowledge: [`../../knowledge-base/probability/concentration-inequality.en.md` (high-dimensional concentration inequalities provide theoretical explanation for attention collapse), `../../knowledge-base/probability/entropy.en.md` (entropy of attention distributions as a quality metric)]
 
 ## Required Mathematical Knowledge
 - **Johnson-Lindenstrauss Lemma**: High-dimensional point sets can be projected onto an $O(\log n / \epsilon^2)$-dimensional subspace while approximately preserving distances
@@ -31,7 +31,7 @@ scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
 attn = softmax(scores) @ V
 ```
 
-2. **Random Fixed Projection** (guaranteed by the JL Lemma):
+2. **Random Fixed Projection** (distance-preservation bound when JL conditions hold):
 ```
 P = random_gaussian(d, r) / sqrt(r)  # fixed, not trained
 scores = (Q @ P) @ (K @ P).T / sqrt(r)
@@ -61,7 +61,7 @@ scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
 - **D8[~] Retrofittable, needs kernel-level validation**: Projection may be integrated into a FlashAttention-style kernel, but requires kernel-level verification
 
 ## Paper Phrasing
-"We decompose attention computation into two steps -- low-dimensional subspace projection and projected-space attention -- achieving a $d/r$-fold Key-cache compression (full KV-Cache compression requires separate V compression/reconstruction) while preserving attention quality and guaranteeing Johnson-Lindenstrauss distance preservation."
+"We decompose attention computation into low-dimensional subspace projection and projected-space attention to reduce Key dimensionality; V-cache must be handled separately. With an independent random projection and the sample-size / target-dimension conditions of the Johnson-Lindenstrauss lemma, Euclidean distances of the projected objects can be approximately preserved. This does not automatically guarantee softmax-attention quality, so attention/output error and task metrics must be reported."
 
 ## Risks
 - **Projection Direction Degeneracy**: Learnable projections may collapse onto a few directions (deterioration of the condition number of $P^T P$), causing attention distributions to degenerate. Orthogonal regularization $\|P^T P - I\|_F^2$ is required.

package/design-patterns/attention/projection-attention.md

@@ -31,7 +31,7 @@ scores = (Q @ P_Q) @ (K @ P_K).T / sqrt(r)
 attn = softmax(scores) @ V
 ```
 
-2. **随机固定投影**（JL 引理保证）：
+2. **随机固定投影**（满足 JL 条件时可给距离保持界）：
 ```
 P = random_gaussian(d, r) / sqrt(r)  # 固定不训练
 scores = (Q @ P) @ (K @ P).T / sqrt(r)
@@ -61,7 +61,7 @@ scores = (Q @ eigenvecs) @ (K @ eigenvecs).T / sqrt(r)
 - **D8[~]可改造需验证**：投影可融入 FlashAttention 的 online softmax 循环中
 
 ## 论文表述方式
-"我们将注意力计算分解为低维子空间投影与投影空间注意力两步，在保持注意力质量的同时将 Key-cache 压缩 $d/r$ 倍（V-cache 需独立处理），并保证 Johnson-Lindenstrauss 距离保持性。"
+"我们将注意力计算分解为低维子空间投影与投影空间注意力两步，以降低 Key 表示维度；V-cache 需独立处理。若使用独立随机投影并满足 Johnson-Lindenstrauss 的样本规模与维度条件，可近似保持投影对象的欧氏距离；这不自动保证 softmax attention 质量，需报告 attention/output 误差与任务指标。"
 
 ## 风险
 - **投影方向退化**：可学习投影可能坍缩到少数方向（$P^T P$ 条件数恶化），导致注意力分布退化。需加正交正则化 $\|P^T P - I\|_F^2$。

package/design-patterns/attention/spectral-attention.en.md

@@ -2,21 +2,23 @@
 > **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Applicable Problems
-When the input signal exhibits **frequency-domain/spectral structure** (periodicity, cyclic symmetry, graph structure), computing attention in the spectral domain rather than the spatial domain can dramatically reduce complexity while exploiting the signal's intrinsic structure. Typical scenarios include: time series forecasting (periodic signals), graph neural networks (graph Laplacian spectral decomposition), positional encoding (frequency-domain interpretation of RoPE/ALiBi), and $O(n \log n)$ acceleration of long-sequence attention.
+When the input signal exhibits **frequency-domain/spectral structure** (periodicity, cyclic symmetry, graph structure), computing attention in the spectral domain rather than the spatial domain can dramatically reduce complexity while exploiting the signal's intrinsic structure. Typical scenarios include: time series forecasting (periodic signals), graph neural networks (graph Laplacian spectral decomposition), positional encoding (frequency-domain interpretation of RoPE/ALiBi), and $O(n \log n)$ acceleration of long-sequence **position-dependent** attention (requires circulant/Toeplitz structure assumption; does not apply to general content-dependent softmax attention).
 
 ## Mathematical Inspiration
 - Lenses: [duality (frequency-domain transform), symmetry (cyclic/translation invariance)]
-- Knowledge: [`../../knowledge-base/probability/entropy.md` (spectral entropy for measuring signal complexity), `../../knowledge-base/probability/concentration-inequality.md` (frequency-domain concentration inequalities)]
+- Knowledge: [`../../knowledge-base/probability/entropy.en.md` (spectral entropy for measuring signal complexity), `../../knowledge-base/probability/concentration-inequality.en.md` (frequency-domain concentration inequalities)]
 
 ## Required Mathematical Knowledge
 - **Discrete Fourier Transform (DFT/FFT)**: $O(n \log n)$ frequency-domain transform and the cyclic convolution theorem
 - **Graph Laplacian Spectral Decomposition**: $L = U \Lambda U^T$, where $U$ is the graph Fourier basis
-- **Irreducible Representations of the Cyclic Group $\mathbb{Z}_n$**: The DFT matrix is precisely the representation matrix of the cyclic group (see `references/books/abstract-algebra.md` Ch.4, Ch.11)
+- **Irreducible Representations of the Cyclic Group $\mathbb{Z}_n$**: The DFT matrix is precisely the representation matrix of the cyclic group (see `../../references/books/abstract-algebra.en.md` Ch.4, Ch.11)
 
 ## AI Module Form
 
 **Core Idea**: Transform attention from the spatial-domain $Q K^T$ into diagonal/sparse operations in the spectral domain:
 
+> **Critical Prerequisite**: The equivalence "spectral-domain diagonalization = attention" **holds only when the attention matrix is circulant or Toeplitz**, i.e., attention weights depend solely on relative position $a_{ij} = f(i-j)$ and are independent of token content. In this case, the cyclic convolution theorem guarantees that FFT diagonalizes the convolution kernel. For standard content-dependent softmax attention $\text{softmax}(QK^T/\sqrt{d})$, the attention matrix is determined by query/key content and is generally **not** circulant or Toeplitz; therefore FFT element-wise multiplication **cannot** equivalently replace the $QK^T$ computation. Schemes A/C are essentially position-dependent convolutional attention approximations, not exact substitutes for general softmax attention.
+
 **Scheme A: FFT-Accelerated Attention (Time Series)**:
 ```python
 # Express cyclic convolution as element-wise multiplication in the frequency domain
@@ -56,16 +58,28 @@ attn = ifft(scores_freq)
 ## GPU Feasibility
 - **D1**: FFT and matrix multiplication are both standard tensor operations
 - **D2**: Spectral projection $U^T Q$ is a standard GEMM; although FFT is not GEMM, highly optimized cuFFT implementations are available
-- **D3**: FFT attention $O(n \log n \cdot d)$, far superior to $O(n^2 d)$
+- **D3[~]**: FFT attention $O(n \log n \cdot d)$, far superior to $O(n^2 d)$ -- **but only when attention has translation-invariant / position-dependent structure** (e.g., convolutional attention). General content-dependent softmax attention remains $O(n^2 d)$; linear attention approximations can achieve $O(n d^2)$, but via a different mechanism unrelated to FFT spectral methods.
 - **D4**: Frequency-domain representation introduces no extra dimensions; spectral projection can reduce to $k \ll n$ dimensions
 - **D5**: Complex-valued FFT suffers precision loss under fp16; fp32 or real-valued FFT (RFFT) is required
 - **D6**: FFT can be parallelized across batch/head; cuFFT supports multi-stream execution
 - **D7**: High-frequency components can be truncated in the spectral domain (structured sparsity), retaining only top-k frequencies
 - **D8**: Fusing FFT with attention requires custom kernels; no ready-made fusion exists in standard libraries
 
+**Quantitative assessment example** (standard transformer, d=128, n=2048, h=16):
+- D3: FFT path FLOPs ≈ 2·n·log₂(n)·d ≈ 2·2048·11·128 ≈ 5.8M vs standard attention 2·n²·d ≈ 1.1G (only when attention is convolutional)
+- D4: No n×n attention matrix materialization; FFT intermediates O(n·d) ≈ 1MB
+- D5: FFT under bf16 has twiddle factor error ~10⁻³, acceptable
+- D8: FFT + pointwise can be fused into a single CUDA kernel
+
 ## Paper Phrasing
-"We propose a spectral-domain attention mechanism that transforms attention computation into the Fourier/Laplacian spectral domain, leveraging the cyclic convolution theorem to reduce sequence attention complexity from $O(n^2)$ to $O(n \log n)$ while preserving the ability to model dependencies at multiple scales through frequency-adaptive weights."
+"We propose a spectral-domain attention mechanism that transforms attention computation into the Fourier/Laplacian spectral domain, leveraging the cyclic convolution theorem to reduce **translation-invariant** sequence attention complexity from $O(n^2)$ to $O(n \log n)$ while preserving the ability to model dependencies at multiple scales through frequency-adaptive weights. Note: this acceleration **requires position-dependent (not content-dependent) attention structure**; for general content-dependent softmax attention, the spectral-domain equivalence does not hold."
+
+## Applicability Conditions
+- **Core restriction of FFT spectral attention**: The FFT acceleration in Schemes A/C **requires the attention matrix to be circulant or Toeplitz**, i.e., attention weights depend only on relative position $a_{ij} = f(i-j)$ and are independent of the specific token content (query/key vectors).
+- **When the condition is NOT satisfied**: Standard softmax attention $\text{softmax}(QK^T/\sqrt{d})$ in most NLP tasks is content-dependent -- attention weights are jointly determined by query and key content, and the resulting attention matrix generally lacks circulant/Toeplitz structure. In this case, FFT element-wise multiplication **cannot** replace the $QK^T$ operation, and the $O(n \log n)$ complexity advantage does not hold.
+- **When the condition IS satisfied**: Positional encoding (e.g., frequency components of RoPE/ALiBi), periodic convolutional kernels for time series, fixed-pattern attention with translation-invariant priors. Scheme B (graph spectral attention) is not subject to this restriction, as it projects onto graph Laplacian eigenbases rather than exploiting the cyclic convolution theorem.
 
 ## Risks
+- **[x] Content-dependent attention incompatibility**: The core assumption of FFT spectral methods (circulant/Toeplitz structure) is **fundamentally incompatible** with the mainstream content-dependent softmax attention used in NLP. Applying FFT methods indiscriminately to general attention mechanisms constitutes a mathematical error -- they compute different quantities. Any usage must explicitly declare the position-dependence assumption being made.
 - **Violation of Periodicity Assumption**: FFT implicitly assumes periodic boundary conditions, causing spectral leakage for non-periodic signals such as natural language. Windowing functions or zero-padding are needed.
 - **Graph Laplacian Precomputation Cost**: Eigendecomposition $O(n^3)$ is infeasible for large-scale graphs, requiring approximations (Nystrom/Lanczos), and dynamic graphs necessitate recomputation.

package/design-patterns/attention/spectral-attention.md

@@ -2,7 +2,7 @@
 > **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
-当输入信号具有**频域/谱结构**（周期性、循环对称性、图结构）时，在谱域而非空域计算注意力可大幅降低复杂度并利用信号的内在结构。典型场景：时间序列预测（周期性信号）、图神经网络（图 Laplacian 谱分解）、位置编码（RoPE/ALiBi 的频率解释）、长序列注意力的 $O(n \log n)$ 加速。
+当输入信号具有**频域/谱结构**（周期性、循环对称性、图结构）时，在谱域而非空域计算注意力可大幅降低复杂度并利用信号的内在结构。典型场景：时间序列预测（周期性信号）、图神经网络（图 Laplacian 谱分解）、位置编码（RoPE/ALiBi 的频率解释）、长序列中**位置依赖型**注意力的 $O(n \log n)$ 加速（需循环/Toeplitz 结构假设，不适用于通用 content-dependent softmax 注意力）。
 
 ## 数学思想来源
 - 透镜：[duality（对偶透镜 — 频域变换）, symmetry（对称透镜 — 循环/平移不变性）]
@@ -11,12 +11,14 @@
 ## 需要的数学知识
 - **离散傅里叶变换 DFT / FFT**：$O(n \log n)$ 频域变换，循环卷积定理
 - **图 Laplacian 谱分解**：$L = U \Lambda U^T$，$U$ 为图 Fourier 基
-- **循环群 $\mathbb{Z}_n$ 的不可约表示**：DFT 矩阵即循环群的表示矩阵（参见 `references/books/abstract-algebra.md` Ch.4, Ch.11）
+- **循环群 $\mathbb{Z}_n$ 的不可约表示**：DFT 矩阵即循环群的表示矩阵（参见 `../../references/books/abstract-algebra.md` Ch.4, Ch.11）
 
 ## AI 模块形式
 
 **核心思路**：将注意力从空域的 $Q K^T$ 转换到谱域的对角/稀疏运算：
 
+> **关键前提条件**：上述「谱域对角化 = 注意力」的等价性**仅在注意力矩阵为循环矩阵（circulant）或 Toeplitz 矩阵时成立**，即注意力权重仅依赖于 token 之间的相对位置 $a_{ij} = f(i-j)$，而与 token 内容无关。此时循环卷积定理保证 FFT 可将卷积核对角化。对于标准的 content-dependent softmax 注意力 $\text{softmax}(QK^T/\sqrt{d})$，注意力矩阵由 query/key 内容决定，一般**不是**循环矩阵或 Toeplitz 矩阵，因此 FFT 逐元素乘法**不能**等价替代 $QK^T$ 计算。方案 A/C 本质上是位置依赖的卷积式注意力近似，而非通用 softmax 注意力的精确替代。
+
 **方案 A：FFT 加速注意力（时间序列）**：
 ```python
 # 将循环卷积写成谱域逐元素乘
@@ -56,16 +58,28 @@ attn = ifft(scores_freq)
 ## GPU 可行性
 - **D1[v]**：FFT 和矩阵乘均为标准张量运算
 - **D2[v]**：谱投影 $U^T Q$ 为标准 GEMM；FFT 虽非 GEMM 但有高度优化的 cuFFT 实现
-- **D3[v]**：FFT 注意力 $O(n \log n \cdot d)$，远优于 $O(n^2 d)$
+- **D3[~]**：FFT 注意力 $O(n \log n \cdot d)$，远优于 $O(n^2 d)$——**但仅当注意力具有平移不变/位置依赖结构时成立**（如卷积式注意力）。通用 content-dependent softmax 注意力仍为 $O(n^2 d)$；linear attention 近似可达 $O(n d^2)$，但其机制与 FFT 谱方法不同。
 - **D4[v]**：频域表示不增加额外维度，谱投影可降低到 $k \ll n$ 维
 - **D5[~]**：复数 FFT 在 fp16 下精度损失，需 fp32 或使用实数 FFT (RFFT)
 - **D6[v]**：FFT 可跨 batch/head 并行，cuFFT 支持多流
 - **D7[v]**：谱域中高频分量可截断（结构化稀疏），只保留 top-k 频率
 - **D8[~]**：FFT 与 attention 融合需自定义 kernel，标准库无现成融合
 
+**量化评估示例**（标准 transformer, d=128, n=2048, h=16）：
+- D3: FFT 路径 FLOPs ≈ 2·n·log₂(n)·d ≈ 2·2048·11·128 ≈ 5.8M vs 标准 attention 2·n²·d ≈ 1.1G（仅当 attention 为卷积核时）
+- D4: 无需物化 n×n attention matrix；FFT 中间结果 O(n·d) ≈ 1MB
+- D5: FFT 在 bf16 下 twiddle factor 误差 ~10⁻³，可接受
+- D8: FFT + pointwise 可融合为单个 CUDA kernel
+
 ## 论文表述方式
-"我们提出谱域注意力机制，通过将注意力计算转换到 Fourier/Laplacian 谱域，利用循环卷积定理将序列注意力复杂度从 $O(n^2)$ 降至 $O(n \log n)$，同时通过频率自适应权重保留对不同尺度依赖关系的建模能力。"
+"我们提出谱域注意力机制，通过将注意力计算转换到 Fourier/Laplacian 谱域，利用循环卷积定理将**具有平移不变结构的**序列注意力复杂度从 $O(n^2)$ 降至 $O(n \log n)$，同时通过频率自适应权重保留对不同尺度依赖关系的建模能力。注意：此加速**要求注意力模式具有位置依赖（而非内容依赖）结构**；对于通用 content-dependent softmax 注意力，该谱域等价性不成立。"
+
+## 适用条件
+- **FFT 谱注意力的核心限制**：方案 A/C 的 FFT 加速**要求注意力矩阵为循环矩阵或 Toeplitz 矩阵**，即注意力权重仅依赖于相对位置 $a_{ij} = f(i-j)$，而与 token 的具体内容（query/key 向量）无关。
+- **不满足条件的情形**：大多数 NLP 任务中的标准 softmax 注意力 $\text{softmax}(QK^T/\sqrt{d})$ 是 content-dependent 的——注意力权重由 query 和 key 的内容共同决定，产生的注意力矩阵一般不具有循环/Toeplitz 结构。在此情形下，FFT 逐元素乘法**不能**替代 $QK^T$ 运算，$O(n \log n)$ 复杂度优势也不成立。
+- **满足条件的情形**：位置编码（如 RoPE/ALiBi 的频率分量）、时间序列的周期性卷积核、具有平移不变先验的固定模式注意力。方案 B（图谱注意力）不受此限制，因为它在图 Laplacian 特征基上做投影而非利用循环卷积定理。
 
 ## 风险
+- **[x] Content-dependent 注意力不兼容**：FFT 谱方法的核心假设（循环/Toeplitz 结构）与 NLP 中主流的 content-dependent softmax 注意力**根本矛盾**。将 FFT 方法不加区分地应用于通用注意力机制是数学上的错误——两者计算的是不同的量。使用时必须明确声明所做的位置依赖假设。
 - **非周期性假设违反**：FFT 隐式假设周期性边界条件，对自然语言等非周期信号产生谱泄漏，需加窗函数或零填充。
 - **图 Laplacian 预计算成本**：大规模图的特征分解 $O(n^3)$ 不可行，需近似（Nyström/Lanczos），且动态图需重算。

package/design-patterns/compression/leverage-score-selection.en.md

@@ -2,15 +2,15 @@
 > **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Target Problem
-Use when selecting the most representative rows / columns / tokens from a large-scale matrix while guaranteeing the precision of downstream linear algebra operations: KV-Cache token selection, data coreset construction, Nystrom landmark sampling, distributed gradient compression. Core objective: **sample based on statistical leverage scores derived from subspace projections, with probabilistic guarantees of approximating full-computation accuracy**.
+Use when selecting representative rows / columns / tokens from a large-scale matrix while seeking verifiable error bounds for downstream linear-algebra tasks: KV-Cache token selection, data coreset construction, Nystrom landmark sampling, distributed gradient compression. Core objective: **sample using statistical leverage scores derived from subspace projections, approximating full computation with high probability under matrix-sampling assumptions**.
 
 ## Mathematical Foundations
-- Lenses: lenses/spectral.md (leverage scores = projection energy of row vectors onto the principal subspace), lenses/probabilistic.md (probabilistic sampling and concentration inequality guarantees), lenses/algorithmic.md (complexity--accuracy trade-offs of randomized algorithms)
-- Knowledge: knowledge-base/matrix-analysis/low-rank-approximation.md (randomized SVD, nuclear norm), knowledge-base/matrix-analysis/projection.md (diagonal entries of the projection matrix = leverage scores), knowledge-base/probability/concentration-inequality.md (Bernstein matrix concentration bound)
+- Lenses: ../../lenses/spectral.en.md (leverage scores = projection energy of row vectors onto the principal subspace), ../../lenses/probabilistic.en.md (probabilistic sampling and concentration inequality guarantees), ../../lenses/algorithmic.en.md (complexity--accuracy trade-offs of randomized algorithms)
+- Knowledge: ../../knowledge-base/matrix-analysis/low-rank-approximation.en.md (randomized SVD, nuclear norm), ../../knowledge-base/matrix-analysis/projection.en.md (diagonal entries of the projection matrix = leverage scores), ../../knowledge-base/probability/concentration-inequality.en.md (Bernstein matrix concentration bound)
 
 ## Required Mathematical Background
 - **Statistical Leverage Scores**: $\ell_i = \|(V_k V_k^T)_i\|^2 = (V_k V_k^T)_{ii}$, the projection energy of the $i$-th row onto the rank-$k$ subspace; $\sum_i \ell_i = k$
-- **Leverage Score Sampling Guarantee**: sampling $s = O(k \log k / \epsilon^2)$ rows with probabilities $p_i = \ell_i / k$ yields a $(1+\epsilon)$ approximation to full least squares (Drineas--Mahoney)
+- **Leverage Score Sampling Bound**: in least-squares / subspace-embedding settings with independent reweighted sampling, sampling $s = O(k \log k / \epsilon^2)$ rows with probabilities $p_i = \ell_i / k$ can yield a $(1+\epsilon)$ approximation (Drineas--Mahoney-style results)
 - **Bernstein Matrix Bound**: after sampling, $\|\hat{A}^T \hat{A} - A^T A\|_2 \leq \epsilon \|A\|_F^2$ with probability $\geq 1-\delta$
 - **Fast Approximation**: $\tilde{\ell}_i = \|(A\Omega)_i\|^2$ ($\Omega$ random Gaussian), avoids full SVD, $O(Ndk)$
 - **DPP Extension**: Determinantal Point Process $P(S) \propto \det(L_S)$ adds a diversity guarantee on top of leverage scores
@@ -48,20 +48,20 @@ Method 4 - DPP greedy diverse selection:
 
 ## Implementable Architectures
 - **Random projection leverage layer**: 1 GEMM + 1 QR yields approximate leverage scores in $O(Ndk)$
-- **KV-Cache eviction policy**: evict by leverage score ranking, with stronger theoretical guarantees than attention-score-based eviction
-- **Coreset constructor**: leverage score sampling + importance reweighting, ensuring empirical risk approximates the full-data risk
+- **KV-Cache eviction policy**: evict by leverage score ranking; the theory is mainly for matrix subspace approximation, so transfer to attention quality requires separate validation
+- **Coreset constructor**: leverage score sampling + importance reweighting, controlling empirical-risk approximation under the relevant sampling assumptions
 - **DPP greedy extension**: leverage scores + exclusion penalty, balancing importance and diversity
 
 ## GPU Feasibility
-- Tensorization / GEMM: $A\Omega$ is a GEMM; QR via cuSOLVER; leverage scores = elementwise row-wise sum of squares
-- Complexity: $O(Ndk)$ is far superior to $O(Nd^2)$ full SVD; top-s selection in $O(N \log s)$
-- Memory: $\Omega$ is only $d \times k$ (KB-scale); $Q$ is the same size as $A$ but can be computed in batches
-- Low precision: QR recommended in fp32 (small matrix with $k+p$ columns, negligible overhead); leverage scores can be cast back to bf16
-- Parallelism: the $A\Omega$ GEMM is highly parallelizable; top-s can use parallel radix sort
-- Operator fusion: $A\Omega$ + QR + row-norm² can be fused to avoid materializing intermediate matrices
+- **D1[v]/D2[v]**: $A\Omega$ is a GEMM; QR via cuSOLVER; leverage scores = elementwise row-wise sum of squares
+- **D3[v]**: $O(Ndk)$ is far superior to $O(Nd^2)$ full SVD; top-s selection in $O(N \log s)$
+- **D4[v]**: $\Omega$ is only $d \times k$ (KB-scale); $Q$ is the same size as $A$ but can be computed in batches
+- **D5[~]**: QR recommended in fp32 (small matrix with $k+p$ columns, negligible overhead); leverage scores can be cast back to bf16
+- **D6[v]**: the $A\Omega$ GEMM is highly parallelizable; top-s can use parallel radix sort
+- **D8[v]**: $A\Omega$ + QR + row-norm² can be fused to avoid materializing intermediate matrices
 
 ## Paper-Worthy Formulation
-"We adopt statistical leverage scores as the importance metric for token selection: approximating rank-$k$ subspace leverage scores via random projection in $O(Ndk)$, with the Drineas--Mahoney theory guaranteeing that $O(k \log k / \epsilon^2)$ samples suffice for a $(1+\epsilon)$ approximation of the full subspace."
+"We adopt statistical leverage scores as the importance metric for token / row selection: approximate rank-$k$ subspace leverage scores via random projection in $O(Ndk)$, and use Drineas--Mahoney-style bounds to choose sample counts when the target is subspace / least-squares approximation under independent reweighted sampling and effective-rank diagnostics. For KV-cache eviction, attention/output error and task metrics must still be reported."
 
 ## Risks
 - **Rank parameter $k$ selection**: leverage scores depend on the rank-$k$ subspace; an incorrect $k$ leads to biased sampling -- the effective rank must be diagnosed first

package/design-patterns/compression/leverage-score-selection.md

@@ -2,15 +2,15 @@
 > **严谨性声明**：本文件中涉及复杂度、显存、FlashAttention 融合、Tensor Core、KV-Cache 压缩的结论均标注为「[v] 已验证 / [~] 可改造需验证 / [x] 不可行」。未标注的视为理论可行，需工程验证。
 
 ## 适用问题
-当需要从大规模矩阵中选取最有代表性的行/列/token，且需保证下游线性代数运算精度时使用：KV-Cache token 选择、数据 coreset 构建、Nyström landmark 采样、分布式梯度压缩。核心诉求：**基于子空间投影的统计杠杆分数做采样，以概率保证逼近全量计算精度**。
+当需要从大规模矩阵中选取最有代表性的行/列/token，并希望在下游线性代数任务中获得可检验误差界时使用：KV-Cache token 选择、数据 coreset 构建、Nyström landmark 采样、分布式梯度压缩。核心诉求：**基于子空间投影的统计杠杆分数做采样，在矩阵采样理论假设下以高概率逼近全量计算**。
 
 ## 数学思想来源
-- 透镜：lenses/spectral.md（杠杆分数 = 行向量在主子空间上的投影能量）、lenses/probabilistic.md（概率采样与浓度不等式保证）、lenses/algorithmic.md（随机化算法的复杂度与精度权衡）
-- 知识：knowledge-base/matrix-analysis/low-rank-approximation.md（随机化 SVD、核范数）、knowledge-base/matrix-analysis/projection.md（投影矩阵对角元 = 杠杆分数）、knowledge-base/probability/concentration-inequality.md（Bernstein 矩阵浓度界）
+- 透镜：../../lenses/spectral.md（杠杆分数 = 行向量在主子空间上的投影能量）、../../lenses/probabilistic.md（概率采样与浓度不等式保证）、../../lenses/algorithmic.md（随机化算法的复杂度与精度权衡）
+- 知识：../../knowledge-base/matrix-analysis/low-rank-approximation.md（随机化 SVD、核范数）、../../knowledge-base/matrix-analysis/projection.md（投影矩阵对角元 = 杠杆分数）、../../knowledge-base/probability/concentration-inequality.md（Bernstein 矩阵浓度界）
 
 ## 需要的数学知识
 - **统计杠杆分数**：$\ell_i = \|(V_k V_k^T)_i\|^2 = (V_k V_k^T)_{ii}$，第 $i$ 行在 rank-$k$ 子空间的投影能量；$\sum_i \ell_i = k$
-- **杠杆分数采样保证**：以 $p_i = \ell_i / k$ 采样 $s = O(k \log k / \epsilon^2)$ 行，$(1+\epsilon)$ 近似全量最小二乘（Drineas-Mahoney）
+- **杠杆分数采样界**：在最小二乘/子空间嵌入设定和独立重加权采样条件下，以 $p_i = \ell_i / k$ 采样 $s = O(k \log k / \epsilon^2)$ 行可得到 $(1+\epsilon)$ 近似（Drineas-Mahoney 类结果）
 - **Bernstein 矩阵界**：采样后 $\|\hat{A}^T \hat{A} - A^T A\|_2 \leq \epsilon \|A\|_F^2$，概率 $\geq 1-\delta$
 - **快速近似**：$\tilde{\ell}_i = \|(A\Omega)_i\|^2$（$\Omega$ 随机高斯），避免完整 SVD，$O(Ndk)$
 - **DPP 扩展**：行列式点过程 $P(S) \propto \det(L_S)$ 在杠杆分数基础上增加多样性保证
@@ -48,20 +48,20 @@
 
 ## 可实现结构
 - **随机投影杠杆层**：1 次 GEMM + 1 次 QR 即得近似杠杆分数，$O(Ndk)$
-- **KV-Cache 驱逐策略**：按杠杆分数排序驱逐，比 attention score 驱逐更有理论保证
-- **Coreset 构建器**：杠杆分数采样 + 重要性重加权，保证经验风险逼近全量
+- **KV-Cache 驱逐策略**：按杠杆分数排序驱逐；理论界主要来自矩阵子空间近似，迁移到 attention 质量需单独验证
+- **Coreset 构建器**：杠杆分数采样 + 重要性重加权，在相应采样假设下控制经验风险近似误差
 - **DPP 贪心扩展**：杠杆分数 + 互斥惩罚，兼顾重要性与多样性
 
 ## GPU 可行性
-- 张量化/GEMM：$A\Omega$ 为 GEMM；QR 有 cuSOLVER；杠杆分数 = elementwise 逐行平方和
-- 复杂度：$O(Ndk)$ 远优于 $O(Nd^2)$ 完整 SVD；top-s 选择 $O(N \log s)$
-- 显存：$\Omega$ 仅 $d \times k$（KB 级）；$Q$ 与 $A$ 同尺寸但可分批计算
-- 低精度：QR 建议 fp32（小矩阵 $k+p$ 列，开销可忽略）；杠杆分数可回 bf16
-- 并行：$A\Omega$ 的 GEMM 高度并行；top-s 可用 radix sort 并行
-- 算子融合：$A\Omega$ + QR + row-norm² 可融合避免物化中间矩阵
+- **D1[v]/D2[v]**：$A\Omega$ 为 GEMM；QR 有 cuSOLVER；杠杆分数 = elementwise 逐行平方和
+- **D3[v]**：$O(Ndk)$ 远优于 $O(Nd^2)$ 完整 SVD；top-s 选择 $O(N \log s)$
+- **D4[v]**：$\Omega$ 仅 $d \times k$（KB 级）；$Q$ 与 $A$ 同尺寸但可分批计算
+- **D5[~]**：QR 建议 fp32（小矩阵 $k+p$ 列，开销可忽略）；杠杆分数可回 bf16
+- **D6[v]**：$A\Omega$ 的 GEMM 高度并行；top-s 可用 radix sort 并行
+- **D8[v]**：$A\Omega$ + QR + row-norm² 可融合避免物化中间矩阵
 
 ## 论文表述方式
-"采用统计杠杆分数作为 token 选择的重要性度量：通过随机投影在 $O(Ndk)$ 内近似 rank-$k$ 子空间杠杆分数，Drineas-Mahoney 理论保证 $O(k \log k / \epsilon^2)$ 次采样即可 $(1+\epsilon)$ 近似全量子空间。"
+"采用统计杠杆分数作为 token/行选择的重要性度量：通过随机投影在 $O(Ndk)$ 内近似 rank-$k$ 子空间杠杆分数，并在独立重加权采样、有效秩诊断和目标为子空间/最小二乘近似的前提下，使用 Drineas-Mahoney 类界选择样本数。对 KV-cache eviction，仍需报告 attention/output 误差与任务指标。"
 
 ## 风险
 - **秩参数 $k$ 选择**：杠杆分数依赖 rank-$k$ 子空间，$k$ 选错导致采样偏差；需先诊断有效秩

package/design-patterns/compression/low-rank-kv-cache.en.md

@@ -2,16 +2,16 @@
 > **Rigor disclaimer**: Claims about complexity, memory, FlashAttention fusion, Tensor Core, and KV-Cache compression are marked as [v] verified / [~] retrofittable (needs validation) / [x] infeasible. Unmarked claims are theoretically possible but require engineering validation.
 
 ## Target Problem
-Use when KV-Cache memory consumption becomes the bottleneck during LLM inference: long-context inference ($L > 8K$), multi-turn conversation accumulation, edge deployment, speculative decoding / beam search. Core objective: **compress the KV-Cache from $O(Ld)$ to $O(kd)$ with $k \ll L$, under controlled information loss**.
+Use when KV-Cache memory consumption becomes the bottleneck during LLM inference: long-context inference ($L > 8K$), multi-turn conversation accumulation, edge deployment, speculative decoding / beam search. Core objective: **compress the KV-Cache from $O(Ld)$ to low-rank factor storage $O(Lk + kd)$ with $k \ll d$, under controlled information loss**.
 
 ## Mathematical Foundations
-- Lenses: lenses/spectral.md (spectral component identification and truncation), lenses/variational.md (Pareto trade-off between compression ratio and reconstruction error), lenses/duality.md (nuclear norm--spectral norm duality)
-- Knowledge: knowledge-base/matrix-analysis/low-rank-approximation.md (Eckart--Young optimal approximation, randomized SVD), knowledge-base/matrix-analysis/projection.md (orthogonal projection onto principal subspace), knowledge-base/matrix-analysis/matrix-perturbation.md (Weyl perturbation bound)
+- Lenses: ../../lenses/spectral.en.md (spectral component identification and truncation), ../../lenses/variational.en.md (Pareto trade-off between compression ratio and reconstruction error), ../../lenses/duality.en.md (nuclear norm--spectral norm duality)
+- Knowledge: ../../knowledge-base/matrix-analysis/low-rank-approximation.en.md (Eckart--Young optimal approximation, randomized SVD), ../../knowledge-base/matrix-analysis/projection.en.md (orthogonal projection onto principal subspace), ../../knowledge-base/matrix-analysis/matrix-perturbation.en.md (Weyl perturbation bound)
 
 ## Required Mathematical Background
 - **Eckart--Young--Mirsky Theorem**: $A_k = U_k \Sigma_k V_k^H$, $\|A - A_k\|_F = \sqrt{\sum_{i>k} \sigma_i^2}$; truncated SVD = optimal rank-$k$ approximation
 - **Randomized SVD**: $Y = A\Omega$ ($\Omega$ random Gaussian), $Y = QR$, $B = Q^H A$, perform SVD on $B$; complexity $O(Ldk)$, dominated by matrix multiplications
-- **Weyl Perturbation Bound**: $\|A - A_k\|_2 = \sigma_{k+1}$; maximum deviation of attention scores after compression $\leq \sigma_{k+1}$
+- **Eckart--Young spectral-norm error**: $\|A - A_k\|_2 = \sigma_{k+1}$ (bounds only the optimal rank-$k$ compression error of the matrix itself; Weyl is for singular-value changes under perturbation). Extending this to attention output error requires additional conditions: bounding $\|\text{Attn}(Q,K,V) - \text{Attn}(Q,K_k,V_k)\|$ requires (a) bounded $\|Q\|$, (b) the softmax Lipschitz constant (which depends on temperature and score range), and (c) compression errors for **both** K and V. Roughly: error $\lesssim C \cdot (\|Q\| \cdot \|K - K_k\| \cdot \|V\| + \|Q\| \cdot \|K_k\| \cdot \|V - V_k\|) / \tau$, where $C$ depends on the softmax Lipschitz constant and $\tau$ is the temperature parameter. Directly using $\sigma_{k+1}$ to bound attention score deviation **holds only under the simplifying assumptions of fixed Q and ignoring V compression error**.
 - **Effective Rank**: $r_{\text{eff}}(K) = \|K\|_F^2 / \|K\|_2^2$, guides adaptive selection of $k$
 
 ## AI Module Specification
@@ -21,11 +21,32 @@ Input: K ∈ R^{L×d}, V ∈ R^{L×d}    Parameters: target rank k << L, update
 
 Method 1 - Offline periodic compression (most practical):
   Omega = randn(d, k+p)                    // random projection, p=5 oversampling
-  Q_k = qr(K @ Omega)[0]                   // L×(k+p) orthonormal basis (GEMM + QR)
-  B_k = Q_k^T @ K                          // (k+p)×d small matrix GEMM
+  Q_base = qr(K @ Omega)[0]                // L×(k+p) oversampled orthonormal basis (GEMM + QR)
+  B_k = Q_base^T @ K                       // (k+p)×d small matrix GEMM
   U_r, S_r, Vt_r = svd(B_k)               // small matrix SVD
-  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d compressed Key
-  // Attention: softmax(Q @ K_comp^T / √d) @ V_comp, sequence dimension L → k
+  Q_final = Q_base @ U_r[:, :k]            // L×k final left factor
+  K_comp = S_r[:k] * Vt_r[:k, :]           // k×d compressed Key (low-rank factor)
+  // V_comp definition: apply analogous truncated SVD to V, V_comp = Σ_k^{(V)} · Vt_k^{(V)} (k×d compressed Value)
+  // Alternatively, if K and V share the left factor Q_final, then V_comp = Q_final^T @ V (project onto the same low-rank subspace)
+  //
+  // ⚠ Critical distinction -- low-rank factors CANNOT directly replace the original sequence in softmax attention:
+  //   K ≈ Q_final @ K_comp (L×d reconstruction); softmax is a nonlinear operation, so
+  //   softmax(Q @ K_comp^T / √d) @ V_comp ≠ softmax(Q @ K^T / √d) @ V
+  //   The "k compressed tokens" interpretation is only valid for linear attention, not softmax attention.
+  //
+  // Mode A - Standard softmax attention (saves memory; may reduce inner dimension, not sequence length):
+  //   K_recon need not be materialized. If K ≈ Q_final @ K_comp, then
+  //   logits = (Q @ K_comp^T) @ Q_final^T / √d // for Q in R^{m×d}: O(mdk + mLk)
+  //   attn = softmax(logits)                   // softmax is still normalized over L positions
+  //   If V ≈ Q_final @ V_comp, output = (attn @ Q_final) @ V_comp
+  //   // Advantage: storage reduced from O(Ld) to O(Lk + kd); QK/AV inner dimension goes from d to k
+  //   // Limitation: the attention matrix is still m×L; the k factors are not k tokens
+  //
+  // Mode B - Linear attention (kernel feature map φ; compress additive statistics of φ(K), V):
+  //   Replace softmax with kernel feature map φ: Attn = φ(Q) @ (φ(K)^T @ V) / (φ(Q) @ φ(K)^T @ 1)
+  //   History can shrink from L tokens to k statistical factors only if the compressed objects are
+  //   additive statistics such as φ(K)^T V and φ(K)^T 1, or if low-rank/aggregation is performed
+  //   directly in φ(K) space. Applying φ directly to K_comp is not generally equivalent.
 
 Method 2 - Streaming incremental compression (low latency):
   Maintain basis (U_basis ∈ R^{k×d}), on new token arrival:
@@ -37,7 +58,7 @@ Method 3 - Layer-wise adaptive: allocate per-layer, per-head k based on effectiv
 ```
 
 ## Implementable Architectures
-- **Periodic compression layer**: trigger randomized SVD every $M=64$ steps, $L \times d \to k \times d$
+- **Periodic compression layer**: trigger randomized SVD every $M=64$ steps, $L \times d \to (L \times k) + (k \times d)$
 - **Double buffering**: compressed basis + recent $w$ raw tokens, balancing accuracy and compression ratio
 - **Shared basis**: multi-head sharing of the Key column-space basis, each head stores only coefficients
 - **Quantized basis**: further INT8/FP8 quantization after compression, achieving dual compression
@@ -45,13 +66,19 @@ Method 3 - Layer-wise adaptive: allocate per-layer, per-head k based on effectiv
 ## GPU Feasibility
 - Tensorization / GEMM: randomized SVD = 3 GEMMs + 1 small SVD, maps perfectly onto Tensor Cores
 - Complexity: $O(Ldk)$ is far superior to $O(Ld^2)$ full SVD; overhead negligible for $k \sim 256$
-- Memory: KV-Cache reduced from $O(Ld)$ to $O(kd)$; $k/L \sim 1/16$ yields 16x compression
+- Memory: Key-Cache stored in low-rank factor form ($Q_{\text{final}} \in \mathbb{R}^{L \times k}$ + $B_k \in \mathbb{R}^{k \times d}$), total parameters $Lk + kd$, compression ratio $Ld/(Lk+kd) \approx d/k$ (when $L \gg k$). Note $Q_{\text{final}}$ still has $L$ dimension, so sequence length is NOT reduced; softmax attention can compute length-$L$ logits with factorized GEMMs without materializing a full $L \times d$ reconstruction. V-Cache requires independent compression. End-to-end compression ratio depends on K/V storage format and rank selection
 - Low precision: SVD recommended in fp32 (acceptable for small matrices); compressed KV can be stored back in bf16
 - Parallelism: compression across layers / heads is fully independent; incremental update $O(kd)$ with very low latency
-- Operator fusion: $K\Omega$ + QR can be partially fused; the QK^T dimension in attention is already reduced
+- Operator fusion: $K\Omega$ + QR can be partially fused; in the softmax path, QK/AV inner dimension can drop from $d$ to $k$, but softmax is still normalized over length $L$; linear attention only shrinks history to $k$ when compressing additive statistics
+
+**Quantitative assessment example** (standard transformer, d=128, n=2048, rank k=64):
+- D3: SVD computation O(n·d·k) ≈ 2048·128·64 ≈ 16.8M FLOPs (one-time); softmax inference per query with factors is about O(dk + n·k), with a length-n softmax still required; linear attention removes n only when compressed statistics are used
+- D4: KV-Cache from O(n·d) = 2048·128·2B ≈ 512KB; factor format O(n·k + k·d) ≈ 2048·64·2B + 64·128·2B ≈ 278KB (compression ratio ~1.8x); final-left-factor + coefficient format ($Q_{\text{final}} \in \mathbb{R}^{n \times k}$ columns + $B_k \in \mathbb{R}^{k \times d}$) total parameters $nk + kd$, compression ratio $nd/(nk+kd) = d/k \cdot 1/(1+d/n) \approx d/k = 2x$, V-Cache requires independent compression
+- D5: Truncated SVD under bf16 amplifies singular value errors near σ_k by ~κ(A), requires caution
+- D8: SVD → matmul can be fused; online updates use incremental SVD to avoid full recomputation
 
 ## Paper-Worthy Formulation
-"Building on the Eckart--Young--Mirsky theorem, we employ randomized SVD to project the KV-Cache onto the optimal rank-$k$ subspace, achieving $L/k$-fold memory compression at $O(Ldk)$ complexity, with the Weyl perturbation bound guaranteeing that attention score deviations do not exceed $\sigma_{k+1}$."
+"Building on the Eckart--Young--Mirsky theorem, we employ randomized SVD to project the KV-Cache onto a rank-$k$ subspace, compressing storage from $O(Ld)$ to $O(Lk + kd)$ (basis+coefficient format) at $O(Ldk)$ complexity. For standard softmax attention, low-rank factors cannot be interpreted as $k$ compressed tokens; softmax is still normalized over length $L$, but factorized GEMMs can compute logits and value aggregation without materializing a full $L \times d$ reconstruction, reducing the QK/AV inner dimension from $d$ to $k$. Only in linear attention, and only when compressing additive statistics such as $\phi(K)^T V$ and $\phi(K)^T\mathbf{1}$, can the historical state truly shrink from $L$ tokens to $k$ statistical factors. The Eckart--Young spectral-norm error gives $\sigma_{k+1}$ as the optimal rank-$k$ compression error for the K/V matrices themselves; the end-to-end attention output error bound further depends on query norms, the softmax Lipschitz constant, and the temperature parameter. The actual memory compression ratio depends on storage format and the independent V-Cache compression strategy."
 
 ## Risks
 - **Improper rank selection**: $k$ too small causes $\sigma_{k+1}$ to be non-negligible, degrading long-range recall; singular value decay curves must be monitored