dsh-context-compression-improved 0.4.0-beta.1 → 0.5.0

package/docs/installation.md

@@ -1,78 +1,103 @@
-# Installing dsh-context-compression-improved
-
-> [English](installation.md) · [中文](installation.zh.md) · [日本語](installation.ja.md) · [한국어](installation.ko.md)
-
-This guide installs the fork from source. The fork is not yet published to npm; the internal
-package name intentionally stays upstream's (`dsh-context-compression-improved`). The runtime
-that used to be a second package is now part of it, so one install brings the whole stack.
-
-## Prerequisites
-
-- Node `^22.19.0 || >=24` and pnpm `11.7.0` (`corepack enable` picks the pinned version from `packageManager`).
-- A DeepSeek Harness installation compatible with the `0.1.1-rc.2` peer range (verified against the official `dsh-v0.1.2-alpha.5` release).
-- A DeepSeek V4 model route (`deepseek-v4-flash`, `deepseek-v4-pro`, or `deepseek-v4-flash-vision-exp`). Lossy compression — including the code-skeleton gate — requires the exact bundled tokenizer; other routes fail open and keep original tool results.
-- Git.
-
-## 1. Build from source
-
-```sh
-git clone https://github.com/drscrewdriver/dsh-context-compression-improved.git
-cd dsh-context-compression-improved
-pnpm install --frozen-lockfile
-pnpm build
-```
-
-`pnpm build` bundles both library faces of every package (`tsdown`). Run `pnpm test` first if you want the full suite on your machine before installing.
-
-## 2. Pack the Bundle entry package
-
-The selector package is the single Bundle entry; the runtime comes along as its exact-version dependency:
-
-```sh
-cd packages/selector
-pnpm pack
-# → dsh-context-compression-improved-0.1.0.tgz
-cd ../..
-```
-
-`pnpm pack` runs the bundle through the `prepack` hook, so the tarball always matches your checkout.
-
-## 3. Add it to a Harness profile
-
-The selector package declares the Harness Bundle manifest field `dsh.bundle.patch`, so `dsh plugin add` is the standard out-of-tree Bundle installation path:
-
-```sh
-dsh plugin --profile web add packages/selector/dsh-context-compression-improved-0.1.0.tgz
-dsh --profile web --dump-config
-```
-
-Restart the selected profile after installation. The config dump should list the selector Bundle as active. Do **not** install or wire the selector and runtime packages separately — the runtime is installed automatically.
-
-## 4. Turn on the code-skeleton gate
-
-Open DeepSeek Harness settings → **Context compression selector**:
-
-1. Pick a compression profile (the gate is orthogonal to all of them).
-2. Optionally adjust the Auto Compact trigger level (50–90%, default 80%).
-3. Set **Code skeleton compression** to **On**. The toggle saves on change.
-
-Like all selector settings, the value is frozen when a session first observes it — the gate affects newly observed sessions, never a task that is already running.
-
-## 5. Update or remove
-
-```sh
-# update: pull, rebuild, repack, and add the new tarball again
-git pull && pnpm install --frozen-lockfile && pnpm build
-cd packages/selector && pnpm pack && cd ../..
-dsh plugin --profile web add packages/selector/dsh-context-compression-improved-0.1.0.tgz
-
-# remove
-dsh plugin --profile web remove dsh-context-compression-improved
-```
-
-## Troubleshooting
-
-- **Bundle not active in the dump**: restart the profile; confirm you added the selector entry package (not the runtime) and that the Harness version is in the compatible peer range.
-- **Tool results are never skeleton-compressed**: the gate is off by default; check the toggle. Compression only applies to fresh, oversized source-code tool results on exact-tokenizer model routes, and every skip is recorded with a reason in the audit trail.
-- **The toggle shows as unreadable**: the stored `codeSkeleton` section failed the strict browser decode (it must be exactly `{ enabled: boolean }`). Removing the malformed section restores defaults.
-- **Updating fails on the upgrade step**: the plugin follows npm package semantics; remove the old version first if a tarball-to-tarball upgrade is refused by your Harness build.
+# Installing dsh-context-compression-improved
+
+> [English](installation.md) · [中文](installation.zh.md) · [日本語](installation.ja.md) · [한국어](installation.ko.md)
+
+This guide installs the fork from source. The fork is published to npm as
+`dsh-context-compression-improved` under the `dsh-0.1.5` dist-tag; the package name
+intentionally stays upstream's. The runtime that used to be a second package is now part of
+it, so one install brings the whole stack.
+
+## Prerequisites
+
+- Node `^22.19.0 || >=24` and pnpm `11.7.0` (`corepack enable` picks the pinned version from `packageManager`).
+- A DeepSeek Harness installation compatible with the `0.1.1-rc.2` peer range (verified against the official `dsh-v0.1.2-alpha.5` release).
+- A DeepSeek model route. Lossy compression — including the code-skeleton gate — decides on the character basis, so no bundled-tokenizer route requirement remains; when a bundled tokenizer exists its exact counts are recorded as telemetry, and other routes fail open and keep original tool results.
+- Git.
+
+## 1. Build from source
+
+```sh
+git clone https://github.com/drscrewdriver/dsh-context-compression-improved.git
+cd dsh-context-compression-improved
+pnpm install --frozen-lockfile
+pnpm build
+```
+
+`pnpm build` bundles both library faces of every package (`tsdown`). Run `pnpm test` first if you want the full suite on your machine before installing.
+
+## 2. Pack the Bundle entry package
+
+The selector package is the single Bundle entry; the runtime comes along as its exact-version dependency:
+
+```sh
+cd packages/selector
+pnpm pack
+# → dsh-context-compression-improved-0.1.0.tgz
+cd ../..
+```
+
+`pnpm pack` runs the bundle through the `prepack` hook, so the tarball always matches your checkout.
+
+## 3. Add it to a Harness profile
+
+The selector package declares the Harness Bundle manifest field `dsh.bundle.patch`, so `dsh plugin add` is the standard out-of-tree Bundle installation path:
+
+```sh
+dsh plugin --profile web add packages/selector/dsh-context-compression-improved-0.1.0.tgz
+dsh --profile web --dump-config
+```
+
+Restart the selected profile after installation. The config dump should list the selector Bundle as active. Do **not** install or wire the selector and runtime packages separately — the runtime is installed automatically.
+
+## 4. Turn on the code-skeleton gate
+
+Open DeepSeek Harness settings → **Context compression selector**:
+
+1. Pick a compression profile (the gate is orthogonal to all of them).
+2. Optionally adjust the Auto Compact trigger level (50–90%, default 80%).
+3. Set **Code skeleton compression** to **On**. The toggle saves on change.
+
+Like all selector settings, the value is frozen when a session first observes it — the gate affects newly observed sessions, never a task that is already running.
+
+## 5. Optional: the advisory relevance advisor
+
+The plugin can keep statistics about how relevant the session's history still is — advisory only, it never decides or blocks anything. It is off by default; turn it on by editing the `presetOptions` section of the context-compression settings (settings JSON, no UI card this round):
+
+```json
+"presetOptions": {
+  "advisorMode": "host",
+  "advisorRefreshTurns": 8,
+  "advisorScoreThreshold": 0.35,
+  "advisorSampleLimit": 16,
+  "advisorMinTokens": 250,
+  "advisorTimeoutMs": 8000
+}
+```
+
+`advisorMode: "host"` calls the harness `llm` service; `"direct"` reuses the estimator's
+`estimatorBaseUrl` / `estimatorApiKey` / `estimatorModel` endpoint. At every turn boundary
+the advisor (1) summarizes the current task from the most recent `todo/write` event,
+(2) incrementally scores historical tool results for content-and-comment relevance, and
+(3) records a prefix-decay figure. Results surface as `advisor-outcome` audit records and,
+with the deployment flag `advisorReportRoute: true`, a read-only
+`GET .../advisor-report?sessionId=` HTTP route. What the advisor reports can never
+suppress, delay, or rewrite any reduction that would land.
+
+## 6. Update or remove
+
+```sh
+# update: pull, rebuild, repack, and add the new tarball again
+git pull && pnpm install --frozen-lockfile && pnpm build
+cd packages/selector && pnpm pack && cd ../..
+dsh plugin --profile web add packages/selector/dsh-context-compression-improved-0.1.0.tgz
+
+# remove
+dsh plugin --profile web remove dsh-context-compression-improved
+```
+
+## Troubleshooting
+
+- **Bundle not active in the dump**: restart the profile; confirm you added the selector entry package (not the runtime) and that the Harness version is in the compatible peer range.
+- **Tool results are never skeleton-compressed**: the gate is off by default; check the toggle. Compression only applies to fresh, oversized source-code tool results (decisions run on the character basis; no exact-tokenizer route requirement), and every skip is recorded with a reason in the audit trail.
+- **The toggle shows as unreadable**: the stored `codeSkeleton` section failed the strict browser decode (it must be exactly `{ enabled: boolean }`). Removing the malformed section restores defaults.
+- **Updating fails on the upgrade step**: the plugin follows npm package semantics; remove the old version first if a tarball-to-tarball upgrade is refused by your Harness build.

package/docs/installation.zh.md

@@ -1,77 +1,100 @@
-# 安装 dsh-context-compression-improved
-
-> [English](installation.md) · [中文](installation.zh.md) · [日本語](installation.ja.md) · [한국어](installation.ko.md)
-
-本教程从源码安装 fork。fork 尚未发布到 npm；包名有意与上游保持一致：
-`dsh-context-compression-improved`（单一包 —— 原先独立发布的 runtime 包已并入其中）。
-
-## 前置条件
-
-- Node `^22.19.0 || >=24` 与 pnpm `11.7.0`（`corepack enable` 会按 `packageManager` 字段使用固定版本）。
-- 兼容 `0.1.1-rc.2` peer 范围的 DeepSeek Harness（已针对官方 `dsh-v0.1.2-alpha.5` 验证）。
-- DeepSeek V4 模型路由（`deepseek-v4-flash`、`deepseek-v4-pro` 或 `deepseek-v4-flash-vision-exp`）。有损压缩——包括代码骨架门——依赖内置的精确 tokenizer；其他路由 fail-open 并保留原始工具结果。
-- Git。
-
-## 1. 从源码构建
-
-```sh
-git clone https://github.com/drscrewdriver/dsh-context-compression-improved.git
-cd dsh-context-compression-improved
-pnpm install --frozen-lockfile
-pnpm build
-```
-
-`pnpm build` 会打包所有包的两套产物（`tsdown`）。如需在安装前先跑全量测试，可执行 `pnpm test`。
-
-## 2. 打包 Bundle 入口包
-
-selector 包是唯一的 Bundle 入口；runtime 作为其精确版本依赖自动随行：
-
-```sh
-cd packages/selector
-pnpm pack
-# → dsh-context-compression-improved-0.1.0.tgz
-cd ../..
-```
-
-`pnpm pack` 会通过 `prepack` 钩子执行打包，因此 tarball 始终与你的检出内容一致。
-
-## 3. 安装到 Harness Profile
-
-selector 包声明了 Harness Bundle manifest 字段 `dsh.bundle.patch`，因此 `dsh plugin add` 是标准的树外 Bundle 安装方式：
-
-```sh
-dsh plugin --profile web add packages/selector/dsh-context-compression-improved-0.1.0.tgz
-dsh --profile web --dump-config
-```
-
-安装后重启对应 Profile。配置导出中应显示 selector Bundle 已激活。**不要**分别安装或手动连接 selector 与 runtime 两个包——runtime 会自动安装。
-
-## 4. 打开代码骨架门
-
-打开 DeepSeek Harness 设置 → **上下文压缩选择器*：
-
-1. 选择一个压缩 Profile（这道门与所有 Profile 正交）。
-2. 按需调整 Auto Compact 触发水位（50–90%，默认 80%）。
-3. 将**代码骨架压缩**设为**开**。开关即改即存。
-
-与所有选择器设置一致，取值在会话首次观察时冻结——这道门只影响新观察的会话，不会改变正在运行的任务。
-
-## 5. 更新或卸载
-
-```sh
-# 更新：拉取、重建、重新打包、再次添加新 tarball
-git pull && pnpm install --frozen-lockfile && pnpm build
-cd packages/selector && pnpm pack && cd ../..
-dsh plugin --profile web add packages/selector/dsh-context-compression-improved-0.1.0.tgz
-
-# 卸载
-dsh plugin --profile web remove dsh-context-compression-improved
-```
-
-## 故障排除
-
-- **配置导出中 Bundle 未激活**：重启 Profile；确认添加的是 selector 入口包（而非 runtime），且 Harness 版本在兼容的 peer 范围内。
-- **工具结果从未被骨架化压缩**：该门默认关闭，请检查开关。压缩只作用于精确 tokenizer 路由上新鲜、超大、源码类的工具结果，且每次跳过都会在审计记录中留有原因。
-- **开关显示为不可读**：已存的 `codeSkeleton` 段未通过严格的浏览器解码（必须恰好是 `{ enabled: boolean }`）。删除畸形段即可恢复默认。
-- **升级步骤失败**：插件遵循 npm 包语义；如果你的 Harness 构建拒绝 tarball 到 tarball 的升级，请先移除旧版本再安装。
+# 安装 dsh-context-compression-improved
+
+> [English](installation.md) · [中文](installation.zh.md) · [日本語](installation.ja.md) · [한국어](installation.ko.md)
+
+本教程从源码安装 fork。fork 亦已发布到 npm，dist-tag 为 `dsh-0.1.5`；包名有意与上游保持一致：
+`dsh-context-compression-improved`（单一包 —— 原先独立发布的 runtime 包已并入其中）。
+
+## 前置条件
+
+- Node `^22.19.0 || >=24` 与 pnpm `11.7.0`（`corepack enable` 会按 `packageManager` 字段使用固定版本）。
+- 兼容 `0.1.1-rc.2` peer 范围的 DeepSeek Harness（已针对官方 `dsh-v0.1.2-alpha.5` 验证）。
+- DeepSeek 模型路由。有损压缩——包括代码骨架门——按字符基准决策，不再要求内建精确 tokenizer；存在内建 tokenizer 时其 exact 计数仅作遥测记录，其他路由 fail-open 并保留原始工具结果。
+- Git。
+
+## 1. 从源码构建
+
+```sh
+git clone https://github.com/drscrewdriver/dsh-context-compression-improved.git
+cd dsh-context-compression-improved
+pnpm install --frozen-lockfile
+pnpm build
+```
+
+`pnpm build` 会打包所有包的两套产物（`tsdown`）。如需在安装前先跑全量测试，可执行 `pnpm test`。
+
+## 2. 打包 Bundle 入口包
+
+selector 包是唯一的 Bundle 入口；runtime 作为其精确版本依赖自动随行：
+
+```sh
+cd packages/selector
+pnpm pack
+# → dsh-context-compression-improved-0.1.0.tgz
+cd ../..
+```
+
+`pnpm pack` 会通过 `prepack` 钩子执行打包，因此 tarball 始终与你的检出内容一致。
+
+## 3. 安装到 Harness Profile
+
+selector 包声明了 Harness Bundle manifest 字段 `dsh.bundle.patch`，因此 `dsh plugin add` 是标准的树外 Bundle 安装方式：
+
+```sh
+dsh plugin --profile web add packages/selector/dsh-context-compression-improved-0.1.0.tgz
+dsh --profile web --dump-config
+```
+
+安装后重启对应 Profile。配置导出中应显示 selector Bundle 已激活。**不要**分别安装或手动连接 selector 与 runtime 两个包——runtime 会自动安装。
+
+## 4. 打开代码骨架门
+
+打开 DeepSeek Harness 设置 → **上下文压缩选择器*：
+
+1. 选择一个压缩 Profile（这道门与所有 Profile 正交）。
+2. 按需调整 Auto Compact 触发水位（50–90%，默认 80%）。
+3. 将**代码骨架压缩**设为**开**。开关即改即存。
+
+与所有选择器设置一致，取值在会话首次观察时冻结——这道门只影响新观察的会话，不会改变正在运行的任务。
+
+## 5. 可选：建议型相关度 advisor
+
+插件可以对"会话历史还有多相关"做统计——仅建议性质，不做任何决策、也不阻断任何流程。默认关闭；本轮没有设置卡片，请直接编辑上下文压缩设置中的 `presetOptions` 段（settings JSON）：
+
+```json
+"presetOptions": {
+  "advisorMode": "host",
+  "advisorRefreshTurns": 8,
+  "advisorScoreThreshold": 0.35,
+  "advisorSampleLimit": 16,
+  "advisorMinTokens": 250,
+  "advisorTimeoutMs": 8000
+}
+```
+
+`advisorMode: "host"` 走 harness `llm` 服务；`"direct"` 复用 estimator 的
+`estimatorBaseUrl` / `estimatorApiKey` / `estimatorModel` 端点。每个 turn 边界，advisor
+会：(1) 从最近的 `todo/write` 事件总结当前任务语义；(2) 对历史 tool result 做"内容+注释
+语义 ↔ 当前任务"的增量相关度打分；(3) 记录前缀腐化度（prefix-decay）。结果以
+`advisor-outcome` 审计记录呈现；部署配置打开 `advisorReportRoute: true` 后，还可经只读
+HTTP 路由 `GET .../advisor-report?sessionId=` 读取。advisor 报告的任何内容都不会抑制、
+延迟或改写任何本应落地的 reduction。
+
+## 6. 更新或卸载
+
+```sh
+# 更新：拉取、重建、重新打包、再次添加新 tarball
+git pull && pnpm install --frozen-lockfile && pnpm build
+cd packages/selector && pnpm pack && cd ../..
+dsh plugin --profile web add packages/selector/dsh-context-compression-improved-0.1.0.tgz
+
+# 卸载
+dsh plugin --profile web remove dsh-context-compression-improved
+```
+
+## 故障排除
+
+- **配置导出中 Bundle 未激活**：重启 Profile；确认添加的是 selector 入口包（而非 runtime），且 Harness 版本在兼容的 peer 范围内。
+- **工具结果从未被骨架化压缩**：该门默认关闭，请检查开关。压缩只作用于新鲜、超大、源码类的工具结果（决策按字符基准执行，无精确 tokenizer 路由要求），且每次跳过都会在审计记录中留有原因。
+- **开关显示为不可读**：已存的 `codeSkeleton` 段未通过严格的浏览器解码（必须恰好是 `{ enabled: boolean }`）。删除畸形段即可恢复默认。
+- **升级步骤失败**：插件遵循 npm 包语义；如果你的 Harness 构建拒绝 tarball 到 tarball 的升级，请先移除旧版本再安装。

package/docs/repair-log.md

@@ -357,6 +357,59 @@ here for reasons the project's own `upgrade-pitfalls` §2.1 records independentl
 
 ---
 
+## D8 — The search reducer dropped 21.2% of hits and never reported where they went
+
+**Symptom.** Large `grep`/`rg` results compressed by `search-by-file` lost every hit the
+per-file "first 4 + last" rule did not keep, and the file header only said `(592 matches)`
+with no line numbers. The model could not tell which hits were dropped or where to re-read.
+
+**Root cause.** `reduceSearch` filled each file's keep-set from index 0 (`keep.size < 5`)
+and reported only counts. Nothing in the output identified the omitted hit positions, so the
+loss was invisible and unrecoverable except by re-running the search.
+
+**Evidence.** 14 real sessions (18.6M characters, DSH 0.1.5-rc.2): 186 search events /
+5,301 hits; per-file median 18, p90 55, max 592; 1,122 hits (21.2%) discarded silently.
+
+**Affected surface.** `packages/selector/src/runtime/reducers.ts` (`reduceSearch`).
+
+**Fix.** Two-tier folding (R10): L1 is a lossless per-file locator
+(`## <path> (N matches)  L12,L15,…`, one line number per hit, budget reserved first); L2 is
+the content quota, water-filled round-robin so no file vanishes. When L1 itself cannot fit
+the shortfall is announced (withheld file/match counts), never silently truncated.
+
+**Verification.** `packages/selector/tests/runtime/search-reducer.spec.ts`: a 592-hit file
+reports 592 locator numbers; water-filling serves every file before any file takes a second
+row; withheld locators are announced in the output.
+
+---
+
+## D9 — HTML fell into `pi-head`, which kept the `<head>` and dropped the body
+
+**Symptom.** Fetched/compressed HTML pages kept `<!DOCTYPE>`, `<meta>`, `<link>`,
+`<script>` and `<style>` — the metadata — while the entire body content disappeared.
+
+**Root cause.** `looksLikeSourceCode` matches no HTML tags (its patterns cover
+declarations/imports/decorators only), so pages fell through to the head/tail fallbacks, and
+`reduceHead` takes from the top — exactly the worst segment for HTML.
+
+**Evidence.** `CODE_STRUCTURE_PATTERN` / `CODE_IMPORT_PATTERN` / `CODE_DECORATOR_PATTERN`
+match none of `<html|<div|<script|…`; the repository carried zero HTML-input tests (31
+`<html|<div|<script` hits, all in TSX sources and tokenizer vocabularies).
+
+**Affected surface.** `packages/selector/src/runtime/reducers.ts` (candidate chain).
+
+**Fix.** Two-stage reduction (R13): `html-slim` strips comments, script/style/noscript/svg/
+head elements, data URIs, non-whitelisted attributes (`href/src/alt/title/id` survive) and
+inline-tag markup, line-aligned so original-event line numbers survive; `html-skeleton`
+keeps the heading hierarchy, section first lines and table header rows under tighter budgets.
+Classification is pure form (≥3 markup-tag lines in the first 400).
+
+**Verification.** `packages/selector/tests/runtime/html-reducer.spec.ts`: body paragraphs
+and headings survive; script/style/comment/data-URI content is gone; the skeleton fallback
+cites original line ranges; TS generics/comparisons are not misclassified as HTML.
+
+---
+
 ## U1 — The estimator card vanished off TokenPilot-inspired instead of explaining its gate
 
 **Symptom.** With any profile other than TokenPilot-inspired selected, the Settings page showed
@@ -601,3 +654,4 @@ caused.
 | 2026-09-15 | doc corruption — mechanism | **Diagnosed.** A damaged spot is the 2-byte prefix of a three-byte UTF-8 character followed by `0x3F`: the character lost its third byte and, in most spots, the byte that followed it was consumed too (a double-byte-code-page decode/write pair collapse; 0 or 1 bytes lost per spot). The damage is **inherited, not produced here**: the newest valid blob of every affected file is `e337bf5`, while the same files are already defective at the `compat/0.1.5` baseline `d7c592d` and at `04f86e4`. Ten files carry it, not five — `README.{zh,ja,ko}.md` and `scripts/packed-install-e2e.mjs` were missed by the earlier note |
 | 2026-09-15 | doc corruption — repaired | **Batch I / T-I2.** All ten files repaired by restoring each damaged spot from `e337bf5`, with three independent checks: the restored character must carry the surviving 2-byte prefix; re-corrupting the repair reproduces the previous bytes exactly (so the edit touches nothing but the damage, and no line, no EOL and no other character moves); and the repair must agree with the valid ancestor everywhere outside the restored spots. Every affected file is now valid UTF-8. The English `README.md` is the clean case: ten spots, all `—`/quote characters plus their following space, zero other differences from the ancestor |
 | 2026-09-15 | ledger encoding | The twelve `—` characters in this file had been mangled to `鈥?` by the PowerShell port of the ledger (the 0.1.2 source has none); restored. Same class as the doc corruption above, introduced by that one-time port rather than inherited |
+| 2026-09-19 | D8/D9 | Compression-quality defects ledgered on `feat/ctx-compression-v2-compat015`: search hits were dropped unlocatably (21.2%), HTML kept its `<head>` and lost its body; both fixed with two-tier search folding and two-stage HTML reduction, each with counter-proof tests |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-context-compression-improved",
-  "version": "0.4.0-beta.1",
+  "version": "0.5.0",
   "publishConfig": {
     "registry": "https://registry.npmjs.org",
     "access": "public",