dsh-context-toolkit 0.3.3 → 0.3.5

package/CHANGELOG.md

@@ -4,6 +4,12 @@
 
 User-visible changes, newest first. Commit hashes in brackets.
 
+## 0.3.5
+- **Fixed: four gate bypasses.** The gate asks whether the *command word* of a segment is a flood tool, so anything that legally precedes that word has to be peeled first — otherwise it inspects a variable assignment and calls it safe. Previously missed: `FOO=1 curl …`, `PATH=/x curl …` (assignment prefixes were never peeled), `nice -n 5 curl …`, `timeout 30 curl …` (a wrapper's own options were read as the command word), and `sudo FOO=1 curl …` (the prefix loop capped at two tokens). Prefixes are now peeled as assignment → wrapper → wrapper options → assignment, with seven must-block and six must-allow assertions; two negative controls knock out the assignment peel and the option skip respectively. [`TBD`]
+
+## 0.3.4
+- **Fixed**: output that a tool already truncated is no longer compressed a second time. An upstream tool (`ops_bash`) truncates its own output and spills the rest to a file, and the sandbox (`ctk_execute`) caps a run at 256 KB and closes with a `~~~` line; this layer used to compress those already-handled bodies again — saving nothing, storing a second copy of the same original, and counting the same traffic twice. Both markers (`[output truncated; full output: …]` and a standalone `~~~` line) are recognised and skipped, and the reason is reported in a new "skip reasons" ledger line. [`038447b`]
+
 ## 0.3.3
 - **Docs**: adds this bilingual changelog, listed in the manifest and gated by release-check; the README and repository description were refreshed too. The npm package now carries the changelog with each release. [`01d7107`]
 

package/CHANGELOG.zh.md

@@ -4,6 +4,12 @@
 
 本文件按版本记录用户可感知的变化。方括号内为对应提交。
 
+## 0.3.5
+- **修复：门禁四处绕过**。门禁判断的是「某段的命令词是不是 curl/wget」，因此凡合法地排在命令词前面的东西都必须先剥掉，否则它检查的是变量赋值并被判为安全。此前漏掉的：`FOO=1 curl …`、`PATH=/x curl …`（赋值前缀从未剥离）、`nice -n 5 curl …`、`timeout 30 curl …`（包装器自身选项被当成命令词）、`sudo FOO=1 curl …`（前缀上限 2 个 token）。现按「赋值 → 包装器 → 包装器选项 → 再赋值」的顺序剥离，并补 7 条必须拦、6 条必须放的断言；两条负向对照分别精确打掉赋值剥离与选项跳过。[`TBD`]
+
+## 0.3.4
+- **修复**：对**已被上游截断**的输出不再重复压缩。上游工具（如 `ops_bash`）会自行截断并落盘，沙箱（`ctk_execute`）也有 256 KB 捕获上限并以 `~~~` 收尾；此前本层会对这些**已经处理过**的产物再压一次 —— 省不了任何字节，却把同一份原文**又存一遍**，等于把同一份流量记两次收益。现识别这两种截断标记（`[output truncated; full output: …]` 与独立的 `~~~` 行）并跳过，同时把跳过原因计入台账新增的「跳过原因」一行。[`038447b`]
+
 ## 0.3.3
 - **文档**：新增中英双语更新说明（本文件），并纳入发布清单与发布门；README 与仓库首页说明同步更新。npm 包开始随版本携带 CHANGELOG。[`01d7107`]
 

package/README.md

@@ -115,7 +115,7 @@ Tool switches: `tools.execute`, `tools.executeFile`, `tools.batchExecute`, `tool
 The offline suites need no DSH runtime:
 
 ```bash
-node smoke.mjs        # 85 checks: contract, runtime discovery, tools, gate (segment matching, host-aligned path base, script_path and host-fetcher coverage), output slimming with reversible retrieval and retention, resource disposal, knowledge base, observability, version reporting
+node smoke.mjs        # 88 checks: contract, runtime discovery, tools, gate (segment matching, host-aligned path base, script_path and host-fetcher coverage), output slimming with reversible retrieval and retention, resource disposal, knowledge base, observability, version reporting
 node test-client.mjs  # 27 checks: browser-half contract and bilingual dictionary parity
 node test-package.mjs # 20 checks: publish manifest and the zero-dependency constraint
 node release-check.mjs # 30 checks: release readiness (manifest, deps, tool count, gate matching, version, SHA coverage, no hardcoded paths)
@@ -147,6 +147,7 @@ At `tools/post-execute`, the strategy follows the shape:
 - **`ctk_execute` timeouts are hard kills**: the child is terminated and `timedOut` is reported.
 - **A single run captures at most 256 KB**; anything beyond is counted but not retained. The text handed back is additionally capped at 4000 characters, so even a 1 MB single-line output cannot enter the conversation whole.
 - **The knowledge base deduplicates by path**: filing the same path again replaces the earlier copy.
+- **Output a tool already truncated is not compressed again.** A body that ends in `[output truncated; full output: …]`, or in a standalone `~~~` line from the sandbox capture cap, is treated as work already done: this layer skips it and counts the reason in the ledger's "skip reasons" line.
 - **The gate is precise, not exhaustive.** A flood tool is refused only when it is the command word of some segment — after exec prefixes (`sudo`, `exec`, …) and path/extension normalisation are stripped — and `script_path` is read up to the first 256 KB. A tool a program invokes internally (`python -c 'import wget'`) is outside its remit.
 - **Paths resolve the way the host resolves them.** For `script_path` and for `read`'s `file_path`, a relative path is resolved against the session working directory — the same base the host uses to open the file. A gate reading that string against its own cwd would miss anything the host can still run.
 

package/README.zh.md

@@ -116,7 +116,7 @@ await ctk_search({ queries: ['rate limit', 'auth'] });  // 只取相关片段
 离线套件不依赖 DSH 运行时：
 
 ```bash
-node smoke.mjs        # 85 项：契约、运行时探测、工具、门禁（分段匹配 + 路径基准对齐 + script_path 与取网页工具覆盖）、输出瘦身与可逆取回及保留期、资源释放、知识库、观测、版本自述
+node smoke.mjs        # 88 项：契约、运行时探测、工具、门禁（分段匹配 + 路径基准对齐 + script_path 与取网页工具覆盖）、输出瘦身与可逆取回及保留期、资源释放、知识库、观测、版本自述
 node test-client.mjs  # 27 项：浏览器半体契约与双语字典一致性
 node test-package.mjs # 20 项：发布清单与零依赖约束
 node release-check.mjs # 30 项：发布就绪（清单/依赖/工具数/门禁匹配/版本/SHA 覆盖/无硬编码路径）
@@ -148,6 +148,7 @@ node release-check.mjs # 30 项：发布就绪（清单/依赖/工具数/门禁
 - **`ctk_execute` 的超时是硬杀**：超时会终止子进程并把 `timedOut` 标出来。
 - **单次捕获上限 256 KB**，超出部分只计数不保留；回给对话的文本另有 4000 字符硬帽，因此 1 MB 的单行输出也不会整段进入上下文。
 - **知识库按路径去重**：同一路径再次归档是替换，不是追加。
+- **已被上游截断的输出不再压缩**：上游工具自行截断并落盘的产物（`[output truncated; full output: …]`）与沙箱捕获上限的收尾（独立的 `~~~` 行）都视为已完成的工作，本层跳过并在台账「跳过原因」中计数。
 - **门禁精确而非穷尽**：只有当洪水工具是某个分段的「命令词」时才拒绝（会先剥掉 `sudo`/`exec` 等前缀、归一化路径与 `.exe` 等扩展名），`script_path` 最多读取前 256 KB。程序内部发起的调用（如 `python -c 'import wget'`）不在其职责内。
 - **路径基准与宿主一致**：门禁读取 `script_path` 与 `read` 的 `file_path` 时，按**会话工作目录**解析相对路径（与宿主实际打开文件的基准相同）；否则「宿主能跑、门禁看不见」就是一条现成的绕过路径。
 

package/index.js

@@ -271,6 +271,8 @@ export function apply(ctx, rawConfig) {
     slimmedBytes: 0,
     slimmedTokens: 0,
     slimSkipped: 0,
+    /** Why a body was left alone — "already truncated upstream" is a saving, not a miss. */
+    slimSkips: {},
     slimStrategies: {},
     blobsStored: 0,
     calls: {},
@@ -356,7 +358,23 @@ const SHELL_TOOL_NAMES = new Set(['bash', 'pwsh', 'shell', 'ops_bash', 'bash_per
  * The list stays short on purpose: anything longer turns into a shell parser, and a
  * parser that gets it wrong either blocks an innocent call or waves a real one past.
  */
-const EXEC_PREFIXES = new Set(['!', 'sudo', 'command', 'exec', 'nohup', 'time', 'env', 'busybox'])
+const EXEC_PREFIXES = new Set([
+  '!', 'sudo', 'command', 'exec', 'nohup', 'time', 'env', 'busybox',
+  // Wrappers that take their own options before the real command word. Without
+  // these, "timeout 30 curl …" and "nice -n 5 curl …" read as a command called
+  // "timeout"/"nice" and sailed through.
+  'nice', 'ionice', 'timeout', 'stdbuf', 'setsid', 'chrt', 'taskset', 'xvfb-run',
+])
+
+/**
+ * A shell assignment prefix: `FOO=1`, `PATH=/x`, `LC_ALL=C`.
+ *
+ * `FOO=1 curl …` is an ordinary way to run curl and the assignment is not a command
+ * word at all, so the gate used to read "FOO=1" as the command and wave the call
+ * through. Only the leading run of assignments is skipped; a bare `=` later in the
+ * line is an argument and must not be treated as one.
+ */
+const ASSIGNMENT_PREFIX = /^[A-Za-z_][A-Za-z0-9_]*=/
 
 /**
  * Split a command line at every operator that starts a new command.
@@ -378,8 +396,17 @@ function commandSegments(command) {
 function leadingWord(segment) {
   const tokens = segment.split(/\s+/).filter(Boolean)
   let i = 0
-  while (i < tokens.length && i < 2 && EXEC_PREFIXES.has(tokens[i].toLowerCase().replace(/^!+/, ''))) i += 1
-  return (tokens[i] || '').replace(/^[\"']+|[\"']+$/g, '')
+  // Peel assignment prefixes first: FOO=1 sudo curl has both, in that order.
+  while (i < tokens.length && ASSIGNMENT_PREFIX.test(tokens[i])) i += 1
+  // Then exec wrappers, each possibly carrying its own options and their values
+  // (nice -n 5, timeout 30, ionice -c 3). Consume what is recognisable and stop
+  // at the first token that is neither; that token is the command word.
+  while (i < tokens.length && EXEC_PREFIXES.has(tokens[i].toLowerCase().replace(/^!+/, ''))) {
+    i += 1
+    while (i < tokens.length && (tokens[i].startsWith('-') || /^\d+[smhd]?$/.test(tokens[i]))) i += 1
+    while (i < tokens.length && ASSIGNMENT_PREFIX.test(tokens[i])) i += 1
+  }
+  return (tokens[i] || '').replace(/^["']+|["']+$/g, '')
 }
 
 /** Normalise a command word to the plain tool name (`C:\bin\curl.exe` -> `curl`). */
@@ -544,7 +571,15 @@ function registerOutputSlimmer(ctx, config, stats, store, log) {
           rewritten.push(block)
           continue
         }
-        const outcome = slimText(block.text, { limits, isError })
+        // onSkip records why a body was exempted. "The tool already truncated it" is a
+        // legitimate outcome and belongs in the ledger beside the ones that saved bytes;
+        // otherwise the only visible number is how often this layer did nothing.
+        const outcome = slimText(block.text, {
+          limits,
+          isError,
+          label: name,
+          onSkip: (why) => { stats.slimSkips[why] = (stats.slimSkips[why] || 0) + 1 },
+        })
         if (!outcome) { stats.slimSkipped += 1; rewritten.push(block); continue }
 
         let marker = ''
@@ -1447,6 +1482,7 @@ function registerObservabilityTools(ctx, config, stats, store, storageDir, log)
           '- 沙箱保留 bytesSandboxed: ' + stats.bytesSandboxed + ' 字节（子进程产出但只回结论的部分）',
           '- 已归档 bytesIndexed: ' + stats.bytesIndexed + ' 字节（存入知识库、未留在对话里）',
           '- 输出瘦身 slimmed: ' + stats.slimmed + ' 块（' + stats.slimmedBytes + ' 字节 ≈ ' + stats.slimmedTokens + ' tokens 未进入上下文；跳过的块 ' + stats.slimSkipped + '）',
+          '- 跳过原因: ' + (Object.keys(stats.slimSkips).length ? Object.keys(stats.slimSkips).map((k) => k + '×' + stats.slimSkips[k]).join(' · ') : '（无）'),
           '- 合计 kept out of context: ' + keptOut + ' 字节 ≈ ' + Math.round(keptOut / 4) + ' tokens（按 4 字节/token 估算）',
           '',
           '### 知识库 / knowledge base',

package/lib/compress.js

@@ -47,6 +47,26 @@ function looksLikeNumberedSource(text) {
   return lines.length > 4 && numbered / lines.length > 0.5
 }
 
+/**
+ * Was this output already truncated upstream, by the tool that produced it?
+ *
+ * Two markers are in the wild on this machine. A torn subprocess output ends with a
+ * bracket notice naming the spill file; the ctk_* sandbox caps what it hands back and
+ * ends with a tilde line. In both cases the bytes that would have flooded the context
+ * are already gone and already stored elsewhere, so compressing again cannot save
+ * anything — it only stores a second copy of the same original and claims the same
+ * saving twice. Telling them apart is the difference between a gate and a gate that
+ * bills the same traffic twice.
+ */
+export function isAlreadyTruncated(text) {
+  const source = String(text == null ? '' : text)
+  // The notice is a whole line, so anchor it: a passing mention of the phrase inside
+  // prose must not exempt the body it appears in.
+  if (/^\[output truncated; full output: .+\]$/m.test(source)) return 'upstream spill notice'
+  if (/^~+$/m.test(source)) return 'sandbox capture cap'
+  return null
+}
+
 /** What kind of output is this? The strategy follows the shape, not a guess. */
 export function classify(text) {
   const source = String(text || '')
@@ -132,12 +152,41 @@ function windowTable(lines, header, limits) {
   return header ? [header].concat(kept) : kept
 }
 
+/**
+ * Cut a string at a character boundary that does not split a surrogate pair.
+ *
+ * `String.prototype.slice` counts UTF-16 code units, so a naive cut through an
+ * astral character (an emoji, an uncommon CJK ideograph) leaves a lone surrogate
+ * behind. That lone surrogate then goes to the store, is written to SQLite as
+ * UTF-8, and reads back as U+FFFD — the text a user is shown has been corrupted
+ * at exactly the cut, while the ledger still claims the original is retrievable
+ * byte-for-byte. The original *is* retrievable; the copy in the transcript is
+ * what rotted, which is the one thing a reversible slimmer must never do.
+ *
+ * @param text - the source string.
+ * @param index - a UTF-16 offset that may land inside a surrogate pair.
+ * @returns the same offset, moved one code unit left when it would split a pair.
+ */
+function safeCut(text, index) {
+  if (index <= 0) return 0
+  if (index >= text.length) return text.length
+  const before = text.charCodeAt(index - 1)
+  const at = text.charCodeAt(index)
+  // A high surrogate followed by a low one is a pair; cutting between them is the bug.
+  const splitsPair = before >= 0xd800 && before <= 0xdbff && at >= 0xdc00 && at <= 0xdfff
+  return splitsPair ? index - 1 : index
+}
+
 /** Keep a head and a tail of a plain-text body, with an explicit elision marker. */
 function windowText(text, limits) {
   const { headChars, tailChars } = limits
   if (text.length <= headChars + tailChars) return null
-  const head = text.slice(0, headChars)
-  const tail = text.slice(text.length - tailChars)
+  // Both cuts go through safeCut: a lone surrogate here would corrupt the transcript
+  // copy while the stored original stays intact, which is the worst of both worlds.
+  const headEnd = safeCut(text, headChars)
+  const tailStart = safeCut(text, text.length - tailChars)
+  const head = text.slice(0, headEnd)
+  const tail = text.slice(tailStart)
   const dropped = text.length - headChars - tailChars
   return head + '\n… ' + dropped + ' characters elided …\n' + tail
 }
@@ -156,6 +205,14 @@ export function slimText(text, options = {}) {
   const floor = options.isError ? limits.errorMinBytes : limits.minBytes
   if (originalBytes < floor) return null
 
+  // Exempt before any strategy runs: work already done upstream is not work saved here.
+  // Reported through onSkip so the caller can count it instead of silently ignoring it.
+  const truncatedBy = isAlreadyTruncated(source)
+  if (truncatedBy) {
+    if (typeof options.onSkip === 'function') options.onSkip(truncatedBy)
+    return null
+  }
+
   const kind = classify(source)
   if (kind === 'code' || kind === 'empty') return null
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-context-toolkit",
-  "version": "0.3.3",
+  "version": "0.3.5",
   "description": "DeepSeek Harness bundle: native ctk_* sandbox tools, a routing gate that keeps bulk output out of the transcript, and a local full-text knowledge base. No third-party runtime dependencies.",
   "repository": {
     "type": "git",