billion-context-dsh 0.2.8 → 0.2.9

package/README.en.md

@@ -3,7 +3,7 @@
 [English](./README.en.md) | [中文](./README.md)
 
 > **⚠️ Beta notice — not for production use**
-> This project (**v0.2.8**) is a work-in-progress beta. The [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) itself is also in **public beta**. **Do not use either in engineering / production environments** — expect breaking changes and rough edges.
+> This project (**v0.2.9**) is a work-in-progress beta. The [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) itself is also in **public beta**. **Do not use either in engineering / production environments** — expect breaking changes and rough edges.
 
 <p align="center">
 <strong>Built with gratitude on top of these projects</strong> — please give them a ⭐:
@@ -133,6 +133,7 @@ DSH derives every model request from its append-only session log (the *surface*)
 | `acp_status` | CONTEXT BREAKDOWN (tool/text/summaries shares of the visible total) + compressed-block ledger + nudge decision line; no context-window rows; scope/view/tool/sort/limit drilldown supported |
 | block state | in-memory kernel state + **log-rebuilt ledger** (no sidecar files) |
 | tiered distillation (T2/T3) | re-compressing a block's summary node distills that block (tier 2); distilling a tier-2 block yields tier 3. Tier + kernel block ids are persisted to the log, so kernel state rehydrates from the log after a restart and stays distillable |
+| compression accounting (shadow price) | `shadowedTokenCount` (what the host occupancy display deducts) is priced in the **host token-meter's vocabulary** (`ctx.tokenMeter.measure` preferred; exact mirror in `src/host-tokens.ts` as fallback) — never the plugin's internal CJK-aware estimate (that is display currency; mixing it into the host ledger can drive `messageTokens` negative and brick a CJK-heavy session, issue #54) |
 
 The load-bearing compression guidance (tools, philosophy, summary rules, tier rules) is registered as a one-time system-prompt section; each nudge carries a condensed version (efficiency note + philosophy + context breakdown + HOW_TO_COMPRESS_RULES + range table + batch tip). There is deliberately **no automatic summarization**: automatic policy only nudges the model (`compactIfNeeded` returns null).
 

package/README.md

@@ -3,7 +3,7 @@
 [中文](./README.md) | [English](./README.en.md)
 
 > **⚠️ 测试版声明——请勿用于生产环境**
-> 本项目（**v0.2.8**）仍处于开发中的测试版。[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) 本身也处于**公开测试版**阶段。**请勿将两者用于工程化 / 生产环境**——预期会有破坏性变更与粗糙之处。
+> 本项目（**v0.2.9**）仍处于开发中的测试版。[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) 本身也处于**公开测试版**阶段。**请勿将两者用于工程化 / 生产环境**——预期会有破坏性变更与粗糙之处。
 
 <p align="center">
 <strong>衷心感谢以下项目——请给它们一个 ⭐：</strong>
@@ -134,6 +134,7 @@ DSH 的每个模型请求都派生自其 append-only 会话日志（*surface*）
 | `acp_status` | CONTEXT BREAKDOWN（tool/text/summaries 占可见总量）+ 压缩块账本 + nudge 决策行；不含上下文窗口；支持 scope/view/tool/sort/limit 钻取 |
 | 块状态 | 内存内核状态 + **日志重建账本**（无旁车文件） |
 | 分层蒸馏（T2/T3） | 再次压缩某块的摘要节点 = 蒸馏该块（tier 2），蒸馏 tier-2 块得 tier 3；tier 与内核块 id 持久化进日志，重启后内核状态从日志再水合、可继续蒸馏 |
+| 压缩记账（影子价格） | `shadowedTokenCount`（宿主占用率据此扣减）**用宿主 token-meter 的词汇计价**（`ctx.tokenMeter.measure` 优先，`src/host-tokens.ts` 精确镜像兜底）——绝不混用插件内部的 CJK 感知估算（那是展示货币，混用会把宿主账本扣成负数、卡死中文会话，issue #54） |
 
 承载性的压缩指引（工具、哲学、摘要规则、tier 蒸馏/浓缩规则）注册为一次性系统提示段；每条 nudge 携带精简版（效率提示 + 哲学 + 上下文分解 + 压缩规则 + 范围表 + 批量提示）。刻意**不做自动摘要**：自动策略只 nudge 模型（`compactIfNeeded` 返回 null）。
 

package/dist/host-tokens.d.ts

@@ -0,0 +1,72 @@
+/**
+ * Host-vocabulary token pricing for the durable shadow-price protocol.
+ *
+ * The host token-meter prices every appended message with a fixed flat-4
+ * heuristic (`estimateContent` / `estimateMessage` in `dsh-token-meter`) and
+ * the producer contract requires every `compaction/summary`/`compaction/prune`
+ * `shadowedTokenCount` claim to be derived from the SAME estimator. Writing
+ * claims with the engine's CJK-aware `defaultCountTokens` overdraws the meter
+ * on CJK-heavy sessions and permanently bricks them (live session
+ * `session-3aa366c3`, issue #54; AGENTS.md rule 12 — `defaultCountTokens` is
+ * display currency, NEVER event currency).
+ *
+ * This module prices claims in the host's vocabulary: it prefers the live
+ * meter's own per-node prices (`ctx.tokenMeter.measure(session).nodes` —
+ * exact by construction, follows host estimator changes automatically, the
+ * same path the host's own `compaction-basic` uses) and falls back to an
+ * exact mirror of the host's estimator when the meter is unreachable.
+ */
+import type { Session, SessionEvent } from '@deepseek-ai/dsh-session';
+/** The host's model-visible content block union (structural, mirror-side only). */
+export type HostBlock = {
+    type: 'text';
+    text: string;
+} | {
+    type: 'reasoning';
+    text: string;
+} | {
+    type: 'tool-call';
+    name: string;
+    arguments: string;
+} | {
+    type: 'tool-result';
+    toolCallId: string;
+    content: HostContent;
+} | ({
+    type?: string;
+} & Record<string, unknown>);
+/** A content block list, or a bare string (`tool-result` content may be either). */
+export type HostContent = readonly HostBlock[] | string;
+/**
+ * Exact mirror of the host's `estimateContent`
+ * (`@deepseek-ai/dsh-token-meter/lib/types/estimate.js`): text/reasoning
+ * `ceil(len/4)+4`, tool-call `ceil(name/4)+ceil(arguments/4)+4`, tool-result
+ * recursive over its content, unknown blocks `4+ceil(JSON.stringify/4)` over
+ * the ORIGINAL block object. A string content is iterated as an iterable, so
+ * every CHARACTER falls to the default branch (`4+ceil(JSON.stringify(char)/4)`
+ * — 5 tokens for any single unescaped character).
+ */
+export declare function estimateHostContent(blocks: HostContent): number;
+/** Exact mirror of the host's `estimateMessage` (content + role framing). */
+export declare function estimateHostMessage(message: {
+    content: HostContent;
+}): number;
+/**
+ * Host price of ONE session event under the mirror: project it through the
+ * host's `deriveEventMessage` (null for non-surface events and empty-content
+ * assistant messages) and price the derived message; null derives to 0.
+ */
+export declare function hostPriceEvent(event: SessionEvent): number;
+/** Mirror price of a set of surface seqs (the fallback claim computation). */
+export declare function shadowedHostTokens(session: Session, seqs: readonly number[]): number;
+/**
+ * Claim price for `seqs` in the host's vocabulary. Prefers the live meter's
+ * own per-node prices when `ctx.tokenMeter` is reachable and covers every
+ * shadowed seq (exact by construction, follows host estimator changes); ANY
+ * failure — meter absent, `measure` throwing (e.g. a step-less log), or a seq
+ * missing from the measurement — falls back to the exact mirror. Never returns
+ * a `defaultCountTokens` price (rule 12).
+ */
+export declare function shadowedTokensViaMeter(session: Session, seqs: readonly number[], ctx?: {
+    get?(name: string): unknown;
+} | null): number;

package/dist/index.js

@@ -2775,6 +2775,84 @@ function extractEventText(event) {
   }
 }
 
+// src/host-tokens.ts
+import { deriveEventMessage } from "@deepseek-ai/dsh-session";
+var CHARS_PER_TOKEN = 4;
+var BLOCK_OVERHEAD = 4;
+var ROLE_OVERHEAD = 4;
+function blockType(block) {
+  if (typeof block !== "object" || block === null) return void 0;
+  const type = block.type;
+  return typeof type === "string" ? type : void 0;
+}
+function estimateHostContent(blocks) {
+  if (typeof blocks === "string") {
+    let tokens2 = 0;
+    for (const char of blocks) {
+      tokens2 += BLOCK_OVERHEAD + Math.ceil(JSON.stringify(char).length / CHARS_PER_TOKEN);
+    }
+    return tokens2;
+  }
+  let tokens = 0;
+  for (const block of blocks) {
+    switch (blockType(block)) {
+      case "text":
+      case "reasoning": {
+        tokens += Math.ceil(block.text.length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD;
+        break;
+      }
+      case "tool-call": {
+        const call = block;
+        tokens += Math.ceil(call.name.length / CHARS_PER_TOKEN) + Math.ceil(call.arguments.length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD;
+        break;
+      }
+      case "tool-result": {
+        tokens += estimateHostContent(block.content) + BLOCK_OVERHEAD;
+        break;
+      }
+      default:
+        tokens += BLOCK_OVERHEAD + Math.ceil(JSON.stringify(block).length / CHARS_PER_TOKEN);
+    }
+  }
+  return tokens;
+}
+function estimateHostMessage(message) {
+  return estimateHostContent(message.content) + ROLE_OVERHEAD;
+}
+function hostPriceEvent(event) {
+  const message = deriveEventMessage(event);
+  return message === null ? 0 : estimateHostMessage(message);
+}
+function shadowedHostTokens(session, seqs) {
+  let total = 0;
+  for (const seq of seqs) {
+    const event = session.events[seq];
+    if (event !== void 0) total += hostPriceEvent(event);
+  }
+  return total;
+}
+function shadowedTokensViaMeter(session, seqs, ctx) {
+  try {
+    const meter = ctx?.get?.("tokenMeter");
+    if (meter?.measure !== void 0) {
+      const bySeq = new Map(meter.measure(session).nodes.map((node) => [node.seq, node.tokens]));
+      let total = 0;
+      let missing = false;
+      for (const seq of seqs) {
+        const tokens = bySeq.get(seq);
+        if (tokens === void 0) {
+          missing = true;
+          break;
+        }
+        total += tokens;
+      }
+      if (!missing) return total;
+    }
+  } catch {
+  }
+  return shadowedHostTokens(session, seqs);
+}
+
 // src/region.ts
 function findOpenTurn(events) {
   let open = null;
@@ -3027,14 +3105,14 @@ function assistantProviderModel(event) {
   }
   return { provider: "billion-context-dsh", model: "surface-prune" };
 }
-function hideSurfaceSeqs(session, seqs, provider, model, text) {
+function hideSurfaceSeqs(session, seqs, provider, model, text, priceEvent = hostPriceEvent) {
   if (seqs.length === 0) return;
   const start = seqs[0];
   const end = seqs[seqs.length - 1];
   let shadowedTokenCount = 0;
   for (const seq of seqs) {
     const event = session.events[seq];
-    if (event !== void 0) shadowedTokenCount += defaultCountTokens(extractEventText(event));
+    if (event !== void 0) shadowedTokenCount += priceEvent(event);
   }
   session.append("compaction/prune", {
     shadowedRange: { start, end },
@@ -3536,7 +3614,7 @@ var DEFAULT_PROMPTS = {
   tools: {
     compress: "Replace older conversation ranges with dense summaries you write. Each message seq is a surface reference. Single range: compress({ content: [{ startSeq, endSeq, summary }] }). Batch multiple unrelated ranges in one call (each content entry becomes its own block); keep ranges disjoint. Never compress content the current step is actively using. Compress boundaries are SURFACE SEQS (acp_status Surface: row, latest nudge table) \u2014 NOT the block refs (bN, e.g. b1) that acp_status COMPRESSED BLOCKS shows, which are for decompress only. Drilldown mN refs (e.g. m00306) are ALSO accepted as startSeq/endSeq \u2014 they are auto-mapped to the live surface seq; an unknown mN (never assigned on the current surface) fails with guidance. Seq refs must come from the CURRENT surface (acp_status or the latest nudge): a span whose edges were shadowed by an earlier compress is auto-remapped to its still-live content, a fully compressed span is reported as already compressed, and invented/other-session seqs fail with guidance. Good compression moments: stage or subtask completion, strategy switches, intermediate milestones, and wrapping up failed exploration \u2014 when the details are consumed and no longer critical for the task ahead. When you write a summary, turn dead-end exploration into a conclusion (what was tried, why it failed, the next step) \u2014 not a blow-by-blow; and keep the summary the ONLY record: self-contained, so a later reader (or you, after decompress) can continue without the original.",
     decompress: "Recover the original content of a compressed block by its blockId \u2014 the kernel block ref `bN` shown by acp_status (e.g. b1), or a compaction id from search_context (read-only; does not unshadow the range).",
-    searchContext: "Search inside compressed blocks (summaries and original content) for information the model no longer sees in context.",
+    searchContext: "Search inside compressed blocks (summaries and original content) for information the model no longer sees in context. When a summary lacks a detail you need (exact values, error strings, decisions, verbatim code), SEARCH the compressed blocks FIRST \u2014 never guess or reconstruct from memory: search_context(query) locates the right block, then decompress only that block to recover the original.",
     acpStatus: 'Context status: overview of the current context \u2014 CONTEXT BREAKDOWN (tool/text/summaries token shares of the visible total), COMPRESSED BLOCKS ledger, and the nudge decision. No args = overview. Percentages are shares of the visible content, not the context window. Note: the block refs in COMPRESSED BLOCKS (bN, e.g. b1) are for decompress; compress uses the Surface: seq range, not bN. Drilldown: pass scope:"compressed" for a per-block list, or scope:"uncompressed" with view:"messages" (every visible message) / view:"ranges" (merged ranges); tool filters to one tool name, sort reorders (size/time/tool; age for compressed), limit caps rows (default 30). Drilldown row refs are kernel ids (mN) \u2014 feed them straight to compress as startSeq/endSeq (auto-mapped to the live surface seq); bN is for decompress, Surface: seqs also work in compress.'
   },
   systemPromptTemplate: `Active Context Pruning \u2014 model-driven context management
@@ -3564,7 +3642,7 @@ WHEN NOT TO COMPRESS:
 Compression tools (refs are SURFACE SEQS, not ids):
 - compress: replace one or more seq ranges, each with your own dense summary. Single range: compress({ content: [{ startSeq, endSeq, summary }] }). Batch multiple unrelated segments in one call (each entry becomes its own block): compress({ content: [{ startSeq: 1, endSeq: 5, summary: '...' }, { startSeq: 12, endSeq: 18, summary: '...' }] }). Keep ranges disjoint \u2014 overlapping entries in one batch are skipped. Edges are auto-balanced to tool-call/result boundaries; a trailing #callId fragment in a seq is ignored. Seq refs must be on the current surface: seqs from older nudges or earlier compresses go stale as the surface moves, so a stale span is auto-remapped to its still-live remainder (the result reports the adjusted span), a fully compressed span is reported as already compressed, and invented/other-session seqs fail with guidance. The block refs (bN, e.g. b1) in acp_status COMPRESSED BLOCKS are for decompress, NOT compress boundaries.
 - decompress: recover a compressed block's original content, read-only. decompress({ blockId }) \u2014 accept the bN ref shown by acp_status (e.g. b1) or a compaction id.
-- search_context: find information inside compressed blocks BEFORE decompressing. search_context({ query }).
+- search_context: when a summary lacks the details you need (exact values, error strings, decisions, verbatim code), SEARCH the compressed blocks FIRST \u2014 never guess or reconstruct from memory; search_context(query) locates the right block, decompress only that block.
 - acp_status: current context usage and the live compressible-range list. Run it right before compressing \u2014 the only seqs that never go stale are the ones you just read. Drilldown (scope/view/tool/sort/limit) lists per-message or per-block sizes; drilldown rows are kernel ids (mN) \u2014 compress accepts them directly (auto-mapped to the live surface seq).
 
 Tiered compression: each compressed block appears on the surface as one summary node. Compressing that node again DISTILLS the block (tier 2): the parent summary folds into your new summary and the original messages are freed. Distilling a tier-2 block yields tier 3. Distill when a summary itself is consumed \u2014 decompress on the tier-2 block recovers the full originals.
@@ -3968,11 +4046,7 @@ async function handleCompress(env, args, exec) {
     }
     const { start, end } = range;
     const shadowed = shadowedSeqsOf(session, start, end);
-    let shadowedTokens = 0;
-    for (const seq of shadowed) {
-      const event = session.events[seq];
-      if (event !== void 0) shadowedTokens += defaultCountTokens(extractEventText(event));
-    }
+    const shadowedTokens = shadowedTokensViaMeter(session, shadowed, agent.ctx);
     const tier = block.tier === 2 || block.tier === 3 ? block.tier : 1;
     const parentBlockIds = compactionIdsOfKernelBlocks(session, block.directBlockIds);
     const { compactionId } = runCompactionTransaction(session, {
@@ -4288,12 +4362,8 @@ function compressText(env, agent, args) {
   if (blockRefForSummarySeq(session, start) !== null || blockRefForSummarySeq(session, end) !== null) {
     return "/acp compress: the range touches a compressed block summary node \u2014 distill it with the compress tool (seq-based batch), not /acp compress";
   }
-  const shadowed = shadowedSeqsOf(session, startSeq, endSeq);
-  let shadowedTokens = 0;
-  for (const seq of shadowed) {
-    const event = session.events[seq];
-    if (event !== void 0) shadowedTokens += defaultCountTokens(extractEventText(event));
-  }
+  const shadowed = shadowedSeqsOf(session, start, end);
+  const shadowedTokens = shadowedTokensViaMeter(session, shadowed, agent.ctx);
   const { compactionId } = runCompactionTransaction(session, {
     start,
     end,