pi-midcompact 0.5.3 → 0.6.0

package/README.md

@@ -55,7 +55,7 @@ The raw session JSONL still contains:
 
 ### A reviewed draft can reclaim meaningful context
 
-The earlier browser and TUI captures below illustrate a draft with **2 ranges** covering **42 of 73 atoms**, while the other 31 atoms remain verbatim. The current UI reports Pi-provided anchor usage separately from factual content chars and image counts; it does not derive projected token savings from local character estimates. Click either image to open it at full resolution.
+The earlier browser and TUI captures below illustrate a draft with **2 ranges** covering **42 of 73 atoms**, while the other 31 atoms remain verbatim. The current UI reports Pi-provided anchor usage as the baseline and derives a **display-only** projection of post-commit usage from documented char-class assumptions (labeled `est.`, shown as a range, never used for gating); factual content chars and image counts stay alongside it. Click either image to open it at full resolution.
 
 <p align="center">
   <a href="./figures/review-webui.png">
@@ -222,6 +222,23 @@ Enter/Esc/q        close
 
 The extension shows planning status in Pi's footer only while a transaction is active. It disappears after commit or abort.
 
+## Web UI Development
+
+From a source checkout, run the browser workbench against in-memory fixtures
+without starting Pi:
+
+```bash
+npm run dev:webui
+npm run dev:webui -- --port=4180 --no-open
+```
+
+The command opens a fixture router for `review-ready`, `review-pending`,
+`selection-mixed`, `no-telemetry`, and `wide-content`. Each button opens an
+isolated workbench with its own in-memory draft. Fixture pages survive browser
+refresh and the page's Close action, reload HTML changes automatically, and
+restart for imported TypeScript changes. Stop the router with `Ctrl+C`. Use
+`dev/midcompact-debug-ui.ts` only when validating against an actual Pi session.
+
 ## Guarantees and Limits
 
 - **Original history is retained.** Compression changes what later model requests see, not the stored Pi messages.

package/README.zh-CN.md

@@ -55,7 +55,7 @@ Pi 内置的 `/compact` 可理解为**前缀压缩**（prefix compaction）：
 
 ### 实际压缩效果
 
-下面的早期浏览器和 TUI 截图展示了一份包含 **2 个区段**、覆盖 **73 个 atom 中 42 个**的草案，其余 31 个 atom 保留原文。当前 UI 会把 Pi 提供的锚点 usage 与扩展统计的 content chars、图片数量分开显示，不再根据本地字符估算推导预计 token 节省量。点击图片可查看原图。
+下面的早期浏览器和 TUI 截图展示了一份包含 **2 个区段**、覆盖 **73 个 atom 中 42 个**的草案，其余 31 个 atom 保留原文。当前 UI 以 Pi 上报的锚点 usage 为基线，并根据文档化的字符分类假设推导**仅供展示**的 commit 后占用预计（标注 `est.`、以区间呈现、绝不参与门禁）；事实性的 content chars 与图片数量仍然并列展示。点击图片可查看原图。
 
 <p align="center">
   <a href="./figures/review-webui.png">
@@ -222,6 +222,21 @@ Enter/Esc/q        关闭
 
 扩展只在事务进行期间在 Pi 页脚显示规划状态；提交或放弃后会自动清除。
 
+## Web UI 开发
+
+在源码 checkout 中，无需启动 Pi 即可用内存 fixture 运行浏览器工作台：
+
+```bash
+npm run dev:webui
+npm run dev:webui -- --port=4180 --no-open
+```
+
+命令会打开一个 fixture router，其中包含 `review-ready`、`review-pending`、
+`selection-mixed`、`no-telemetry` 和 `wide-content`。每个按钮都会打开一套独立的
+工作台和内存草案。刷新浏览器或点击 Close 不会终止 fixture；HTML 修改会自动刷新
+页面，导入的 TypeScript 修改会触发进程重启。用 `Ctrl+C` 停止 router。只有在验证
+真实 Pi 会话集成时，才需要使用 `dev/midcompact-debug-ui.ts`。
+
 ## 保证与限制
 
 - **保留原始历史。** 压缩只改变后续模型请求看到的内容，不改写存储的 Pi 消息。

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-midcompact",
-  "version": "0.5.3",
+  "version": "0.6.0",
   "description": "Branch-aware mid-context compression for the Pi coding agent",
   "author": "frostime",
   "repository": {
@@ -32,7 +32,8 @@
   ],
   "scripts": {
     "typecheck": "tsc -p tsconfig.json --noEmit",
-    "test": "rm -rf .test-dist && tsc -p tsconfig.test.json && node test/install-mocks.mjs && node --test test/schema-contract.test.mjs test/core.test.mjs test/runtime-start.test.mjs test/runtime-agent.test.mjs test/runtime-user.test.mjs test/runtime-transaction.test.mjs test/runtime-lock.test.mjs",
+    "dev:webui": "tsx watch dev/review-webui-preview.ts",
+    "test": "rm -rf .test-dist && tsc -p tsconfig.test.json && node test/install-mocks.mjs && node --test test/schema-contract.test.mjs test/core.test.mjs test/runtime-start.test.mjs test/runtime-agent.test.mjs test/runtime-user.test.mjs test/runtime-transaction.test.mjs test/runtime-lock.test.mjs && node test/review-webui-dom.test.mjs",
     "typecheck:contract": "tsc -p tsconfig.test.json --noEmit",
     "pack:check": "npm pack --dry-run"
   },
@@ -45,6 +46,7 @@
     "@earendil-works/pi-ai": "^0.84.1",
     "@earendil-works/pi-coding-agent": "^0.84.1",
     "typescript": "^5.9.3",
+    "tsx": "^4.23.13",
     "@earendil-works/pi-tui": "^0.84.1"
   },
   "pi": {

package/src/SPEC.md

@@ -32,7 +32,12 @@ package.json → `pi.extensions`).
   then appends state, abort appends nothing. Both refuse to run while the
   Agent holds the planning lock.
 - `anchorUsage` is informational only — Pi-reported awareness, never an
-  optimization target (same rule as `projectedTokens` / estimates).
+  optimization target. The web workbench may additionally render a
+  **display-level** post-commit projection derived from the documented
+  char-class assumption table in `content-metrics` (`TOKEN_ESTIMATE`): always
+  labeled `est.`, shown as a propagated band, and never feeding commit
+  gating, range validation, or any decision (same rule as `projectedTokens`
+  / `approxTokens`).
 - Atom refs are transaction-local: re-run inspect/locate in a later
   transaction; group refs (`g...`) are never locate refs.
 
@@ -78,6 +83,24 @@ package.json → `pi.extensions`).
   `skill:<name>`).
 - The tool never starts a transaction and never commits; both are command-
   or user-gated. Recall is the only action valid without a transaction.
+- Web workbench (`review-webui.html` + `review-webui.ts`): the state payload
+  carries per-atom char-class counts (`narrowChars`/`wideChars`), per-range
+  replacement char-class counts (including the actual wrapper), and the
+  assumption table (`est`), so the page renders the projection band and can
+  update estimates while a summary is edited; `GET /api/atom/:ref` serves the
+  frozen atom's full text for the original-text drawer (read-only, snapshot-local).
+  `startReviewWebUiServer` owns the Pi-independent loopback HTTP contract;
+  `showReviewWebUi` adapts it to Pi notification, browser launch, and page-bound
+  lifetime. Development may opt into a persistent server without changing the
+  production defaults.
+  User-facing copy says "can't compress" for protected atoms; "protected"
+  stays the agent/tool-side term.
+  Page invariants that broke once and must hold: the `<!--MIDCOMPACT_STATE-->`
+  script tag is the server's template injection point (renaming it breaks
+  state loading); `selectionRefs` initialization depends on helpers defined
+  later in the page script (order is load-bearing, TDZ); `gbody` visibility
+  is driven by the render-time `hidden` attribute, so collapse handlers must
+  sync that attribute, not just a class.
 
 ## Change rules
 

package/src/content-metrics.ts

@@ -1,6 +1,8 @@
-// Sole owner of factual message content statistics. Never converts local char or
-// image byte counts into token claims. Image base64 never contributes to text
-// char counts.
+// Sole owner of message content statistics. Factual char/image counts are the
+// authority; the only token conversion allowed is the explicitly scoped,
+// display-level estimator at the bottom of this file (web UI presentation
+// only — it never gates decisions, commits, or range validity). Image base64
+// never contributes to text char counts.
 
 import type { ContentMetrics, ImageFact, MessageLike } from "./types.js";
 
@@ -183,3 +185,73 @@ export function aggregateMetrics(parts: readonly ContentMetrics[]): ContentMetri
   }
   return { contentChars, imageCount, images };
 }
+
+// ---- Display-level token estimation (web UI presentation only) ----
+//
+// The web UI shows a projected post-commit usage band. Because the consumer
+// model is provider-dependent, tokens are estimated from char classes with a
+// documented assumption table instead of a real tokenizer: the [low, high]
+// pairs express published tokenizer spread and are propagated as a band, never
+// collapsed into a single authoritative number. The band is derived from the
+// content mix of the atoms involved (ASCII-heavy content lands near the tight
+// end, CJK-heavy near the wide end). Nothing here feeds commit gating, range
+// validation, or any decision.
+
+/** Per-char-class token-cost assumptions: [low, high] tokens per char. */
+export const TOKEN_ESTIMATE = {
+  /** ASCII/code: roughly 3.3–4.5 chars per token across common tokenizers. */
+  narrowTokPerChar: [0.22, 0.3],
+  /** Non-ASCII scripts (CJK, kana, hangul, emoji, …): the dominant spread. */
+  wideTokPerChar: [0.5, 1.0],
+  /** Images: provider- and resolution-dependent flat allowance. */
+  imageTok: [700, 1600],
+} as const;
+
+/** Char classes the estimator distinguishes. */
+export interface CharMix {
+  /** ASCII code points. */
+  narrowChars: number;
+  /** Everything non-ASCII (counted conservatively at the wide rate). */
+  wideChars: number;
+}
+
+// Classification is deliberately coarse: ASCII is estimated at the narrow
+// rate; every non-ASCII code point (CJK, kana, hangul, emoji, Cyrillic, …) is
+// counted at the wide rate, which is the conservative side for CJK-heavy
+// content and keeps the table honest without per-script modeling.
+
+/** Split a text into narrow (ASCII) and wide (everything else) code points. */
+export function charClassCounts(text: string): CharMix {
+  let narrowChars = 0;
+  let wideChars = 0;
+  for (const ch of text) {
+    const cp = ch.codePointAt(0)!;
+    if (cp <= 0x7f) {
+      narrowChars += 1;
+      continue;
+    }
+    wideChars += 1;
+  }
+  return { narrowChars, wideChars };
+}
+
+export interface TokenEstimate {
+  point: number;
+  low: number;
+  high: number;
+}
+
+/** Estimate tokens for a char mix plus images, as a propagated band. */
+export function estimateTokens(mix: CharMix, imageCount: number): TokenEstimate {
+  const [aLo, aHi] = TOKEN_ESTIMATE.narrowTokPerChar;
+  const [cLo, cHi] = TOKEN_ESTIMATE.wideTokPerChar;
+  const [iLo, iHi] = TOKEN_ESTIMATE.imageTok;
+  const mid = (lo: number, hi: number) => (lo + hi) / 2;
+  return {
+    point: Math.round(mix.narrowChars * mid(aLo, aHi)
+      + mix.wideChars * mid(cLo, cHi)
+      + imageCount * mid(iLo, iHi)),
+    low: Math.round(mix.narrowChars * aLo + mix.wideChars * cLo + imageCount * iLo),
+    high: Math.round(mix.narrowChars * aHi + mix.wideChars * cHi + imageCount * iHi),
+  };
+}

package/src/projection.ts

@@ -74,8 +74,8 @@ export function replacementMetrics(block: CompressionBlock): ContentMetrics {
  * Factual replacement content chars for a range, given its topic and summary.
  * Computed from the actual summary message wrapper text that projection emits.
  */
-export function replacementContentChars(summary: string, topic?: string): number {
-  const block: CompressionBlock = {
+function draftReplacementBlock(summary: string, topic?: string): CompressionBlock {
+  return {
     id: "draft",
     summary,
     topic,
@@ -89,7 +89,16 @@ export function replacementContentChars(summary: string, topic?: string): number
     originalApproxTokens: 0,
     compressedApproxTokens: 0,
   };
-  return replacementMetrics(block).contentChars;
+}
+
+/** Render the exact replacement content used for draft display estimates. */
+export function replacementContentText(summary: string, topic?: string): string {
+  const content = summaryMessage(draftReplacementBlock(summary, topic)).content;
+  return typeof content === "string" ? content : "";
+}
+
+export function replacementContentChars(summary: string, topic?: string): number {
+  return replacementMetrics(draftReplacementBlock(summary, topic)).contentChars;
 }
 
 /** @deprecated legacy heuristic; kept only for old approximate-token field compat. */