pi-midcompact 0.3.0 → 0.5.0

package/skills/midcompact/SKILL.md

@@ -1,117 +1,132 @@
 ---
 name: midcompact
-description: Use during an active /midcompact transaction to plan and draft selective compression of stale middle sections of a long Pi conversation, or independently to retrieve details from previously compressed blocks. Covers how to choose compression ranges, how to negotiate compression depth with the user, and the midcompact tool interface.
+description: Use when a midcompact runtime prompt starts or hands off a compression-planning transaction, or when current work needs details from an active committed midcompact block. Guides user-aligned candidate discovery, effort-scaled inspect/locate/plan work, replacement-summary writing, and recall.
 ---
 
 # Midcompact
 
-Use this skill when a `/midcompact start` transaction is active, or when information must be retrieved from a previously compressed block.
+## Route the activation
 
-## What compression does
+This skill handles two independent tasks: planning compression and recalling committed content. Choose the route before calling the tool.
 
-Mid-context compaction is **selective replacement** inside one linear conversation, not a restart:
+| Signal | Immediate duty | First action |
+|--------|----------------|--------------|
+| Runtime prompt says `FINAL STATE: USER MANUAL` | Let the user create the initial DraftPlan | Reply exactly `OK`; call no midcompact tool |
+| Runtime prompt says `FINAL STATE: AGENT DIRECT` | Start from the new empty draft | `action="inspect"` |
+| A handoff reports a persisted DraftPlan and the user asks to continue | Read the shared selection and determine what help the user wants | `action="plan", op="show"` |
+| The user or a projected summary needs detail from a committed block | Retrieve that history only | Follow **Recall workflow** |
 
-```
-A → B → C → ... → NOW
-      └─ selected slices become summaries; everything else stays verbatim
-```
+The state-specific runtime prompt is authoritative. Recall does not enter planning or mutate the DraftPlan. During the User-manual acknowledgement turn, the no-tool instruction overrides every other route.
+
+## Plan compression
+
+Selected ranges become summaries in future model context; content outside them stays verbatim. Originals remain stored, but recall is a recovery path, not a substitute for a sufficient summary. Agent and user edit one DraftPlan; the user retains final control through review and `/midcompact commit`.
+
+The start mode controls how the first draft is created. It does not determine whether a user selection is final, how much initiative the Agent should take, or how deeply the history should be investigated.
+
+Apply one invariant:
+
+> Every fact that can still affect future work must remain with sufficient fidelity, either outside the selected ranges or in their replacement summaries.
+
+### Planning workflow
 
-Three mechanical facts shape how you work:
+#### 1. Read the entry state and user intent
 
-- **KEEP by omission.** Anything outside a draft range stays verbatim. You declare only what to compress, never what to preserve.
-- **Originals survive.** Session entries stay on disk, and `action="recall"` brings an active committed block back into view. Compression is reversible at the information-access level, not a deletion — but recall returns a readable rendering, not a byte-exact replay, so it is no substitute for keeping something verbatim.
-- **Projection is layered.** Compression applies to what future models see, not to stored history.
+For a fresh Agent-direct transaction, begin with `action="inspect"`. For a handed-off DraftPlan, begin with `action="plan", op="show"`; inspect the anchor only if the requested work needs broader context.
 
-## Who decides what
+Establish the user's desired scope, fidelity, and planning effort from their words, current selection, `User focus: ...`, and surrounding interaction. Treat answers collected through question, questionnaire, or similar tools as user-originated input even when represented as tool results.
 
-| Actor | Owns |
-|-------|------|
-| Extension | Session-tree mechanics, projection, protocol safety |
-| You | Semantic judgment: which content may yield to a summary, and what each summary must carry |
-| User | Compression scope and depth; the only actor that can commit |
+A manual selection records what the user selected, not whether they mean "only this" or "help me refine it." Follow a clear intent directly. If different interpretations would materially change the work, ask briefly whether to limit work to the selection or inspect and suggest changes. Do not force clarification when the user has already expressed a preference or delegated the judgment.
 
-You cannot commit. `/midcompact commit` is the user's gate. Your output is a proposal.
+Match effort to the requested fidelity. A quick or approximate request calls for bounded planning; a precision-sensitive request may justify deeper work after alignment.
 
-## What may be compressed
+#### 2. Form a bounded semantic view
 
-One conservation law decides every case:
+`inspect` returns factual structure with bounded user landmarks, grouped by user message. Use it for structure and volume, not to invent semantics. Ground candidates in visible conversation or a few targeted `locate` calls, and stop paging after the potentially relevant region. When exact candidate spans are known and their relative volume could change the choice, use read-only `inspect` spans to compare them without mutating the DraftPlan.
 
-> Compression must conserve every fact that can still affect the work **in the projected context**. After a range is replaced, each such fact must remain available with sufficient fidelity, either in text left verbatim outside the ranges or in the replacement summary.
+Segment by completed work phase, not message count. User-originated input and concluding Agent responses are useful landmarks for intent and outcome, but are not automatically KEEP. Intermediate tool exchanges may also contain decisions or evidence absent from the final response.
 
-Not age. Not token count. Not whether it is a tool call or prose. Two consequences follow:
+| Often able to yield to a summary | Often load-bearing |
+|-----------------------------------|--------------------|
+| Repetitive or superseded exploration | Current intent and active constraints |
+| Tool output absorbed into a conclusion | Exact errors, evidence, or wording whose form matters |
+| A completed subtask's intermediate process | Decisions and rationale still governing the work |
+| A rejected attempt's mechanics | Its still-relevant failure reason |
 
-- **The summary is load-bearing.** When a fact lives only inside a compressed range, omitting it from the summary removes it from the projected context. Recall is a recovery path, not a default carrier.
-- **Some information must not be entrusted to paraphrase.** Exact requirement wording, literal error text, decision-critical evidence, protocol structure. When exact form or provenance matters, keep the source verbatim unless the replacement preserves it with equivalent fidelity.
+Use little or no `locate` during this reconnaissance. If a phase cannot yet be described faithfully, omit it from the proposal or mark it as needing confirmation rather than exploring the anchor broadly.
 
-Illustrative cases, derived from the law:
+#### 3. Present semantic options and align
 
-- Work that reached a conclusion or resolved result, with nothing downstream depending on how it got there → the summary states the outcome, or the concluding atom stays outside the range → compressible.
-- Tool output fully absorbed into the answer that follows it → the answer sits outside the range → a thin summary suffices.
-- A constraint, correction, or approval the user stated once → if the wording itself carries the constraint, keep it verbatim; otherwise the summary must state it explicitly.
-- A rejected approach whose rejection reason still constrains current work → the summary must carry the reason; the exploration around it can go.
+Before deep `locate` work or substantial DraftPlan mutation, establish the user's compression preference through explicit instruction, a reliable implication, or concise clarification.
 
-**Reverse failure.** Some content looks stale but is the only record of an environment quirk, a version-specific behavior, or a failure mode that will resurface. If you cannot state faithfully what a segment established, you cannot summarize it conservingly — keep it verbatim.
+Describe each proposal in recognizable conversation terms:
 
-**Your summary becomes the successor.** The next working Agent — likely you, after commit — sees only your summary. Write what it needs to avoid redoing or breaking work: user intent and constraints, decisions and their rationale, file paths and signatures, validation state, unresolved issues, and the next useful state. Cutting repetitive exploration is the goal; shortening prose is not.
+> From `<semantic start>` to `<semantic end>`, `<replace the whole phase / keep the endpoints and compress the work between>`; preserve `<load-bearing information>` in the summary.
 
-## Phase 1 — read, segment, propose, align
+Use short recognizable excerpts when available and clear paraphrases otherwise. State whether the endpoint messages remain verbatim, what intermediate work disappears, and what the summary carries. Do not identify a user-facing range primarily by atom refs, DraftPlan ids, or arbitrary item numbers.
 
-**Do not call `plan` until you have presented the semantic candidates and the user has confirmed or adjusted the direction**, unless the user explicitly instructs you to skip the proposal.
+When treatments involve a meaningful tradeoff, present concise alternatives and recommend one. A clear quick request may need only one proportionate proposal. Add factual content or image measurements only when they help the choice, and do not convert them into unsupported token-savings claims.
 
-1. Read back over the conversation in your current context and segment it semantically — by phase of work, not by message count.
-2. Judge each segment against the conservation law. For each candidate, be able to say where its load-bearing facts would end up.
-3. Present candidates: where each begins and ends, roughly how large it is, and what its summary would carry. Name segments you deliberately excluded when the exclusion is non-obvious.
-4. If scope or depth is still unresolved, ask how deep to go and which regions matter. Depth is the user's decision, not a number you optimize.
+#### 4. Resolve boundaries and build the DraftPlan
 
-`locate` is read-only and may be used sparingly here to confirm a boundary you are about to propose, but not to scan exploratorily. A session that has reached compression is already near its limit: speculative calls, and a plan rebuilt after review, both consume what you are trying to reclaim.
+After the intended treatment is clear, use `action="locate"` for targeted content and boundary checks. An atom is the smallest selectable unit; a tool call and its matching results form one indivisible `tool_exchange` atom. Keep source text outside a range when exact wording or provenance matters and a summary cannot preserve it equivalently.
 
-If `/midcompact start` carried an instruction (it arrives as `User focus: ...`), treat it as guidance for whichever of scope and depth it specifies. Still propose, but briefly; ask only what the instruction leaves open.
+Choose boundaries from the information that must survive, not from a fixed category. A range may replace a whole semantic phase, including its initiating and concluding messages. It may instead retain a load-bearing user instruction and concluding Agent response while replacing only the execution between them. It may split around important material to leave KEEP holes. These are reasoning patterns, not rules tied to start mode, message age, or one prescribed kind of work.
 
-## Phase 2 — locate, draft, review
+Build or refine the shared DraftPlan with `plan add`, `update`, `remove`, and `show`. A handed-off selection may be preserved or revised as the user's intent permits. Use separate ranges for non-contiguous phases; fill pending summaries before commit. If deeper inspection would materially change the agreed treatment, surface the change instead of silently applying it.
 
-1. `action="locate"` to resolve the landmarks you agreed on into atom refs. Results include previews; request `detail="full"` when a boundary is ambiguous.
-2. `action="plan", op="add"` per range — several ranges for non-contiguous compression. To keep one important atom verbatim inside a broader phase, add ranges around it; that is KEEP by omission in practice.
-3. `action="plan", op="show"`, then present the complete plan described by content rather than atom IDs.
-4. Recommend `/midcompact review` when the user wants to inspect the anchor timeline, ranges, summaries, and KEEP holes visually. Apply changes with `op="update"`, `op="remove"`, or new ranges.
-5. Ask the user to run `/midcompact commit` when satisfied.
+Read `references/tool-interface.md` before retrying a rejected operation or when exact parameters, protected-atom causes, or measurements matter.
 
-## Tool interface
+#### 5. Write replacement summaries
+
+A replacement summary is successor context for a future Agent, not a transcript or a prompt to perform compression. State the resulting knowledge and working state directly.
+
+Preserve, when applicable:
+
+- user intent and active constraints;
+- decisions, conclusions, and necessary rationale;
+- relevant files, symbols, interfaces, commands, or configuration;
+- completed changes and validation results;
+- rejected approaches only when their failure reason still matters;
+- unresolved issues and any established next step or continuation state.
+
+Remove repetitive exploration, superseded hypotheses, raw output captured by a conclusion, and chronology with no remaining consequence. Do not turn uncertainty into fact, invent a next action, or use references that only make sense inside the removed text. If retained endpoint messages already carry part of the intended context, do not duplicate them mechanically; use the summary to preserve what would otherwise be lost.
+
+Organize by future utility rather than original chronology. When useful, use this compact frame without forcing empty fields:
+
+```text
+Goal and constraints:
+Established state and decisions:
+Artifacts and validation:
+Open issues and established next state:
+```
 
-**Atoms are not messages.** An atom is the smallest compressible unit. One assistant message containing one or more tool calls, plus its immediately following matching results, forms a single `tool_exchange` atom; you cannot compress half of one.
+Length follows the information that must survive, not a target ratio. Final test: could a fresh Agent continue correctly from this summary plus retained context, without repeating work or violating a prior decision?
 
-**Two ref namespaces.** `a0001` is an atom ref, valid only within the current transaction's anchor snapshot — indices shift after every commit, so never reuse one across transactions; re-run `locate`. `c0001` is a compressed block id, stable while that block stays active on the current branch, used by `recall`.
+#### 6. Verify and hand off
 
-`action="locate"` — pass either `ref` for a direct lookup, or at least one real filter: `pattern`, `tool_name`, or a `source` other than `any`. With none it returns nothing rather than an error. Optional: `direction` (`oldest`/`newest`), `limit` (default 5, max 20), `detail` (`brief`/`full`).
+Call `action="plan", op="show"`. Check that the intended semantic phases are covered, KEEP holes remain outside ranges, every range has a summary, and the summaries conserve the future working state.
 
-`action="plan"` — `op` defaults to `show`.
+Describe the completed proposal with the same recognizable landmarks used during alignment. Direct the user to `/midcompact select` or `/midcompact select-webui` for boundaries and KEEP holes, and to `/midcompact review` or `/midcompact review-webui` for summary inspection or rejection. Use browser variants when the TUI is unavailable or preferred. Ask the user to run `/midcompact commit` when ready; never commit for them.
 
-| op | Requires |
-|----|----------|
-| `show` | — |
-| `add` | `start`, `end`, `summary`; `topic` optional |
-| `update` | `draft_id` and at least one of `summary`, `topic` |
-| `remove` | `draft_id` |
+## Recall compressed content
 
-`op="add"` rejects a range when any of the following holds. These are mechanical constraints, independent of semantic value — a range must satisfy both.
+Recall works with or without an active planning transaction. It reads committed blocks active on the current branch without changing projection or DraftPlan.
 
-| Condition | Meaning |
-|-----------|---------|
-| Range crosses a protected atom | Split the plan around it. An atom is protected when its tool-call protocol is still open (an incomplete `tool_exchange`, or an orphaned tool result), when it is an existing compressed block, when its message kind is not one the extension can compress, or when its messages have no persistent session entry to anchor to. |
-| Range overlaps an existing draft range | Remove or update that range instead. |
-| `start` occurs after `end` | Refs are positional; order them. |
-| Unknown atom ref | Usually a typo or a ref carried over from an earlier transaction. Re-run `locate` against the current snapshot. |
+### Recall workflow
 
-**Telemetry** accompanies every `plan` result: anchor usage at start, approximate raw and summary tokens for the draft, and projected whole-context usage if committed now. Use it to check the draft against the depth agreed in Phase 1. It is awareness, not a target, and the projections are estimates.
+#### 1. Find the block
 
-## Repeated compression
+If its id is unknown, call `action="recall", pattern="..."` to search active topics and summaries. A projected summary also states its block id and exact recall call.
 
-A session may be compacted multiple times. Committed blocks stay active and appear as protected atoms in later snapshots, so they cannot be recompressed. A later transaction compresses raw history accumulated around them.
+#### 2. Retrieve the detail
 
-## Recall
+Call `action="recall", ref="c0001"`. If the readable, structure-flattened result ends with a truncation marker, retry with `detail="full"`. Retrieve only what the current task needs; do not start or change a plan merely to recall history.
 
-`action="recall"` works whether or not a transaction is active, does not change the projection, and reads blocks active on the current branch.
+## Tool conventions
 
-- `pattern="..."` searches topics and summaries of active blocks (`limit` default 8, max 20).
-- `ref="c0001"` returns a readable, structure-flattened rendering of that block's messages, truncated if long. On a truncation marker, retry the same ref with `detail="full"`; if that still truncates, inspect the session tree. There is no paging.
+- `g0001` labels an inventory group; use its `a...` span as internal landmarks.
+- `a0001` is a transaction-local atom ref; `d1` is a DraftPlan range id; `c0001` is an active committed-block id.
+- Protected atoms cannot enter a range. Split around them.
 
-Every summary in context states its own block id and the exact recall call for it. Use recall when a summary lacks a detail the current work needs.
+These refs are planning handles, not the primary way to explain compression to the user. Read `references/tool-interface.md` for exact defaults and limits, rejected operations, protected-atom causes, measurements, repeated compression, or recall truncation.

package/skills/midcompact/references/tool-interface.md

@@ -0,0 +1,83 @@
+# Midcompact Tool Interface
+
+Read this reference when exact call requirements, limits, rejection behavior, repeated compression, or recall truncation matters. The main workflow remains in `../SKILL.md`.
+
+## Inspect
+
+Without `spans`, `action="inspect"` inventories the frozen anchor. It returns factual structure and bounded user landmarks, not full message bodies, assistant/tool previews, summaries, or image base64.
+
+- `page_size`: default 20 groups, maximum 50.
+- `cursor`: opaque value returned by the previous page.
+- Results include group refs, atom spans, content chars, image facts, protected/compressible counts, and Pi-reported anchor usage.
+
+Stop paging after the candidate regions are covered.
+
+To compare explicit candidates without mutating the DraftPlan, pass one or more possibly overlapping spans:
+
+```text
+midcompact(action="inspect", spans=[
+  {"start":"a0006","end":"a0014"},
+  {"start":"a0006","end":"a0020"}
+])
+```
+
+Span inspection reports bounded endpoint landmarks, atom/message and role counts, tool exchanges and calls, factual content share, images, and protected/compressible counts. It has a 12,000-character total output budget and reports how many requested spans fit. It does not report per-span tokens: Pi supplies usage for the whole anchor, not token attribution by range. Do not combine `spans` with `page_size` or `cursor`.
+
+## Locate
+
+`action="locate"` returns atoms from the frozen anchor. Supply either:
+
+- `ref`: one direct atom lookup; or
+- at least one real filter: `pattern`, `tool_name`, or `source` other than `any`.
+
+With no lookup or filter it returns no matches rather than an error. Filtered searches return at most three brief candidates; when more match, the result reports the total and asks for a more specific pattern or additional filter. `direction` is `oldest` by default or `newest`; `limit` may request one to three results.
+
+Brief direct lookups preserve both ends of an atom landmark. Pattern searches show a bounded excerpt around the match rather than the atom prefix. `detail="full"` is allowed only with one direct `ref` and preserves both ends with an explicit middle-omission marker when the 12,000-character atom limit is exceeded.
+
+A `g...` inventory ref is not a locate ref; use the group's `a...` start/end landmarks.
+
+## Plan
+
+`action="plan"` uses `op="show"` by default. Agent and user mutate the same DraftPlan.
+
+| op | Required fields |
+|----|-----------------|
+| `show` | none; optional `draft_id` for one range |
+| `add` | `start`, `end`; optional `summary`, `topic` |
+| `update` | `draft_id` and at least one of `summary`, `topic` |
+| `remove` | `draft_id` |
+
+Default show lists each range with bounded `from`, `to`, and `summary` landmarks plus factual metrics. Use `op="show", detail="full", draft_id="d1"` for one stored summary and endpoint previews under a 40,000-character total budget; full detail without `draft_id` is rejected. Show is read-only and does not persist a duplicate DraftPlan entry.
+
+Add/update return the changed range in brief form; remove returns its id and compact draft totals. Mutation responses omit the Pi-awareness header and unchanged ranges. Use explicit show when complete awareness is needed.
+
+An omitted or empty `summary` creates a pending range. Review can open it, but commit requires at least one range and a non-empty summary for every range. Changing boundaries requires removing the old range and adding the replacement.
+
+### Add rejection conditions
+
+| Condition | Response |
+|-----------|----------|
+| The range crosses a protected atom | Split the range around that atom |
+| The range overlaps an existing draft range | Remove or replace the existing range first |
+| `start` occurs after `end` | Correct the positional order |
+| An atom ref is unknown | Re-run inspect/locate against the current transaction snapshot |
+
+An atom is protected when its tool protocol is incomplete or orphaned, it represents an existing compressed block, its message kind is unsupported, or it lacks the persistent session entry needed to anchor compression.
+
+## Telemetry
+
+Explicit `plan show` includes Pi-reported anchor usage when available and factual draft measurements: original/replacement content chars, image count, and decoded payload bytes. Mutation results omit awareness and report only compact totals plus the changed range where one remains. Local character or image measurements are not converted into token savings or projected-token percentages. Use measurements to compare the proposal with user-directed depth, not as an optimization target.
+
+## Repeated compression
+
+Committed blocks appear as protected atoms in later transaction snapshots and cannot be compressed again. A later transaction may compress newly accumulated raw history around those blocks. Re-run inspect/locate because atom refs are transaction-local.
+
+## Recall
+
+`action="recall"` works independently of a planning transaction and reads committed blocks active on the current branch.
+
+- Without `ref`, `pattern` searches block topics and summaries; `limit` defaults to 8 and has a maximum of 20.
+- With `ref="c0001"`, the tool renders that block's stored messages.
+- `detail="full"` raises the rendering cap when the normal result is truncated.
+
+Recall has no paging. The truncation marker is `[truncated; refine the recall request or inspect the source session for more]`. If `detail="full"` still truncates before the needed detail, report that recall cannot expose the omitted content; do not infer it.

package/src/atoms.ts

@@ -1,5 +1,6 @@
 import type { Atom, LocateQuery, MessageLike, MessageRef, SessionEntryLike } from "./types.js";
-import { approxTokens, mapEntryIds, messageKey, renderMessage, toolCalls, truncate } from "./messages.js";
+import { aggregateMetrics, measureMessage } from "./content-metrics.js";
+import { approxTokens, excerptAround, mapEntryIds, messageKey, renderMessage, toolCalls, truncateMiddle } from "./messages.js";
 
 export function buildAtoms(messages: MessageLike[], branch: readonly SessionEntryLike[]): Atom[] {
   const entryIds = mapEntryIds(messages, branch);
@@ -70,6 +71,11 @@ export function buildAtoms(messages: MessageLike[], branch: readonly SessionEntr
   return atoms;
 }
 
+/** A protected atom cannot be part of any compressible range. */
+export function isProtectedAtom(atom: Atom): boolean {
+  return !atom.compressible || !atom.protocolClosed || atom.kind === "compressed";
+}
+
 function hasEntry(ref: MessageRef): boolean {
   return typeof ref.entryId === "string" && ref.entryId.length > 0;
 }
@@ -103,8 +109,9 @@ function makeAtom(
     messages,
     entryIds: messages.flatMap((ref) => (ref.entryId ? [ref.entryId] : [])),
     messageKeys: messages.map((ref) => ref.key),
-    preview: truncate(fullText, 700),
+    preview: truncateMiddle(fullText, 700),
     fullText,
+    metrics: aggregateMetrics(messages.map((ref) => measureMessage(ref.message))),
     approxTokens: approxTokens(fullText),
     compressible,
     protocolClosed,
@@ -114,10 +121,17 @@ function makeAtom(
   };
 }
 
-export function locateAtoms(atoms: Atom[], query: LocateQuery): Atom[] {
+export const MAX_LOCATE_MATCHES = 3;
+
+export interface LocatedAtoms {
+  atoms: Atom[];
+  totalMatches: number;
+}
+
+export function locateAtomMatches(atoms: Atom[], query: LocateQuery): LocatedAtoms {
   if (query.ref) {
     const atom = atoms.find((candidate) => candidate.ref === query.ref);
-    return atom ? [atom] : [];
+    return { atoms: atom ? [atom] : [], totalMatches: atom ? 1 : 0 };
   }
   const pattern = query.pattern?.toLocaleLowerCase();
   const toolName = query.toolName?.toLocaleLowerCase();
@@ -129,7 +143,13 @@ export function locateAtoms(atoms: Atom[], query: LocateQuery): Atom[] {
     return Boolean(pattern || toolName || source !== "any");
   });
   if ((query.direction ?? "oldest") === "newest") matches = matches.reverse();
-  return matches.slice(0, Math.max(1, Math.min(query.limit ?? 5, 20)));
+  const totalMatches = matches.length;
+  const limit = Math.max(1, Math.min(query.limit ?? MAX_LOCATE_MATCHES, MAX_LOCATE_MATCHES));
+  return { atoms: matches.slice(0, limit), totalMatches };
+}
+
+export function locateAtoms(atoms: Atom[], query: LocateQuery): Atom[] {
+  return locateAtomMatches(atoms, query).atoms;
 }
 
 function matchesSource(atom: Atom, source: NonNullable<LocateQuery["source"]>): boolean {
@@ -141,9 +161,13 @@ function matchesSource(atom: Atom, source: NonNullable<LocateQuery["source"]>):
   return false;
 }
 
-export function formatLocatedAtom(atom: Atom, detail: "brief" | "full" = "brief"): string {
+export function formatLocatedAtom(atom: Atom, detail: "brief" | "full" = "brief", pattern?: string): string {
   const flags = [atom.kind, atom.compressible ? "compressible" : "protected", atom.protocolClosed ? "closed" : "open"].join(", ");
-  const text = detail === "full" ? truncate(atom.fullText, 12_000) : atom.preview;
+  const text = detail === "full"
+    ? atom.fullText.length <= 12_000 ? atom.fullText : truncateMiddle(atom.fullText, 12_000)
+    : pattern
+      ? excerptAround(atom.fullText, pattern, 700)
+      : atom.preview;
   return [
     `${atom.ref} | position ${atom.index + 1} | ${flags}`,
     atom.toolNames.length ? `tools: ${atom.toolNames.join(", ")}` : "",

package/src/content-metrics.ts

@@ -0,0 +1,185 @@
+// Sole owner of factual message content statistics. Never converts local char or
+// image byte counts into token claims. Image base64 never contributes to text
+// char counts.
+
+import type { ContentMetrics, ImageFact, MessageLike } from "./types.js";
+
+/** Count Unicode code points of a string. */
+export function codePointCount(text: string): number {
+  let count = 0;
+  // for..of iterates by code point, not UTF-16 code unit.
+  for (const _ of text) count += 1;
+  return count;
+}
+
+/** Decode a base64 string into a Uint8Array without depending on Node Buffer. */
+function decodeBase64(data: string): Uint8Array {
+  const binary = atob(data);
+  const bytes = new Uint8Array(binary.length);
+  for (let i = 0; i < binary.length; i += 1) bytes[i] = binary.charCodeAt(i);
+  return bytes;
+}
+
+/** Best-effort pixel dimension read from decoded image bytes. Returns undefined on failure. */
+export function readImageDimensions(bytes: Uint8Array): { width?: number; height?: number } {
+  if (bytes.length < 8) return {};
+  // PNG: 89 50 4E 47 0D 0A 1A 0A; width/height are big-endian at offsets 16/20.
+  if (bytes[0] === 0x89 && bytes[1] === 0x50 && bytes[2] === 0x4e && bytes[3] === 0x47) {
+    if (bytes.length < 24) return {};
+    const view = new DataView(bytes.buffer, bytes.byteOffset + 16, 8);
+    return { width: view.getUint32(0), height: view.getUint32(4) };
+  }
+  // GIF: 47 49 46 38; width/height little-endian at offsets 6/8.
+  if (bytes[0] === 0x47 && bytes[1] === 0x49 && bytes[2] === 0x46 && bytes[3] === 0x38) {
+    return { width: bytes[6]! | (bytes[7]! << 8), height: bytes[8]! | (bytes[9]! << 8) };
+  }
+  // JPEG: scan SOF0 (0xFFC0) segment for dimensions.
+  if (bytes[0] === 0xff && bytes[1] === 0xd8) {
+    let i = 2;
+    while (i < bytes.length - 9) {
+      if (bytes[i] !== 0xff) { i += 1; continue; }
+      const marker = bytes[i + 1];
+      if (marker === undefined) break;
+      // SOF0..SOF15 carry dimensions; skip DHT (0xC4) and the reserved JPG marker (0xC8).
+      if (marker >= 0xc0 && marker <= 0xcf && marker !== 0xc4 && marker !== 0xc8) {
+        const height = (bytes[i + 5]! << 8) | bytes[i + 6]!;
+        const width = (bytes[i + 7]! << 8) | bytes[i + 8]!;
+        return { width, height };
+      }
+      const len = (bytes[i + 2]! << 8) | bytes[i + 3]!;
+      i += 2 + len;
+      continue;
+    }
+    return {};
+  }
+  // WebP: RIFF....WEBP; VP8/VP8L/VP8X variants.
+  if (
+    bytes.length >= 30 &&
+    bytes[0] === 0x52 && bytes[1] === 0x49 && bytes[2] === 0x46 && bytes[3] === 0x46 &&
+    bytes[8] === 0x57 && bytes[9] === 0x45 && bytes[10] === 0x42 && bytes[11] === 0x50
+  ) {
+    const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+    const chunk = (bytes[12]! << 16) | (bytes[13]! << 8) | bytes[14]!;
+    if (chunk === 0x56503820) { // "VP8 "
+      return { width: view.getUint16(26, true) & 0x3fff, height: view.getUint16(28, true) & 0x3fff };
+    }
+    if (chunk === 0x56503858) { // "VP8X"
+      const w = 1 + (view.getUint32(24, true) >>> 0);
+      return { width: w & 0xffffff, height: view.getUint32(27, true) & 0xffffff };
+    }
+    return {};
+  }
+  return {};
+}
+
+function safeJson(value: unknown): string {
+  try {
+    return JSON.stringify(value) ?? "";
+  } catch {
+    return "[unserializable]";
+  }
+}
+
+interface PartLike {
+  type?: string;
+  text?: string;
+  thinking?: string;
+  name?: string;
+  arguments?: unknown;
+  data?: string;
+  mimeType?: string;
+  id?: string;
+}
+
+function isObject(value: unknown): value is Record<string, unknown> {
+  return typeof value === "object" && value !== null;
+}
+
+/**
+ * Measure the content parts of a message (its `content` field). Image parts are
+ * recorded as ImageFacts and never contribute to contentChars.
+ */
+export function measureContentParts(content: unknown, startIndex = 0): ContentMetrics {
+  if (typeof content === "string") {
+    return { contentChars: codePointCount(content), imageCount: 0, images: [] };
+  }
+  if (!Array.isArray(content)) return { contentChars: 0, imageCount: 0, images: [] };
+
+  let contentChars = 0;
+  const images: ImageFact[] = [];
+  let imageIndex = 0;
+
+  for (const raw of content) {
+    if (!isObject(raw)) continue;
+    const part = raw as PartLike;
+    if (part.type === "text" && typeof part.text === "string") {
+      contentChars += codePointCount(part.text);
+    } else if (part.type === "thinking" && typeof part.thinking === "string") {
+      contentChars += codePointCount(part.thinking);
+    } else if (part.type === "toolCall") {
+      if (typeof part.name === "string") contentChars += codePointCount(part.name);
+      // Normalized arguments representation; id is metadata and not counted.
+      contentChars += codePointCount(safeJson(part.arguments));
+    } else if (part.type === "image") {
+      const mimeType = typeof part.mimeType === "string" ? part.mimeType : "application/octet-stream";
+      const data = typeof part.data === "string" ? part.data : "";
+      let payloadBytes = 0;
+      let dimensions: { width?: number; height?: number } = {};
+      if (data.length > 0) {
+        try {
+          const bytes = decodeBase64(data);
+          payloadBytes = bytes.length;
+          dimensions = readImageDimensions(bytes);
+        } catch {
+          payloadBytes = 0;
+        }
+      }
+      images.push({
+        index: startIndex + imageIndex,
+        mimeType,
+        payloadBytes,
+        ...(dimensions.width !== undefined ? { width: dimensions.width } : {}),
+        ...(dimensions.height !== undefined ? { height: dimensions.height } : {}),
+      });
+      imageIndex += 1;
+    }
+  }
+
+  return { contentChars, imageCount: images.length, images };
+}
+
+/**
+ * Measure a single message, including role-specific content fields
+ * (bash command/output, custom summary). Image payload bytes are never added to
+ * contentChars.
+ */
+export function measureMessage(message: MessageLike): ContentMetrics {
+  const base = measureContentParts(message.content);
+  let contentChars = base.contentChars;
+  const images = [...base.images];
+
+  if (message.role === "bashExecution") {
+    if (typeof message.command === "string") contentChars += codePointCount(message.command);
+    if (typeof message.output === "string") contentChars += codePointCount(message.output);
+  }
+  if (message.role === "custom" && typeof message.summary === "string") {
+    contentChars += codePointCount(message.summary);
+  }
+
+  return { contentChars, imageCount: images.length, images };
+}
+
+/** Re-index images sequentially across the aggregated list. */
+export function aggregateMetrics(parts: readonly ContentMetrics[]): ContentMetrics {
+  let contentChars = 0;
+  let imageCount = 0;
+  const images: ImageFact[] = [];
+  for (const part of parts) {
+    contentChars += part.contentChars;
+    imageCount += part.imageCount;
+    for (const image of part.images) {
+      images.push({ ...image, index: images.length });
+    }
+  }
+  return { contentChars, imageCount, images };
+}