pi-fold 4.2.3 → 4.2.4

package/README.md

@@ -40,12 +40,16 @@ In an interactive session one compact, full-height bar sits directly above Pi's
   <img alt="A single full-height usage bar. Batlow colours group consolidated folds, span folds, tool folds, marked content, raw context and pinned content in that order. The suffix shows percentage, commit point, and fold, mark, and pin counts." src="https://raw.githubusercontent.com/shaneconner/fold/main/media/fold-bar-light.svg">
 </picture>
 
-The forty-column bar uses full-height half-blocks for eighty steps of fill, reading Pi's `getContextUsage()` directly on every render. Its percentage uses Pi's full context window, the same scale as Pi's footer, including Pi's estimate of messages since the last provider response. Within that fill, colours show the estimated visible composition, grouped by distance from compression: **Consolidated Folds, Span Folds, Tool Folds, Marked, Raw, Pinned**. The first three are compressed now, deepest first; marked content folds at the next commit; raw content is eligible later; pinned content is held against every commit until released. The six shades, from consolidated to pinned, are read off a colour map, [Crameri's Batlow](https://www.fabiocrameri.ch/colourmaps/) end to end by default, so the bar reads as a gradient from most compressed to most settled. `/fold-settings` offers Batlow, viridis, plasma, magma and cividis, and where on the chosen map the two ends sit, so a slice of a map or a reversed one is a keypress away; the bar repaints on each step. Whatever the choice, each shade's lightness is moved away from the background, hue held, until it clears 4.5:1 on dark and light terminals, and label colours match the bar. Each nonempty category keeps at least one half-cell when the measured fill has room; remaining width follows relative mass. This visibility floor makes small compressed categories visible, so individual category widths are not token-exact. At very low occupancy, the largest shares take the available space. There are no individual fold edges, scores, or height differences.
+The forty-column bar uses full-height half-blocks for eighty steps of fill, reading Pi's `getContextUsage()` directly whenever the session moves, on the same key Pi's footer re-reads on (session, leaf entry, entry count, model), so a repaint never pays for a fresh reading. Its percentage uses Pi's full context window, the same scale as Pi's footer, including Pi's estimate of messages since the last provider response. Within that fill, colours show the estimated visible composition, grouped by distance from compression: **Consolidated Folds, Span Folds, Tool Folds, Marked, Raw, Pinned**. The first three are compressed now, deepest first; marked content folds at the next commit; raw content is eligible later; pinned content is held against every commit until released. The six shades, from consolidated to pinned, are read off a colour map, [Crameri's Batlow](https://www.fabiocrameri.ch/colourmaps/) end to end by default, so the bar reads as a gradient from most compressed to most settled. `/fold-settings` offers Batlow, viridis, plasma, magma and cividis, and where on the chosen map the two ends sit, so a slice of a map or a reversed one is a keypress away; the bar repaints on each step. Whatever the choice, each shade's lightness is moved away from the background, hue held, until it clears 4.5:1 on dark and light terminals, and label colours match the bar. Each nonempty category keeps at least one half-cell when the measured fill has room; remaining width follows relative mass. This visibility floor makes small compressed categories visible, so individual category widths are not token-exact. At very low occupancy, the largest shares take the available space. There are no individual fold edges, scores, or height differences.
 
 The suffix reads `34% · commit at 55% · N Folds (X Cons., Y Span, Z Tool, X Mark, Y Pin)`, naming only the kinds with something to count, in the bar's order; a fresh session reads `0 Folds`. `N Folds` counts visible compressed folds; `Mark` counts pending marks and `Pin` counts protected entries, not additional compressed folds. A tool fold is lossless compression of tool-result context, not a count of tool calls. `/fold-status` retains the detailed token and nesting figures. Its `Budget` row and the editor's `Budget` header report the separate serving budget and the tokens remaining before a commit.
 
 Two dotted marks stand in the band's edge cells: one where a commit aims to land and one where automatic folding is triggered. `minTarget` and `maxTarget` remain shares of the serving budget; the marks and displayed commit point are converted to Pi's full-window scale, so the trigger does not move. With a 272,000-token window and the default 255,616-token serving budget, the 80% trigger appears at about 75% of Pi's window. The usage label rounds down to a whole percentage, matching pi-cc's footer; bar fill and folding decisions keep the unrounded value. The displayed commit point rounds to the nearest whole percentage. Each is drawn over the fill's own colour in a deeper tone of that same hue, so the fill continues behind the mark; on the empty track they stand in muted ink. Neither takes a colour of its own or a warning colour. When Pi reports unknown usage, including immediately after native compaction, the row says `not measured yet`; no saved count or local estimate fills the gap. After a fold commit it identifies the old reading until the provider measures again. Crossing the target is routine runtime work, not a user action item. On reload, saved session context supplies the fold counts and composition without a new model message; the next real context event replaces that read-only view. Unavailable composition is stated as `mapping`; if folding stops, `FOLDING STOPPED` replaces the bar. Non-truecolor terminals use the theme's own colours.
 
+### The chat
+
+The chat shows the model's window. Pi paints its chat from every session entry and only native compaction clears it, so without this a long folded session would keep its whole transcript on screen. Each fold the model holds collapsed appears as one muted row, `▸ consolidation · 59 entries · <brief>`, in place of the messages it covers. When the model expands a fold, its messages return where they were. The view follows the model one way: it changes when a commit, an expansion or a refold changes what the model sees, or when Pi repaints the chat, and nothing done in the chat changes the model's context. `/fold-editor` remains the place to read inside a fold.
+
 ## Results
 
 One 64-stage assignment over the curl C repository, run as a single agentic turn. Same model, same 251,520-token serving budget, every run. After all 64 stages landed, one withheld message asked sixteen questions about material from across the whole session.

package/extensions/active-context.ts

@@ -35,6 +35,7 @@ import {
   requireActiveFold,
   selectAutomaticRung,
   setFoldProjectionState,
+  visibleCollapsedFolds,
   withExpandLease,
 } from "./lib/folding.ts";
 import {
@@ -206,6 +207,12 @@ import type {
 import { buildActiveContextCommands, buildActiveContextTool } from "./lib/tool-surface.ts";
 import { buildFoldEditorData, FoldEditorView } from "./lib/editor-ui.ts";
 import { publishLiveSettings } from "./lib/live-settings.ts";
+import {
+  foldedChildren,
+  locateChatContainer,
+  type MirrorFold,
+  unfoldedChildren,
+} from "./lib/transcript-mirror.ts";
 import {
   entryText,
   mapActiveContext,
@@ -740,8 +747,8 @@ export function registerActiveContext(pi: any, options: {
    * THE FOLD BAR. One coloured composition row directly above the footer,
    * drawn by `renderFoldBar` from a model this function rebuilds on every status update.
    * The component reads Pi's context usage and the LIVE theme at render time. Neither
-   * reading is cached between status events; the host is asked to render after
-   * each rebuild. Installed once per host UI, on the first update that finds one.
+   * reading waits for a status event; the host is asked to render after each rebuild.
+   * Installed once per host UI, on the first update that finds one.
    */
   const foldBar = {
     model: null as FoldBarModel | null,
@@ -752,14 +759,106 @@ export function registerActiveContext(pi: any, options: {
     staleSince: null as unknown,
     /** Display-only reconstruction while waiting for the first real context event. */
     startupSnapshot: null as ActiveContextSnapshot | null,
+    /** Pi's last usage reading and the session state it was read against. */
+    usageKey: null as unknown[] | null,
+    usage: undefined as unknown,
+  };
+  // MIRROR THE FOOTER'S READING, ON THE FOOTER'S KEY. getContextUsage rebuilds the whole
+  // session projection per call (29 ms on a 16,218-entry session) and the bar renders
+  // every frame. Pi's own footer re-reads only when the session, session id, leaf, entry
+  // count or model moves (FooterComponent.getSessionStats), since entries are append-only
+  // and every append moves the leaf. Read on exactly those moves and we show the same
+  // value at the same moments, without paying for it per frame.
+  const piContextUsage = (ctx: any): unknown => {
+    let key: unknown[] | null = null;
+    try {
+      // No leaf to key on means no way to know the reading moved: read every time.
+      const manager = ctx.sessionManager;
+      if (typeof manager?.getLeafId !== "function") throw new Error("no session leaf");
+      key = [manager, manager.getSessionId?.(), manager.getLeafId(),
+        manager.getEntryCount?.(), ctx.model];
+      const cached = foldBar.usageKey;
+      if (cached && cached.length === key.length && cached.every((value, i) => value === key![i])) {
+        return foldBar.usage;
+      }
+    } catch { key = null; }
+    let usage: unknown;
+    try { usage = ctx.getContextUsage?.(); } catch { }
+    foldBar.usageKey = key;
+    foldBar.usage = usage;
+    return usage;
+  };
+  /**
+   * THE CHAT SHOWS THE MODEL'S WINDOW (2026-10-04, lib/transcript-mirror.ts). The PLAN
+   * is the folds the model holds collapsed, rebuilt on every status update and keyed by
+   * which folds those are, so a commit, an expansion and a refold each move it. The
+   * chat is re-folded only when the key moved or Pi repainted the chat (a repaint swaps
+   * the container's children array), checked on the fold bar's frame: two comparisons
+   * per frame, one pass over the chat per change. DISPLAY ONLY: it reads runtime state
+   * and writes nothing but Pi's chat container, and a failure leaves Pi's own transcript.
+   */
+  const transcriptMirror = {
+    tui: null as unknown,
+    chat: null as { children: unknown[] } | null,
+    ctx: null as any,
+    plan: [] as MirrorFold[],
+    planKey: "",
+    applied: null as unknown[] | null,
+    appliedKey: "",
+  };
+  const planTranscript = (ctx: any): void => {
+    transcriptMirror.ctx = ctx;
+    const state = persistence.state;
+    let snapshot = lifecycle.latestSnapshot ?? foldBar.startupSnapshot;
+    try { snapshot = authoritativeSnapshotFor(ctx); } catch { }
+    const folds = state && snapshot ? visibleCollapsedFolds(state, snapshot) : [];
+    const key = folds.map((fold) => fold.id).join(",");
+    if (key === transcriptMirror.planKey) return;
+    const plan = folds.map((fold) => {
+      const row = foldIndexRow(fold, state!, snapshot!);
+      return {
+        id: fold.id,
+        kind: String(row.kind ?? ""),
+        brief: String(row.brief ?? ""),
+        sourceCount: Number(row.sourceCount ?? 0),
+        entryIds: flattenFoldRefs(fold, state!).map((ref) => ref.entryId),
+      };
+    });
+    transcriptMirror.plan = plan;
+    transcriptMirror.planKey = key;
+  };
+  const syncTranscript = (): void => {
+    const mirror = transcriptMirror;
+    if (!mirror.tui || !mirror.ctx) return;
+    try {
+      mirror.chat ??= locateChatContainer(mirror.tui);
+      const chat = mirror.chat;
+      if (!chat) return;
+      if (chat.children === mirror.applied && mirror.planKey === mirror.appliedKey) return;
+      // Settle the attempt BEFORE making it: a failure is retried when something moves,
+      // never on every frame.
+      mirror.appliedKey = mirror.planKey;
+      mirror.applied = chat.children;
+      const theme = () => mirror.ctx?.ui?.theme;
+      const next = foldedChildren(chat.children, mirror.ctx.sessionManager.getBranch(), mirror.plan,
+        (text) => { try { return theme()?.fg?.("muted", text) ?? text; } catch { return text; } });
+      chat.children = next;
+      mirror.applied = next;
+      foldBar.requestRender?.();
+    } catch { }
+  };
+  const restoreTranscript = (): void => {
+    const chat = transcriptMirror.chat;
+    try { if (chat) chat.children = unfoldedChildren(chat.children); } catch { }
+    Object.assign(transcriptMirror, { chat: null, ctx: null, plan: [], planKey: "",
+      applied: null, appliedKey: "" });
   };
   // DISPLAY ONLY. Pi owns the live reading, including its unmeasured trailing messages
   // and post-compaction unknown state. Never substitute our saved provider measurement
   // or projection estimate. The band remains a share of the SERVING budget: convert its
   // points onto Pi's full-window axis, without changing any scheduling arithmetic.
   const foldBarUsage = (ctx: any): Pick<FoldBarModel, "share" | "percent" | "commitShare" | "aimShare"> => {
-    let usage: unknown;
-    try { usage = ctx.getContextUsage?.(); } catch { }
+    const usage = piContextUsage(ctx);
     const reportedWindow = ownValue(usage, "contextWindow");
     const modelWindow = ownValue(ctx.model, "contextWindow");
     const validWindow = (value: unknown): value is number =>
@@ -787,12 +886,14 @@ export function registerActiveContext(pi: any, options: {
     foldBar.installed = true;
     ctx.ui.setWidget(entryTypePrefix, (tui: any) => {
       foldBar.requestRender = () => { try { tui.requestRender(); } catch { } };
+      transcriptMirror.tui = tui;
       return {
         /** The model behind the row, for the lab and the gates. */
         get model(): FoldBarModel | null {
           return foldBar.model ? { ...foldBar.model, ...foldBarUsage(ctx) } : null;
         },
         render(width: number): string[] {
+          syncTranscript();
           const model = this.model;
           if (!model) return [];
           try { return [renderFoldBar(model, width, ctx.ui.theme)]; }
@@ -907,6 +1008,7 @@ export function registerActiveContext(pi: any, options: {
             weighed: ladder.bandTopMeasurement !== null &&
               ladder.bandTopMeasurement === measurements.lastProviderMeasurement,
           });
+          planTranscript(ctx);
           foldBar.requestRender?.();
         }
       } catch { }
@@ -5071,6 +5173,7 @@ export function registerActiveContext(pi: any, options: {
     try { ctx.ui?.setStatus?.(entryTypePrefix, undefined); } catch { }
     foldBar.model = null;
     foldBar.startupSnapshot = null;
+    restoreTranscript();
     foldBar.requestRender?.();
   });
 

package/extensions/lib/transcript-mirror.ts

@@ -0,0 +1,207 @@
+// THE TRANSCRIPT MIRRORS THE MODEL'S WINDOW (2026-10-04).
+//
+// Pi paints the chat from raw session entries and walks every component on every frame,
+// and only a native compaction ever clears it. pi-fold cancels native compaction, so the
+// chat grew for the whole life of a session: 16,218 entries painted per frame on
+// session 01a10161. The model, meanwhile, sees each fold as one placeholder.
+//
+// The chat now shows what the model sees: every fold the model holds collapsed becomes
+// ONE row, and the components it covers leave the chat until the model expands it.
+// Read-only and one-way, as in memex's applyTranscriptFolds: the model's structure moves
+// the view, nothing in the view moves the model. Pi gives an extension no transcript API,
+// so this edits Pi's chat container in place. It never builds a message component: the
+// rows Pi painted stay Pi's own objects, live references and all, and the ones a fold
+// covers are held by its row so an expansion hands them back unchanged.
+
+import {
+	AssistantMessageComponent,
+	CustomMessageComponent,
+	ToolExecutionComponent,
+	UserMessageComponent,
+} from "@earendil-works/pi-coding-agent";
+import { Spacer, truncateToWidth } from "@earendil-works/pi-tui";
+
+export interface MirrorFold {
+	id: string;
+	kind: string;
+	brief: string;
+	sourceCount: number;
+	/** Every session entry the fold covers, nested folds included. */
+	entryIds: string[];
+}
+
+interface ChatContainer {
+	children: unknown[];
+}
+
+/** One collapsed fold in the chat. Holds the components it covers, in order. */
+export class FoldRow {
+	readonly foldId: string;
+	readonly hidden: unknown[] = [];
+	private readonly text: string;
+	private readonly style: (text: string) => string;
+
+	constructor(fold: MirrorFold, style: (text: string) => string) {
+		this.foldId = fold.id;
+		this.style = style;
+		const [head] = fold.brief.split("\n").map((line) => line.trim()).filter(Boolean);
+		const count = `${fold.sourceCount} ${fold.sourceCount === 1 ? "entry" : "entries"}`;
+		this.text = `\u25b8 ${fold.kind ? `${fold.kind} · ` : ""}${count}${head ? ` · ${head}` : ""}`;
+	}
+
+	render(width: number): string[] {
+		return ["", this.style(truncateToWidth(` ${this.text}`, Math.max(1, width)))];
+	}
+
+	invalidate(): void { }
+}
+
+/** Pi's chat container: the container holding the most message components directly.
+ *  Pi moves the same container between its regular and fullscreen hosts, so it is found
+ *  once. Null until the chat holds a message, which is before any fold exists. */
+export function locateChatContainer(root: unknown): ChatContainer | null {
+	let best: ChatContainer | null = null;
+	let bestCount = 0;
+	const seen = new Set<unknown>();
+	const visit = (node: unknown, depth: number): void => {
+		const children = (node as { children?: unknown })?.children;
+		if (depth > 4 || seen.has(node) || !Array.isArray(children)) return;
+		seen.add(node);
+		let count = 0;
+		for (const child of children) {
+			// A message component's own parts are never the chat: do not descend into one.
+			if (anchorKind(child) !== null) count += 1;
+			else visit(child, depth + 1);
+		}
+		if (count > bestCount) { best = node as ChatContainer; bestCount = count; }
+	};
+	visit(root, 0);
+	return best;
+}
+
+function anchorKind(component: unknown): "assistant" | "tool" | "user" | "custom" | null {
+	if (component instanceof AssistantMessageComponent) return "assistant";
+	if (component instanceof ToolExecutionComponent) return "tool";
+	if (component instanceof UserMessageComponent) return "user";
+	if (component instanceof CustomMessageComponent) return "custom";
+	return null;
+}
+
+function userText(content: unknown): string {
+	if (typeof content === "string") return content;
+	if (!Array.isArray(content)) return "";
+	return content.filter((part) => part?.type === "text").map((part) => String(part.text ?? "")).join("");
+}
+
+/** What a component shows, in the terms its session entry can be found by. */
+function componentSignature(component: unknown): string | null {
+	const any = component as Record<string, any>;
+	switch (anchorKind(component)) {
+		case "assistant": return `a:${any.lastMessage?.timestamp}`;
+		case "user": return `u:${any.text}`;
+		case "custom": return `c:${any.message?.customType}`;
+		default: return null;
+	}
+}
+
+/**
+ * Which session entry painted each chat component. Anchors are found by what they show:
+ * an assistant message by its timestamp, a tool call by its call id, a user message by
+ * its text and a custom message by its type, each searched FORWARD from the last match so
+ * a repeated text binds to the right occurrence. Everything else (spacers, notices)
+ * belongs to its neighbour: a run of spacers directly before an anchor leads that anchor,
+ * anything else trails the anchor before it. Unowned components are never folded.
+ */
+export function chatOwners(children: unknown[], branch: any[]): Array<string | null> {
+	const positions = new Map<string, number[]>();
+	const callOwner = new Map<string, number>();
+	branch.forEach((entry, index) => {
+		const message = entry?.type === "message" ? entry.message : null;
+		let signature: string | null = null;
+		if (message?.role === "assistant") {
+			signature = `a:${message.timestamp}`;
+			for (const part of Array.isArray(message.content) ? message.content : []) {
+				if (part?.type === "toolCall" && typeof part.id === "string") callOwner.set(part.id, index);
+			}
+		} else if (message?.role === "user") signature = `u:${userText(message.content)}`;
+		else if (entry?.type === "custom_message") signature = `c:${entry.customType}`;
+		if (signature === null) return;
+		const list = positions.get(signature);
+		if (list) list.push(index); else positions.set(signature, [index]);
+	});
+	const nextAt = (signature: string, from: number): number => {
+		const list = positions.get(signature);
+		if (!list) return -1;
+		let low = 0;
+		let high = list.length;
+		while (low < high) {
+			const mid = (low + high) >> 1;
+			if (list[mid] < from) low = mid + 1; else high = mid;
+		}
+		return low < list.length ? list[low] : -1;
+	};
+
+	const owners: Array<string | null> = new Array(children.length).fill(null);
+	let cursor = 0;
+	let previous: string | null = null;
+	let pending: number[] = [];
+	const settle = (lead: string | null): void => {
+		// Spacers immediately before an anchor lead it; the rest trail the previous anchor.
+		let split = pending.length;
+		while (split > 0 && children[pending[split - 1]] instanceof Spacer) split -= 1;
+		for (let i = 0; i < pending.length; i += 1) owners[pending[i]] = i < split ? previous : lead;
+		pending = [];
+	};
+	children.forEach((component, index) => {
+		let at = -1;
+		if (component instanceof ToolExecutionComponent) {
+			at = callOwner.get(String((component as unknown as { toolCallId: unknown }).toolCallId)) ?? -1;
+		} else {
+			const signature = componentSignature(component);
+			if (signature === null) { pending.push(index); return; }
+			at = nextAt(signature, cursor);
+		}
+		const owner = at >= 0 && typeof branch[at]?.id === "string" ? branch[at].id as string : null;
+		settle(owner);
+		owners[index] = owner;
+		if (at >= 0) cursor = Math.max(cursor, at);
+		previous = owner;
+	});
+	settle(previous);
+	return owners;
+}
+
+/** Hand every row's components back, in place: the chat as Pi painted it. */
+export function unfoldedChildren(children: unknown[]): unknown[] {
+	return children.flatMap((child) => child instanceof FoldRow ? child.hidden : [child]);
+}
+
+/** The chat with each collapsed fold reduced to its row. A fold none of whose entries
+ *  are on screen gets no row. */
+export function foldedChildren(
+	children: unknown[],
+	branch: any[],
+	folds: MirrorFold[],
+	style: (text: string) => string,
+): unknown[] {
+	const raw = unfoldedChildren(children);
+	if (!folds.length) return raw;
+	const foldOf = new Map<string, MirrorFold>();
+	for (const fold of folds) for (const entryId of fold.entryIds) foldOf.set(entryId, fold);
+	const owners = chatOwners(raw, branch);
+	const rows = new Map<string, FoldRow>();
+	const output: unknown[] = [];
+	raw.forEach((component, index) => {
+		const owner = owners[index];
+		const fold = owner === null ? undefined : foldOf.get(owner);
+		if (!fold) { output.push(component); return; }
+		let row = rows.get(fold.id);
+		if (!row) {
+			row = new FoldRow(fold, style);
+			rows.set(fold.id, row);
+			output.push(row);
+		}
+		row.hidden.push(component);
+	});
+	return output;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-fold",
-  "version": "4.2.3",
+  "version": "4.2.4",
   "description": "Lossless context folding for the Pi coding agent: only the oldest context changes; the fresh window stays untouched.",
   "keywords": [
     "pi-package",