pi-posthorse 0.4.5 → 0.4.6

package/README.md

@@ -8,6 +8,8 @@ Native, no-summary context windows for the [`fitchmultz/pi`](https://github.com/
 
 Pi owns the persisted boundary. Posthorse owns the policy: stable window guidance, one best-effort checkpoint reminder, `new_context`, `get_context_remaining`, durable `notes`, and window-aware `history`. A rollover removes the old window from active model context while the JSONL transcript stays append-only and complete.
 
+An isolated [Codex prototype](adapters/codex-posthorse/README.md) tests local notes and recovery with Codex's native context reset. It is not ready for everyday use and is not included in the Pi package.
+
 ## Requirements
 
 - Node `>=22.19.0`.
@@ -52,7 +54,7 @@ Keep exactly one copy loaded. `pi list` shows every package source; if an older
 4. **`new_context`.** Requests an atomic rollover after the complete tool batch succeeds. An optional handoff is persisted and becomes the first state of the fresh window. If a sibling tool in the same batch fails, Pi does not commit the boundary; the checkpoint reminder still applies.
 5. **Automatic rollover without summaries.** With a supported context budget and room for a recovery record, Posthorse claims Pi's automatic threshold and overflow trigger through `session_before_auto_compact`, before Pi resolves summarization credentials or prepares a summary. Oversized first turns and tool results can then roll over even without summarization credentials. Otherwise Pi's own compaction remains in control. Manual `/compact` is unchanged.
 6. **Bounded recovery record.** The automatic handoff keeps direct user inputs, `ask_question` outcomes, visible coordination messages, and the trailing tool batch that no model has consumed yet (call arguments, bounded result text, and the entry ids to recover the rest). A clearly labeled, possibly stale older checkpoint comes last, after the current inputs and unseen results. Older assistant prose and consumed tool results are not treated as state. Newly submitted input stays separate and is saved after the boundary, not copied into the handoff.
-7. **`notes` and `history`.** Notes live with the repository root, shared across linked worktrees. History searches normalized transcript text and returns stored images for a requested entry.
+7. **`notes` and `history`.** Notes are shared across linked worktrees, at the main checkout for conventional Git layouts or inside the common Git directory when metadata is stored separately. History searches normalized transcript text and returns stored images for a requested entry.
 
 At turn end, Posthorse checks whether usage is in the reminder band before explicitly looking up the full branch. Context filtering skips its branch lookup when model input contains neither `posthorse-reminder` nor legacy `headroom-reminder` messages. History searches, reads, and recovery remain available with no new limits.
 
@@ -79,20 +81,20 @@ In the TUI, tools use Pi's native expandable cards. Collapsed cards show the ope
 
 Committed context-window messages and checkpoint reminders are compact, expandable cards too. The `new_context` tool card describes a request; only the committed context-window message says a fresh window has started.
 
-Read pages, including returned images, shrink to the context that is actually left. Before usage is known, they reserve prompt/tool overhead and leave half the rest free. Unsafe pages are refused with the offset preserved; call `new_context` and retry.
+Read pages, including returned images, shrink to the context that is actually left. Parallel note and history reads share that budget; pages already counted by Pi are not counted twice. Before usage is known, they reserve prompt/tool overhead and leave half the rest free. Unsafe pages are refused with the offset preserved in the call; call `new_context` and retry. Refusal text uses the same budget, so later failures omit repeated guidance once no more fits.
 
 `history search` puts matching original content before recovery material: handoffs, compaction and branch summaries, checkpoint reminders, and `notes`, `new_context`, and `history` calls/results. Ordinary prose or another tool call in the same assistant entry keeps its priority when that content matches. Every entry remains searchable; `history read` returns the complete normalized entry, including any recovery content omitted from a search excerpt.
 
 Within each group, current-branch matches are newest first. With `all: true`, Posthorse searches every session file in the active Pi session directory, newest-modified sessions first and newest entries within each session; this is not a global timestamp sort. The result limit applies after priority, so newer echoes cannot displace older original matches. Entries copied by a fork are reported once.
 
-Notes live in `.pi/notes/` at the repository root (the main checkout for a linked worktree, the current directory outside Git). Add the directory to `.gitignore` when the project should not track it.
+Notes live in `.pi/notes/` at the main checkout for conventional Git layouts, or the current directory outside Git. When Git metadata is stored separately, including in submodules, all checkouts use `.pi/notes/` inside Git's common directory instead; no configuration is needed. If that metadata is unavailable, notes stay local to the checkout. Old checkout-local notes are imported when that checkout is accessed, without overwriting shared files or deleting the originals. Writes report the actual storage path. Add `.pi/notes/` to `.gitignore` when the project should not track it.
 
 ## Data and privacy
 
 - Posthorse makes no network requests.
 - Notes are plaintext files under `.pi/notes`. They survive package removal and may be committed unless ignored.
 - `history` with `all: true` scans nested JSONL files in the active Pi session directory, including subagent sessions.
-- History can return user text, assistant text and thinking, tool arguments and results, handoffs, custom messages, and images. Direct shell entries Pi marked `excludeFromContext` come back as a placeholder only.
+- History can return user text, assistant text and thinking, assistant failure status and error messages, tool arguments and results, handoffs, custom messages, and images. Direct shell entries Pi marked `excludeFromContext` come back as a placeholder only.
 - Returned history content enters the currently selected model and provider context.
 - Removing Posthorse does not remove notes or Pi session history.
 

package/index.ts

@@ -8,16 +8,18 @@
 
 import {
 	appendFileSync,
+	constants,
+	copyFileSync,
 	createReadStream,
 	existsSync,
 	mkdirSync,
 	readdirSync,
 	readFileSync,
+	realpathSync,
 	statSync,
 	writeFileSync,
 } from "node:fs";
-import { dirname, isAbsolute, join, normalize, relative, resolve, sep } from "node:path";
-import { createInterface } from "node:readline";
+import { basename, dirname, isAbsolute, join, normalize, relative, resolve, sep } from "node:path";
 import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
 import { Type } from "typebox";
 import { registerPosthorseMessages, toolCards, type PosthorseDisplay } from "./ui.ts";
@@ -64,6 +66,7 @@ type ImageLike = { type: "image"; data: string; mimeType: string };
 type MessageLike = {
 	role?: string;
 	stopReason?: string;
+	errorMessage?: string;
 	content?: unknown;
 	toolName?: string;
 	toolCallId?: string;
@@ -164,21 +167,49 @@ function textResult(text: string, images: ImageLike[] = [], display?: PosthorseD
 	return { content: [{ type: "text" as const, text }, ...images], details: display };
 }
 
-/**
- * Notes belong to the repository root, walking up from cwd. A linked worktree's `.git` is a file
- * ("gitdir: <main>/.git/worktrees/<name>"); its notes belong to the main checkout so every worktree
- * shares them and they outlive the worktree. Outside Git, cwd is the root.
- */
+/** Import old checkout-local notes without overwriting shared notes, including concurrent writes. */
+function importLegacyNotes(source: string, target: string): void {
+	if (!existsSync(source)) return;
+	if (existsSync(target) && (!statSync(target).isDirectory() || realpathSync(source) === realpathSync(target))) return;
+	mkdirSync(target, { recursive: true });
+	for (const name of readdirSync(source)) {
+		const from = join(source, name);
+		const to = join(target, name);
+		if (statSync(from).isDirectory()) {
+			importLegacyNotes(from, to);
+		} else {
+			try {
+				copyFileSync(from, to, constants.COPYFILE_EXCL);
+			} catch (error) {
+				if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error;
+			}
+		}
+	}
+}
+
+/** Conventional repos use the main checkout; separate Git directories are their own shared root. */
 function notesRoot(cwd: string): string {
 	for (let dir = cwd; ; dir = dirname(dir)) {
 		const marker = join(dir, ".git");
 		if (existsSync(marker)) {
+			let root: string;
 			try {
-				const gitdir = readFileSync(marker, "utf8").match(/^gitdir:\s*(.+?)\s*$/m)?.[1];
-				const main = gitdir && resolve(dir, gitdir).match(/^(.+)[\\/]\.git[\\/]worktrees[\\/][^\\/]+$/)?.[1];
-				if (main) return main;
-			} catch {}
-			return dir;
+				let target = marker;
+				if (!statSync(marker).isDirectory()) {
+					const gitdir = readFileSync(marker, "utf8").match(/^gitdir:\s*(.+?)\s*$/m)?.[1];
+					if (!gitdir) return dir;
+					target = resolve(dir, gitdir);
+				}
+				target = realpathSync(target);
+				const commonFile = join(target, "commondir");
+				const common = realpathSync(existsSync(commonFile) ? resolve(target, readFileSync(commonFile, "utf8").trim()) : target);
+				root = basename(common) === ".git" ? dirname(common) : common;
+			} catch {
+				// Orphaned/copied worktrees can outlive their Git metadata; local notes still work.
+				return dir;
+			}
+			if (root !== dir) importLegacyNotes(join(dir, ".pi", "notes"), join(root, ".pi", "notes"));
+			return root;
 		}
 		if (dirname(dir) === dir) return cwd;
 	}
@@ -195,6 +226,12 @@ function excerptAround(text: string, index: number, before: number, length: numb
 	return `${start ? "…" : ""}${text.slice(start, end)}${end < text.length ? "…" : ""}`;
 }
 
+function assistantFailure(message: MessageLike): string {
+	if (message.role !== "assistant") return "";
+	if (message.stopReason !== "error" && message.stopReason !== "aborted" && !message.errorMessage) return "";
+	return `[${message.stopReason ?? "error"}]${message.errorMessage ? ` ${message.errorMessage}` : ""}`;
+}
+
 function flattenEntry(entry: EntryLike): string | undefined {
 	if (entry.type === "message") {
 		const message = entry.message ?? {};
@@ -203,7 +240,7 @@ function flattenEntry(entry: EntryLike): string | undefined {
 			if (message.excludeFromContext === true) return "[bashExecution] (excluded from model context by Pi)";
 			return `[bashExecution] $ ${message.command ?? ""}\n${message.output ?? ""}`;
 		}
-		return `[${message.role ?? "message"}] ${[textOf(message), imageSummary(imagesOf(message.content))].filter(Boolean).join("\n")}`;
+		return `[${message.role ?? "message"}] ${[textOf(message), assistantFailure(message), imageSummary(imagesOf(message.content))].filter(Boolean).join("\n")}`;
 	}
 	if (entry.type === "compaction" || entry.type === "branch_summary") {
 		return `[${entry.type}] ${entry.summary ?? ""}`;
@@ -254,8 +291,8 @@ function historyHit(item: WindowedEntry, query: string, source = ""): HistoryHit
 				if (partText) originals[originals.length - 1] += `${originals.at(-1) ? "\n" : ""}${partText}`;
 			}
 		}
-		const images = imageSummary(item.images);
-		if (images) originals[originals.length - 1] += `${originals.at(-1) ? "\n" : ""}${images}`;
+		const metadata = [assistantFailure(entry.message), imageSummary(item.images)].filter(Boolean).join("\n");
+		if (metadata) originals[originals.length - 1] += `${originals.at(-1) ? "\n" : ""}${metadata}`;
 		if (originals[0]) originals[0] = `[assistant] ${originals[0]}`;
 		const original = originals.find((part) => part.toLowerCase().includes(query));
 		priority = original ? 0 : 1;
@@ -291,25 +328,30 @@ function* windowEntries(entries: Iterable<EntryLike>): Generator<WindowedEntry>
 	}
 }
 
+/** JSONL splits on LF, not the Unicode separators that Node's readline also recognizes. */
+async function* jsonlLines(file: string, signal?: AbortSignal): AsyncGenerator<string> {
+	const stream = createReadStream(file, { encoding: "utf8", signal });
+	let pending = "";
+	for await (const chunk of stream) {
+		const lines = (pending + chunk).split("\n");
+		pending = lines.pop()!;
+		yield* lines;
+	}
+	if (pending) yield pending;
+}
+
 async function* sessionWindowEntries(file: string, signal?: AbortSignal): AsyncGenerator<WindowedEntry> {
 	if (!existsSync(file)) return;
-	const stream = createReadStream(file, { encoding: "utf8", signal });
-	const lines = createInterface({ input: stream, crlfDelay: Number.POSITIVE_INFINITY });
 	const windows = new Map<string, string>();
-	try {
-		for await (const line of lines) {
-			let entry: EntryLike;
-			try {
-				entry = JSON.parse(line) as EntryLike;
-			} catch {
-				continue;
-			}
-			const item = toWindowedEntry(entry, windows);
-			if (item) yield item;
+	for await (const line of jsonlLines(file, signal)) {
+		let entry: EntryLike;
+		try {
+			entry = JSON.parse(line) as EntryLike;
+		} catch {
+			continue;
 		}
-	} finally {
-		lines.close();
-		stream.destroy();
+		const item = toWindowedEntry(entry, windows);
+		if (item) yield item;
 	}
 }
 
@@ -636,26 +678,13 @@ function unsupportedMessage(budget: Budget): string {
 	return `Posthorse: unsupported configuration. The model's context window (${n(budget.contextWindow)} tokens) minus Pi's compaction.reserveTokens (${n(budget.reserveTokens)}) leaves ${n(budget.usable)} usable tokens; Posthorse needs at least ${n(MIN_USABLE_TOKENS)}. Automatic rollover and checkpoint reminders are off for this model. Lower compaction.reserveTokens in Pi settings or use a larger-context model. new_context remains available with a model-aware handoff limit.`;
 }
 
-/** Characters that fit before Pi's automatic line (or the hard limit), capped at the absolute ceiling. */
-function requirePage(ctx: NativeContext, offset: number, imageCount: number, toolTokens: number): number {
+/** Capacity before Pi's automatic line (or hard limit), including the margin for page metadata and refusals. */
+function pageCapacity(ctx: NativeContext, toolTokens: number): number {
 	const usage = ctx.getContextUsage();
-	let chars: number;
-	if (!usage || usage.tokens == null) {
-		chars = Math.max(0, freshPayloadChars(ctx, toolTokens) - imageCount * ESTIMATED_IMAGE_CHARS);
-	} else {
-		const budget = budgetFor(ctx, usage.contextWindow);
-		const line = budget?.enabled && budget.supported ? budget.rolloverAt : usage.contextWindow;
-		chars = Math.min(
-			MAX_HANDOFF_CHARS,
-			Math.max(0, line - usage.tokens - PAGE_MARGIN_TOKENS) * 4 - imageCount * ESTIMATED_IMAGE_CHARS,
-		);
-	}
-	if (chars < MIN_PAGE_CHARS) {
-		throw new Error(
-			`Too little context remains to read a page safely. Call new_context first, then retry with offset ${offset}.`,
-		);
-	}
-	return chars;
+	if (!usage || usage.tokens == null) return freshPayloadChars(ctx, toolTokens) / 4 + PAGE_MARGIN_TOKENS;
+	const budget = budgetFor(ctx, usage.contextWindow);
+	const line = budget?.enabled && budget.supported ? budget.rolloverAt : usage.contextWindow;
+	return line - usage.tokens;
 }
 
 function buildGuidance(ctx: NativeContext): string {
@@ -698,7 +727,42 @@ export default function (pi: ExtensionAPI) {
 			);
 	};
 
+	let pendingPageTokens = 0;
+	let previousPageUsage: number | null | undefined;
+	const pageSize = (ctx: NativeContext, offset: number, imageCount: number) => {
+		const usage = ctx.getContextUsage()?.tokens;
+		// Parallel siblings are not in native usage yet; sequential results must not be counted twice.
+		if (usage != null && previousPageUsage != null) {
+			pendingPageTokens = Math.max(0, pendingPageTokens - Math.max(0, usage - previousPageUsage));
+		}
+		previousPageUsage = usage;
+		const remaining = pageCapacity(ctx, activeToolTokens()) - pendingPageTokens;
+		const chars = Math.min(
+			MAX_HANDOFF_CHARS,
+			Math.max(0, remaining - PAGE_MARGIN_TOKENS) * 4 - imageCount * ESTIMATED_IMAGE_CHARS,
+		);
+		if (chars < MIN_PAGE_CHARS) {
+			const message = `Too little context remains to read a page safely. Call new_context first, then retry with offset ${offset}.`;
+			// Refusals consume context too. Once even the guidance no longer fits, omit repeated text;
+			// the failed call still carries its original offset and earlier refusals explain the retry.
+			const text = Math.ceil(message.length / 4) <= remaining ? message : "";
+			pendingPageTokens += Math.ceil(text.length / 4);
+			throw new Error(text);
+		}
+		return chars;
+	};
+	const pageResult = (text: string, images: ImageLike[], display: PosthorseDisplay) => {
+		pendingPageTokens += Math.ceil(text.length / 4) + images.length * (ESTIMATED_IMAGE_CHARS / 4);
+		return textResult(text, images, display);
+	};
+	pi.on("turn_start", () => {
+		pendingPageTokens = 0;
+		previousPageUsage = undefined;
+	});
+
 	pi.on("session_start", (_event, ctx) => {
+		pendingPageTokens = 0;
+		previousPageUsage = undefined;
 		nativeContext(ctx);
 	});
 
@@ -854,7 +918,7 @@ export default function (pi: ExtensionAPI) {
 		label: "Notes",
 		...toolCards("notes"),
 		description:
-			"Persistent notes in .pi/notes/ that survive context resets. Ops: list, read (paged; pass offset to continue), write (create/replace; empty content clears), append, search (case-insensitive substring over note lines). Inside a Git repository, including nested directories and linked worktrees, notes belong to the main checkout.",
+			"Persistent notes in .pi/notes/ that survive context resets. Ops: list, read (paged; pass offset to continue), write (create/replace; empty content clears), append, search (case-insensitive substring over note lines). Git worktrees share notes: at the main checkout for conventional .git layouts, or in the common Git directory when metadata is stored separately. Old checkout-local notes are imported without overwriting shared notes.",
 		promptSnippet: "save and recall durable state that survives context resets",
 		promptGuidelines: [
 			"Use notes for durable state too large for a new_context handoff",
@@ -903,10 +967,10 @@ export default function (pi: ExtensionAPI) {
 					if (offset && offset >= text.length) {
 						throw new Error(`Offset ${offset} is past the end of ${relative} (${text.length} chars).`);
 					}
-					const end = Math.min(text.length, offset + requirePage(nativeContext(ctx), offset, 0, activeToolTokens()));
+					const end = Math.min(text.length, offset + pageSize(nativeContext(ctx), offset, 0));
 					const more =
 						end < text.length ? `\n[chars ${offset}-${end} of ${text.length}; continue with offset ${end}]` : "";
-					return textResult(`${text.slice(offset, end)}${more}`, [], { kind: "note-read", offset, end, total: text.length });
+					return pageResult(`${text.slice(offset, end)}${more}`, [], { kind: "note-read", offset, end, total: text.length });
 				}
 				case "write": {
 					const relative = requireValue(params.path, "path", params.op);
@@ -914,7 +978,7 @@ export default function (pi: ExtensionAPI) {
 					const path = safeJoin(relative);
 					mkdirSync(dirname(path), { recursive: true });
 					writeFileSync(path, params.content);
-					return textResult(`Wrote .pi/notes/${relative}`, [], { kind: "note-write" });
+					return textResult(`Wrote ${path}`, [], { kind: "note-write" });
 				}
 				case "append": {
 					const relative = requireValue(params.path, "path", params.op);
@@ -927,7 +991,7 @@ export default function (pi: ExtensionAPI) {
 					const existing = existsSync(path) ? readFileSync(path, "utf8") : "";
 					const separator = existing && !existing.endsWith("\n") ? "\n" : "";
 					appendFileSync(path, `${separator}${content.replace(/\n?$/, "\n")}`);
-					return textResult(`Appended to .pi/notes/${relative}`, [], { kind: "note-append" });
+					return textResult(`Appended to ${path}`, [], { kind: "note-append" });
 				}
 				case "search": {
 					const query = requireValue(params.query, "query", params.op).toLowerCase();
@@ -1019,12 +1083,12 @@ export default function (pi: ExtensionAPI) {
 				}
 				const end = Math.min(
 					item.text.length,
-					offset + requirePage(nativeContext(ctx), offset, offset === 0 ? item.images.length : 0, activeToolTokens()),
+					offset + pageSize(nativeContext(ctx), offset, offset === 0 ? item.images.length : 0),
 				);
 				const more = end < item.text.length ? `\nMore remains; call history read with id "${id}" and offset ${end}.` : "";
 				const header = `${source ? `${source} ` : ""}${item.entry.timestamp ?? ""} [window ${item.windowId}] [${id}] [chars ${offset}-${end} of ${item.text.length}] `;
 				// Stored images ride along with the first page only.
-				return textResult(
+				return pageResult(
 					`${header}${item.text.slice(offset, end)}${more}`,
 					offset === 0 ? item.images : [],
 					{ kind: "history-read", headerLength: header.length, offset, end, total: item.text.length },

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "pi-posthorse",
-	"version": "0.4.5",
+	"version": "0.4.6",
 	"type": "module",
 	"description": "Posthorse: fresh context, same journey. Native no-summary context windows for the fitchmultz/pi fork, with sparse budget reminders, handoffs, notes, and history.",
 	"keywords": ["pi-package", "pi-extension", "context", "posthorse", "rollover"],