@narumitw/pi-codex-compact 0.47.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Narumi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,108 @@
+# @narumitw/pi-codex-compact
+
+> [!WARNING]
+> This extension is experimental. It depends on an undocumented OpenAI Codex Responses wire
+> contract and stores provider-specific opaque checkpoints. Keep backups of important sessions.
+
+Codex Remote Compaction V2 for Pi's built-in `openai-codex` OAuth provider. It replaces Pi's
+plaintext summary-generation call with a server-generated opaque compaction item, then replays
+that item in later OpenAI Codex Responses requests.
+
+Pi still owns compaction thresholds, `/compact`, overflow retries, retained-message selection, and
+the append-only session tree. This package does not reproduce Codex core's context-window lineage
+or exact pre-turn and mid-turn lifecycle.
+
+## Install
+
+```bash
+pi install npm:@narumitw/pi-codex-compact
+```
+
+From this repository:
+
+```bash
+just try codex-compact
+```
+
+Loading the package enables it. Use Pi normally, including its built-in `/compact` command. Open
+the extension's TUI control menu with:
+
+```text
+/codex-compact
+```
+
+The menu shows the active model and whether compaction will use Codex Remote V2 or Pi's native
+fallback. Choose **Compact now** to close the menu and compact immediately, or **Settings** to edit
+user settings in `~/.pi/agent/pi-codex-compact.json` (or the active Pi agent directory). Invalid
+files are never overwritten; manual compaction remains available, while Settings opens read-only
+repair guidance until the file is fixed and `/reload` is run.
+
+| Setting | Default | Accepted values |
+| --- | ---: | --- |
+| `enabled` | `true` | Boolean |
+| `requestTimeoutMs` | `300000` | Integer from 30,000 to 600,000 ms |
+| `maxRetries` | `2` | Integer from 0 to 2 |
+| `replacementTokenBudget` | `64000` | Integer from 8,000 to 128,000 tokens |
+| `notifyOnFallback` | `true` | Boolean |
+
+Missing fields use built-in defaults; the file has no environment-variable or project-level
+override. Settings reload on every `session_start`, including `/reload`, resume, and fork. Menu
+writes apply immediately, preserve unknown JSON fields, serialize within the current Pi process, and
+use a conflict check plus atomic rename. Separate Pi processes are not coordinated by a shared lock;
+a detected concurrent edit is rejected for the user to reopen and retry.
+
+## Requirements
+
+- Pi `0.83.0`-compatible extension APIs.
+- A model using provider `openai-codex` and API `openai-codex-responses`.
+- A working OpenAI Codex OAuth login in Pi.
+
+OpenAI API-key, Azure, Copilot, proxies, and generic Responses-compatible providers are not
+supported. Unsupported models continue to use Pi's native compaction.
+
+## How it works
+
+1. Pi decides when compaction is needed.
+2. The extension sends the current Codex Responses input with a final `compaction_trigger`.
+3. It validates and persists the server's encrypted `compaction` item in the Pi
+   `CompactionEntry.details` field.
+4. Later Codex requests replace Pi's fallback summary and kept suffix with the opaque replacement
+   history before dispatch.
+
+The SSE response is limited to 8 MiB, the opaque item to 2 MiB, and the persisted replacement
+history to the configured text budget (64,000 tokens by default) and 8 MiB. Oversized media
+retention is dropped rather than creating an unbounded session entry.
+
+## Failure and portability
+
+Remote auth, transport, protocol, or validation failures fall back to Pi's native plaintext
+compaction. User cancellation does not start that fallback.
+
+After a successful remote compaction, full older history can be replayed only when this extension
+is loaded with the same `openai-codex` model. If the extension is removed or the model/provider is
+changed, Pi exposes a warning summary plus its retained recent messages. Switching back restores
+opaque replay. Repeated compaction after an opaque checkpoint also requires the checkpoint to be
+projected safely; otherwise Pi falls back rather than guessing.
+
+## Privacy and storage
+
+Remote compaction sends the active conversation context, system prompt, and active tool schemas to
+the same OpenAI Codex backend used by the selected model. The extension stores the encrypted
+compaction item and bounded recent user-role Responses items in the Pi session. It never persists
+the OAuth token or request headers.
+
+## Development
+
+```bash
+npm --workspace @narumitw/pi-codex-compact run check
+npm test
+just pack codex-compact
+```
+
+See
+[`docs/implementation-notes/codex-compaction-mechanism.md`](../../docs/implementation-notes/codex-compaction-mechanism.md)
+for the underlying Codex mechanism research and the extension boundary.
+
+## License
+
+MIT

package/package.json

@@ -0,0 +1,52 @@
+{
+	"name": "@narumitw/pi-codex-compact",
+	"version": "0.47.0",
+	"description": "Experimental Codex Remote Compaction V2 for Pi's OpenAI Codex provider.",
+	"type": "module",
+	"license": "MIT",
+	"private": false,
+	"keywords": [
+		"pi-package",
+		"pi-extension",
+		"pi",
+		"codex",
+		"compaction",
+		"openai"
+	],
+	"files": [
+		"src",
+		"README.md",
+		"LICENSE"
+	],
+	"pi": {
+		"extensions": [
+			"./src/index.ts"
+		]
+	},
+	"scripts": {
+		"check": "biome check --vcs-use-ignore-file=false src test package.json tsconfig.json README.md && npm run typecheck",
+		"format": "biome check --write --vcs-use-ignore-file=false src test package.json tsconfig.json README.md",
+		"typecheck": "tsc --noEmit"
+	},
+	"dependencies": {
+		"@narumitw/pi-tui-kit": "^0.45.0"
+	},
+	"peerDependencies": {
+		"@earendil-works/pi-agent-core": "*",
+		"@earendil-works/pi-ai": "*",
+		"@earendil-works/pi-coding-agent": "*"
+	},
+	"devDependencies": {
+		"@biomejs/biome": "2.5.6",
+		"@earendil-works/pi-agent-core": "0.83.0",
+		"@earendil-works/pi-ai": "0.83.0",
+		"@earendil-works/pi-coding-agent": "0.83.0",
+		"@types/node": "26.1.2",
+		"typescript": "7.0.2"
+	},
+	"repository": {
+		"type": "git",
+		"url": "https://github.com/narumiruna/pi-extensions",
+		"directory": "experimental/pi-codex-compact"
+	}
+}

package/src/checkpoint.ts

@@ -0,0 +1,257 @@
+import { createHash, randomUUID } from "node:crypto";
+import type { AgentMessage } from "@earendil-works/pi-agent-core";
+import type { CompactionEntry, SessionEntry } from "@earendil-works/pi-coding-agent";
+import { type JsonObject, validateCompactionItem } from "./protocol.js";
+
+export const CHECKPOINT_KIND = "pi-codex-remote-compaction";
+export const CHECKPOINT_VERSION = 1;
+export const REPLACEMENT_TOKEN_BUDGET = 64_000;
+export const REPLACEMENT_BYTE_BUDGET = 8 * 1024 * 1024;
+const MAX_MEDIA_ITEM_BYTES = 2 * 1024 * 1024;
+
+export interface CodexCheckpointDetails {
+	kind: typeof CHECKPOINT_KIND;
+	version: typeof CHECKPOINT_VERSION;
+	checkpointId: string;
+	provider: "openai-codex";
+	api: "openai-codex-responses";
+	modelId: string;
+	protocol: "remote-compaction-v2";
+	replacementHistory: JsonObject[];
+	keptMessageFingerprints: string[];
+	createdAt: string;
+}
+
+function isObject(value: unknown): value is JsonObject {
+	return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+
+function stableValue(value: unknown): unknown {
+	if (Array.isArray(value)) return value.map(stableValue);
+	if (!isObject(value)) return value;
+	return Object.fromEntries(
+		Object.entries(value)
+			.sort(([left], [right]) => left.localeCompare(right))
+			.map(([key, child]) => [key, stableValue(child)]),
+	);
+}
+
+function serializedBytes(value: unknown): number {
+	return Buffer.byteLength(JSON.stringify(value), "utf8");
+}
+
+export function fingerprintMessage(message: AgentMessage): string {
+	return createHash("sha256")
+		.update(JSON.stringify(stableValue(message)))
+		.digest("hex");
+}
+
+export function checkpointMarker(checkpointId: string): string {
+	return [
+		`[PI_CODEX_REMOTE_CHECKPOINT:${checkpointId}]`,
+		"Opaque checkpoint injection failed. Do not infer missing history; tell the user to re-enable",
+		"@narumitw/pi-codex-compact with an openai-codex model.",
+	].join(" ");
+}
+
+export function fallbackSummary(checkpointId: string): string {
+	return [
+		`OpenAI Codex Remote Compaction V2 checkpoint ${checkpointId} stores the older history opaquely.`,
+		"Full replay requires @narumitw/pi-codex-compact and an openai-codex model.",
+		"Without them, only Pi's retained recent messages remain available.",
+	].join(" ");
+}
+
+function markerMessage(checkpointId: string, timestamp: number): AgentMessage {
+	return {
+		role: "user",
+		content: [{ type: "text", text: checkpointMarker(checkpointId) }],
+		timestamp,
+	};
+}
+
+export function parseCheckpointDetails(value: unknown): CodexCheckpointDetails | undefined {
+	if (!isObject(value)) return undefined;
+	if (
+		value.kind !== CHECKPOINT_KIND ||
+		value.version !== CHECKPOINT_VERSION ||
+		typeof value.checkpointId !== "string" ||
+		value.checkpointId.length < 8 ||
+		value.provider !== "openai-codex" ||
+		value.api !== "openai-codex-responses" ||
+		typeof value.modelId !== "string" ||
+		value.protocol !== "remote-compaction-v2" ||
+		!Array.isArray(value.replacementHistory) ||
+		!Array.isArray(value.keptMessageFingerprints) ||
+		typeof value.createdAt !== "string"
+	) {
+		return undefined;
+	}
+	if (
+		value.replacementHistory.length === 0 ||
+		!value.replacementHistory.every(isObject) ||
+		!value.keptMessageFingerprints.every(
+			(fingerprint) => typeof fingerprint === "string" && /^[a-f0-9]{64}$/.test(fingerprint),
+		) ||
+		serializedBytes(value.replacementHistory) > REPLACEMENT_BYTE_BUDGET
+	) {
+		return undefined;
+	}
+	const last = value.replacementHistory.at(-1);
+	try {
+		validateCompactionItem(last);
+	} catch {
+		return undefined;
+	}
+	return structuredClone(value) as unknown as CodexCheckpointDetails;
+}
+
+export function latestCheckpoint(entries: readonly SessionEntry[]):
+	| {
+			entry: CompactionEntry<CodexCheckpointDetails>;
+			details: CodexCheckpointDetails;
+	  }
+	| undefined {
+	for (let index = entries.length - 1; index >= 0; index--) {
+		const entry = entries[index];
+		if (entry.type !== "compaction") continue;
+		const details = parseCheckpointDetails(entry.details);
+		return details
+			? { entry: entry as CompactionEntry<CodexCheckpointDetails>, details }
+			: undefined;
+	}
+	return undefined;
+}
+
+export function projectCheckpointContext(
+	messages: readonly AgentMessage[],
+	details: CodexCheckpointDetails,
+): AgentMessage[] | undefined {
+	const summary = fallbackSummary(details.checkpointId);
+	const summaryIndex = messages.findIndex(
+		(message) => message.role === "compactionSummary" && message.summary === summary,
+	);
+	if (summaryIndex < 0) return undefined;
+	const keptStart = summaryIndex + 1;
+	const keptEnd = keptStart + details.keptMessageFingerprints.length;
+	if (keptEnd > messages.length) return undefined;
+	for (let index = keptStart; index < keptEnd; index++) {
+		if (
+			fingerprintMessage(messages[index]) !== details.keptMessageFingerprints[index - keptStart]
+		) {
+			return undefined;
+		}
+	}
+	const timestamp = messages[summaryIndex].timestamp;
+	return [
+		...messages.slice(0, summaryIndex),
+		markerMessage(details.checkpointId, timestamp),
+		...messages.slice(keptEnd),
+	];
+}
+
+function rawText(item: JsonObject): string {
+	if (!Array.isArray(item.content)) return "";
+	return item.content
+		.flatMap((part) =>
+			isObject(part) && typeof part.text === "string" && part.type === "input_text"
+				? [part.text]
+				: [],
+		)
+		.join("\n");
+}
+
+function hasMedia(item: JsonObject): boolean {
+	return (
+		Array.isArray(item.content) &&
+		item.content.some((part) => isObject(part) && part.type === "input_image")
+	);
+}
+
+function truncateTextItem(item: JsonObject, maxChars: number): JsonObject | undefined {
+	if (!Array.isArray(item.content) || maxChars <= 32) return undefined;
+	let remaining = maxChars - 16;
+	const content = [...item.content].reverse().flatMap((part) => {
+		if (
+			!isObject(part) ||
+			part.type !== "input_text" ||
+			typeof part.text !== "string" ||
+			remaining <= 0
+		) {
+			return [];
+		}
+		const text = part.text.slice(-remaining);
+		remaining -= text.length;
+		return [{ ...part, text: `[truncated]\n${text}` }];
+	});
+	if (content.length === 0) return undefined;
+	return { ...item, content: content.reverse() };
+}
+
+export function buildReplacementHistory(
+	input: readonly unknown[],
+	compactionItem: JsonObject,
+	options: { tokenBudget?: number; byteBudget?: number } = {},
+): JsonObject[] {
+	const tokenBudget = options.tokenBudget ?? REPLACEMENT_TOKEN_BUDGET;
+	const byteBudget = options.byteBudget ?? REPLACEMENT_BYTE_BUDGET;
+	const opaque = validateCompactionItem(compactionItem);
+	let remainingBytes = byteBudget - serializedBytes(opaque);
+	let remainingChars = tokenBudget * 4;
+	if (remainingBytes <= 0)
+		throw new Error("Opaque compaction item exceeds replacement history budget");
+	const retainedNewestFirst: JsonObject[] = [];
+	const candidates = input.filter(
+		(item): item is JsonObject =>
+			isObject(item) && item.role === "user" && item.type !== "compaction_trigger",
+	);
+	for (let index = candidates.length - 1; index >= 0; index--) {
+		const candidate = candidates[index];
+		const bytes = serializedBytes(candidate);
+		if (hasMedia(candidate) && bytes > MAX_MEDIA_ITEM_BYTES) continue;
+		const text = rawText(candidate);
+		let retained = candidate;
+		if (text.length > remainingChars) {
+			if (hasMedia(candidate)) continue;
+			const truncated = truncateTextItem(candidate, remainingChars);
+			if (!truncated) continue;
+			retained = truncated;
+		}
+		if (serializedBytes(retained) > remainingBytes) {
+			if (hasMedia(retained)) continue;
+			const maxCharsByBytes = Math.max(0, remainingBytes - 128);
+			const truncated = truncateTextItem(retained, Math.min(remainingChars, maxCharsByBytes));
+			if (!truncated || serializedBytes(truncated) > remainingBytes) continue;
+			retained = truncated;
+		}
+		retainedNewestFirst.push(structuredClone(retained));
+		remainingBytes -= serializedBytes(retained);
+		remainingChars -= Math.min(remainingChars, rawText(retained).length);
+		if (remainingBytes <= 128 || remainingChars <= 32) break;
+	}
+	return [...retainedNewestFirst.reverse(), opaque];
+}
+
+export function createCheckpointDetails(input: {
+	modelId: string;
+	replacementHistory: JsonObject[];
+	keptMessages: readonly AgentMessage[];
+	checkpointId?: string;
+	createdAt?: string;
+}): CodexCheckpointDetails {
+	const details: CodexCheckpointDetails = {
+		kind: CHECKPOINT_KIND,
+		version: CHECKPOINT_VERSION,
+		checkpointId: input.checkpointId ?? randomUUID(),
+		provider: "openai-codex",
+		api: "openai-codex-responses",
+		modelId: input.modelId,
+		protocol: "remote-compaction-v2",
+		replacementHistory: structuredClone(input.replacementHistory),
+		keptMessageFingerprints: input.keptMessages.map(fingerprintMessage),
+		createdAt: input.createdAt ?? new Date().toISOString(),
+	};
+	const parsed = parseCheckpointDetails(details);
+	if (!parsed) throw new Error("Created an invalid Codex checkpoint");
+	return parsed;
+}