pi-openai-codex-compat 0.0.14 → 0.0.15

package/CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## Unreleased
 
+## 0.0.15 - 2026-09-23
+
+### Fixed
+
+- Preserve earlier message omissions and replacements during overflow-recovery
+  compaction. Recover only the latest attempt Pi omitted for recovery, keeping
+  its committed progress without reviving unrelated content. Stored native
+  responses no longer override later assistant-message replacements.
+- Rebuild the compaction request's tool declarations when a tool's parameter
+  schema changes under an unchanged name and description, instead of reusing a
+  stale schema from the previous turn.
+
 ## 0.0.14 - 2026-09-22
 
 ### Changed

package/README.md

@@ -328,9 +328,21 @@ The extension handles native compaction for `openai-codex`. It follows the Codex
 1. Send normal Responses history followed by `{ "type": "compaction_trigger" }`.
 2. Validate the returned opaque `compaction` item.
 3. Retain approximately 64,000 tokens of recent user messages, plus every developer and
-   system message outside that budget.
+   system message outside that budget. Repeated prompt updates (for example several
+   `/reload` runs or prompt-section changes) each remain in place as their own inline
+   developer item; they accumulate across checkpoints and are never dropped, merged, or
+   deduplicated, because each one is a distinct instruction the model already saw.
 4. Persist the opaque checkpoint in the Pi session and replay it on later requests.
 
+Overflow and length recovery follow Pi 0.87's `context_edit` semantics. Every request
+reads Pi's session projection, so earlier omissions and replacements (an elided tool
+output, an omitted reply) apply exactly as they do for Pi's own adapters. Before an
+overflow-recovery compaction, Pi omits the latest failed or truncated attempt and its tool
+results; the checkpoint request projects only those entries as they were before that
+omission, so a truncated response and its tool results survive as committed progress while
+every unrelated earlier edit stays in force. A failed attempt contributes only the native
+output committed before the overflowing subrequest.
+
 Ordinary responses and compaction use the same extension-managed SSE/WebSocket transport. Before the first WebSocket turn for a session and model, the provider performs a best-effort v2 `generate: false` prewarm of the static instruction/tool prefix, then generates the dynamic conversation input from its continuation. With `responsesLite: false`, supported models use the ordinary Responses envelope and receive their own ordinary-prefix prewarm.
 
 Healthy session WebSockets remain available until the server closes them or Pi tears down the session. Retryable WebSocket failures before model-visible output receive up to five fresh-connection retries before the session switches to sticky SSE. Retryable SSE HTTP failures and dropped streams receive up to five same-request resampling attempts before model-visible output. Both transports use Codex-style exponential backoff with ±10% jitter and preserve the same prompt-cache, session, account, installation, and window identities. Server metadata is not considered model-visible output, so a routing-state-only response can still be retried safely. Transport failures after model-visible output fail closed rather than risk duplicate text or tool calls. Explicit retryable `response.failed`/`response.incomplete` protocol terminals instead return to the provider-owned sampling loop, which preserves completed output items as the next request's history.
@@ -576,6 +588,15 @@ mise run check
 npm run pack:dry
 ```
 
+`mise run check` runs the linters, formatters, type checks, and the offline
+test suite; `mise run test` runs only the offline suite. Both tasks bind
+`PI_PACKAGE_DIR` to `node_modules/@earendil-works/pi-coding-agent`, so tests
+that load Pi in-process read this repository's Pi 0.87.0 resources even when a
+global `PI_PACKAGE_DIR` points at another installation. Ordinary `pi` launches
+outside these tasks are unaffected. Offline tests also exercise
+`scripts/release.ts` with every child process mocked: they never run Git, npm,
+Mise, or provider calls.
+
 Run the credentialed Pi/Codex integration tests separately. They load the real
 extension into headless Pi sessions, use the real WebSocket service, and ask
 the model to report all prior history markers after each text and tool
@@ -593,6 +614,21 @@ prompt reload, native compaction, and persisted-session resume against Codex.
 Test credentials stay in memory and child-process environments. Test sessions
 and configuration are isolated from the user's agent directory.
 
+The packaged-CLI test selects its archive and executable as follows:
+
+- Without `PI_CODEX_PACKAGE_ARCHIVE`, it packs the current worktree into a
+  temporary directory. With `PI_CODEX_PACKAGE_ARCHIVE`, it loads exactly that
+  archive: the value resolves against the working directory, must be an
+  existing regular file that `tar` can list, and an empty, missing, directory,
+  or invalid value fails the test instead of falling back to a fresh pack.
+- `PI_CODEX_CLI_PATH` selects another Pi `cli.js`; the default is this
+  repository's Pi 0.87.0 dependency. The test asserts that the selected
+  executable reports version 0.87.0, the only Pi version the packaged CLI test
+  is run against. The supported range is `>=0.87.0 <0.88.0`.
+- Each Pi child process receives `PI_PACKAGE_DIR` bound to the selected
+  executable's package directory, so the runtime under test reads its own
+  metadata.
+
 The public package entrypoint is `extensions/index.ts`; implementation modules remain under
 `extensions/openai-codex-compat/`. The focused Pi AI serializer copy lives under
 `extensions/openai-codex-compat/vendor/pi-ai/`. The custom Codex provider transport and stream
@@ -606,10 +642,74 @@ search, and compaction continuation.
 1. Run `npm run release -- X.Y.Z` from a clean, synchronized `main`.
 2. The command builds the exact package locally and runs live CLI tests against that archive. It also runs the existing live SDK tests against checkout source. Both suites must pass before it records the archive's SHA-256 in an SSH-signed release commit, proves a clean rebuild is reproducible, and creates a lightweight tag. Missing credentials or failing live tests stop the release.
 3. Inspect the result, then push atomically with `git push --atomic origin main vX.Y.Z`.
-4. A read-only GitHub Actions job validates and packs the package. After approval in the tag-restricted `npm-publish` environment, a separate GitHub-owned job verifies the signature and signed digest before attesting and staging that exact archive through npm trusted publishing.
-5. A final job creates the immutable GitHub release for the tag from the same verified archive, its
-   checksum, and the version's `CHANGELOG.md` section (`Unreleased` for prereleases).
-6. Approve the staged package on npmjs.com, or with `npm stage approve <stage-id>`.
+4. A read-only GitHub Actions job validates and packs the package. After approval
+   in the tag-restricted `npm-publish` environment, a separate GitHub-owned job
+   verifies the signature and signed digest before attesting and staging that
+   exact archive through npm trusted publishing.
+5. A final job creates the immutable GitHub release from the verified archive,
+   its checksum, and the version's changelog section (`Unreleased` for prereleases).
+6. Approve the staged package on npmjs.com or with `npm stage approve <stage-id>`.
+
+The release command runs these steps in order:
+
+1. Refuse to start unless the branch is `main`, `git status --porcelain` is
+   empty, `CHANGELOG.md` has a section for the version, `HEAD` equals
+   `origin/main` after a fetch, and tag `vX.Y.Z` does not exist. Nothing has
+   changed when one of these checks fails.
+2. Write the version into `package.json` and `package-lock.json` and stage
+   only those two files.
+3. Check out the staged index into a temporary directory, run
+   `npm ci --ignore-scripts` and `npm pack` there, validate the package
+   identity, file list, and manifest, then run `mise run test:live:codex` in
+   the repository with `PI_CODEX_PACKAGE_ARCHIVE` set to that archive. This is
+   the exact archive release gate: the live CLI tests load that archive and
+   nothing else. A live process that cannot start, a nonzero exit, or a failed
+   package validation stops the release here.
+4. Create the SSH-signed `release: vX.Y.Z` commit with the archive's SHA-256
+   as its `Npm-Artifact-SHA256` trailer and verify the signature and trailer.
+5. Rebuild the package from the committed tree without repeating the live
+   tests and require the same digest.
+6. Create the lightweight tag and verify that it points at the release commit.
+
+### Recovering from a failed release command
+
+Do not run any blanket `git restore`, `git reset`, `git checkout --`, or
+`git clean`. Inspect first, then undo only what the failed command produced.
+
+If the command fails during the version update, version changes can remain
+in the worktree. Once the update and `git add` succeed, package, live-test,
+and signing failures leave those changes staged. No release commit or tag
+was created by that attempt. Inspect the state:
+
+```sh
+git status --short
+git diff -- package.json package-lock.json
+git diff --cached -- package.json package-lock.json
+```
+
+Undo only this attempt's version edits in the worktree and index. Preserve
+concurrent changes, including edits in those same files. Do not stage whole
+files containing unrelated edits. Fix the cause and confirm that `main` is
+clean and synchronized before rerunning the release command.
+
+If the command fails after the release commit (steps 4 to 6), a local signed
+`release: vX.Y.Z` commit exists on `main`, and a local tag may exist. Do not rerun
+the release command: it would refuse the unsynchronized `HEAD`, and a second
+run on top of the existing commit would produce a duplicate release commit.
+Do not push the commit or tag until signature verification, the reproducibility
+check, and tag verification are complete. Inspect the commit and the digest:
+
+```sh
+git log -1 --format='%H%n%s%n%(trailers:key=Npm-Artifact-SHA256,valueonly)' HEAD
+git diff origin/main..HEAD -- package.json package-lock.json
+git tag --list 'vX.Y.Z'
+```
+
+Removing the local release commit or tag changes local refs. Confirm that
+they were never pushed, record their hashes and a recovery path, and obtain
+explicit approval before changing them. Never replace a published tag.
+Fix the underlying cause before starting a new release from clean,
+synchronized `main`.
 
 Stable releases use `latest`; prereleases derive their npm dist-tag from the first prerelease identifier.
 

package/extensions/openai-codex-compat/codex-provider/codex-provider-contracts.ts

@@ -47,6 +47,8 @@ export type RuntimeScope = {
 export type RequestTemplate = {
   modelId: string;
   payload: JsonRecord;
+  /** Fingerprint of the raw tool definitions that produced `payload.tools`. */
+  toolFingerprint: string;
   grammarToolInputProperties: GrammarToolInputProperties;
   requestOptions: OpenAICodexResponsesOptions;
 };

package/extensions/openai-codex-compat/codex-provider/codex-provider-runtime.ts

@@ -32,6 +32,7 @@ import {
   resolveRequestTranscript,
   responsesCompatibility,
   searchCheckpoint,
+  toolDefinitionFingerprint,
   type CheckpointData,
   type CompactionDecision,
   type GrammarToolInputProperties,
@@ -871,6 +872,7 @@ export class CodexProviderRuntime {
         this.templates.set(runtimeSessionId, {
           modelId: model.id,
           payload: replacementValues ? {} : withoutConversationInput(body),
+          toolFingerprint: toolDefinitionFingerprint(getCurrentTools(context.messages)),
           grammarToolInputProperties,
           requestOptions: { ...requestOptions },
         });

package/extensions/openai-codex-compat/compaction-checkpoint.ts

@@ -1,10 +1,11 @@
 import { isBoolean, isString } from "./value-contracts.ts";
-import { randomUUID } from "node:crypto";
+import { createHash, randomUUID } from "node:crypto";
 import {
   buildSessionContext,
   buildSessionProjection,
   convertToLlm,
   sessionEntryToContextMessages,
+  type ContextEditEntry,
   type SessionEntry,
   type ToolInfo,
 } from "@earendil-works/pi-coding-agent";
@@ -31,6 +32,7 @@ import {
   type JsonRecord,
 } from "./codex-protocol.ts";
 import { nativeCommittedPrefixBeforeOverflow, nativeResponseOverrides } from "./native-history.ts";
+import { stableResponsesJson } from "./responses-replay.ts";
 import {
   CODEX_NAMESPACED_TOOL_NAMES,
   CODEX_TEXT_CONTENT_ITEM_TOOL_RESULT_NAMES,
@@ -181,9 +183,45 @@ export function activeResponsesTools(
     : undefined;
 }
 
+/** The raw tool definition fields that both Pi AI's `Tool` and Pi's `ToolInfo` carry. */
+export type ToolDeclarationSource = Pick<ToolInfo, "name" | "description" | "parameters">;
+
 /**
- * Whether a serialized Responses `tools` list declares exactly the active tools,
- * by name and description. Namespace members count as `namespace.member`.
+ * Fingerprint raw tool definitions by name, description, and parameter schema,
+ * independent of declaration order. Both the transcript tools a turn request
+ * declared and the registry's active `ToolInfo` list produce the same value for
+ * the same definitions, so a cached turn template can be compared with the
+ * current tools without reversing the strict schema conversion applied on the
+ * wire.
+ */
+export function toolDefinitionFingerprint(tools: readonly ToolDeclarationSource[]): string {
+  const declarations = tools
+    .map((tool) => {
+      // The JSON round-trip drops typebox symbol keys and undefined fields, as
+      // Pi AI's own declaration comparison does.
+      const parameters: unknown = JSON.parse(JSON.stringify(tool.parameters));
+      return { name: tool.name, description: tool.description, parameters };
+    })
+    .sort((left, right) => left.name.localeCompare(right.name));
+  return createHash("sha256").update(stableResponsesJson(declarations)).digest("hex");
+}
+
+/** Fingerprint of the registry tools that are currently active. */
+export function activeToolFingerprint(
+  allTools: readonly ToolInfo[],
+  activeNames: readonly string[],
+): string {
+  const enabled = new Set(activeNames);
+  return toolDefinitionFingerprint(allTools.filter((tool) => enabled.has(tool.name)));
+}
+
+/**
+ * Whether a cached turn template still declares exactly the active tools.
+ *
+ * The serialized `tools` list must name the active set (namespace members
+ * count as `namespace.member`), and the raw definitions that produced it must
+ * fingerprint identically to the active registry tools. Comparing names and
+ * descriptions alone missed parameter-schema changes under an unchanged name.
  *
  * A cached turn template carries the exact declarations the turn requests sent,
  * including strict schema conversion that `ToolInfo` cannot reproduce. Reusing
@@ -191,9 +229,11 @@ export function activeResponsesTools(
  */
 export function declaresActiveTools(
   serialized: readonly JsonRecord[],
+  cachedToolFingerprint: string,
   allTools: readonly ToolInfo[],
   activeNames: readonly string[],
 ): boolean {
+  if (cachedToolFingerprint !== activeToolFingerprint(allTools, activeNames)) return false;
   const declared = new Map<string, string | undefined>();
   for (const item of serialized) {
     if (item.type === "namespace" && isString(item.name) && Array.isArray(item.tools)) {
@@ -415,18 +455,69 @@ export function checkpointData(
   return checkpoint;
 }
 
+/**
+ * The entries Pi 0.87 omits for overflow or length recovery: the latest
+ * assistant attempt and the tool results of that turn, each targeted by a
+ * `context_edit` omission appended after the attempt. `leafId` is the parent of
+ * the first such omission, so a projection built up to it reflects every edit
+ * that existed before Pi omitted the attempt.
+ *
+ * Pi only selects a recovery attempt while it is still projected, so an
+ * omission of the latest assistant found here is Pi's recovery omission and
+ * not an unrelated edit.
+ */
+function recoveryOmissions(
+  branch: readonly SessionEntry[],
+): { entryIds: ReadonlySet<string>; leafId: string } | undefined {
+  const attemptIndex = branch.findLastIndex(
+    (entry) => entry.type === "message" && entry.message.role === "assistant",
+  );
+  const attempt = branch[attemptIndex];
+  if (attempt?.type !== "message" || attempt.message.role !== "assistant") return undefined;
+
+  const toolCallIds = new Set(
+    attempt.message.content.flatMap((block) => (block.type === "toolCall" ? [block.id] : [])),
+  );
+  const candidates = new Set([attempt.id]);
+  for (const entry of branch.slice(attemptIndex + 1)) {
+    if (entry.type !== "message") continue;
+    if (entry.message.role !== "toolResult" || !toolCallIds.has(entry.message.toolCallId)) break;
+    candidates.add(entry.id);
+  }
+
+  const latestEdits = new Map<string, ContextEditEntry>();
+  for (const entry of branch.slice(attemptIndex + 1)) {
+    if (entry.type === "context_edit" && candidates.has(entry.targetId)) {
+      latestEdits.set(entry.targetId, entry);
+    }
+  }
+  if (latestEdits.get(attempt.id)?.replacement !== null) return undefined;
+  const omissions = new Set(
+    [...latestEdits.values()].filter((entry) => entry.replacement === null),
+  );
+  const first = branch.find((entry) => entry.type === "context_edit" && omissions.has(entry));
+  if (first?.type !== "context_edit" || first.parentId === null) return undefined;
+  return {
+    entryIds: new Set([...omissions].map((entry) => entry.targetId)),
+    leafId: first.parentId,
+  };
+}
+
 /**
  * Materialize provider history for the active branch. Checkpoint history
  * replaces everything before its entry; later branch entries form the tail.
  *
- * Ordinary requests read the session projection, so `context_edit` omissions
- * and replacements apply exactly as they do for Pi's own adapters. Recovery
- * compaction reads the raw entries instead: Pi 0.87 omits the failed or
- * truncated attempt through a `context_edit` child of that entry before it
- * runs `session_before_compact`. The serializer already skips error and
- * aborted assistants, and a truncated boundary is committed progress that the
- * checkpoint must keep. Entries are never removed from the branch copy because
- * the omission entry's parent chain would break and the history would vanish.
+ * Every request reads the session projection, so `context_edit` omissions and
+ * replacements apply exactly as they do for Pi's own adapters. Recovery
+ * compaction differs only for the latest attempt: Pi 0.87 omits the failed or
+ * truncated assistant and its tool results through `context_edit` entries
+ * before it runs `session_before_compact`. Those entries are projected as they
+ * were before Pi's omission, so a truncated boundary stays as committed
+ * progress while every earlier omission or replacement remains in force. The
+ * serializer skips error and aborted assistants; a failed attempt contributes
+ * only the native prefix committed before the overflowing subrequest. Entries
+ * are never removed from the branch copy because the omission entry's parent
+ * chain would break and the history would vanish.
  */
 export function providerHistory(options: {
   branch: readonly SessionEntry[];
@@ -459,13 +550,30 @@ export function providerHistory(options: {
   const projection = buildSessionProjection(branch);
   const projected = new Map<string, AgentMessages>();
   for (const entry of projection.entries) projected.set(entry.sourceEntry.id, entry.messages);
-  const entryMessages = (entry: SessionEntry): AgentMessages =>
-    options.recoverLatestOverflowPrefix
-      ? sessionEntryToContextMessages(entry)
-      : (projected.get(entry.id) ?? []);
+  const recovery = options.recoverLatestOverflowPrefix ? recoveryOmissions(branch) : undefined;
+  if (recovery) {
+    for (const entry of buildSessionProjection(branch, recovery.leafId).entries) {
+      if (recovery.entryIds.has(entry.sourceEntry.id)) {
+        projected.set(entry.sourceEntry.id, entry.messages);
+      }
+    }
+  }
+  const entryMessages = (entry: SessionEntry): AgentMessages => projected.get(entry.id) ?? [];
 
   const checkpoint = searchCheckpoint(branch);
-  const nativeAssistantItems = nativeResponseOverrides(branch, options.wireModel.id);
+  const nativeAssistantItems = new Map(nativeResponseOverrides(branch, options.wireModel.id));
+  for (const entry of branch) {
+    if (
+      entry.type === "message" &&
+      entry.message.role === "assistant" &&
+      entry.message.responseId &&
+      entryMessages(entry).some((message) => message !== entry.message)
+    ) {
+      // A replacement keeps the response ID but changes the model-visible content.
+      // Its original native output must not override the canonical replacement.
+      nativeAssistantItems.delete(entry.message.responseId);
+    }
+  }
   if (checkpoint.kind === "corrupt") {
     throw new Error("The latest Codex compaction checkpoint is corrupt.");
   }

package/extensions/openai-codex-compat/remote-compaction.ts

@@ -258,15 +258,18 @@ export default function registerRemoteCompaction(
       });
       // Reuse the last turn's exact declarations when they still describe the
       // active tools, so the compaction request shares the turn requests'
-      // prompt-cache prefix. Rebuild from the registry only when the active
-      // set changed since that turn or no turn template exists.
+      // prompt-cache prefix. Rebuild from the registry when the active set or
+      // any definition (including a parameter schema under an unchanged name)
+      // changed since that turn, or when no turn template exists.
       const template: JsonRecord = { ...matching?.payload };
       const activeNames = pi.getActiveTools();
       const cachedTools = Array.isArray(matching?.payload.tools)
         ? matching.payload.tools.filter(isObject)
         : undefined;
       template.tools =
-        cachedTools && declaresActiveTools(cachedTools, allTools, activeNames)
+        cachedTools &&
+        matching &&
+        declaresActiveTools(cachedTools, matching.toolFingerprint, allTools, activeNames)
           ? cachedTools
           : (activeResponsesTools(allTools, activeNames, grammarToolInputProperties) ?? []);
       const requestOptions: OpenAICodexResponsesOptions = {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-openai-codex-compat",
-  "version": "0.0.14",
+  "version": "0.0.15",
   "description": "OpenAI Codex compatibility for Pi with native compaction, fast mode, and Codex-optimized capabilities",
   "keywords": [
     "pi-package"