pi-openai-codex-compat 0.0.6 → 0.0.7

package/APPLY_PATCH_INSTRUCTION_FEEDBACK.md

@@ -0,0 +1,617 @@
+# `apply_patch` Instruction Feedback
+
+## Status
+
+This document is the standalone implementation reference for model and TUI
+feedback produced by `apply_patch`. It defines the agreed result structure,
+terminology, rendering, failure inspection, and test requirements.
+
+`APPLY_PATCH_SEMANTIC_OPERATIONS.md` remains authoritative for filesystem and
+matching semantics. This document is authoritative for presenting those
+semantics to the model and user.
+
+## Goals
+
+Feedback must:
+
+1. retain the aggregate changed-file summary;
+2. when an instruction ledger is needed, report every instruction without a
+   model-facing limit;
+3. attribute filesystem effects, errors, and matching evidence to the
+   instruction that produced them;
+4. report failed and not-run instructions;
+5. use concise, simplified technical English;
+6. report deterministic facts rather than speculative outcomes; and
+7. avoid repeating old or replacement text supplied in the patch.
+
+## Aggregate summary
+
+Successful results retain the current summary:
+
+```text
+Success. Updated the following files:
+A added.txt
+M updated.txt
+D deleted.txt
+```
+
+When every instruction has no filesystem effect:
+
+```text
+Success. No files were changed.
+```
+
+Failed results identify the failed instruction and list confirmed changed
+paths:
+
+```text
+Patch failed at instruction 3 of 5.
+Files changed:
+M updated.txt
+A destination.txt
+```
+
+The changed-file list includes completed effects from failed instructions. A
+path whose final state is not verified is not listed as changed unless another
+completed, verified effect on that path is available.
+
+When a failed instruction creates only a directory or leaves only a temporary
+entry, use:
+
+```text
+Filesystem changed.
+```
+
+The directory or temporary-entry path remains on its instruction instead of
+being mislabelled as an added file.
+
+`No files were changed.` is valid only when every relevant path was verified
+unchanged. When no changed path is confirmed and at least one relevant final
+state is not verified, omit the global changed/unchanged statement and report
+the unverified state on its instruction.
+
+Keep the existing exit-code and wall-time envelope where the tool currently
+uses it.
+
+## Patch instruction results
+
+Append an instruction ledger when at least one instruction is not `APPLIED` or
+an applied instruction has a feedback clause. Omit the ledger when every
+instruction is `APPLIED` without feedback.
+
+When included, the aggregate summary is followed by this exact heading:
+
+```text
+Patch instruction results:
+```
+
+Every instruction is then listed in source order, including ordinary applied
+instructions without feedback. Model feedback has no instruction limit and no
+omitted-count message.
+
+Visible statuses are:
+
+- `APPLIED`
+- `NO CHANGE`
+- `SKIPPED`
+- `FAILED`
+- `NOT RUN`
+
+The TUI may additionally use `PLANNED` while an `apply_patch` call is still in
+progress. Pi renders this read-only preview after the arguments are complete
+and before execution begins. `PLANNED` is not sent to the model and does not
+appear in completed tool results.
+
+Example:
+
+```text
+Patch instruction results:
+1. [APPLIED] Update a.txt
+2. [NO CHANGE] Delete missing.txt - Path already absent.
+3. [FAILED] Update & Move source.txt -> destination.txt - Created destination.txt; source.txt remains; The updated content was written to the destination, but removing the source failed: permission denied; The file at destination.txt contains the requested content byte-for-byte despite the reported error.
+4. [NOT RUN] Update other.txt - Instruction 3 failed.
+```
+
+Model-facing separators and move arrows are ASCII. The TUI may use styled
+status labels and Unicode arrows.
+
+## Concision rules
+
+When the ledger is included because another instruction needs explanation, an
+ordinary applied instruction needs only its status and operation:
+
+```text
+1. [APPLIED] Add a.txt
+2. [APPLIED] Update b.txt
+3. [APPLIED] Update & Move c.txt -> d.txt
+4. [APPLIED] Move e.txt -> f.txt
+5. [APPLIED] Delete g.txt
+```
+
+Add a short clause only for:
+
+- a no-change or skipped reason;
+- a failure;
+- a non-obvious externally visible filesystem effect;
+- a partial effect;
+- a final state that is not verified; or
+- concise matcher evidence.
+
+Do not include:
+
+- old or replacement patch blocks;
+- a diff-availability warning;
+- verbose proof sections;
+- a duplicated raw error section;
+- a detached matcher section; or
+- prose that merely restates an ordinary successful operation.
+
+## Terminology
+
+Do not expose these terms in model or TUI feedback:
+
+- `Committed prefix`
+- `exact`
+- `inexact`
+- `dead`
+- `dominated`
+- `no-op`
+- `preflight`
+- `installed`
+
+Prefer:
+
+- `created`
+- `replaced`
+- `updated`
+- `deleted`
+- `removed`
+- `moved`
+- `present`
+- `absent`
+- `remains`
+- `not verified`
+- `validation`
+- `patch format`
+- `symlink`
+
+Generated explanatory text must not speculate with words such as `may`,
+`might`, `possibly`, `probably`, or `likely`. When inspection cannot determine
+a path's state, say:
+
+```text
+Final state not verified for a.txt.
+```
+
+## No-change results
+
+Render semantic no-ops as `NO CHANGE`:
+
+```text
+1. [NO CHANGE] Update a.txt - The instruction contains no changes.
+2. [NO CHANGE] Update b.txt - Old and replacement content are identical.
+3. [NO CHANGE] Update c.txt - Applying the update would not change the file.
+4. [NO CHANGE] Add d.txt - The file already contains the requested content byte-for-byte.
+5. [NO CHANGE] Delete e.txt - Path already absent.
+6. [NO CHANGE] Move f.txt -> f.txt - Source and destination identify the same entry.
+7. [NO CHANGE] Move a.txt -> b.txt - Instruction 2 already moved this entry.
+```
+
+The fulfilled-move result records and references the earlier instruction that
+performed the move. A missing related instruction number is an internal
+invariant failure; there is no generic earlier-instruction fallback.
+
+## Skipped results
+
+Render a semantically eliminated operation as `SKIPPED`, not `DEAD`.
+
+```text
+2. [SKIPPED] Update a.txt - Instruction 4 deletes a.txt before another instruction reads it.
+```
+
+For a shared hard-linked file:
+
+```text
+2. [SKIPPED] Update a.txt - Instructions 4 and 6 delete every link before another instruction reads the file.
+```
+
+The concise reason and related instruction numbers remain on the skipped
+instruction. Do not create a separate proof section.
+
+## Per-instruction filesystem effects
+
+Each instruction result stores:
+
+- its status;
+- an optional concise reason;
+- completed externally visible filesystem effects;
+- relevant final path states;
+- an optional error;
+- optional matcher evidence; and
+- related instruction numbers.
+
+The operation label is sufficient for ordinary successful effects. Explicit
+effects are required for:
+
+- a partial effect from a failed instruction;
+- destination replacement;
+- symlink entry versus target behavior, including the raw target pathname
+  stored in the symlink;
+- a parent directory that remains after failure;
+- deletion of a destination before replacement failure;
+- a temporary entry remaining after cleanup failure;
+- a source remaining after destination creation or replacement; and
+- a final state that is not verified.
+
+Effects from a failed instruction remain attached to that instruction. They
+must not be called earlier changes.
+
+Use only `symlink` terminology. Symlink feedback records the raw target
+pathname returned by `readlink`; it does not substitute a resolved absolute
+path.
+
+Every replacement effect records both the verified previous entry type and the
+verified resulting entry type. A resulting symlink also records its raw target
+pathname. Replacement feedback must not omit either entry type. If
+neither execution nor post-failure inspection can verify the result, report
+the final state as not verified instead of claiming a replacement.
+
+Examples:
+
+```text
+1. [APPLIED] Add alias.txt - alias.txt, previously a symlink to target.txt, is now a regular file.
+```
+
+```text
+1. [APPLIED] Update alias.txt - Modified file content through the symlink at alias.txt (target: target.txt); the symlink was not modified.
+```
+
+```text
+1. [APPLIED] Move alias.txt -> moved/alias.txt - Moved the symlink alias.txt; moved/alias.txt is now a symlink to target.txt.
+```
+
+Deletion and move differ only because deletion removes the symlink while a
+move preserves it and its stored target pathname. Feedback reports the
+resulting path directly rather than making an unconditional claim about the
+filesystem path named by the target.
+
+Failure examples:
+
+```text
+3. [FAILED] Update & Move a.txt -> b.txt - Created b.txt; a.txt remains; The updated content was written to the destination, but removing the source failed: permission denied; The file at b.txt contains the requested content byte-for-byte despite the reported error.
+```
+
+```text
+3. [FAILED] Move a.txt -> b.txt - b.txt is still a regular file; a.txt remains; Move failed: permission denied.
+```
+
+```text
+3. [FAILED] Move a.txt -> b.txt - Deleted the previous b.txt; b.txt is absent; replacement failed: permission denied.
+```
+
+```text
+3. [FAILED] Move a.txt -> dir/b.txt - Created directory dir; move failed: permission denied.
+```
+
+## Post-failure inspection
+
+After a low-level filesystem operation fails, inspect every relevant source,
+destination, parent, followed target, and temporary path before constructing
+feedback.
+
+Path-state feedback uses only deterministic states:
+
+- present as a regular file;
+- present as a symlink;
+- absent;
+- unchanged;
+- contains the requested content byte-for-byte;
+- differs from the requested content;
+- differs from the previously observed content;
+- differs from both the requested and previously observed content;
+- is a different filesystem entry;
+- entry type changed; or
+- not verified.
+
+Examples:
+
+```text
+3. [FAILED] Update a.txt - Write failed: permission denied; a.txt is unchanged.
+```
+
+```text
+3. [FAILED] Update a.txt - Write reported an error; The file at a.txt contains the requested content byte-for-byte despite the reported error.
+```
+
+```text
+3. [FAILED] Update a.txt - Write failed: input/output error; final state not verified for a.txt.
+```
+
+An empty recorded-change list is not proof that no file changed.
+
+## Unreadable previous content
+
+Replacing a file without reading its previous content produces ordinary
+success feedback:
+
+```text
+Success. Updated the following files:
+A file.txt
+
+Patch instruction results:
+1. [APPLIED] Add file.txt
+```
+
+Do not tell the model that previous content was unreadable, a diff is
+unavailable, history is incomplete, or the result is inexact. Diff
+availability is a TUI/history concern, not an operation result.
+
+## Matcher feedback
+
+Matcher feedback belongs to the failed update instruction. Do not emit a
+detached `Matcher diagnostics:` section or repeat old/replacement patch text.
+
+Examples:
+
+```text
+3. [FAILED] Update file.ts - Old content was not found. Read the current file and use apply_patch again with updated instructions if needed.
+```
+
+```text
+3. [FAILED] Update file.ts - The requested changes match in reverse source-file order at lines 10-12 and lines 30-32. Use apply_patch again with the requested changes in source-file order if needed.
+```
+
+```text
+3. [FAILED] Update file.ts - Matching locations at lines 10-12 and lines 40-42 produce different results. Use apply_patch again with more specific surrounding context or smaller changes if needed.
+```
+
+```text
+3. [FAILED] Update file.ts - More than 256 possible ways to apply the requested changes were found. Use apply_patch again with more specific surrounding context or smaller changes if needed.
+```
+
+```text
+3. [FAILED] Update file.ts - Requested replacement found at lines 40-44, but old content was not found. Inspect the reported lines and use apply_patch again with updated instructions if needed.
+```
+
+The 64-location limit, different-result ambiguity, and 256-way limit use the
+same guidance because each requires more specific surrounding context or
+smaller requested changes. Reverse-order and overlap failures instead tell the
+model to reorder or separate the requested changes.
+
+## Not-run results
+
+Every not-run instruction identifies why it did not run:
+
+```text
+4. [NOT RUN] Update other.txt - Instruction 3 failed.
+```
+
+For errors not owned by an instruction:
+
+```text
+1. [NOT RUN] Update a.txt - Patch format error.
+```
+
+```text
+1. [NOT RUN] Update a.txt - apply_patch setup failed before this instruction was executed.
+```
+
+Cancellation uses
+`apply_patch was cancelled before this instruction was executed.` A
+non-cancellation stop without an owning instruction uses
+`apply_patch stopped before this instruction was executed.` Patch-level
+cancellation says `apply_patch was cancelled before execution.` or identifies
+the last completed instruction.
+
+## Patch-level failures
+
+Use a patch-level statement only when no instruction owns the error.
+
+```text
+Patch format error at line 1: the patch must begin with "*** Begin Patch".
+
+Patch instruction results:
+1. [NOT RUN] Update a.txt - Patch format error.
+```
+
+```text
+apply_patch setup failed: filesystem access failed for path.
+
+Patch instruction results:
+1. [NOT RUN] Update a.txt - apply_patch setup failed before this instruction was executed.
+```
+
+Rejected tool input uses `apply_patch request rejected: <cause>` once at patch
+level and `The apply_patch request was rejected before this instruction was
+executed.` on each affected instruction.
+
+If cancellation or an integration callback stops the patch after an
+instruction completed and before another instruction became active:
+
+```text
+apply_patch stopped after instruction 3.
+```
+
+Completed instruction effects remain listed normally.
+
+## TUI
+
+The TUI retains its aggregate visual summary. It follows that summary with
+`Patch instruction results:` and every instruction row under the same
+conditional rule as model feedback.
+
+Collapsed rendering keeps each row concise:
+
+```text
+• Edited 2 files (+3 -1)
+
+Patch instruction results:
+1. [APPLIED] Update a.txt
+2. [NO CHANGE] Delete missing.txt — Path already absent.
+3. [FAILED] Update & Move source.txt → destination.txt — Created destination.txt; source.txt remains; The updated content was written to the destination, but removing the source failed; The file at destination.txt contains the requested content byte-for-byte despite the reported error.
+4. [NOT RUN] Update other.txt — Instruction 3 failed.
+```
+
+Expanded rendering nests these details beneath the relevant instruction:
+
+- applied textual diffs;
+- non-obvious filesystem effects;
+- final-state verification;
+- concise matcher locations; and
+- useful system-error details.
+
+Partial effects are not rendered as detached successful operations. Failed
+matching feedback does not repeat old or replacement patch blocks.
+
+When the default-off `applyPatchDebug` setting is enabled, the title becomes
+`apply_patch (debug)` and a completed collapsed result renders the exact text
+content returned to the model without an extra renderer-only heading. Expanding
+the same result continues to render the normal aggregate visual summary,
+instruction rows, and complete diffs. Partial tool-call previews are unchanged
+because they do not yet have final model-facing feedback.
+
+## Canonical data
+
+Model and TUI formatters consume one canonical instruction-result structure:
+
+```ts
+type InstructionResult = {
+  index: number;
+  operation: OperationLabel;
+  status: "applied" | "planned" | "no-change" | "skipped" | "failed" | "not-run";
+  reason?: InstructionReason;
+  effects: InstructionEffect[];
+  finalStates: FinalPathState[];
+  error?: InstructionError;
+  matching?: MatchingEvidence;
+  relatedInstructions?: number[];
+};
+```
+
+Replacement effects are complete:
+
+```ts
+type FileEntry = { entryType: "regular-file" } | { entryType: "symlink"; target: string };
+
+type ReplacementEffect = {
+  kind: "replaced";
+  path: string;
+  previousEntry: FileEntry;
+  replacementEntry: FileEntry;
+};
+```
+
+Only presentation differs:
+
+- styled status labels;
+- path presentation;
+- ASCII versus Unicode move arrows and separators;
+- expanded TUI diffs;
+- syntax highlighting; and
+- expanded system-error details.
+
+Semantic wording and facts remain shared.
+
+## Matcher candidate deduplication
+
+Before applying candidate and mapping limits:
+
+1. canonicalize candidates by complete byte-edit effect;
+2. remove byte-identical duplicate candidates;
+3. retain every semantically distinct candidate; and
+4. do not rank candidates or use heuristic preference.
+
+This prevents duplicate line-level and structural candidates from consuming
+the exhaustive mapping bound.
+
+## Required tests
+
+### Completeness
+
+- model and TUI results with 1, 8, 9, 100, and 500 instructions;
+- every instruction appears whenever the ledger is included;
+- all-applied instructions without feedback omit the ledger;
+- one applied instruction with feedback includes every instruction;
+- no omitted-count message exists.
+
+### Summary and attribution
+
+- successful A/M/D summary remains;
+- all-no-change summary remains;
+- failed summary lists confirmed changed paths;
+- effects from failed instructions appear under that instruction;
+- an unverified final state does not produce `No files were changed.`.
+
+### Terminology
+
+Generated model and TUI feedback does not contain:
+
+- `Committed prefix`
+- `exact`
+- `inexact`
+- `installed`
+- `dead`
+- `dominated`
+- `Matcher diagnostics`
+- the heading `Instructions:`
+
+When the conditional ledger is present, its heading is
+`Patch instruction results:`.
+
+### Deterministic failures
+
+Cover:
+
+- patch-level input and format errors appear once;
+- cancelled execution, no-owner stops, setup failures, rejected requests, and
+  patch-format failures use distinct reasons;
+- every replacement records both entry types and every resulting symlink
+  records its raw target;
+- unchanged after failure;
+- requested content present after a reported write failure;
+- different content after failure;
+- same-content entry replacement after failure;
+- entry-type change after failure;
+- unchanged planned content from an earlier instruction;
+- final state not verified;
+- a confirmed partial move remains in the changed-file summary when later
+  inspection cannot verify the destination;
+- destination created while source remains;
+- destination replaced while source remains;
+- destination deleted and replacement absent;
+- parents created without a false no-change statement;
+- temporary entry remains without a false no-change statement; and
+- post-operation verification failure.
+
+Every formatter-matcher failure reason is covered with its direct retry
+guidance.
+
+### No patch-content repetition
+
+Model feedback does not reproduce old or replacement hunk blocks.
+
+### Model/TUI parity
+
+Every fixture has the same statuses, concise reasons, filesystem effects,
+final path states, and instruction attribution in model and TUI output.
+
+With `applyPatchDebug` enabled:
+
+- a completed collapsed result contains the complete model-facing text;
+- a completed expanded result contains the normal visual summary and diffs;
+- changing the setting updates existing result components; and
+- partial previews remain unchanged.
+
+### Matcher deduplication
+
+A multi-group formatter-recovery fixture proves duplicate candidates do not
+consume the mapping limit.
+
+## Documentation
+
+Implementation changes must also update:
+
+- `APPLY_PATCH_SEMANTIC_OPERATIONS.md`
+- `APPLY_PATCH_REMAINING_WORK.md`
+- `README.md`
+- `CHANGELOG.md`

package/CHANGELOG.md

@@ -2,6 +2,50 @@
 
 ## Unreleased
 
+## 0.0.7 - 2026-08-16
+
+### Added
+
+- Add a default-off `applyPatchDebug` setting, environment override, and `/codex-settings` control that labels calls `apply_patch (debug)` and shows exact model-facing feedback without a redundant renderer-only heading in collapsed completed results while preserving normal diffs when expanded.
+- Persist content-free provider response decisions for nontrivial continuations and record the compaction reason and retry intent in native checkpoint details.
+
+### Changed
+
+- Omit Pi's auto-compaction state from the compatibility footer until extensions can observe its live value.
+- Evaluate grammar-valid `apply_patch` operations against a sequential virtual filesystem, accepting harmless no-ops and provably dead updates while rejecting conflicts before writes.
+- Recover uniquely determined `apply_patch` edits after whitespace-only formatter line reflow using the official `web-tree-sitter` runtime and packaged `@2h2d/tree-sitter-wasms` grammars for JavaScript, JSX, TypeScript, TSX, Python, Go, Java, and Scala, including typed Markdown code fences and formatter-aligned tables; require exact old-side punctuation and apply requested replacement lines without reinterpretation.
+- Move regular files and symlink entries opaquely for move-only `apply_patch` hunks, preserving arbitrary binary bytes and rendering path-only move history.
+- Preserve the aggregate `apply_patch` changed-file summary and, when any instruction is not applied or an applied instruction has feedback, list every concise source-ordered result under `Patch instruction results:` as `N. [STATUS] operation` in model and TUI feedback while omitting the redundant ledger for ordinary all-applied patches.
+- Label combined operations `Update & Move`, require verified previous and resulting entry types for replacement feedback, describe regular-file-over-regular-file replacements as remaining regular files, report raw symlink targets in straightforward move results, distinguish byte-identical update outcomes from identical adds, and use explicit metadata, partial-move, and post-failure content wording.
+- Add direct retry guidance to every formatter-matcher failure, including clearer wording when more than 256 possible applications are found.
+
+### Fixed
+
+- Return complete function and custom tool calls to Pi before provider-owned continuation, recover incomplete call batches without execution, and retry failed call-bearing responses without replaying calls that have no tool output.
+- Preserve each matched source region's local line endings when strict `apply_patch` matching edits CRLF or mixed-line-ending files.
+- Model native renames and cross-filesystem copy-and-unlink moves with distinct hard-link topology, and account for earlier planned unlinks when proving later edits dead.
+- Record symlink deletion as a path-only entry operation so history and rendering never claim that the link target's bytes were deleted.
+- Serialize same-process `apply_patch` calls across case, Unicode, symlink-parent, and hard-link aliases while retaining Pi's ordinary path queues.
+- Render semantic no-change and skipped results in simplified technical English, without instruction limits, detached proof sections, or repeated patch text.
+- Attribute completed filesystem effects, deterministic post-failure path states, matcher evidence, and errors to the instruction that produced them.
+- Preserve confirmed partial-move effects in changed-file summaries when later inspection is unavailable, without rendering the partial operation as a completed rename.
+- Report parent-directory and temporary-entry failure effects without a false no-change summary.
+- Report patch-level errors once and distinguish cancelled or stopped execution, `apply_patch` setup failures, rejected requests, and patch-format failures.
+- Keep unreadable previous content and diff availability out of model feedback.
+- Execute the move strategy proven during validation, with an injectable filesystem boundary covering native rename, cross-filesystem replacement, Windows overwrite fallback, and partial-failure inspection.
+- Retry Tree-sitter parser and grammar initialization after transient failures, and cover cancellation across queue, matching, pre-mutation, and committed-prefix phases.
+- Reject FIFOs, Unix sockets, directories, and available character or block devices during side-effect-free preflight.
+- Avoid dereferencing symlink targets while queuing entry-only operations and no-op updates, including cyclic links.
+- Replay later add/delete entry state when proving stale hard-link updates dead.
+- Preserve cancellation raised during Tree-sitter work and classify pure-move filesystems from the source entry rather than its parent.
+- Re-resolve relative symlink targets from their destination after native or cross-filesystem pure moves.
+- Deduplicate byte-identical line and structural matcher candidates before applying candidate and mapping limits.
+
+### Security
+
+- Require npm releases to match a locally built SHA-256 recorded in an SSH-signed release commit before trusted publishing can stage the package.
+- Require code-owner review for release policy, protect `main` and `v*` refs, and gate npm OIDC behind a reviewed tag-only environment.
+
 ## 0.0.6 - 2026-08-10
 
 ### Added

package/LICENSES/tree-sitter-wasms-MIT.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kaan Ozdokmeci
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.