@astrosheep/pi-context 0.13.0 → 0.15.0

package/README.md

@@ -14,46 +14,13 @@ Or load it for a single invocation without installing:
 pi -e npm:@astrosheep/pi-context
 ```
 
-## How it works
+## What you get
 
-The extension composes Pi's public `session_before_compact` / `session_compact` hooks, custom session entries, and the `context` hook to approximate Codex's experimental context management:
-
-- **`new_context` tool** — the model requests a fresh context window. The extension waits for the current run to settle, compacts with a short deterministic reset message (old conversation is excluded from the new provider context but stays in the session), then sends one hidden continuation turn once compaction has fully completed, unless another prompt is already queued or running.
-- **`<context_window>` boot block** — the head of every fresh window. For a reset it IS the summary returned from `session_before_compact` (position 0, persisted, no extra message); for the root window `session_start` persists it once as a hidden custom message (`display: false`, kept out of the TUI). It carries the agent name and first/current/previous window IDs, the recent-notes index, and a `<context_window_protocol>` teaching block. The notes index lists up to three most-recent notes, each with its `X lines, Y UTF-8 bytes` metadata plus a local-time ISO 8601 `updated` timestamp (explicit UTC offset, never `Z`) and an inline preview. A note of 320 Unicode characters or fewer is shown whole; a longer one shows its first 80 and last 240 Unicode characters joined by an ellipsis, so the two ends never overlap and no text is repeated. The notes index is a window-open snapshot with the same frozen-at-write semantics as the reminder count. Nothing is injected transiently per request: the boot block is static once-per-window content, so the head of the window stays cache-stable. Codex diverges here — its `<context_window>` block carries only the agent path and window IDs, while the notes index is our own addition.
-- **Low-budget guidance** — when estimated remaining context first drops to the reminder threshold (by default **40,960 tokens**: Pi's default 16,384 `reserveTokens` plus a 24,576 reminder margin; see [Reminder timing](#reminder-timing)), a `<context_window_guidance>` reminder is **persisted once per window** into history (hidden from the TUI transcript — one ephemeral `ui.notify` warning tells the user instead, and the model never sees it; no extra turn; `sendMessage` safely defers mid-stream). There is deliberately no transient copy: a bridge would make the model meet the same text twice at shifted positions, because history records the persisted copy after the crossing request's assistant reply. The reminder is an early warning, so arriving from the next request on costs nothing and keeps the model's view identical to recorded history. The measured remaining budget excludes `reserveTokens` and is frozen into the text at the threshold crossing, so the persisted reminder is a snapshot true at write time; `get_context_remaining` remains the live source for the current figure. The text is appended rather than prepended; existing history is not rewritten.
-- **Two-phase automatic fallback** — while Pi is streaming, the first automatic `threshold`/`overflow` crossing of the reserve line does not reset immediately. `session_before_compact` queues the final note-taking instruction with `pi.sendMessage(..., { triggerTurn: true })` — which Pi routes to `agent.steer()` while streaming, so the message is queued synchronously and reaches the model before any pending user input — and returns `{ cancel: true }`. Pi records that as an aborted compaction, spends no summary, and continues the same run with the borrowed turn; no user text or images are copied, intercepted, or replayed, and no `input` handler is registered. Once the borrowed run has finished, `agent_end` arms the real reset and `agent_settled` requests it through `ctx.compact()` for both `threshold` and `overflow`, unless another compaction has already completed. Pi can perform another automatic check after a run, and overflow recovery has a one-shot guard. The settled scheduler ensures the borrowed turn has one reset owner even when no native check resets it. Requesting the reset after the run settles avoids issuing it from `agent_end` while Pi is still finishing the run. A phase flag makes the cancel happen at most once per window — it re-arms only after a completed reset starts a fresh window. A failed reset clears the pending request, warns the user, and leaves history intact; another prompt can retry without borrowing another turn. Extension-requested fallback resets send a continuation after the compaction completion callback. The borrow is skipped when the crossing arrives while Pi is idle (the pre-prompt check in `AgentSession.prompt()`, where `triggerTurn` would start a nested run and make the pending `Agent.prompt()` reject); that crossing resets directly. `manual` `/compact` and `new_context` never take the borrowed-turn path. The reminder asks the model to write notes early; if it misses that opportunity, old history remains searchable.
-- **Runtime toggle** — `/pi-context off` disables the boot block, guidance, and reset-style compaction (Pi's default compaction, including `keepRecentTokens`, applies again). `/pi-context on` re-enables; a bare `/pi-context` reports the current state.
-- **History tools** — the model searches pre-reset conversation with case-sensitive literal substring search, exactly like Codex's `history.*` namespace.
-- **Notes tools** — persistent, session-scoped virtual files that survive window resets.
-
-## Reminder timing
-
-The reminder threshold derives from Pi's compaction reserve plus a margin configured in `settings.json` under the top-level `pi-context` key:
-
-```json
-{
-  "compaction": { "reserveTokens": 16384 },
-  "pi-context": {
-    "reminderMarginTokens": 24576
-  }
-}
-```
-
-The margin is measured in **remaining context tokens** added on top of Pi's `compaction.reserveTokens`:
-
-- `reminder = reserveTokens + reminderMarginTokens` (default margin `24576`)
-
-Put the key in the global settings (`~/.pi/agent/settings.json`) or the project settings (`<cwd>/.pi/settings.json`); project values win per key, mirroring Pi's own settings merge. With Pi's default `reserveTokens: 16384` the default gives reminder `40960`, leaving 24,576 tokens of working room above Pi's reset line no matter how you set `reserveTokens`. The reminder is an early warning; once fallback guidance has been persisted in the current window, no later early reminder is appended. This also applies after session reload. The borrowed fallback turn needs no threshold of its own: it is driven by Pi's automatic `threshold`/`overflow` compaction request, so `before_agent_start`/`turn_end` send nothing and no `fallbackMarginTokens` setting exists.
-
-Pi's `reserveTokens` and the `pi-context` reminder margin are re-read from disk at every `session_start` and cached for that session. An invalid margin — not a positive integer — is ignored with one TUI warning naming the key and the default used instead; session handling never throws.
-
-This is a file-backed read through Pi's public `SettingsManager`. It sees committed `settings.json` only: SDK-level ephemeral `applyOverrides()` calls and the unreleased `compaction.modelOverrides` are not seen by this extension.
-
-The extension does not change Pi settings or reserve additional context. Large tool outputs or user inputs can jump over the reminder. For small context windows, tune the reminder margin (or Pi's reserve) to fit the model.
-
-## Tools
-
-The nine Codex History/Notes actions are flattened because Pi tools have one global name space:
+- **`new_context`** — the model can start a fresh context window. The old conversation leaves the provider context but stays in the session, so nothing is lost. Call it on its own, not inside a parallel tool batch.
+- **A boot block at every window head** — static once-per-window content (cache-stable) carrying the window identity, the recent-notes index, and a short protocol that teaches the model how to recover: notes for its own bookkeeping, history tools for everything before the reset.
+- **Low-budget guidance** — one persisted early warning per window when the estimated remaining budget crosses the reminder line, so the model checkpoints before the lights go out.
+- **`get_context_remaining`** — the live, reserve-adjusted estimate of the context budget left before Pi's compaction reserve.
+- **Nine history/notes tools** — Codex's History/Notes actions flattened into Pi's single tool namespace:
 
 | Codex action | Pi tool |
 | --- | --- |
@@ -61,42 +28,42 @@ The nine Codex History/Notes actions are flattened because Pi tools have one glo
 | `history.list_items` | `history_list_items` |
 | `history.read_item` | `history_read_item` |
 | `history.search_contents` | `history_search_contents` |
-| `notes.list_files_by_prefix` | `notes_list_files_by_prefix` |
+| `notes.list_files` | `notes_list_files` |
 | `notes.read_file` | `notes_read_file` |
 | `notes.search_contents` | `notes_search_contents` |
 | `notes.append_to_file` | `notes_append_to_file` |
 | `notes.write_file` | `notes_write_file` |
 
-`history_*` reads the current branch's actual Pi session entries, including entries hidden by earlier compaction. Window IDs are extension-owned: the root window is `pcw:<session8>:root`, and each reset mints an 8-hex id baked into the compaction entry's `details.windowId` as `pcw:<session8>:<minted>`. Pi-native compactions (extension toggled off) fall back to `pcw:<session8>:<compaction-entry-id>`. Here `<session8>` is the first 8 characters of the session id; baked window ids stay opaque on read, so reset windows minted under the older full-session form remain resolvable. Item IDs are the persisted Pi entry IDs. No transcript copy or volatile archive is maintained. Ordering is newest-first by default: `recent_first` omitted or `true` returns newest-first, and only an explicit `false` returns oldest-first; `history_list_windows`, `history_list_items`, and `history_search_contents` share that switch.
-
-`history_*` and paged `notes_*` read/search results share a **32 KiB (32 * 1024 UTF-8 bytes) per-result budget**. List/search tools return whole items only and include `next_cursor` (an integer cursor, `null` only when the set is exhausted); pass it back unchanged as the `cursor` parameter, with the same filters and ordering, to continue. Page caps (`limit`, `max_results`, `max_files`) bound a single page, never the enumerable set; with no cap, a page holds as many items as the budget fits. A live window keeps growing while it is paged, so enumerate a closed window when a stable, complete set is needed. `history_read_item` defaults to `limit_chars: 12000`, accepts at most 50000, and returns `total_chars` plus `next_offset_chars` when more content remains. `notes_read_file` returns whole lines only, includes `total_lines`, and uses `next_start_line` for a bounded continuation. When one indivisible unit is larger than the whole budget (a single note line, search match, or history item), it is returned middle-truncated — head and tail joined by a `…[truncated N chars]…` marker, mirroring Codex's `truncate_middle` — instead of being dropped, so a page is never empty and its cursor always advances. `max_chars_per_item` limits Unicode code points including the truncation ellipsis (`…`). The 32 KiB choice matches Aider's default and Claude Code's roughly 30k-character limit while keeping worst-case CJK output (about 11k tokens) below Pi's default `reserveTokens` of 16384.
-
-`notes_*` stores operation entries in the same append-only Pi session under `pi-context/note`. They are session-scoped, survive JSONL reload, never enter provider context, and use safe relative virtual paths only (no absolute paths, `..`, `.`, empty components, or backslashes). Searches are literal and case-sensitive. `notes_read_file` accepts inclusive 1-based line ranges; negative line numbers count from the last line. Staleness is explicit and never inferred from timestamps. `notes_write_file` and `notes_append_to_file` accept an optional `mark_stale` boolean, and `notes_list_files_by_prefix` reports each file's `stale` flag. Stale notes leave the boot notes index — which omits itself once no fresh note remains — but stay listed, readable, and searchable. Each call must carry `text`, `mark_stale`, or both: `mark_stale: true` alone flags a note without touching its content, `mark_stale: false` alone clears the flag without touching content, and either call may carry `text` too — so final content can be written and marked stale in one call, and a closing log line can be appended and marked stale in one call. Carrying `text` without `mark_stale` revives a stale note, and marking a path that does not exist is an error. `notes_read_file` success results and every matched file object from `notes_search_contents` also carry `created_at` and `updated_at`, the same fields `notes_list_files_by_prefix` returns. All note timestamps are local-time ISO 8601 strings with an explicit UTC offset (for example `2026-09-15T17:31:45.392+08:00`; a UTC host renders `+00:00`, never `Z`), while the persisted `NoteFile`/`NoteOperation` metadata keeps plain epoch milliseconds. Error results carry no timestamps. Writes are capped at 1,000,000 UTF-8 bytes. `notes_write_file` and `notes_append_to_file` declare `executionMode: "sequential"` (the Pi equivalent of Codex's `supports_parallel_tool_calls = false`), so a tool batch containing either runs its calls one at a time and note read-modify-write cannot race with itself.
-
-Unlike Codex, the history tools do not advertise `agent_name`: Pi has no cross-agent session routing, so the parameter is omitted from the schemas entirely (strict `additionalProperties: false` still rejects it) instead of costing schema tokens on every request.
-
-Two extra controls compose Pi public APIs:
+The tool descriptions the model sees are the behavioral documentation: search is case-sensitive literal substring; reads are character windows whose cursors reconstruct the original exactly; anything a response does not deliver is named by an explicit field.
 
-- `get_context_remaining` returns `{ "remaining_tokens": number | null }`, computed as `max(0, contextWindow - usedTokens - reserveTokens)`: the estimated budget available before Pi's compaction reserve. `null` means Pi itself cannot make a reliable estimate (notably immediately after compaction). The low-budget guidance reports this same reserve-adjusted budget and stays silent when the estimate is unknown. Its trigger still compares physical remaining capacity against `reserveTokens + reminderMarginTokens`, so subtracting reserve from the reported number does not change reminder timing.
-- `new_context` returns terminal tool output, then waits for Pi's `agent_settled`, triggers public `ctx.compact()`, installs a short deterministic reset compaction, and sends exactly one hidden continuation turn after compaction succeeds. Call it by itself in a tool batch. Pi only ends a tool turn when every parallel tool result is terminal, so Pi 0.85.1 cannot force an atomic rollover from the middle of a mixed parallel tool batch.
+- **Runtime toggle** — `/pi-context off` restores Pi's default compaction (including `keepRecentTokens`); `/pi-context on` re-enables; bare `/pi-context` reports the current state.
 
-## Reset behavior and limits
+## Configuration
 
-On `session_before_compact`, the extension appends a persistent custom reset marker through public `pi.appendEntry`, reads that real marker ID from the readonly session manager, and returns it as `firstKeptEntryId`. Pi's `buildContextEntries()` then keeps the compaction envelope plus that custom marker; custom markers are excluded from LLM context. Thus the subsequent provider context contains the boot block summary and the marker, not old conversation messages. The boot summary also carries the extension-minted window id in its `details.windowId`, so the fresh window names itself with an id Pi could not have supplied at bake time. Both explicit `new_context` and the reset requested after a borrowed fallback run add a hidden continuation from `ctx.compact`'s `onComplete` callback. Native automatic resets and user `/compact` keep Pi's native scheduling. Explicit and fallback reset requests share the `agent_settled` scheduler: if the model calls `new_context` in a borrowed fallback run, the explicit request consumes that fallback allowance, producing one reset and at most one continuation. Continuation is suppressed if another prompt is queued or already running. The old entries remain only in the session tree for `history_*`.
+The reminder threshold is Pi's compaction reserve plus a margin, configured under the top-level `pi-context` key in `~/.pi/agent/settings.json` or `<cwd>/.pi/settings.json` (project values win per key):
 
-The scheduler lives in [`src/reset-lifecycle.ts`](src/reset-lifecycle.ts); `src/index.ts` composes the features and builds reset boundaries. History and notes projections, tool adapters, budget policy, and prompt rendering have separate ownership described in [Architecture](docs/architecture.md). See the [lifecycle event table](docs/reset-lifecycle.md) for ownership and cancellation rules. Shutdown, tree navigation, and toggling off invalidate outstanding callbacks. An aborted run clears pending rollover work. Queued steering/follow-up messages may continue before the run settles; the extension does not clear or replay that queue.
+```json
+{
+  "compaction": { "reserveTokens": 16384 },
+  "pi-context": { "reminderMarginTokens": 24576 }
+}
+```
 
-Pi's built-in “compacted into the following summary” envelope is left intact. Its content says, in the extension's wake-up voice: “You wake up. Your head is empty — no memories, the past a blank. But nothing is lost: the notes you wrote and the recorded history still remember for you.” No context filtering or TUI override is used to hide that envelope.
+`reminder = reserveTokens + reminderMarginTokens`; with the defaults the early warning fires 24,576 tokens above Pi's reset line.
 
-The same handler is used for native automatic compaction. When Pi marks an overflow compaction `willRetry`, Pi core performs its single retry itself and this extension deliberately sends no second continuation. While the extension is enabled, its custom reset keeps nothing after the boundary marker, so Pi's `keepRecentTokens` setting has no effect; with `/pi-context off`, Pi's default compaction (and `keepRecentTokens`) applies again.
+## Documentation
 
-This is a composition of public `session_before_compact`, `session_compact`, `pi.appendEntry`, `ctx.compact`, and `pi.sendMessage`; it is not a Pi-core `newSession` call. A manual Pi compaction is only eligible when Pi's own `prepareCompaction()` accepts the session. Therefore a `new_context` request in a too-small/uncompactable session fails cleanly without default-summary fallback or continuation. A core change would be needed only to guarantee a force-reset at arbitrary small context sizes or to atomically interrupt a mixed parallel tool batch.
+Implementation architecture and the reset lifecycle live in [docs/](docs/).
 
-## Verification
+## Development
 
 ```sh
+npm test            # build from a clean dist, then run the suite
 npm run typecheck
-npm test
 ```
 
-The integration harness uses the installed Pi `SessionManager` and `SettingsManager` (global and project settings fixtures in temp directories, so the real `~/.pi` is never touched), including an on-disk JSONL reload. It verifies note persistence/Unicode/path rules, the boot block contents and extension-minted window ids on reset and root paths, that the `context` hook never injects, provider context exclusion after the real `firstKeptEntryId` boundary while history remains searchable, completed tool-result boundary placement, settings-derived reminder threshold resolution and margin validation, that the removed `before_agent_start`/`turn_end` fallback is gone, one early reminder per window, one continuation only, the two-phase automatic ordering (one cancel, one steer, one real reset, re-armed per window, `ctx.compact()` for both threshold and overflow, idle crossings and manual/`new_context` resets bypassing the borrow), and cancellation/failure/no-double-retry behavior. Additional lifecycle tests cover duplicate callbacks, stale callback identity, shutdown/tree/toggle changes, user abort, synchronous errors, native retry ownership, and queued prompts. SDK tests run the real Pi agent loop, extension hooks, manual compaction, and continuation with scripted provider responses: explicit and fallback resets each produce one boundary, exclude old provider context, and preserve durable history. A real uncompactable-session case verifies no automatic retry and recovery on the next user prompt. Further SDK cases verify fallback note-writing while usage remains above the reserve line, with no obsolete early reminder before reset; steering/follow-up delivery before reset without replay; consecutive distinct windows; and user cancellation without automatic continuation. Persisted-data tests verify boot/reminder deduplication after JSONL reload and branch navigation, and reject malformed note timestamps without overwriting valid notes. These tests use temporary settings and fake credentials and make no model or network call.
+The harness runs against the real installed Pi `SessionManager`/`SettingsManager` in temporary directories with fake credentials — no model or network calls, and the real `~/.pi` is never touched.
+
+## License
+
+MIT

package/docs/architecture.md

@@ -8,10 +8,12 @@ pi-context uses Pi's session branch as the durable source of truth. It does not
 | --- | --- | --- |
 | `index.ts` | Compose features, expose toggle/reset tool, construct reset boundary | Pi extension API |
 | `history.ts` | Project branch entries into windows/items; identify active window | `SessionReader`, read-only branch and session ID |
-| `notes.ts` | Replay note operations, validate paths/timestamps, slice lines | `SessionReader`; no scheduling or writes |
+| `notes.ts` | Replay note operations, validate paths/timestamps | `SessionReader`; no scheduling or writes |
 | `history-tools.ts`, `note-tools.ts` | Public schemas and tool results; append validated note operations | Pi tool API plus read projections |
 | `budget.ts` | Resolve settings, report usable budget, persist guidance once | Pi settings/context hooks |
-| `prompts.ts` | Render static boot block, note index and reminder | Read projections and protocol text |
+| `thresholds.ts` | Derive the reminder/reserve/warning lines from Pi's reserve plus the pi-context margins | Pi `SettingsManager`, read-only; session-scoped cache |
+| `warning.ts` | Steer the final checkpoint warning once per window | Pi context hook |
+| `prompts.ts` | Render static boot block, note index, reminder and warning | Read projections and protocol text |
 | `reset-lifecycle.ts` | Own reset requests, completion and continuation | Pi lifecycle hooks and injected boundary builder |
 | `protocol.ts` | Persisted entry tags, protocol text and defaults | No imports or effects |
 | `tool-schema.ts`, `tool-output.ts` | Shared wire-schema primitives and JSON result encoding | No session state |
@@ -22,7 +24,7 @@ Dependencies flow from the composition root and tool adapters to projections and
 
 Reset requests are a discriminated union: `idle`, `requested`, or `compacting` with an identified attempt. A request cannot simultaneously be pending and in flight. Each attempt records its originating session, whether it was explicit, and whether a matching boundary completed. Callback identity prevents an old attempt from consuming a newer one. See [reset lifecycle](reset-lifecycle.md).
 
-Fallback allowance is a separate per-window state: `available`, `borrowed`, `ready`, `spent`. It answers whether another note-taking turn may be borrowed, independently of whether an explicit request exists. Success re-arms the allowance; failure does not create a retry loop. Runtime shutdown and tree navigation invalidate outstanding requests.
+Manual, threshold and overflow compactions all build the reset boundary on the spot, idle or streaming: `session_before_compact` returns the reset immediately, never cancels and never takes a model turn, and only an aborted signal cancels. The final checkpoint warning was already steered from the context hook (see [reset lifecycle](reset-lifecycle.md)), so the model had its chance to write a note; what crosses the line now is the wipe itself. `agent_settled` services only explicit `new_context` requests, whose `ctx.compact` route needs the completion callback.
 
 Boot and reminder deduplication inspect messages in the current persisted window. Reloading the extension or the JSONL file therefore does not duplicate either message. Reminder reservation in memory covers Pi's deferred message write; navigation clears that reservation, while persisted branch-local messages remain authoritative. A sibling branch cannot suppress a reminder it never received.
 
@@ -34,6 +36,6 @@ Reset IDs are opaque strings tagged with `reset-v2`; newly minted IDs use `pcw:<
 
 The integration suite uses real SessionManager and SettingsManager instances, including JSONL restoration, branch navigation, Unicode content, malformed note operations and settings precedence. Lifecycle event tests cover duplicate/stale callbacks, native scheduling, disabled state, abort and failed compaction.
 
-Scripted SDK tests execute the real Pi agent loop with no model network request. They cover explicit and fallback reset, rejected compaction, steering and follow-up delivery before reset without replay, consecutive distinct windows, and cancellation followed by a new user prompt. They inspect actual provider contexts and durable entries. They do not establish reliability of an external provider or every possible interleaving between unrelated extensions.
+Scripted SDK tests execute the real Pi agent loop with no model network request. They cover explicit reset, instant automatic reset, rejected compaction, steering and follow-up delivery before reset without replay, consecutive distinct windows, and cancellation followed by a new user prompt. They inspect actual provider contexts and durable entries. They do not establish reliability of an external provider or every possible interleaving between unrelated extensions.
 
 Pi decides compaction eligibility before the boundary hook. A short uncompactable session therefore cannot be force-reset with the public API. Mixed tool batches and queued messages may finish before `agent_settled`; the extension preserves their delivery rather than clearing the queue. Native compaction owns its subsequent scheduling, while extension-requested compaction resumes from `onComplete` after Pi clears compaction state.

package/docs/reset-lifecycle.md

@@ -1,21 +1,22 @@
 # Reset lifecycle
 
-`src/reset-lifecycle.ts` owns requests, fallback allowance, compaction attempts, and continuation. `src/index.ts` composes the features and constructs reset boundaries; projections, tools, budget policy, and prompt rendering have separate modules described in [Architecture](architecture.md).
+`src/reset-lifecycle.ts` owns reset requests, compaction attempts, and continuation. `src/index.ts` composes the features and constructs reset boundaries; projections, tools, budget policy, and prompt rendering have separate modules described in [Architecture](architecture.md).
 
 | Event | Transition / owner |
 | --- | --- |
 | `new_context` | Mark explicit request; repeated calls report already pending. Tool returns terminal output. |
-| Streaming automatic `session_before_compact` | Available → borrowed; send one note-taking steer and cancel this compaction. |
-| Idle automatic or manual compaction | Build reset directly; no borrowed turn. |
-| `agent_end` | Borrowed → ready. Aborted run clears explicit request and spends borrowed allowance. |
-| `agent_settled` | If idle and explicit/ready, create one identified attempt and request `ctx.compact`. |
-| Matching `session_compact` | Confirm boundary, persist window state, re-arm allowance. Native compaction retains its own scheduling. |
+| Manual, threshold or overflow `session_before_compact`, idle or streaming | Build the reset boundary immediately and return it. Never cancel and never take a model turn; an aborted signal returns `{ cancel: true }`. |
+| `agent_end` | No-op for an instant reset. |
+| `agent_settled` | If idle and an explicit request is pending, create one identified attempt and request `ctx.compact`. |
+| Matching `session_compact` | Confirm boundary, persist window state. Native compaction retains its own scheduling. |
 | Attempt `onComplete` | Consume attempt; send continuation only for a confirmed boundary when idle with no queued messages. |
 | Attempt `onError` or synchronous throw | Clear attempt/request, warn, retain history. No automatic retry loop. |
 | Shutdown / start / tree / toggle off | Invalidate outstanding attempt. Identity checks reject callbacks from older attempts. |
 
-The completion callback is the scheduling boundary: `session_compact` fires before Pi clears manual compaction state. Sending a prompt inside that hook is too early. Both explicit and borrowed-fallback resets use the manual `ctx.compact` route and therefore need the same completion logic. Native compaction/retry already has a caller responsible for subsequent work.
+The final checkpoint warning is steered earlier from the context hook (`warning.ts`) once per window at reserve+8192 tokens remaining. After it, the model either ends the window itself with `new_context` or rides into Pi's automatic compaction, which resets on the spot with no turn.
+
+The completion callback is the scheduling boundary: `session_compact` fires before Pi clears manual compaction state. Sending a prompt inside that hook is too early. An explicit reset uses the manual `ctx.compact` route and therefore needs this completion logic; an automatic compaction is already the reset and resumes through Pi's own caller.
 
 Public APIs cannot guarantee immediate reset inside mixed tool batches or before queued steering/follow-up messages finish. `terminate` ends the tool-followup path; `agent_settled` remains the safe point to request compaction. The scheduler does not manipulate user queues. Pi also determines compaction eligibility before the extension hook; an uncompactable session produces a warning and waits for a new prompt.
 
-Validation is split into persisted-data integration tests, isolated lifecycle event tests, and scripted SDK tests running Pi's actual agent loop. The SDK tests cover explicit success, fallback success, and core compaction rejection followed by a user prompt. Lifecycle tests cover callback races and queue guards without pretending to exercise provider/network behavior.
+Validation is split into persisted-data integration tests, isolated lifecycle event tests, and scripted SDK tests running Pi's actual agent loop. The SDK tests cover explicit success, instant automatic reset, and core compaction rejection followed by a user prompt. Lifecycle tests cover callback races and queue guards without pretending to exercise provider/network behavior.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@astrosheep/pi-context",
-  "version": "0.13.0",
+  "version": "0.15.0",
   "type": "module",
   "description": "Codex-style context windows for Pi: reset-style compaction, durable session history tools, and persistent notes.",
   "license": "MIT",
@@ -25,7 +25,7 @@
     "README.md"
   ],
   "scripts": {
-    "build": "tsc -p tsconfig.json",
+    "build": "node -e \"require('node:fs').rmSync('dist',{recursive:true,force:true})\" && tsc -p tsconfig.json",
     "typecheck": "tsc -p tsconfig.json --noEmit",
     "test": "npm run build && node --test dist/test/*.test.js",
     "prepublishOnly": "npm run typecheck"

package/src/budget.ts

@@ -1,113 +1,49 @@
 import { Type } from "@earendil-works/pi-ai";
-import { defineTool, SettingsManager, type ExtensionAPI, type ExtensionContext } from "@earendil-works/pi-coding-agent";
-import { PI_CONTEXT_SETTINGS_KEY, DEFAULT_RESERVE_TOKENS, DEFAULT_REMINDER_MARGIN_TOKENS, GUIDANCE_TYPE, FALLBACK_TYPE } from "./protocol.js";
+import { defineTool, type ExtensionAPI, type ExtensionContext } from "@earendil-works/pi-coding-agent";
+import { GUIDANCE_TYPE, WARNING_TYPE } from "./protocol.js";
+import { thresholdsFor, resetThresholds, deriveThresholds, mergePiContextSettings } from "./thresholds.js";
 import { currentWindowId, hasWindowMessage } from "./history.js";
 import { tokenBudgetGuidance } from "./prompts.js";
 import { output } from "./tool-output.js";
 
-type ResolvedThresholds = { reminder: number; reserve: number };
-type PiContextMargins = { reminderMarginTokens: unknown };
-
-function isSettingsObject(value: unknown): value is Record<string, unknown> {
-	return typeof value === "object" && value !== null && !Array.isArray(value);
-}
-
-/** Read the raw "pi-context" object from one parsed settings scope. */
-function piContextSettings(settings: unknown): Record<string, unknown> {
-	if (!isSettingsObject(settings)) return {};
-	const value = settings[PI_CONTEXT_SETTINGS_KEY];
-	return isSettingsObject(value) ? value : {};
-}
-
-/** Merge the global and project "pi-context" objects per key; project wins, mirroring Pi's deep merge. */
-export function mergePiContextSettings(globalSettings: unknown, projectSettings: unknown): PiContextMargins {
-	const merged = { ...piContextSettings(globalSettings), ...piContextSettings(projectSettings) };
-	return { reminderMarginTokens: merged.reminderMarginTokens };
-}
-
-/** A margin is usable only as a positive integer; anything else is ignored. */
-function validMargin(raw: unknown): number | undefined {
-	if (typeof raw !== "number" || !Number.isSafeInteger(raw) || raw <= 0) return undefined;
-	return raw;
-}
-
-/**
- * Pure derivation of the reminder threshold from Pi's reserve plus the pi-context
- * reminder margin. An invalid margin degrades to the default and reports one warning.
- * The borrowed fallback turn has no token threshold of its own: it is driven by Pi's
- * automatic threshold/overflow compaction request (see session_before_compact).
- */
-export function deriveThresholds(reserveTokens: number, margins: PiContextMargins): { thresholds: ResolvedThresholds; warnings: string[] } {
-	const warnings: string[] = [];
-	const reminderKey = `${PI_CONTEXT_SETTINGS_KEY}.reminderMarginTokens`;
-	let reminderMargin: number;
-	if (margins.reminderMarginTokens === undefined) reminderMargin = DEFAULT_REMINDER_MARGIN_TOKENS;
-	else {
-		const parsed = validMargin(margins.reminderMarginTokens);
-		if (parsed === undefined) {
-			warnings.push(`pi-context: ${reminderKey} must be a positive integer; using default ${DEFAULT_REMINDER_MARGIN_TOKENS}.`);
-			reminderMargin = DEFAULT_REMINDER_MARGIN_TOKENS;
-		} else reminderMargin = parsed;
-	}
-	return { thresholds: { reminder: reserveTokens + reminderMargin, reserve: reserveTokens }, warnings };
-}
+export { deriveThresholds, mergePiContextSettings } from "./thresholds.js";
 
 export function registerBudget(pi: ExtensionAPI, isEnabled: () => boolean) {
 	let guidancePersistedInWindow: string | undefined;
-	let thresholds: ResolvedThresholds | undefined;
-
-	/**
-	 * Resolve the thresholds for this session from Pi's compaction reserve plus the
-	 * settings.json "pi-context" margins. The file-backed read is cached until the next
-	 * session_start; invalid configuration degrades per offending key with one warning
-	 * and never throws during session operation.
-	 */
-	const resolveThresholds = (ctx: ExtensionContext): ResolvedThresholds => {
-		if (thresholds) return thresholds;
-		try {
-			const settingsManager = SettingsManager.create(ctx.cwd, undefined, { projectTrusted: ctx.isProjectTrusted() });
-			const derived = deriveThresholds(
-				settingsManager.getCompactionSettings().reserveTokens,
-				mergePiContextSettings(settingsManager.getGlobalSettings(), settingsManager.getProjectSettings()),
-			);
-			for (const warning of derived.warnings) ctx.ui.notify(warning, "warning");
-			thresholds = derived.thresholds;
-		} catch (error) {
-			ctx.ui.notify(`pi-context: could not read settings; using defaults (${String(error)}).`, "warning");
-			thresholds = { reminder: DEFAULT_RESERVE_TOKENS + DEFAULT_REMINDER_MARGIN_TOKENS, reserve: DEFAULT_RESERVE_TOKENS };
-		}
-		return thresholds;
-	};
 
-	pi.on("session_start", (_event, ctx) => { thresholds = undefined; guidancePersistedInWindow = undefined; resolveThresholds(ctx); });
-	pi.on("session_tree", () => { guidancePersistedInWindow = undefined; });
+	pi.on("session_start", (_event, ctx) => { guidancePersistedInWindow = undefined; resetThresholds(); thresholdsFor(ctx); });
+	pi.on("session_tree", () => { guidancePersistedInWindow = undefined; resetThresholds(); });
 	pi.on("context", (_event, ctx) => {
-		if (!isEnabled() || hasWindowMessage(ctx, FALLBACK_TYPE)) return undefined;
-		// This hook does exactly one thing: persist the once-per-window low-budget
-		// reminder the first time remaining context crosses the reminder threshold.
-		// It never injects messages into the request.
+		if (!isEnabled() || hasWindowMessage(ctx, GUIDANCE_TYPE)) return undefined;
+		// The early reminder persists once per window the first time remaining crosses
+		// reserve+margin. It never edits the outgoing request.
 		const usage = ctx.getContextUsage();
-		if (usage && usage.tokens !== null) {
-			const remaining = Math.max(0, usage.contextWindow - usage.tokens);
-			const windowId = currentWindowId(ctx);
-			const { reminder, reserve } = resolveThresholds(ctx);
-			if (remaining <= reminder && guidancePersistedInWindow !== windowId && !hasWindowMessage(ctx, GUIDANCE_TYPE)) {
-				guidancePersistedInWindow = windowId;
-				// Persist once per window — no transient copy. A transient bridge would
-				// cover the crossing request, but history would record the reminder after
-				// that request's assistant reply, so across the boundary the model would
-				// meet the same text twice at shifted positions. The reminder is an early
-				// warning, not a per-request instruction: arriving from the next request
-				// on (sendMessage defers safely to end of turn while streaming, queueing
-				// instead of splitting a tool call/result pair) costs nothing, and the
-				// model's view stays identical to recorded history, Codex-style.
-				// The persisted copy stays out of the TUI (display: false); one ephemeral
-				// notify tells the user instead — visible to the human, invisible to the
-				// model, and never recorded, so history and the model's view don't diverge.
-				const left = Math.max(0, remaining - reserve);
-				pi.sendMessage({ customType: GUIDANCE_TYPE, content: tokenBudgetGuidance(left), display: false }, { triggerTurn: false });
-				ctx.ui.notify(`pi-context: context budget low (${left} tokens before reserve) — checkpoint reminder recorded for the model, kept out of the chat view.`, "warning");
-			}
+		if (!usage || usage.tokens === null) return undefined;
+		const remaining = Math.max(0, usage.contextWindow - usage.tokens);
+		const windowId = currentWindowId(ctx);
+		const { reminder, reserve, warning } = thresholdsFor(ctx);
+		// The final warning owns the deep band: when it has fired (or is due now),
+		// the shallow reminder would only repeat the same instruction closer to
+		// the wipe, at a worse position. See warning.ts.
+		if (remaining <= warning || hasWindowMessage(ctx, WARNING_TYPE)) return undefined;
+		if (remaining <= reminder && guidancePersistedInWindow !== windowId && !hasWindowMessage(ctx, GUIDANCE_TYPE)) {
+			guidancePersistedInWindow = windowId;
+			// Persist once per window — no transient copy. A transient bridge would
+			// cover the crossing request, but history would record the reminder after
+			// that request's assistant reply, so across the boundary the model would
+			// meet the same text twice at shifted positions. The reminder is an early
+			// warning, not a per-request instruction: arriving from the next request
+			// on (sendMessage defers safely to end of turn while streaming, queueing
+			// instead of splitting a tool call/result pair) costs nothing, and the
+			// model's view stays identical to recorded history, Codex-style.
+			// The persisted copy stays out of the TUI (display: false); one ephemeral
+			// notify tells the user instead — visible to the human, invisible to the
+			// model, and never recorded, so history and the model's view don't diverge.
+			// The model-facing count ends at the warning line: what lies below is the
+			// runway, invisible by design. The human's notify keeps the honest count.
+			const left = Math.max(0, remaining - warning);
+			pi.sendMessage({ customType: GUIDANCE_TYPE, content: tokenBudgetGuidance(left), display: false }, { triggerTurn: false });
+			ctx.ui.notify(`pi-context: context budget low (${Math.max(0, remaining - reserve)} tokens before reserve) — checkpoint reminder recorded for the model, kept out of the chat view.`, "warning");
 		}
 		return undefined;
 	});
@@ -115,11 +51,13 @@ export function registerBudget(pi: ExtensionAPI, isEnabled: () => boolean) {
 	pi.registerTool(defineTool({
 		name: "get_context_remaining",
 		label: "Get context remaining",
-		description: "Return estimated context tokens available before the compaction reserve, clamped to zero; null when Pi cannot estimate usage.",
+		description: "Return estimated context tokens left before your memory is wiped; null when Pi cannot estimate usage.",
 		parameters: Type.Object({}, { additionalProperties: false }),
 		async execute(_id, _params, _signal, _update, ctx) {
 			const usage = ctx.getContextUsage();
-			const remaining = usage?.tokens === null || usage === undefined ? null : Math.max(0, usage.contextWindow - usage.tokens - resolveThresholds(ctx).reserve);
+			// The countdown the model sees ends at the warning line (reserve + runway);
+			// the runway below it is overdraft the model never sees. See protocol.ts.
+			const remaining = usage?.tokens === null || usage === undefined ? null : Math.max(0, usage.contextWindow - usage.tokens - thresholdsFor(ctx as ExtensionContext).warning);
 			return output({ remaining_tokens: remaining });
 		},
 	}));

package/src/history-tools.ts

@@ -1,9 +1,34 @@
 import { Type } from "@earendil-works/pi-ai";
 import { defineTool, type ExtensionAPI } from "@earendil-works/pi-coding-agent";
-import { output, page, middleTruncate, withinBudget } from "./tool-output.js";
-import { positiveInteger, recentFirst, nullableString, role, cursor } from "./tool-schema.js";
+import { output, page, middleTruncate, prefixFit, earliestMatchOffsetChars, readCharacterWindow } from "./tool-output.js";
+import { positiveInteger, recentFirst, nullableString, role, cursor, searchQuery, searchQueries } from "./tool-schema.js";
 import { historyFromSession, filteredItems, visibleItem, allItems } from "./history.js";
 
+/**
+ * Shrink one page item to fit the wire budget. `truncated`/`total_chars` stay honest: the
+ * payload is only ever cut to a plain prefix of itself, never filled with a marker, and the
+ * flag flips on whenever a shrink actually removed characters. `tool_name` is metadata, not a
+ * payload, and keeps its visible middle-truncation marker. `item_id`, `window_id`, `role`, and
+ * `tool_namespace` are identity or tiny metadata and are never touched.
+ */
+function truncateHistoryItem<T extends { truncated_content: string; truncated: boolean; tool_name: string | null }>(item: T, fits: (candidate: T) => boolean): T {
+	if (fits(item)) return item;
+	const shrinkContent = (base: T): T => ({
+		...base,
+		truncated: true,
+		truncated_content: prefixFit(base.truncated_content, (candidate) => fits({ ...base, truncated: true, truncated_content: candidate } as T)),
+	});
+	const withContent = shrinkContent(item);
+	if (fits(withContent)) return withContent;
+	if (item.tool_name === null) return withContent;
+	// Content could not help even as an empty prefix: tool_name is oversized, so keep the
+	// original payload and truncate the metadata as the last resort.
+	const withName = { ...item, tool_name: middleTruncate(item.tool_name, (candidate) => fits({ ...item, tool_name: candidate } as T)) } as T;
+	if (fits(withName)) return withName;
+	// Both fields are oversized: dissolve the payload against the truncated metadata.
+	return shrinkContent(withName);
+}
+
 export function registerHistoryTools(pi: ExtensionAPI) {
 	pi.registerTool(defineTool({
 		name: "history_list_windows",
@@ -21,41 +46,37 @@ export function registerHistoryTools(pi: ExtensionAPI) {
 	pi.registerTool(defineTool({
 		name: "history_list_items",
 		label: "History list items",
-		description: "List durable session items, including items before compaction, using opaque item and window IDs.",
+		description: "List durable session items, including items before compaction, using opaque item and window IDs. Every item carries truncated and total_chars: when truncated is true, truncated_content is a plain prefix of the item's content with no marker, and total_chars is its full code-point length. max_chars_per_item: 1 therefore yields pure addresses you can resolve with history_read_item.",
 		parameters: Type.Object({ limit: positiveInteger(), cursor: cursor(), recent_first: recentFirst(), tool_namespace: nullableString(), role: Type.Optional(role), tool_name: nullableString(), window_id: nullableString(), max_chars_per_item: positiveInteger() }, { additionalProperties: false }),
 		async execute(_id, params, _signal, _update, ctx) {
 			const items = filteredItems(ctx, params).map((item) => visibleItem(item, params.max_chars_per_item ?? 1200));
-			return output(page(items, params.cursor ?? 0, "items", params.limit, (item, fits) => ({ ...item, truncated_content: middleTruncate(item.truncated_content, (candidate) => fits({ ...item, truncated_content: candidate })) })));
+			return output(page(items, params.cursor ?? 0, "items", params.limit, truncateHistoryItem));
 		},
 	}));
 
 	pi.registerTool(defineTool({
 		name: "history_read_item",
 		label: "History read item",
-		description: "Read a bounded character range from one durable session item.",
-		parameters: Type.Object({ item_id: Type.String(), offset_chars: Type.Optional(Type.Integer({ minimum: 0 })), limit_chars: Type.Optional(Type.Integer({ minimum: 1, maximum: 50000 })), window_id: Type.String() }, { additionalProperties: false }),
+		description: "Read a bounded character range from one session item. Each response delivers the longest contiguous prefix of the requested window that fits the wire budget. next_offset_chars is exactly offset_chars plus the delivered code-point count and is null only once the item ends: follow it to reconstruct the item exactly. A negative offset_chars counts back from the item's end, and the response always echoes the resolved absolute offset. Offsets and counts are code points (an emoji or CJK character counts as one).",
+		parameters: Type.Object({ item_id: Type.String(), offset_chars: Type.Optional(Type.Integer({ description: "Code-point offset to start from. A negative value counts back from the end; the response echoes the resolved absolute offset. Pass the previous next_offset_chars back unchanged to continue." })), limit_chars: Type.Optional(Type.Integer({ minimum: 1, maximum: 50000, description: "Largest requested window in code points (default 12000). A window too large for the wire budget is cut short; next_offset_chars names where the next read resumes." })), window_id: Type.String() }, { additionalProperties: false }),
 		async execute(_id, params, _signal, _update, ctx) {
 			const item = allItems(ctx).find((candidate) => candidate.windowId === params.window_id && candidate.itemId === params.item_id);
 			if (!item) return output({ error: "unknown item_id or window_id" });
-			const chars = Array.from(item.content);
-			const offset = params.offset_chars ?? 0;
-			const limit = Math.min(params.limit_chars ?? 12000, 50000);
-			const result = (content: string) => ({ window_id: item.windowId, item_id: item.itemId, offset_chars: offset, content, total_chars: chars.length, next_offset_chars: offset + limit < chars.length ? offset + limit : null });
-			// One clean middle-truncation replaces the old 0.9 shrink loop: a requested window larger
-			// than the budget comes back with its middle elided, never empty, and the cursor advances.
-			const content = middleTruncate(chars.slice(offset, offset + limit).join(""), (candidate) => withinBudget(result(candidate)));
-			return output(result(content));
+			return output(readCharacterWindow(item.content, params.offset_chars, params.limit_chars, (window) => ({ window_id: item.windowId, item_id: item.itemId, ...window })));
 		},
 	}));
 
 	pi.registerTool(defineTool({
 		name: "history_search_contents",
 		label: "History search",
-		description: "Case-sensitive literal substring search over durable Pi session history; no semantic search.",
-		parameters: Type.Object({ limit: positiveInteger(), cursor: cursor(), query: Type.String(), recent_first: recentFirst(), tool_namespace: nullableString(), role: Type.Optional(role), tool_name: nullableString(), window_id: nullableString(), max_chars_per_item: positiveInteger() }, { additionalProperties: false }),
+		description: "Case-sensitive literal substring search over durable Pi session history; query accepts one string or an array of strings, an item matches when it contains any of them (OR), and each item appears once. No semantic search. Each hit carries truncated and total_chars plus match_offset_chars: the code-point offset of the earliest query occurrence in the item's full content. With max_chars_per_item: 1 the page is an address list; resolve an address with history_read_item at match_offset_chars.",
+		parameters: Type.Object({ limit: positiveInteger(), cursor: cursor(), query: searchQuery(), recent_first: recentFirst(), tool_namespace: nullableString(), role: Type.Optional(role), tool_name: nullableString(), window_id: nullableString(), max_chars_per_item: positiveInteger() }, { additionalProperties: false }),
 		async execute(_id, params, _signal, _update, ctx) {
-			const matching = filteredItems(ctx, params).filter((item) => item.content.includes(params.query)).map((item) => visibleItem(item, params.max_chars_per_item ?? 1200));
-			return output(page(matching, params.cursor ?? 0, "items", params.limit, (item, fits) => ({ ...item, truncated_content: middleTruncate(item.truncated_content, (candidate) => fits({ ...item, truncated_content: candidate })) })));
+			const queries = searchQueries(params.query);
+			const matching = filteredItems(ctx, params)
+				.filter((item) => queries.some((query) => item.content.includes(query)))
+				.map((item) => ({ ...visibleItem(item, params.max_chars_per_item ?? 1200), match_offset_chars: earliestMatchOffsetChars(item.content, queries) }));
+			return output(page(matching, params.cursor ?? 0, "items", params.limit, truncateHistoryItem));
 		},
 	}));
 

package/src/history.ts

@@ -39,6 +39,9 @@ function mapRole(role: AgentMessage["role"]): HistoryItem["role"] | undefined {
 	return undefined;
 }
 
+/** This extension's own custom-entry namespace; entries under it are authored by pi-context. */
+const PI_CONTEXT_ENTRY_PREFIX = "pi-context/";
+
 function messageContent(message: AgentMessage): string {
 	switch (message.role) {
 		case "bashExecution":
@@ -83,7 +86,8 @@ export function historyFromSession(ctx: SessionReader): HistoryWindow[] {
 			window.items.push({
 				windowId: window.windowId,
 				itemId: entry.id,
-				role: "system",
+				// A reset-v2 compaction is authored by this extension; a native Pi compaction is not.
+				role: resetV2WindowId(entry.details) === undefined ? "system" : "developer",
 				content: entry.summary,
 				createdAt: entry.timestamp,
 			});
@@ -106,7 +110,8 @@ export function historyFromSession(ctx: SessionReader): HistoryWindow[] {
 			window.items.push({
 				windowId: window.windowId,
 				itemId: entry.id,
-				role: "user",
+				// Only entries this extension wrote are its own; every foreign custom message stays a user turn.
+				role: entry.customType.startsWith(PI_CONTEXT_ENTRY_PREFIX) ? "developer" : "user",
 				content: contentText(entry.content),
 				createdAt: entry.timestamp,
 			});
@@ -117,13 +122,18 @@ export function historyFromSession(ctx: SessionReader): HistoryWindow[] {
 
 export function visibleItem(item: HistoryItem, maxChars = 1200) {
 	const characters = Array.from(item.content);
+	const truncated = characters.length > maxChars;
 	return {
 		window_id: item.windowId,
 		item_id: item.itemId,
 		role: item.role,
 		tool_namespace: item.toolNamespace ?? null,
 		tool_name: item.toolName ?? null,
-		truncated_content: characters.length > maxChars ? `${characters.slice(0, Math.max(0, maxChars - 1)).join("")}…` : item.content,
+		truncated,
+		total_chars: characters.length,
+		// A truncated payload is a plain prefix: no synthetic marker is appended, and
+		// `total_chars` names exactly how many code points were left out.
+		truncated_content: truncated ? characters.slice(0, maxChars).join("") : item.content,
 	};
 }