pi-condense 2.6.0 → 2.7.0

package/CHANGELOG.md

@@ -7,6 +7,16 @@ Published to npm as [`pi-condense`](https://www.npmjs.com/package/pi-condense) (
 Pushing a `vX.Y.Z` tag triggers `.github/workflows/release.yml`, which runs the tests and
 publishes via OIDC trusted publishing. See `.agents/skills/release/SKILL.md`.
 
+## [2.7.0] - 2026-08-12
+
+- **Single-chain observability + reload trigger repair ([#6](https://github.com/jjuraszek/pi-condense/issues/6)).** A 5-hour, 256-turn session running one uninterrupted tool chain (no text-only assistant reply ever closed it) accumulated ~195k tokens of raw toolResults while `/pruner status` showed near-zero activity: chain compression (Phase 3) requires *closed* chains by design, and a session reload cleared the in-memory pending queue, leaving the automatic flush trigger stranded even though the branch rescan could recover the work.
+  - **Reload rearm.** `session_start` / `session_tree` now probe the branch via the existing capture rescan (no LLM work); when completed unsummarized batches exist past the frontier, a transient `rearmedPending` flag makes the `turn_end` budget/delta gate reachable without a freshly pushed batch - the next threshold crossing flushes automatically. Boolean only: no queue reconstruction, no flush at boot; reload followed by total idleness gets visibility, not silent compaction. Probe failures are reported (`console.error`) and never fail the reload. With the flag unset, `turn_end` behavior is byte-identical to before (regression-pinned).
+  - **Context metrics.** New pure module `src/context-metrics.ts` computes what the pruner cannot (yet) reclaim: **open-cycle thinking tokens** (thinking blocks in the trailing open segment), **largest-chain share** (max of largest closed chain vs the open segment, % of total branch chars), and **frontier gap** (summarization-eligible unsummarized toolResults past the frontier, occurrence-keyed and protection-aware). All `chars/4` estimates over the persisted branch, including persisted summary messages.
+  - **Surfacing.** `/pruner status` grows a `--- context ---` block (plus `rearmed: yes` when armed); the footer status line grows a self-hiding ` · think Nk · gap Nk · chain P%` suffix (rendered only while the frontier gap is non-zero); `agent_end` shows `prune: recovered pending (reload)` instead of a misleading `0 pending`.
+  - **New `context-prune-flush-metrics` session entry.** One per flush attempt - every outcome including `empty` and `error`, single `finally` emit site, delivery-routed writer - recording the trigger (`budget` / `delta` / `message-end` / `manual` / `rearmed`), batch counts, and the pre-flush metrics snapshot. Append-only observability log: never in LLM context, never reconstructed on load; exists so the next incident of this shape is diagnosable post-hoc. `cost:external` payload and `SummarizerStats` semantics unchanged.
+  - **Docs.** `PRUNING.md` documents the single-chain limitation (a design property of Phase 3) with `autoBudgetThreshold` / `budgetTurnDelta` guidance for long autonomous runs; README, `doc/configuration.md`, and the AGENTS.md entry table cover the new surfaces. Spec: `doc/specs/2026-08-12-single-chain-observability-trigger-repair.md`.
+  - Also hardens the test suite's `@earendil-works/pi-ai/compat` mocks to preserve real module exports (they were silently layout-sensitive under flat/deduped node_modules installs).
+
 ## [2.6.0] - 2026-08-12
 
 - **Fix: reused provider tool-call ids could delete a live turn and produce a rejected request ([#8](https://github.com/jjuraszek/pi-condense/issues/8)).** Provider `toolCallId`s (e.g. `bash_23`) are unique only within one response - some providers restart a `${tool}_${n}` counter, so the same bare id recurs across a session denoting different tool calls. `applyChainCompressions` treated the id as session-durable identity: it unioned every persisted chain entry's `droppedToolCallIds` into one session-wide set and dropped any message matching it anywhere in the array, deleting a live assistant turn that happened to reuse a compressed chain's id and orphaning its tool result - rejected outright by Anthropic (`unexpected tool_use_id found in tool_result blocks`) and Kimi K3 (unresolvable tool name), unrecoverable without hand-editing the session JSONL. The same bare ids also mis-keyed the indexer, dedup, `isSummarized`, and batch capture.

package/PRUNING.md

@@ -965,6 +965,33 @@ The `context-prune-chain` session entry carries the matching `protectedToolCallI
 - **Model-driven trigger.** The compressor is autonomous (rolling window). A model-callable compress tool (DCP-style: the model compresses a sub-task as it closes) is not implemented; the earlier scaffolded `agentic-auto` mode + `context_prune` tool were removed in v1.0.0.
 - **Multi-turn span merging.** Each compressed span maps 1:1 to a closed chain (one user -> text-only-assistant round). Merging several consecutive closed spans into one topic summary is future work.
 
+### Single-chain sessions
+
+Phase 3 (chain compression) only ever acts on **closed** chains - a chain closes when a text-only assistant reply follows the tool-call round (see [What a closed chain is](#what-a-closed-chain-is)). A session that runs one long, uninterrupted tool chain - an autonomous agent loop that never emits a text-only reply - never closes a chain, so the rolling window has nothing to compress. The entire run accumulates as one open segment that Phase 3 cannot touch. This is a known design property, not a bug: closed-chain detection is load-bearing for Phase 3's positional-range resolution (`resolveRange`), and there is no forced-close/reopen valve (see [Deferred](#deferred) above and issue #6 - a corpus replay would be needed before any such valve is considered).
+
+Phase 1 (per-batch summarization) is unaffected by chain closure and remains the only lever for this shape of session. For long autonomous runs where no chain is expected to close:
+
+- Lower `autoBudgetThreshold` (e.g. `0.5`-`0.6` instead of the default `0.8`) so the budget trigger fires well before the open segment dominates the window.
+- Set `budgetTurnDelta` so a single turn's sudden context jump force-flushes even between budget-threshold crossings - this catches spikes a static threshold misses until the next turn.
+
+Neither knob makes a chain close; they just keep Phase 1 flushing on schedule so raw toolResults do not pile up unsummarized for the whole run.
+
+**Observability metrics** (`src/context-metrics.ts`, `computeContextMetrics`) exist precisely to make this shape of session visible instead of silently reporting `calls: 1` the way the triggering incident did. All three are chars/4 token estimates (`Math.round`, same convention as the reclaim footer) and surface on `/pruner status` (a `--- context ---` block), the footer status line (a compact suffix, shown only when the frontier gap is non-zero), and a `context-prune-flush-metrics` session entry written once per flush attempt regardless of outcome:
+
+| Metric | Definition |
+|---|---|
+| Open-cycle thinking tokens | Est. tokens of `thinking` content blocks in assistant messages strictly after the last text-only assistant reply (the open segment). Not windowed by the frontier, so a skip/oversized/trivial outcome (which advances the frontier without removing anything from context) never masks stranded thinking as ~0. |
+| Largest-chain share | `max(largest closed chain, open segment)` chars, as a percentage of total branch chars. Interrupted chains count as closed for this purpose - they are retained context regardless of compressibility. A single-chain session's whole branch is its open segment, so this reads near 100% for the incident shape. |
+| Frontier gap | Est. tokens of `ToolResultMessage`s after the persisted prune frontier that are summarization-eligible: not already summarized, not protected (`protectedTools`/`protectedPaths`). 0 when there is nothing left to capture. |
+
+See `doc/specs/2026-08-12-single-chain-observability-trigger-repair.md` for the incident and full design rationale.
+
+### Reload rearm
+
+A reload (`session_start` or `session_tree`) rescans the branch for completed-but-unflushed tool-call batches past the frontier (the same rescan `flushPending` itself uses, no queue reconstruction). If that rescan finds recoverable work, a transient in-memory flag arms: the next `turn_end` evaluates the budget/delta trigger even on a turn that contributes no new tool results itself, instead of silently requiring a fresh batch to reach the gate. No flush runs at reload time - arming only changes what an *existing* trigger sees on the next turn.
+
+The flag clears the moment any flush attempt runs (any outcome) and is re-evaluated on the next reload. A reload followed by total idleness - no further turns at all - gets visibility only: the footer, `/pruner status`, and the `agent_end` pending notice (`prune: recovered pending (reload)`) all reflect the recoverable work, but nothing flushes automatically. This is by design - no immediate flush at boot (print-mode sessions may die under it; a boot-time compact was not requested by the user).
+
 ---
 
 ## Occurrence Identity

package/README.md

@@ -84,6 +84,7 @@ Every summarizer cost update is emitted on the shared `pi.events` channel `cost:
 | Batch vs chain | A batch is one flush's worth of tool calls; a chain is a longer closed sequence eligible for range compression |
 | Prune frontier | The last attempted prune boundary - advances even on a skip, so nothing is reconsidered twice |
 | Diagnostics (`diag u/m/o`) | A self-hiding status-line segment surfacing prune-time degradations: `u` = unresolved chain range, `m` = detection/render id mismatch (informational, does not change what's dropped), `o` = orphan tool-result sweep. Each letter's count is omitted when zero; the whole segment disappears when all three are zero. Backing session entries are `context-prune-diagnostic` - see below |
+| Context metrics (`think`/`gap`/`chain`) | Open-cycle thinking tokens, largest-chain share, frontier gap - what the pruner cannot (yet) reclaim, notably in single-chain sessions. See below and [PRUNING.md § Single-chain sessions](PRUNING.md#single-chain-sessions) |
 | Prompt-cache interaction | Why batching (not per-turn pruning) is the default - see [PRUNING.md](PRUNING.md#how-prefix-caching-works) |
 | `cost:external` | The shared cost-reporting channel pi-condense emits on (see above) |
 
@@ -91,6 +92,16 @@ Every summarizer cost update is emitted on the shared `pi.events` channel `cost:
 
 The status-line `diag u<N>/m<N>/o<N>` segment above is backed by `context-prune-diagnostic` session entries - session-log-only, never added to what the model sees. Full mechanics: [PRUNING.md § Diagnostics](PRUNING.md#diagnostics).
 
+### Context metrics (`context-prune-flush-metrics`)
+
+Three metrics the pruner cannot yet reclaim - open-cycle thinking tokens, largest-chain share (%), frontier gap tokens - surface in three places, all backed by `computeContextMetrics` (`src/context-metrics.ts`):
+
+- `/pruner status` prints a `--- context ---` block: `thinking:`, `chain share:`, `frontier gap:`, plus a `rearmed: yes` line while a reload-rearm probe (below) has recoverable work armed.
+- The footer status line appends `· think Nk · gap Nk · chain P%` - only when the frontier gap is non-zero, so an idle session's footer is unchanged.
+- Each flush attempt (every outcome, including empty/error) writes one `context-prune-flush-metrics` session entry with the pre-flush snapshot - session-log-only, never added to what the model sees, and not reconstructed on reload.
+
+These are most informative for long single-chain sessions where Phase 3 (chain compression) never gets a closed chain to act on - see [PRUNING.md § Single-chain sessions](PRUNING.md#single-chain-sessions) for the limitation and config guidance, and [PRUNING.md § Reload rearm](PRUNING.md#reload-rearm) for how a reload with recoverable pending work re-arms the automatic flush trigger.
+
 ## When to use / when NOT to use
 
 **Use it for:** long coding or research sessions where tool output dominates the prompt; setups deliberately running a smaller/cheaper driver model; pi-cohort fan-outs or pi-gauntlet runs where cost compounds across many turns or many children.