pi-blackhole 0.5.2 → 0.5.4

package/CHANGELOG.md

@@ -2,6 +2,37 @@
 
 ---
 
+## [0.5.4] - 2026-09-13
+
+### Fixed
+
+- **Skip ineligible proactive auto-compaction before core `prepareCompaction` failure.** When provider context reaches the auto-compaction threshold (e.g. via large system prompts, tool definitions, or memory projections) but available session entries after the latest boundary remain below Pi's configured `keepRecentTokens` budget (default 20,000), `prepareCompaction()` returns `undefined`, which previously caused Pi's `AgentSession.compact()` to throw `"Nothing to compact (session too small)"` before extension hooks ran. Settled (`agent_end`) and mid-run (`turn_end` in resume and pause modes) auto-compaction now evaluate session eligibility using Pi's `prepareCompaction` and the captured `AgentSession`'s effective compaction settings, suppressing premature trigger notices, inline failure backoff loops, and unhandled compaction errors while cleanly resuming once session history grows past the keep budget.
+- **Observer now records on sessions that have never compacted.** The observer stage's progress anchor (cursor → observation coverage marker → last compaction entry) zeroed its accumulated-token count when all three were absent — every fresh session, fork, or subagent below the compaction threshold measured 0 tokens and stayed `not_due` forever, while the status line (`anyStageDue`) correctly counted the full history and kept reporting the observer as due. The anchor now falls through to the full-history measurement (`rawTokensAfterIndex` already clamps an index of -1 to index 0), so the observer fires once `observeAfterTokens` accumulate even with no compaction entry in the branch. Sessions with any anchor (compacted, marker-bearing, cursor-bearing) behave identically. Regression from the v0.3.8 cursor work (#28/#29); behavior matches what the trigger path already did. ([#87](https://github.com/k0valik/pi-blackhole/issues/87))
+
+### Dependencies
+
+- Bumped the dev-dependencies group with 9 updates ([#88](https://github.com/k0valik/pi-blackhole/pull/88)). Dependabot now applies a cooldown (2 days default, 7 days for semver-major) so it only opens PRs carrying packages past the 48h `minimumReleaseAge` maturity gate instead of producing lockfiles `pnpm install` rejects.
+
+---
+
+## [0.5.3] - 2026-09-10
+
+### Fixed
+
+- **Recall search no longer misreads filenames as regex.** Any query containing `.` (e.g. `let me check what observer.ts does`) was compiled as one whole-query regex pattern, so natural sentences mentioning a file returned zero hits; bare dotted terms also wildcard-matched (`observer.ts` hit `observerXts`). Queries are now split into terms first: operator-bearing terms (`login|auth`, `Read.*auth`) stay patterns, plain terms match literally, and BM25 ranks the OR of per-term matches (scoring/sort unchanged; single-token regex behavior identical).
+
+- **Recall responses are bounded by a character budget (single expand included).** A five-result search page over long stored lines could exceed 250,000 characters and a 12-entry expansion 600,000, flooding model context: `lineSnippet` had no per-line cap, `expand` rendered fully unbounded, and related observations were appended after pagination with no total limit. New single knob `recallResponseMaxChars` (default 48,000 chars ≈ 12k tokens, `0` = unbounded opt-out; env `PI_BLACKHOLE_RECALL_RESPONSE_MAX_CHARS`): snippet lines clip match-centered at 1000 chars, expanded entries share the budget evenly (never below a 2,000-char readability floor, overhead-reserved so every requested index returns a bounded excerpt), observation/reflection bodies clip at 1200 chars, and an entry-aware assembler drops trailing entries/blocks first — header and `#N` prefixes never cut — with a footer naming the omitted count and continuation (`page:N`, `#N:text`/`#N:path`, 12-hex id). New `#N:text` drill-down pages arbitrary message bodies (user/assistant/toolResult/bash) so clipped content stays fully reachable; when an entry has both message text and `*text*`-named files, the body wins with a note naming the file ops. `/blackhole-recall` command output is intentionally uncapped (renders to the TUI for the human, not into model context). Follow-ups: the budget is resolved per call so settings edits apply without reload; the command also gained the out-of-range page guard and the honest `showing X of Y` truncation note the tool already had (adapted from upstream pi-vcc `f7b80bb`). ([#83](https://github.com/k0valik/pi-blackhole/issues/83))
+- **Observation timestamps are now derived in code from the observation's cited source entries.** The observer model no longer types a `timestamp` field or receives a "current local time" fallback in its prompt: each recorded observation's timestamp is the latest supporting source entry's local time (wall-clock fallback only for entries without a usable timestamp), so the recorded time is guaranteed consistent with the cited evidence. Ledger format and downstream consumers (projections, recall, dropper, reflector) are unchanged.
+- **Consolidation agent prompts rebuilt around session continuity.** The observer now curates for a future session that has no access to the transcript: a per-candidate survival test ("would a future assistant with only this line make a better decision, avoid redoing work, or avoid violating a user constraint?"), grounding rules that forbid conversation-internal labels ("option B", "D10", "m3") and unresolved pronouns, explicit noise exclusions (workflow narration, stateless transient events, already-answered questions, session-local closure), a semantic dedup rule against reworded duplicates, narration-vs-curation few-shots, and a `high` relevance bar that single tool steps can never reach. The reflector must answer "why, not just what" for pattern/decision reflections and reject near-duplicates of existing reflections; the dropper treats paraphrase duplicates as its top drop priority. All three prompts are domain-agnostic (no coding-language or project-specific assumptions).
+
+- **Recall streams large session files instead of crashing past V8's string limit (adapted from upstream pi-vcc [#26](https://github.com/sting8k/pi-vcc/pull/26)).** `loadAllMessages` read the whole session JSONL into one UTF-8 string, which throws `Cannot create a string longer than 0x1fffffe8 characters` on long-running sessions (verified upstream against a ~700 MB file). Session files are now scanned in 64 KiB chunks with line reassembly across chunk boundaries (`src/core/session-lines.ts`, shared with the global index map); message indices, lineage filtering, malformed-line warnings, and final-lines-without-newline are preserved. This also restores treating a not-yet-written session file as empty history instead of throwing `ENOENT` — a case our divergence had lost. `project-recall/corpus.ts` still reads session files whole and is a known follow-up for the same crash class.
+- **Compaction summaries now reference session-global `#N` indices (the recall index space).** Summaries previously numbered the selected window from zero, so after the first compaction or a branch the emitted `(#N)` refs retrieved unrelated operations via recall (or failed lineage checks on branched sessions). The hook now maps each selected entry id to its global message index — shared counting rule with recall in `src/core/global-indices.ts`, aligned through `convertToLlm` per message — and threads it into `compile`/`compileSegment` via `sourceIndices`; unresolvable positions render no ref instead of a wrong one. Covers default and append modes (fresh segments freeze global refs), both tail behaviors, and memory on/off; omission markers and OM `12-hex → #N` annotations already used this space. Summaries minted before this fix keep their window-relative refs until the brief rolls off (default mode) or a `/blackhole rebase` folds them (append segments). ([#82](https://github.com/k0valik/pi-blackhole/issues/82))
+- **Project-memory export de-duplication.** Reflections now share the observation fuzzy + Sørensen-Dice clustering, merged variants render as a `+N variants` count instead of sub-bullets, and observations restating a rendered reflection are suppressed (reflection wins); coverage/topic linkage tracks uncapped member ids so citations through hidden variants still count (~+1.4s export ranking on real corpora).
+- **Append-mode compaction floor: bounded memory and useful rebases.** Observation and reflection caps count complete rendered lines, including newline separators, skip oversized records, and preserve source order and raw recall history. Reflections default to 8000 tokens (`reflectionsPoolMaxTokens`; `0` disables; env `PI_BLACKHOLE_REFLECTIONS_POOL_MAX_TOKENS`), with matching file, environment and settings-modal handling. Historical compact-all full-fold checkpoints remain valid maintenance boundaries.
+- **Comparable append/rebase accounting.** Both candidates use the same bounded memory and retained tail, including persisted tool-output omissions and Pi's provider-visible wrappers. Compatible usage supplies a fixed-overhead residual; missing or inconsistent usage leaves full totals unknown instead of comparing unlike scales. Ordinary rebase requires chain pressure above `floor(W / 8)` or known context pressure above `floor(W / 2)`, plus at least `max(1, min(24000, floor(24000 * W / 272000)))` estimated tokens saved. Manual rebase remains explicit; overflow/capacity recovery selects the smaller candidate without the ordinary saving minimum. Compaction cadence and upstream threshold presets are unchanged. Decision diagnostics report estimates and insufficient recovery; they do not guarantee provider acceptance. Follow-up to [#69](https://github.com/k0valik/pi-blackhole/issues/69).
+
+---
+
 ## [0.5.2] - 2026-09-07
 
 ### Added

package/README.md

@@ -31,13 +31,13 @@ Then `/reload` or restart Pi. The config file at `~/.pi/agent/pi-blackhole/pi-bl
 
 ## ✨ What's new
 
-> **Latest release: [0.5.2](CHANGELOG.md)**
+> **Latest release: [0.5.4](CHANGELOG.md)**
 >
-> - **⚠️ Auto-compaction now scales to your model's context window** — the no-knob default no longer fires at a flat 81,000 tokens: a built-in **`default` preset curve** derives the trigger as `floor(window × ratio)` — 0.90 @ 32,768, 0.80 @ 131,072, 0.70 @ 262,144, 0.40 @ 1,048,576 (piecewise-linear between anchors). Small local windows fill near-full (cheap to resend); 1M-class windows compact early. **Behavior change on upgrade:** a config file containing the literal scaffold value `"compactAfterTokens": 81000` is auto-migrated (treated as residue, never a deliberate pin) so the curve governs — any _other_ explicit value still pins a flat threshold. Prefer a different shape? `compactAfterPreset`, `compactAfterRatio`, and `compactReserveTokens` live under **Compaction** in `/blackhole settings`; thresholds re-derive on every check, so mid-session `/model` switches apply automatically. ([#79](https://github.com/k0valik/pi-blackhole/pull/79))
-> - **Observational memory now works with custom providers** — observer/reflector/dropper resolve their streams through the model registry (`pi.registerProvider` providers like cursor-sdk, CLIProxyAPI, …) instead of the built-in compat path. Custom-provider-only setups no longer crash after a turn, and `observational-memory.model` can stay unset — no second built-in provider required. ([#74](https://github.com/k0valik/pi-blackhole/pull/74))
-> - **Reload-safe memory pipeline** — reloading, forking, or switching a session mid-consolidation now cancels the in-flight observer/reflector/dropper work cleanly instead of appending through the stale session; the fresh session retries the work without losing reflections. ([#74](https://github.com/k0valik/pi-blackhole/pull/74))
-> - **Compact-all compactions keep your memory** — fixed a silent drop of _all_ observations and reflections when compaction fires on a single-prompt or no-user-message session (the compact-all sentinel resolved to an empty fold). ([#74](https://github.com/k0valik/pi-blackhole/pull/74))
-> - **Faster session startup** — the host inline-compaction adapter now prefers Pi's already-loaded bundled runtime chunk over the slow barrel import: adapter setup drops from ~506ms to ~16ms, saving ~490ms per session start.
+> - **Recall responses are now budget-bounded** — one knob (`recallResponseMaxChars`, default 48k chars ≈ 12k tokens) caps search snippets, expansions, and related observations; clipped content stays reachable via the new `#N:text` drill-down and `page:N` continuation. `/blackhole-recall` itself stays uncapped (human TUI output). ([#83](https://github.com/k0valik/pi-blackhole/issues/83))
+> - **Compaction summaries now speak recall's index space** — `(#N)` refs are session-global, so they resolve to the right operation after compactions and branches instead of pointing at unrelated history. ([#82](https://github.com/k0valik/pi-blackhole/issues/82))
+> - **Memory that curates for a future session** — rebuilt observer/reflector/dropper prompts (survival test, grounding rules, noise exclusions) plus observation timestamps derived from cited evidence instead of model-typed dates.
+> - **Observer fires before the first compaction** — a never-compacted session (fresh start, fork, subagent) no longer stays at zero observations forever; the observer runs as soon as the token threshold is hit. ([#87](https://github.com/k0valik/pi-blackhole/issues/87))
+> - **Auto-compaction no longer errors on "session too small"** — proactive triggers pre-check Pi's own compaction eligibility instead of notice-spamming and throwing.
 
 See [`CHANGELOG.md`](CHANGELOG.md) for the full history.
 
@@ -78,19 +78,22 @@ All commands work regardless of `compaction` mode — only _when_ auto-compactio
 
 The agent gets one unified `recall` tool that handles every form of historical lookup. Searches read the raw session file directly, bypassing compaction.
 
-| Input           | What it does                                                                                                                                                      |
-| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `[12-char hex]` | Recover source evidence for a specific observation or reflection ID from the session ledger.                                                                      |
-| `#N`            | Expand a session entry by index (show full content, not truncated).                                                                                               |
-| `#N:path`       | Drill-down into file content from a tool call (e.g. `#42:auth.ts` shows first 30 lines; `#42:auth.ts:30` shows the next 30; `#42:auth.ts:full` shows everything). |
-| Free text       | BM25-ranked search across transcript and/or file content. Rare terms weighted higher.                                                                             |
-| `mode:file`     | Search only write/edit file content.                                                                                                                              |
-| `mode:touched`  | Aggregate all files written/edited across the session, grouped by path.                                                                                           |
-| Regex           | Pattern search (e.g. `fork.*pi-vcc`, `hook\|inject`).                                                                                                             |
-| `scope:all`     | Search across all session lineages (default: active lineage only).                                                                                                |
+| Input           | What it does                                                                                                                                                                         |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `[12-char hex]` | Recover source evidence for a specific observation or reflection ID from the session ledger.                                                                                         |
+| `#N`            | Expand a session entry by index (show full content, bounded by the response budget).                                                                                                 |
+| `#N:path`       | Drill-down into file content from a tool call (e.g. `#42:auth.ts` shows first 30 lines; `#42:auth.ts:30` shows the next 30; `#42:auth.ts:full` shows everything).                    |
+| `#N:text`       | Drill-down into a message body (user/assistant/tool/bash text) with the same paging (`#42:text`, `#42:text:30`, `#42:text:full`) — the continuation path for budget-clipped entries. |
+| Free text       | BM25-ranked search across transcript and/or file content. Rare terms weighted higher.                                                                                                |
+| `mode:file`     | Search only write/edit file content.                                                                                                                                                 |
+| `mode:touched`  | Aggregate all files written/edited across the session, grouped by path.                                                                                                              |
+| Regex           | Pattern search (e.g. `fork.*pi-vcc`, `hook\|inject`).                                                                                                                                |
+| `scope:all`     | Search across all session lineages (default: active lineage only).                                                                                                                   |
 
 When the agent expands a session entry (`#N`), related observations and reflections from the session ledger are automatically shown alongside the expanded content — so the agent gets the raw transcript _and_ the durable fact layer in one call.
 
+Every recall response is capped at `recallResponseMaxChars` (default 48,000 ≈ 12k tokens). Search snippet lines, expanded entries, and related observation bodies are clipped to keep a single huge stored message from flooding the context; a truncation marker names the omitted entries and how to continue (`#N:text` / `#N:path` / `page:N`).
+
 The `/blackhole-recall` command exposes the same engine to the user. Results are shown as a collapsible message and auto-fed to the agent as context.
 
 ---