pi-blackhole 0.5.1 → 0.5.3

package/CHANGELOG.md

@@ -2,6 +2,48 @@
 
 ---
 
+## [0.5.3] - 2026-09-10
+
+### Fixed
+
+- **Recall search no longer misreads filenames as regex.** Any query containing `.` (e.g. `let me check what observer.ts does`) was compiled as one whole-query regex pattern, so natural sentences mentioning a file returned zero hits; bare dotted terms also wildcard-matched (`observer.ts` hit `observerXts`). Queries are now split into terms first: operator-bearing terms (`login|auth`, `Read.*auth`) stay patterns, plain terms match literally, and BM25 ranks the OR of per-term matches (scoring/sort unchanged; single-token regex behavior identical).
+
+- **Recall responses are bounded by a character budget (single expand included).** A five-result search page over long stored lines could exceed 250,000 characters and a 12-entry expansion 600,000, flooding model context: `lineSnippet` had no per-line cap, `expand` rendered fully unbounded, and related observations were appended after pagination with no total limit. New single knob `recallResponseMaxChars` (default 48,000 chars ≈ 12k tokens, `0` = unbounded opt-out; env `PI_BLACKHOLE_RECALL_RESPONSE_MAX_CHARS`): snippet lines clip match-centered at 1000 chars, expanded entries share the budget evenly (never below a 2,000-char readability floor, overhead-reserved so every requested index returns a bounded excerpt), observation/reflection bodies clip at 1200 chars, and an entry-aware assembler drops trailing entries/blocks first — header and `#N` prefixes never cut — with a footer naming the omitted count and continuation (`page:N`, `#N:text`/`#N:path`, 12-hex id). New `#N:text` drill-down pages arbitrary message bodies (user/assistant/toolResult/bash) so clipped content stays fully reachable; when an entry has both message text and `*text*`-named files, the body wins with a note naming the file ops. `/blackhole-recall` command output is intentionally uncapped (renders to the TUI for the human, not into model context). Follow-ups: the budget is resolved per call so settings edits apply without reload; the command also gained the out-of-range page guard and the honest `showing X of Y` truncation note the tool already had (adapted from upstream pi-vcc `f7b80bb`). ([#83](https://github.com/k0valik/pi-blackhole/issues/83))
+- **Observation timestamps are now derived in code from the observation's cited source entries.** The observer model no longer types a `timestamp` field or receives a "current local time" fallback in its prompt: each recorded observation's timestamp is the latest supporting source entry's local time (wall-clock fallback only for entries without a usable timestamp), so the recorded time is guaranteed consistent with the cited evidence. Ledger format and downstream consumers (projections, recall, dropper, reflector) are unchanged.
+- **Consolidation agent prompts rebuilt around session continuity.** The observer now curates for a future session that has no access to the transcript: a per-candidate survival test ("would a future assistant with only this line make a better decision, avoid redoing work, or avoid violating a user constraint?"), grounding rules that forbid conversation-internal labels ("option B", "D10", "m3") and unresolved pronouns, explicit noise exclusions (workflow narration, stateless transient events, already-answered questions, session-local closure), a semantic dedup rule against reworded duplicates, narration-vs-curation few-shots, and a `high` relevance bar that single tool steps can never reach. The reflector must answer "why, not just what" for pattern/decision reflections and reject near-duplicates of existing reflections; the dropper treats paraphrase duplicates as its top drop priority. All three prompts are domain-agnostic (no coding-language or project-specific assumptions).
+
+- **Recall streams large session files instead of crashing past V8's string limit (adapted from upstream pi-vcc [#26](https://github.com/sting8k/pi-vcc/pull/26)).** `loadAllMessages` read the whole session JSONL into one UTF-8 string, which throws `Cannot create a string longer than 0x1fffffe8 characters` on long-running sessions (verified upstream against a ~700 MB file). Session files are now scanned in 64 KiB chunks with line reassembly across chunk boundaries (`src/core/session-lines.ts`, shared with the global index map); message indices, lineage filtering, malformed-line warnings, and final-lines-without-newline are preserved. This also restores treating a not-yet-written session file as empty history instead of throwing `ENOENT` — a case our divergence had lost. `project-recall/corpus.ts` still reads session files whole and is a known follow-up for the same crash class.
+- **Compaction summaries now reference session-global `#N` indices (the recall index space).** Summaries previously numbered the selected window from zero, so after the first compaction or a branch the emitted `(#N)` refs retrieved unrelated operations via recall (or failed lineage checks on branched sessions). The hook now maps each selected entry id to its global message index — shared counting rule with recall in `src/core/global-indices.ts`, aligned through `convertToLlm` per message — and threads it into `compile`/`compileSegment` via `sourceIndices`; unresolvable positions render no ref instead of a wrong one. Covers default and append modes (fresh segments freeze global refs), both tail behaviors, and memory on/off; omission markers and OM `12-hex → #N` annotations already used this space. Summaries minted before this fix keep their window-relative refs until the brief rolls off (default mode) or a `/blackhole rebase` folds them (append segments). ([#82](https://github.com/k0valik/pi-blackhole/issues/82))
+- **Project-memory export de-duplication.** Reflections now share the observation fuzzy + Sørensen-Dice clustering, merged variants render as a `+N variants` count instead of sub-bullets, and observations restating a rendered reflection are suppressed (reflection wins); coverage/topic linkage tracks uncapped member ids so citations through hidden variants still count (~+1.4s export ranking on real corpora).
+- **Append-mode compaction floor: bounded memory and useful rebases.** Observation and reflection caps count complete rendered lines, including newline separators, skip oversized records, and preserve source order and raw recall history. Reflections default to 8000 tokens (`reflectionsPoolMaxTokens`; `0` disables; env `PI_BLACKHOLE_REFLECTIONS_POOL_MAX_TOKENS`), with matching file, environment and settings-modal handling. Historical compact-all full-fold checkpoints remain valid maintenance boundaries.
+- **Comparable append/rebase accounting.** Both candidates use the same bounded memory and retained tail, including persisted tool-output omissions and Pi's provider-visible wrappers. Compatible usage supplies a fixed-overhead residual; missing or inconsistent usage leaves full totals unknown instead of comparing unlike scales. Ordinary rebase requires chain pressure above `floor(W / 8)` or known context pressure above `floor(W / 2)`, plus at least `max(1, min(24000, floor(24000 * W / 272000)))` estimated tokens saved. Manual rebase remains explicit; overflow/capacity recovery selects the smaller candidate without the ordinary saving minimum. Compaction cadence and upstream threshold presets are unchanged. Decision diagnostics report estimates and insufficient recovery; they do not guarantee provider acceptance. Follow-up to [#69](https://github.com/k0valik/pi-blackhole/issues/69).
+
+---
+
+## [0.5.2] - 2026-09-07
+
+### Added
+
+- **One-time migration notice for pinned-threshold users.** On the first session start of the 0.5.2 release, users whose config pins a flat `compactAfterTokens` threshold (the pre-curve legacy behavior) get a single TUI notification explaining that auto-compaction can now derive the threshold from the model's context window and how to switch (`/blackhole settings` → Compaction, or `/blackhole changelog` for details). Users already on a derived knob, the preset curve, manual/off mode, or Pi-default engine are never notified; no state is written to disk (read-only filesystems unaffected), and the notice self-disables from 0.5.3 on. ([#74](https://github.com/k0valik/pi-blackhole/pull/74))
+
+- **Context-window-aware auto-compaction threshold (opt-in).** ([#60](https://github.com/k0valik/pi-blackhole/issues/60)) New `compactAfterRatio` and `compactReserveTokens` config keys let auto-compaction derive its trigger threshold from the **active model's context window** instead of the fixed `compactAfterTokens` default: `ratio` compacts at `floor(window × ratio)` (e.g. 0.65 → ~83k @128k, ~130k @200k, ~650k @1M), `reserve` compacts when only `window − reserve` tokens of headroom remain. The threshold is re-derived on every evaluation, so mid-session `/model` switches take effect on the next check automatically. Precedence: explicit (non-default) `compactAfterTokens` > `compactAfterRatio` > `compactReserveTokens`; when a derived knob is set and `compactAfterTokens` is unset or at its 81000 default, the default is dropped so the derived knob governs (a scaffolded/modal-written default no longer blocks derived mode). Window resolution honors the per-model `contextWindow` override, then Pi's model registry, then a 128k fallback. Env overrides: `PI_BLACKHOLE_COMPACT_AFTER_RATIO`, `PI_BLACKHOLE_COMPACT_RESERVE_TOKENS`. `/blackhole-memory` status shows the effective threshold and its basis; the settings modal exposes both knobs under Compaction (0 = not set).
+- **Window-scaled auto-compaction preset curves (out of the box).** Auto-compaction's no-knob default is no longer a fixed 81,000-token threshold: a built-in `default` preset curve derives the trigger from the active session model's context window — compact at `floor(window × ratio)`, 0.90 @ 32,768, 0.80 @ 131,072, 0.70 @ 262,144, 0.40 @ 1,048,576 (piecewise-linear between anchors, constant outside; e.g. 32k → 29,491, 128k → 104,857, 1M → ~0.42 / ~418,530, ≥1,048,576 → exactly 0.40). New `compactAfterPreset` selects the curve (default `"default"`; settings-modal select under Compaction; env `PI_BLACKHOLE_COMPACT_AFTER_PRESET`); new `compactAfterPresets` holds hand-edited preset _definitions_ — name → sorted `{ window, ratio }` anchors (`window` int > 0, `0 < ratio ≤ 1`). Same-name definitions override the built-in curve, new names extend it; invalid anchors are dropped with a warning; an unknown selected name warns once and falls back to the built-in `default`. Spec: `work_docs/proposal-ratio-presets-by-context-window.md`.
+
+### Changed
+
+- **Ledger `entryIndexById` map is cached per entry-ID list** instead of being rebuilt on every call, removing a full-branch O(n) rebuild from the per-turn consolidation and compaction trigger paths. Includes a canary test documenting why upstream OM PR #57's zero-chunk observer backoff (fix 3) is unnecessary in our architecture ([upstream OM `#57`](https://github.com/elpapi42/pi-observational-memory/pull/57)). ([#74](https://github.com/k0valik/pi-blackhole/pull/74))
+
+- **No-config auto-compaction now follows the built-in `default` preset curve (behavior change on upgrade).** Users who never set a numeric knob previously auto-compacted at a flat 81,000 tokens; they now compact at `floor(window × ratio)` under the `default` preset, whose ratio _falls_ as the window grows — small 32k/64k local windows fill to ~90% and now auto-compact mid-session where the flat 81k was unreachable, while 1M-class windows compact early (~0.40) instead of at ~8% full. Config files scaffolded or modal-written before this change that literally contain `"compactAfterTokens": 81000` are auto-migrated: exactly `81000` is treated as legacy scaffold residue (never a deliberate pin) and dropped, so the preset curve / derived knobs govern — unless the value came from the `PI_BLACKHOLE_COMPACT_AFTER_TOKENS` env var, which stays explicit — an exported env var signals deliberate intent to keep the flat threshold, so unset/remove it to take the new curve. Any other explicit value (e.g. 80,000 or 180,000) still pins a fixed threshold; the flat-81k behavior can no longer be pinned by writing exactly `81000`. Spec: `work_docs/proposal-ratio-presets-by-context-window.md`.
+
+### Fixed
+
+- **Compact-all compactions no longer silently drop every OM observation and reflection.** pi-core's compact-all sentinel (`firstKeptEntryId === ""`) resolved the projection boundary to index −1, producing an empty OM summary on single-prompt and no-user-message sessions; the OM fold now covers the whole branch up to the tip. ([#74](https://github.com/k0valik/pi-blackhole/pull/74))
+- **Consolidation is cancelled across session reloads.** Reloading or replacing a session during active consolidation could append late observer/reflector output through the stale extension instance while reflection work was lost instead of retried. Observer/reflector/dropper stages, model resolution, and deferred compaction are now guarded by a runtime generation + AbortSignal on `session_start`/`session_shutdown`, and a fresh runtime retries the work without accepting stale output ([upstream OM `#58`](https://github.com/elpapi42/pi-observational-memory/pull/58)). ([#74](https://github.com/k0valik/pi-blackhole/pull/74))
+- **Cooldown reasons no longer store HTML error pages and skip toasts no longer dump them.** Cooldown reasons are sanitized to a short `HTTP <status>` line (capped at 200 chars) so an HTML WAF block page never lands in `pi-blackhole-cooldown.json` or the skip toast, which now points at the cooldown log only. ([#80](https://github.com/k0valik/pi-blackhole/issues/80))
+- **OM workers resolve `streamSimple` through the model registry for custom providers.** Observer/reflector/dropper were hard-wired to the pi-ai compat `streamSimple`, which cannot dispatch `pi.registerProvider` streams (cursor-sdk, CLIProxyAPI, …), so custom-provider-only setups crashed after a successful turn. Resolution chain: `modelRegistry.streamSimple` (Pi [`#8964`](https://github.com/earendil-works/pi/issues/8964)) → `getRegisteredProviderConfig()` matching `model.provider`, then `model.api` → global `Symbol.for` map → compat fallback. Custom-provider-only setups can now leave `observational-memory.model` unset ([upstream OM `#60`](https://github.com/elpapi42/pi-observational-memory/pull/60), [`#30`](https://github.com/elpapi42/pi-observational-memory/issues/30)). ([#74](https://github.com/k0valik/pi-blackhole/pull/74))
+
+---
+
 ## [0.5.1] - 2026-09-06
 
 ### Fixed

package/README.md

@@ -31,20 +31,16 @@ Then `/reload` or restart Pi. The config file at `~/.pi/agent/pi-blackhole/pi-bl
 
 ## ✨ What's new
 
-> **Latest release: [0.5.0](CHANGELOG.md#050---2026-09-06)**
+> **Latest release: [0.5.3](CHANGELOG.md)**
 >
-> - **Recall & export ranking upgrade** — BM25+, SimHash64, c-TF-IDF, technical density scoring, and surface-form-preserving topic labels for sharper dedup and more readable exports; export preamble now carries a best-effort heuristic warning.
-> - **`/blackhole-export` — distilled project-memory export** — Export for long-term agent memory tools - scans all project sessions + pending buffers, fuzzy-dedupes, and writes one import-ready Markdown (`Reflections → Critical → High → Medium → Low`) with topic badges and orphan-gated pending. `out:<path>.md` supported.
-> - **Append compaction mode** (`compactionSummaryMode: "append"`) — better prompt caching - keep every auto-compaction summary as an immutable segment visible to the model (`S1 | S2 | …`) instead of rewriting a single summary. `/blackhole` rebases the chain. Opt-in.
-> - **Mid-run auto-compaction** (`midRunCompaction: "resume"` | `"pause"`) — **good for goal/task** opt into transparent compaction during long tool loops without interrupting the agent. Default is `"off"`.
-> - **Robust compaction-failure handling** — unified `session_compact_failed` (pi >=0.84.3) with correct attribution, overflow-retry visibility, and noise filtering; plus bundled-CLI `AgentSession` resolution so inline compaction works from `dist/bundle/cli.js` ([#62](https://github.com/k0valik/pi-blackhole/pull/62)).
+> - **Recall responses are now budget-bounded** — one knob (`recallResponseMaxChars`, default 48k chars ≈ 12k tokens) caps search snippets, expansions, and related observations; clipped content stays reachable via the new `#N:text` drill-down and `page:N` continuation. `/blackhole-recall` itself stays uncapped (human TUI output). ([#83](https://github.com/k0valik/pi-blackhole/issues/83))
+> - **Compaction summaries now speak recall's index space** — `(#N)` refs are session-global, so they resolve to the right operation after compactions and branches instead of pointing at unrelated history. ([#82](https://github.com/k0valik/pi-blackhole/issues/82))
+> - **Memory that curates for a future session** — rebuilt observer/reflector/dropper prompts (survival test, grounding rules, noise exclusions) plus observation timestamps derived from cited evidence instead of model-typed dates.
+> - **Large sessions no longer crash recall** — session files stream in 64 KiB chunks past V8's string limit (adapted from upstream pi-vcc [#26](https://github.com/sting8k/pi-vcc/pull/26)).
+> - **Filenames aren't regex anymore** — natural-language queries mentioning a file (`let me check what observer.ts does`) now match literally instead of returning zero hits.
 
 See [`CHANGELOG.md`](CHANGELOG.md) for the full history.
 
-### ⚠️ Upcoming change
-
-> **Default compaction thresholds will become model-context-window-aware** in an upcoming release. Instead of static absolute tokens (`compactAfterTokens: 81000`), default thresholds will derive from your model's effective context window — keeping the same approximate cadence regardless of model size. Existing explicitly-set values will continue to be respected verbatim. If you're using the defaults, no action is needed; the migration is automatic.
-
 ---
 
 ## What it does
@@ -82,19 +78,22 @@ All commands work regardless of `compaction` mode — only _when_ auto-compactio
 
 The agent gets one unified `recall` tool that handles every form of historical lookup. Searches read the raw session file directly, bypassing compaction.
 
-| Input           | What it does                                                                                                                                                      |
-| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `[12-char hex]` | Recover source evidence for a specific observation or reflection ID from the session ledger.                                                                      |
-| `#N`            | Expand a session entry by index (show full content, not truncated).                                                                                               |
-| `#N:path`       | Drill-down into file content from a tool call (e.g. `#42:auth.ts` shows first 30 lines; `#42:auth.ts:30` shows the next 30; `#42:auth.ts:full` shows everything). |
-| Free text       | BM25-ranked search across transcript and/or file content. Rare terms weighted higher.                                                                             |
-| `mode:file`     | Search only write/edit file content.                                                                                                                              |
-| `mode:touched`  | Aggregate all files written/edited across the session, grouped by path.                                                                                           |
-| Regex           | Pattern search (e.g. `fork.*pi-vcc`, `hook\|inject`).                                                                                                             |
-| `scope:all`     | Search across all session lineages (default: active lineage only).                                                                                                |
+| Input           | What it does                                                                                                                                                                         |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `[12-char hex]` | Recover source evidence for a specific observation or reflection ID from the session ledger.                                                                                         |
+| `#N`            | Expand a session entry by index (show full content, bounded by the response budget).                                                                                                 |
+| `#N:path`       | Drill-down into file content from a tool call (e.g. `#42:auth.ts` shows first 30 lines; `#42:auth.ts:30` shows the next 30; `#42:auth.ts:full` shows everything).                    |
+| `#N:text`       | Drill-down into a message body (user/assistant/tool/bash text) with the same paging (`#42:text`, `#42:text:30`, `#42:text:full`) — the continuation path for budget-clipped entries. |
+| Free text       | BM25-ranked search across transcript and/or file content. Rare terms weighted higher.                                                                                                |
+| `mode:file`     | Search only write/edit file content.                                                                                                                                                 |
+| `mode:touched`  | Aggregate all files written/edited across the session, grouped by path.                                                                                                              |
+| Regex           | Pattern search (e.g. `fork.*pi-vcc`, `hook\|inject`).                                                                                                                                |
+| `scope:all`     | Search across all session lineages (default: active lineage only).                                                                                                                   |
 
 When the agent expands a session entry (`#N`), related observations and reflections from the session ledger are automatically shown alongside the expanded content — so the agent gets the raw transcript _and_ the durable fact layer in one call.
 
+Every recall response is capped at `recallResponseMaxChars` (default 48,000 ≈ 12k tokens). Search snippet lines, expanded entries, and related observation bodies are clipped to keep a single huge stored message from flooding the context; a truncation marker names the omitted entries and how to continue (`#N:text` / `#N:path` / `page:N`).
+
 The `/blackhole-recall` command exposes the same engine to the user. Results are shown as a collapsible message and auto-fed to the agent as context.
 
 ---
@@ -103,14 +102,14 @@ The `/blackhole-recall` command exposes the same engine to the user. Results are
 
 Two modes, one shared goal: keep your agent's context sharp without manual housekeeping. (`compaction: "off"` is a third escape hatch that hands everything back to Pi.)
 
-|                             | Auto (default)                                | Manual (`compaction: "manual"`)                  | Off (`compaction: "off"`)                                  |
-| --------------------------- | --------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------- |
-| Workers run?                | Yes                                           | Yes                                              | Yes (unless `memory: false`)                               |
-| Observations go to          | Conversation markers (invisible in TUI)       | Per-session disk buffers                         | Conversation markers                                       |
-| Auto-compact on `agent_end` | Yes — blackhole fires at `compactAfterTokens` | No                                               | No (Pi handles it)                                         |
-| `/compact` (Pi built-in)    | Replaced by blackhole                         | Pi handles                                       | Pi handles                                                 |
-| `/blackhole`                | Optional                                      | **Required** to flush + compact                  | Optional, but works                                        |
-| Use case                    | "Install and forget"                          | "I want to control when context gets compressed" | "Let Pi handle it, but I want `/blackhole` when I need it" |
+|                             | Auto (default)                                                         | Manual (`compaction: "manual"`)                  | Off (`compaction: "off"`)                                  |
+| --------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------- |
+| Workers run?                | Yes                                                                    | Yes                                              | Yes (unless `memory: false`)                               |
+| Observations go to          | Conversation markers (invisible in TUI)                                | Per-session disk buffers                         | Conversation markers                                       |
+| Auto-compact on `agent_end` | Yes — fires at the auto-compaction threshold (preset curve by default) | No                                               | No (Pi handles it)                                         |
+| `/compact` (Pi built-in)    | Replaced by blackhole                                                  | Pi handles                                       | Pi handles                                                 |
+| `/blackhole`                | Optional                                                               | **Required** to flush + compact                  | Optional, but works                                        |
+| Use case                    | "Install and forget"                                                   | "I want to control when context gets compressed" | "Let Pi handle it, but I want `/blackhole` when I need it" |
 
 Manual mode is the maintainer's daily driver: workers still run, but observations accumulate in `<sessionId>-pending.json` files instead of cluttering the conversation. `/blackhole` flushes the buffer, runs algorithmic compaction, and injects durable reflections in one shot.