pi-blackhole 0.5.8 → 0.5.10

package/CHANGELOG.md

@@ -2,6 +2,46 @@
 
 ---
 
+## [0.5.10] - 2026-09-29
+
+### Fixed
+
+- **`/blackhole` on a branch Pi considers ineligible now reports nothing-to-compact instead of failing.** Pi's `AgentSession.compact()` runs `prepareCompaction()` _before_ it emits `session_before_compact`, so a branch whose entries after the last boundary fit inside `keepRecentTokens` is refused by the host before any blackhole hook can run. The refusal surfaced twice — as Pi's error card (`reason: "manual"` always renders `Compaction failed: …`) and again as `Compaction failed: Nothing to compact (session too small)` from the command's `onError`, whose benign set covered only `Compaction cancelled` and `Already compacted`. The 0.5.4 eligibility fix gated the auto-compaction trigger but not this call site. The command now asks the host's own question via the new `getCompactionIneligibility()` before it mutates anything: an ineligible branch reports `blackhole: nothing to compact yet — Pi's keep-recent budget still covers this branch` (or `already compacted — …`) as information and returns, and the check sits above the pending-observational-memory flush so a refusal no longer appends entries, clears the pending buffer, and announces a flush that never compacted. `onError` keeps the same messages as a backstop for the case where our probe and the host still disagree.
+- **The eligibility probe now resolves compaction settings with the session model.** `isCompactionEligible` called `settingsManager.getCompactionSettings()` with no argument, while Pi calls `getCompactionSettings(model)`; Pi resolves `compaction.modelOverrides["<provider>/<id>"]` for `keepRecentTokens` and `reserveTokens` only when handed the model, so the probe was asking a different question than the host — with a per-model override configured it could report a branch eligible that `compact()` then refuses (or the reverse). All four call sites now pass `ctx.model`.
+- **A cancelled `/blackhole` no longer adds a second, vaguer toast.** The own-cut guard already names the specific reason (`blackhole: Nothing to compact (no live messages)`, or the `tailBehavior` hint) before returning `{ cancel: true }`, and Pi renders `Compaction cancelled` itself for manual aborts; the command's `onError` was adding a third message on top of those.
+- **`/blackhole` loads the config before reading it.** The handler read `runtime.config.compaction` to decide whether to flush pending observational memory, but never called `runtime.ensureConfig`, which only `agent_start` did. When `/blackhole` was the first command of a session, a configured `compaction: "manual"` install read `DEFAULTS` (`"auto"`) and silently skipped the flush, so memory accumulated in the pending file and the summary missed it.
+- **An observer stream error no longer counts as a completed chunk.** A provider failure after a recorded batch used to return partial work as success, advancing `coversUpToId` past content that was never observed. `runObserver` now throws a `WorkerStreamError` (message unchanged); the stage keeps the cursor and retries through the fallback chain, logging `discardedCount`. A valid `complete: true` close from a run that recorded something is still kept when a later turn fails: the failure is warned with the error class only (per [#80](https://github.com/k0valik/pi-blackhole/issues/80)), logged as `observer.error_after_close`, and cools the model when deterministic. A run the turn cap cuts off before it closes is treated the same way when it recorded something; a cap firing before anything was recorded is still an empty success. ([#137](https://github.com/k0valik/pi-blackhole/pull/137))
+- **A failed reflector or dropper run no longer counts as finished work.** Both stages advance their cursor on success, so a partial result marked that work covered: unreflected observations were never crystallized again, and half-evaluated drop candidates were written under an `OM_OBSERVATIONS_DROPPED` marker. `runReflector` now throws a `WorkerStreamError` unless a valid `complete: true` batch that recorded reflections closed the review (a kept close returns `errorAfterClose`, which the stage cools and warns about exactly like the observer's); `runDropper` always throws, since `drop_observations` has no complete flag to prove the evaluation finished. The turn cap cuts a run that recorded something the same way an error does (a cap before anything was recorded is still an empty success), both stages log `discardedCount`, and neither spends another `agentMaxTurns` budget retrying a session model the cap already exhausted. The cap only reports exhaustion for a turn that did tool work, so a run that stopped naturally on exactly its last allowed turn keeps its result. ([#137](https://github.com/k0valik/pi-blackhole/pull/137))
+- **A stream failure that never reaches `agent_end` still reports what it discarded.** When the stream itself breaks, or `result()` rejects, the observer, reflector and dropper rethrow that error with the count of records already written attached, so the stage error logs no longer show `discardedCount: undefined` for the most common way a run dies. The error keeps its own identity — wrapping it would break the stage's stale-context and timeout checks — and a thrown value that cannot carry a property simply loses the count rather than the failure. A run cut off by the hard `workerAttemptTimeoutMs` deadline is the deliberate exception: the deadline rejects immediately with its own error instead of waiting to learn what the worker had recorded, so that path still logs `discardedCount: undefined`. ([#137](https://github.com/k0valik/pi-blackhole/pull/137))
+
+- **A provider body no longer cools a model because the worker framed it as an error.** `isDeterministicError` only accepts a bare 4xx code alongside an error signal word, but the `<worker> API error: ` framing supplied that word on every message the observer, reflector and dropper emit — so a status-shaped token anywhere in a provider body (`processed 401 rows`) counted as a real status and cooled the session model for the default hour. The framing is stripped before the signal-word check, the three workers build their messages from one prefix shared by the throw path and the kept-close path so the two cannot drift, and the reflector's kept-close debug entry now classifies the same framed message it would throw. ([#137](https://github.com/k0valik/pi-blackhole/pull/137))
+- **The kept-close toast no longer claims a cooldown that was not recorded.** A session model whose resolved config carries no provider/id never reaches the cooldown file (`recordDeterministicError` keys it on both), yet the warning still said `model cooled down; details in cooldown log`. It now reports only what was written (`deterministic error; no cooldown recorded`), alongside the existing `cooldownHours: 0` branch, which the observer and reflector stages share through one helper. ([#137](https://github.com/k0valik/pi-blackhole/pull/137))
+- **Token counts no longer read as retryable provider statuses.** The numeric alternatives in `RETRYABLE_ERROR_RE` matched as substrings, so `1500 tokens`, `5000 rows`, or `14290` classified retryable on the `retryable:`/`cooldownWorthy:` debug axis. The codes (`429`, `500`, `502`–`504`) are now word-boundary anchored; standalone codes and real phrasing (`429 Too Many Requests`, `HTTP 503`) still match. Nothing branches on this axis today (candidate cooldowns are recorded unconditionally), so the change is log-precision only.
+
+---
+
+## [0.5.9] - 2026-09-26
+
+### Added
+
+- **Routine worker notifications are now optional.** New `showWorkerNotifications` boolean (default `true`) gates the observer/reflector/dropper progress toasts — `observer running on ~N-token chunk`, `N observations recorded`, `reflector running`, `dropper running`, and the info-level `no observations` notice. Set it to `false` (settings modal → UI → Worker notifications, `/blackhole configure`, the config file, or `PI_BLACKHOLE_SHOW_WORKER_NOTIFICATIONS`) for quiet sessions. Warnings and errors are unaffected: model fallback/unavailability, context-window skips, no-output warnings, worker failures, compaction notifications, and `/blackhole*` command output stay visible.
+- **Hard elapsed timeout for background worker/model attempts.** Optional `workerAttemptTimeoutMs` bounds the complete Observer, Reflector, or Dropper agent loop for one selected model—including response-header waits, streamed heartbeats, tool turns, and final confirmation. Expiry aborts the attempt; configured candidates enter the normal cooldown and fallback chain, while the final session fallback is attempted once per stage without a persisted cooldown. Direct config and environment values above Node's 2,147,483,647 ms timer maximum are rejected. The setting persists through global, project, and session `/blackhole settings` saves and is also configurable via `PI_BLACKHOLE_WORKER_ATTEMPT_TIMEOUT_MS`; foreground Pi chats are unaffected. This is separate from `providerIdleTimeoutMs`, which remains a response-body inactivity timeout and cannot guarantee wall-clock failover.
+- **Memory workers end a finished run early and honor a prompt-cache preference.** The observer's `record_observations` and the reflector's `record_reflections` gained an optional `complete` flag: when the final batch is flagged `complete` and that batch rejected nothing, the host sends `terminate`, so a fully-processed run stops instead of spending the rest of its `agentMaxTurns` budget. The gate is per batch, so an incomplete batch — or a complete batch that rejected entries — keeps going and a later corrected batch can still close the run; the refusal is spelled out in the receipt so the model corrects instead of re-sending the same batch, and receipts report recorded/duplicate/rejected run totals. A model that omits the flag loses only the early stop — the batch still records. Closing with an empty `complete: true` batch is the single sanctioned "covered, nothing new" outcome and reports as info (`no_new_content`); both observer prompts instruct it, so a plain-text reply without a tool call stays a warning-level protocol slip. The reflector has no empty close (its batch is `minItems: 1`) and says so in the tool description. A new optional `cacheRetention` (`none` \| `short` \| `long`, unset = inherit pi's effective setting) is forwarded to all three worker streams so prompt caching can be tuned per install — case-insensitive on the file, env, and settings-modal paths, and adapters that do not support a value ignore it. `cacheRetention` is configurable in `/blackhole settings`, the config file, or `PI_BLACKHOLE_CACHE_RETENTION`; all three workers already scope their streams to the consolidation `sessionId`. The observer prompt keeps its append-stable prefix and never reintroduces local time ([upstream OM `#89`](https://github.com/elpapi42/pi-observational-memory/pull/89)).
+
+### Removed
+
+- **The dead `configure-overlay.ts` and `status-overlay.ts` TUI overlays are gone.** Neither had a production caller: `/blackhole configure` has routed to the pi-base settings modal (`/blackhole settings`) since the ConfigManager migration, and nothing ever opened the status overlay. Their test files go with them. User-facing behavior is unchanged — the settings modal is the only config UI, and `key-matcher.ts` (`visibleWidth`) stays for `/blackhole cleanup`.
+
+### Fixed
+
+- **Capped recall drill-downs tell the agent where to continue.** File/message drill-downs (`#N:path`, `#N:text`) returned their expansion uncapped, so a minified bundle or a verbose write dumped the whole payload into context, and the cap that did exist named a non-actionable `#N:path:offset:limit` placeholder. Every drill-down response is now bounded by `recallResponseMaxChars`, and `capDrillDownText()` appends a concrete continuation — `continue at #42:src/auth.ts:412:30` — naming the first line the cap did not show. The cut only ever lands on a line boundary, so following the hint neither skips nor repeats content; a line too big for the budget is named explicitly and paged past, and when no lines remain the footer points at a regex query instead. Paths containing colons round-trip through the parser unchanged.
+- **Dropper pool pressure now works end to end.** Pressure uses `observationsPoolMaxTokens` (the footer P-gauge basis), bypasses new-data gates, and evaluates the full live observation pool; `1.0` disables pressure.
+- **Git status annotations no longer follow a leaked `GIT_DIR`.** `loadGitFileTags` spawned `git` with the inherited environment, so an exported `GIT_DIR`/`GIT_WORK_TREE` (common when the agent runs inside a worktree) made `rev-parse --show-toplevel` resolve `cwd` but `status` read the _other_ repo's index — wrong [Files And Changes] tags, and on the test side `git config` actually wrote `user.name`/`user.email` into the ambient repo. Both call sites now run with `gitEnv()`, which strips the repo-location variables so discovery is anchored to `cwd`.
+- **Global and project `/blackhole settings` saves now persist `providerIdleTimeoutMs`.** The key was missing from the default key set the settings modal iterates for those scopes, so edits were silently dropped; session saves already wrote the full config and were unaffected. The setting now round-trips through save and reload at every scope while still inheriting Pi's global provider timeout when unset.
+- **`example-config.json` lists the whole config surface.** The file advertises `_all_settings_in_one_file` but had drifted: `showPreCompactionMessage`, `recallResponseMaxChars`, `compactAfterTokens`/`compactAfterRatio`/`compactReserveTokens`, `fullFoldAlways`, `statusBar`, `showWorkerNotifications`, and `dropperPoolFullnessThreshold` were all missing, so a reader had no reference for them beyond `docs/CONFIG.md`. Every key is now present with its DEFAULTS value, `_notes` and the env-override list cover the additions, and `cacheRetention` stays deliberately unset (documented in `_notes`) because pinning `short` downgrades a host whose effective cache setting is longer. A new `tests/example-config-completeness.test.ts` compares the file against `DEFAULTS`, so a future key cannot land without a reference here.
+
+---
+
 ## [0.5.8] - 2026-09-22
 
 ### Changed

package/README.md

@@ -2,7 +2,19 @@
 
 **Deterministic compaction + session-aware observational memory for [Pi](https://github.com/earendil-works/pi) — in one unified extension.**
 
-`/blackhole` replaces Pi's LLM-based `/compact` with an algorithmic structural summary — fast, zero-cost. Three background workers (Observer, Reflector, Dropper) capture durable facts and decisions that survive across compactions. Per-worker model fallback chains with persisted cooldowns. Manual flush mode. One JSON file to configure it all.
+`/blackhole` replaces Pi's LLM-based `/compact` with an algorithmic structural summary — fast, and the compaction step itself is zero-cost. Three background workers (Observer, Reflector, Dropper) run as separate billed model calls to capture durable facts and decisions that survive across compactions. Per-worker model fallback chains with persisted cooldowns. Manual flush mode. One JSON file to configure it all.
+
+> **A note from the maintainer: this is experimental.**
+>
+> It won't save you tokens by itself. Compacting is free — no model call, just structure. Remembering is not: the background workers wake up as your session grows, read what happened, and write down what seems worth keeping. Those are real billed calls. Untuned, on your main coding model, this costs more than doing nothing. It only earns its keep with the workers on something cheap or free.
+>
+> It also doesn't solve compaction — nobody has, as far as I know. It takes a different approach ([observational memory](https://mastra.ai/blog/observational-memory), via [pi-observational-memory](https://github.com/elpapi42/pi-observational-memory)) and merges it with [pi-vcc](https://github.com/sting8k/pi-vcc), because the two conflicted when installed together. I merged them and vibeslopped a lot on top: robustness work, new behavior, and bugfixes ported from both upstreams. The recap is pattern-matching, not understanding: it guesses at goals and preferences from their shape. The memory is model-written, so it has the opposite weakness: it can keep what sounds right but isn't. Assume both need the search tool as a backstop.
+>
+> Want only one half? Use the project that does just that half. No hard feelings.
+>
+> Scope stays tight on purpose: forks welcome, bug fixes welcome and reviewed, new features only if they serve conversation compaction itself — not the week's AI fad bolted on. Almost everything is a toggle, which I'll admit is tiring; I use a small slice myself. Memory on most days, off some days. Workers on free models, waking rarely, size-based auto-compaction as a backstop. Most days I don't compact at all — fresh sessions beat resumed ones.
+>
+> Best fit is someone who doesn't count every token: code locally and push the workers to free hosted models, or code on a top model and run the workers on a local card — knowing each worker's different instructions wipe the local prompt cache for the next one. No measurements, no comparisons against other methods. Fast and nearly free the way I run it, good enough for what I need.
 
 ---
 
@@ -34,13 +46,12 @@ Then `/reload` or restart Pi. The config file at `~/.pi/agent/pi-blackhole/pi-bl
 
 ## ✨ What's new
 
-> **Latest release: [0.5.8](CHANGELOG.md)**
+> **Latest release: [0.5.10](CHANGELOG.md)**
 >
-> - **Pi loads a prebuilt bundle** — `pi.extensions` points at the tsup `dist/index.js` instead of TypeScript source, cutting startup import time (500–570 ms → 350–530 ms). Registry installs ship `dist/`; git installs need `npmCommand` set, and a missing `dist/` now warns instead of failing silently.
-> - **Every observation-pool readout agrees** — the dropper trigger, `/blackhole-memory` pool lines, and the footer P gauge each summed the pool inline; they now share one helper, and `/blackhole-memory` includes (and labels) manual-mode pending batches so the display matches what the trigger gates on. No threshold or candidate behavior changes. ([#120](https://github.com/k0valik/pi-blackhole/issues/120))
-> - **`agentMaxTurns` is enforced on Pi 0.87** — the worker loops emitted only the removed `shouldStopAfterTurn`, so the turn budget was silently ignored; `createTurnCap` now emits both that hook and 0.87's `finishTurn`, with independent counters. ([upstream OM `#83`](https://github.com/elpapi42/pi-observational-memory/pull/83))
-> - **Extension registration survives class-based hosts** — hooks were invoked detached from their API object, which threw on hosts like oh-my-pi; handlers are now bound before registration. ([#124](https://github.com/k0valik/pi-blackhole/pull/124))
-> - **Custom-provider streams keep their receiver** — captured `streamSimple` handlers are bound to their config, so class-based providers no longer crash or silently fall back to the compat dispatcher. ([upstream OM `#80`](https://github.com/elpapi42/pi-observational-memory/pull/80))
+> - **Memory workers that fail now fall back instead of silently skipping** — observer/reflector/dropper stream errors and turn-cap cutoffs throw into the model fallback chain, so a failed run no longer advances the cursor over work it never finished.
+> - **The turn cap only ends runs that did tool work** — a run that stops naturally on exactly its last allowed turn keeps its result instead of failing as cap-cut, and a capped session model is not retried on an identical budget.
+> - **`/blackhole` on an ineligible branch reports nothing-to-compact** — instead of Pi's error card plus a second failure toast when the branch already fits inside the keep-recent budget.
+> - **A cancelled `/blackhole` stays quiet, and the command loads the config first** — no second vaguer toast on cancel, and a `manual`-mode session flushes pending memory instead of reading defaults.
 >
 > See [`CHANGELOG.md`](CHANGELOG.md) for the full history.
 
@@ -96,7 +107,7 @@ The agent gets one unified `recall` tool that handles every form of historical l
 
 When the agent expands a session entry (`#N`), related observations and reflections from the session ledger are automatically shown alongside the expanded content — so the agent gets the raw transcript _and_ the durable fact layer in one call.
 
-Every recall response is capped at `recallResponseMaxChars` (default 48,000 ≈ 12k tokens). Search snippet lines, expanded entries, and related observation bodies are clipped to keep a single huge stored message from flooding the context; a truncation marker names the omitted entries and how to continue (`#N:text` / `#N:path` / `page:N`).
+Every recall response is capped at `recallResponseMaxChars` (default 48,000 ≈ 12k tokens). Search snippet lines, expanded entries, drill-down bodies, and related observation bodies are clipped to keep a single huge stored message from flooding the context; a truncation marker names the omitted entries and how to continue (`#N:text` / `#N:path` / `page:N`). A capped drill-down cuts only at line boundaries and names the first line it did not show, so the next `#N:path:offset:limit` call continues there without skipping or repeating lines.
 
 The `/blackhole-recall` command exposes the same engine to the user. Results are shown as a collapsible message and auto-fed to the agent as context.