pi-blackhole 0.5.7 → 0.5.9

package/CHANGELOG.md

@@ -2,6 +2,44 @@
 
 ---
 
+## [0.5.9] - 2026-09-26
+
+### Added
+
+- **Routine worker notifications are now optional.** New `showWorkerNotifications` boolean (default `true`) gates the observer/reflector/dropper progress toasts — `observer running on ~N-token chunk`, `N observations recorded`, `reflector running`, `dropper running`, and the info-level `no observations` notice. Set it to `false` (settings modal → UI → Worker notifications, `/blackhole configure`, the config file, or `PI_BLACKHOLE_SHOW_WORKER_NOTIFICATIONS`) for quiet sessions. Warnings and errors are unaffected: model fallback/unavailability, context-window skips, no-output warnings, worker failures, compaction notifications, and `/blackhole*` command output stay visible.
+- **Hard elapsed timeout for background worker/model attempts.** Optional `workerAttemptTimeoutMs` bounds the complete Observer, Reflector, or Dropper agent loop for one selected model—including response-header waits, streamed heartbeats, tool turns, and final confirmation. Expiry aborts the attempt; configured candidates enter the normal cooldown and fallback chain, while the final session fallback is attempted once per stage without a persisted cooldown. Direct config and environment values above Node's 2,147,483,647 ms timer maximum are rejected. The setting persists through global, project, and session `/blackhole settings` saves and is also configurable via `PI_BLACKHOLE_WORKER_ATTEMPT_TIMEOUT_MS`; foreground Pi chats are unaffected. This is separate from `providerIdleTimeoutMs`, which remains a response-body inactivity timeout and cannot guarantee wall-clock failover.
+- **Memory workers end a finished run early and honor a prompt-cache preference.** The observer's `record_observations` and the reflector's `record_reflections` gained an optional `complete` flag: when the final batch is flagged `complete` and that batch rejected nothing, the host sends `terminate`, so a fully-processed run stops instead of spending the rest of its `agentMaxTurns` budget. The gate is per batch, so an incomplete batch — or a complete batch that rejected entries — keeps going and a later corrected batch can still close the run; the refusal is spelled out in the receipt so the model corrects instead of re-sending the same batch, and receipts report recorded/duplicate/rejected run totals. A model that omits the flag loses only the early stop — the batch still records. Closing with an empty `complete: true` batch is the single sanctioned "covered, nothing new" outcome and reports as info (`no_new_content`); both observer prompts instruct it, so a plain-text reply without a tool call stays a warning-level protocol slip. The reflector has no empty close (its batch is `minItems: 1`) and says so in the tool description. A new optional `cacheRetention` (`none` \| `short` \| `long`, unset = inherit pi's effective setting) is forwarded to all three worker streams so prompt caching can be tuned per install — case-insensitive on the file, env, and settings-modal paths, and adapters that do not support a value ignore it. `cacheRetention` is configurable in `/blackhole settings`, the config file, or `PI_BLACKHOLE_CACHE_RETENTION`; all three workers already scope their streams to the consolidation `sessionId`. The observer prompt keeps its append-stable prefix and never reintroduces local time ([upstream OM `#89`](https://github.com/elpapi42/pi-observational-memory/pull/89)).
+
+### Removed
+
+- **The dead `configure-overlay.ts` and `status-overlay.ts` TUI overlays are gone.** Neither had a production caller: `/blackhole configure` has routed to the pi-base settings modal (`/blackhole settings`) since the ConfigManager migration, and nothing ever opened the status overlay. Their test files go with them. User-facing behavior is unchanged — the settings modal is the only config UI, and `key-matcher.ts` (`visibleWidth`) stays for `/blackhole cleanup`.
+
+### Fixed
+
+- **Capped recall drill-downs tell the agent where to continue.** File/message drill-downs (`#N:path`, `#N:text`) returned their expansion uncapped, so a minified bundle or a verbose write dumped the whole payload into context, and the cap that did exist named a non-actionable `#N:path:offset:limit` placeholder. Every drill-down response is now bounded by `recallResponseMaxChars`, and `capDrillDownText()` appends a concrete continuation — `continue at #42:src/auth.ts:412:30` — naming the first line the cap did not show. The cut only ever lands on a line boundary, so following the hint neither skips nor repeats content; a line too big for the budget is named explicitly and paged past, and when no lines remain the footer points at a regex query instead. Paths containing colons round-trip through the parser unchanged.
+- **Dropper pool pressure now works end to end.** Pressure uses `observationsPoolMaxTokens` (the footer P-gauge basis), bypasses new-data gates, and evaluates the full live observation pool; `1.0` disables pressure.
+- **Git status annotations no longer follow a leaked `GIT_DIR`.** `loadGitFileTags` spawned `git` with the inherited environment, so an exported `GIT_DIR`/`GIT_WORK_TREE` (common when the agent runs inside a worktree) made `rev-parse --show-toplevel` resolve `cwd` but `status` read the _other_ repo's index — wrong [Files And Changes] tags, and on the test side `git config` actually wrote `user.name`/`user.email` into the ambient repo. Both call sites now run with `gitEnv()`, which strips the repo-location variables so discovery is anchored to `cwd`.
+- **Global and project `/blackhole settings` saves now persist `providerIdleTimeoutMs`.** The key was missing from the default key set the settings modal iterates for those scopes, so edits were silently dropped; session saves already wrote the full config and were unaffected. The setting now round-trips through save and reload at every scope while still inheriting Pi's global provider timeout when unset.
+- **`example-config.json` lists the whole config surface.** The file advertises `_all_settings_in_one_file` but had drifted: `showPreCompactionMessage`, `recallResponseMaxChars`, `compactAfterTokens`/`compactAfterRatio`/`compactReserveTokens`, `fullFoldAlways`, `statusBar`, `showWorkerNotifications`, and `dropperPoolFullnessThreshold` were all missing, so a reader had no reference for them beyond `docs/CONFIG.md`. Every key is now present with its DEFAULTS value, `_notes` and the env-override list cover the additions, and `cacheRetention` stays deliberately unset (documented in `_notes`) because pinning `short` downgrades a host whose effective cache setting is longer. A new `tests/example-config-completeness.test.ts` compares the file against `DEFAULTS`, so a future key cannot land without a reference here.
+
+---
+
+## [0.5.8] - 2026-09-22
+
+### Changed
+
+- **Observation-pool measurement is centralized.** The dropper trigger, `/blackhole-memory` pool lines, and footer P gauge each summed the active pool with their own inline copy; they now share `observationPoolTokens()` (`ledger/progress.ts`), which sums the live active pool plus — always explicitly — manual-mode pending observation batches. `/blackhole-memory` now includes those pending batches in its `Obs pool` / `Dropper:` percentages (so a manual-only user sees the same pool the trigger gates on) and labels the split (`· branch 0 + pending 1,400`); the footer P gauge deliberately stays branch-only. No trigger, threshold, or dropper-candidate behavior changes ([#120](https://github.com/k0valik/pi-blackhole/issues/120)).
+
+- **Pi loads the prebuilt `dist/index.js`.** `pi.extensions` now points at the tsup bundle instead of `./index.ts`, removing jiti transpilation of the whole module graph on startup (measured import 500 to 570 ms down to 350 to 530 ms, factory time unchanged). Registry installs ship `dist/` in the tarball. Git installs need `npmCommand` set so devDependencies install and `prepare` builds `dist/`; when they don't, `scripts/prepare.mjs` now warns that the extension will not load instead of failing silently, documented next to the GitHub install command in `README.md`.
+
+### Fixed
+
+- **`agentMaxTurns` is enforced on Pi 0.87.** The observer, reflector, and dropper capped their loops with `shouldStopAfterTurn`, which 0.87 removed in favour of `finishTurn` (whose `{ action: "end" }` ends the run), so the configured turn budget was silently ignored and a worker could keep running until it stopped naturally, errored, or was aborted. Both hooks are now emitted through `createTurnCap` with independent counters, so the loaded generation picks the one it knows and a host calling both still ends at `maxTurns`; a turn that already hard-exited (`error`/`aborted`) neither decides the run nor spends budget ([upstream OM `#83`](https://github.com/elpapi42/pi-observational-memory/pull/83)).
+- **Extension registration survives class-based `ExtensionAPI` hosts.** `registerCompactFailedHook` widened and then invoked `pi.on` detached, so a host whose `on` reads instance state (e.g. oh-my-pi's `ConcreteExtensionAPI`) threw `TypeError: undefined is not an object (evaluating 'this.extension')` during registration and took the rest of the extension's hooks down with it; the handler is now bound to `pi` before the widening cast, a no-op on pi's closure-based API ([#124](https://github.com/k0valik/pi-blackhole/pull/124)).
+- **Custom-provider streams keep their receiver when the bridge dispatches them.** Each `streamSimple` handler was stored in the global capture map detached from its provider config, and the api-only fallback for host-aliased provider ids extracted it the same way. A class-based config that reads instance state therefore threw `Cannot read properties of undefined`: the capture-map path crashed the worker stream, while the aliased path swallowed the error and silently degraded to the compat dispatcher. Captured handlers are now bound to their config, and the api-only match is invoked on it ([upstream OM `#80`](https://github.com/elpapi42/pi-observational-memory/pull/80)).
+
+---
+
 ## [0.5.7] - 2026-09-21
 
 ### Added

package/README.md

@@ -12,7 +12,10 @@
 # From npm (recommended)
 pi install npm:pi-blackhole
 
-# Or directly from GitHub
+# Or directly from GitHub.
+# Requires npmCommand to be set in settings.json, otherwise pi runs
+# `npm install --omit=dev`, devDependencies are skipped, and dist/ is not built.
+# Example: "npmCommand": ["npm"] in ~/.pi/agent/settings.json
 pi install git:github.com/k0valik/pi-blackhole
 ```
 
@@ -31,14 +34,16 @@ Then `/reload` or restart Pi. The config file at `~/.pi/agent/pi-blackhole/pi-bl
 
 ## ✨ What's new
 
-> **Latest release: [0.5.7](CHANGELOG.md)**
+> **Latest release: [0.5.9](CHANGELOG.md)**
 >
-> - **Pi 0.87 is supported** — inline compaction, the observer/reflector/dropper system prompts, and auto-compaction all work on `0.87.0`. The compact-shape guard follows 0.87's `_refreshFinalizedContext()` helper, the memory workers build whichever prompt carrier the loaded host reads, and `@earendil-works/*` moves to `0.87.0` with the dependabot hold removed. ([#117](https://github.com/k0valik/pi-blackhole/issues/117), [#118](https://github.com/k0valik/pi-blackhole/issues/118))
-> - **Live footer status bar** — `O` (transcript since the last observer run), `P` (observation pool fill) and `X` (context since the last compaction) gauges with color thresholds, plus worker spinners and compaction notes carrying their trigger reason. Config `statusBar`, default on. Remove the standalone `blackhole-status.ts` extension if you ran it: two writers race on the same status key. ([#113](https://github.com/k0valik/pi-blackhole/pull/113))
-> - **Inline compaction finds Pi's bundled host behind the 0.86 `createRequire` launcher** — the adapter follows the same-package bootstrap file without executing it, so it patches the `AgentSession` the host actually runs instead of the unused modular class, which fell back to settled compaction. ([#116](https://github.com/k0valik/pi-blackhole/pull/116))
-> - **Observer prompts are capped in every compaction mode** — `observerPreambleMaxTokens` now applies in `auto`/`off` too and covers reflections (newest first), and the pre-flight context guard prices the rendered preamble plus system prompt, so an oversized prompt skips the model cleanly instead of failing every attempt with a provider 400.
-
-See [`CHANGELOG.md`](CHANGELOG.md) for the full history.
+> - **Recall drill-downs can no longer flood your context** — every `#N:path` / `#N:text` expansion is capped by `recallResponseMaxChars`, and the cap names where to pick up (`continue at #42:src/auth.ts:412:30`), so a minified bundle or a huge write no longer dumps itself into the prompt.
+> - **Quiet sessions** — `showWorkerNotifications: false` silences routine observer/reflector/dropper progress toasts; warnings, errors, and `/blackhole*` command output stay visible.
+> - **Worker prompt caching is tunable per install** — `cacheRetention: none | short | long` is forwarded to all three worker streams, and unset inherits Pi's effective setting.
+> - **A hung worker model fails on a clock, not a hope** — new `workerAttemptTimeoutMs` bounds the whole agent loop (header waits, heartbeats, tool turns) per attempt; configured candidates then enter their normal cooldown and fallback chain.
+> - **Finished worker runs stop early** — `record_observations` / `record_reflections` accept a `complete` flag, so a fully-processed run ends instead of spending its `agentMaxTurns` budget.
+> - **Dropper pool pressure now works end to end** — gated on `observationsPoolMaxTokens` (the same pool the footer P gauge shows), it bypasses new-data gates and `1.0` disables it.
+>
+> See [`CHANGELOG.md`](CHANGELOG.md) for the full history.
 
 ---
 
@@ -92,7 +97,7 @@ The agent gets one unified `recall` tool that handles every form of historical l
 
 When the agent expands a session entry (`#N`), related observations and reflections from the session ledger are automatically shown alongside the expanded content — so the agent gets the raw transcript _and_ the durable fact layer in one call.
 
-Every recall response is capped at `recallResponseMaxChars` (default 48,000 ≈ 12k tokens). Search snippet lines, expanded entries, and related observation bodies are clipped to keep a single huge stored message from flooding the context; a truncation marker names the omitted entries and how to continue (`#N:text` / `#N:path` / `page:N`).
+Every recall response is capped at `recallResponseMaxChars` (default 48,000 ≈ 12k tokens). Search snippet lines, expanded entries, drill-down bodies, and related observation bodies are clipped to keep a single huge stored message from flooding the context; a truncation marker names the omitted entries and how to continue (`#N:text` / `#N:path` / `page:N`). A capped drill-down cuts only at line boundaries and names the first line it did not show, so the next `#N:path:offset:limit` call continues there without skipping or repeating lines.
 
 The `/blackhole-recall` command exposes the same engine to the user. Results are shown as a collapsible message and auto-fed to the agent as context.
 
@@ -294,6 +299,7 @@ rm -rf ~/.pi/agent/pi-blackhole
 | ------------------------------------------------------------ | ----------------- | ----------------------------------------------------------------------------------------- |
 | **[`README.md`](README.md)**                                 | You, now          | Install, commands, the pitch, the value, the demo.                                        |
 | **[`CHANGELOG.md`](CHANGELOG.md)**                           | You               | Every release, what changed, who contributed.                                             |
+| **[`CONTRIBUTING.md`](CONTRIBUTING.md)**                     | You, if helping   | Branch model, dev setup, PR description format, docs/changelog gates.                     |
 | **[`docs/CONFIG.md`](docs/CONFIG.md)**                       | You, when tuning  | Every config key with type, default, behavior, and env-var overrides.                     |
 | **[`llms.txt`](llms.txt)**                                   | Your agent        | Step-by-step guided setup interview, anti-patterns, exact file paths, internal constants. |
 | **[`docs/MIGRATION-GUIDE.md`](docs/MIGRATION-GUIDE.md)**     | You, if upgrading | Old → new config key mapping, semantic changes, automatic migration behavior.             |