pi-blackhole 0.3.0 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (6) hide show
  1. package/README.md +243 -121
  2. package/example-config.json +6 -2
  3. package/package.json +1 -1
  4. package/src/hooks/before-compact.ts +27 -1
  5. package/src/om/compaction-trigger.ts +59 -9
  6. package/src/om/debug-log.ts +5 -4
package/README.md CHANGED
@@ -1,91 +1,136 @@
1
1
  # pi-blackhole
2
2
 
3
- Algorithmic compaction + session-aware observational memory for [Pi](https://github.com/badlogic/pi-mono), in one extension.
3
+ **Algorithmic compaction + session-aware observational memory for [Pi](https://github.com/badlogic/pi-mono) — in one unified extension.**
4
4
 
5
- Combines [pi-vcc](https://github.com/sting8k/pi-vcc) and [pi-observational-memory](https://github.com/elpapi42/pi-observational-memory) with unified configuration, per-worker model fallback chains, persisted cooldowns, and a manual flush mode.
5
+ Blackhole merges the best ideas from [pi-vcc](https://github.com/sting8k/pi-vcc) and [pi-observational-memory](https://github.com/elpapi42/pi-observational-memory) into something that's become its own beast entirely. Deterministic compaction that costs nothing. A memory layer that survives compactions. Per-worker model fallback chains with persisted cooldowns. Manual flush mode. All configured from one JSON file.
6
6
 
7
- > This is a frankenmerge. I liked both extensions but they were not compatible - observational memory hooked into Pi's default compaction and prevented pi-vcc from working. So I merged them, made them share the same hook and output, and added the things both were missing: fallback chains, cooldowns, and a toggle between them.
8
- > Please also see the [`CHANGELOG.md`](CHANGELOG.md)
7
+ > **Why this exists:** I liked both extensions but they fought each other — OM hooked into Pi's default compaction and blocked vcc from working. So I merged them, made them share a single hook and output, and added everything both were missing: fallback chains, cooldowns, a memory toggle, and a manual mode for people who want to control when context gets compressed.
8
+ >
9
+ > The codebase has since diverged heavily from both upstreams, but tries to keep up-to-date with any fixes from them..
9
10
 
10
- ### Lockstep with upstreams
11
+ 📖 See [`CHANGELOG.md`](CHANGELOG.md) for release history.
12
+ ⚙️ See [`CONFIG.md`](CONFIG.md) for the full configuration reference.
13
+
14
+ ---
15
+
16
+ ## Quick start
17
+
18
+ ```bash
19
+ # Install from npm (recommended)
20
+ pi install npm:pi-blackhole
21
+
22
+ # Or directly from GitHub
23
+ pi install git:github.com/k0valik/pi-blackhole
24
+ ```
11
25
 
12
- pi-blackhole tracks both upstream repositories via a [lockstep audit system](https://github.com/k0valik/pi-blackhole/tree/lockstep/2026-05-27/.pi/skills/lockstep) that classifies every new upstream commit as safe-to-port, modified (needs review), rewritten (skip), or orphan (needs mapping). The goal is to lift bugfixes, prompts improvements, and compatible features without breaking existing users or rolling back intentional divergences. Ported changes are reviewed per-commit with human approval — nothing is blindly merged. See [SKILL.md](.pi/skills/lockstep/SKILL.md) for the full workflow. An example execution (including rationale for skipped changes) is documented in [PR #8](https://github.com/k0valik/pi-blackhole/pull/8).
26
+ If you have standalone pi-vcc or pi-observational-memory installed, remove them first — they conflict and will prevent blackhole from working. You don't loose any features from either extension:
13
27
 
14
- #### For easy setup pass the [`llms.txt`](llms.txt) llms.txt to your agent and it will guide you through the config without needing to read all the docs if you're as lazy as me.
28
+ ```bash
29
+ pi uninstall npm / git:https://github.com/sting8k/pi-vcc
30
+ pi uninstall npm / git:https://github.com/elpapi42/pi-observational-memory
31
+ ```
32
+
33
+ Then `/reload` or restart Pi.
34
+
35
+ ### Automated setup
15
36
 
37
+ Pass [`llms.txt`](llms.txt) to your agent and it will walk you through configuration step by step — no need to read all the docs.
16
38
 
17
- # Demo
39
+ ### Lockstep with upstreams
18
40
 
19
- - /blackhole collapses a 94k context-size conversation into a ~7.8k (for my settings) next input, shows the current stats for /blackhole-memory and /blackhole-recall (can be called by agent as well) augmenting the relevant neighbour observations and exact conversations so the agent can expand when it needs to remember previous conversation
41
+ pi-blackhole tracks both upstream repositories via a [lockstep audit system](.pi/skills/lockstep/SKILL.md). Every new commit from [pi-vcc](https://github.com/sting8k/pi-vcc) and [pi-observational-memory](https://github.com/elpapi42/pi-observational-memory) is classified as safe-to-port, modified (needs review), rewritten (skip), or orphan (needs mapping). Bugfixes and compatible improvements get ported; intentional divergences stay. Nothing is blindly merged — every ported change is reviewed per-commit with human approval. See `.pi/skills/lockstep/` for the full workflow.
42
+
43
+ ---
44
+
45
+ ## Demo
46
+
47
+ `/blackhole` collapses ~94k tokens of conversation into a ~7.8k structured summary (YMMV based on your settings). `/blackhole-memory` shows pipeline status. `/blackhole-recall` searches history the agent can also reach via its `recall` tool.
20
48
 
21
49
  https://github.com/user-attachments/assets/ab7c5787-1bbd-466f-a231-0818e68e9f39
22
50
 
51
+
23
52
  ---
24
53
 
25
- ## The problem
54
+ ## The problem it solves
26
55
 
27
- Long AI engineering sessions inevitably degrade. Pi's native compaction relies on an LLM to generate free-form prose summaries recursively. Pi compacts the conversation — then compacts that summary — then compacts again. After enough cycles, small but load-bearing details disappear: why a decision was made, what approaches were already rejected, what the user clarified earlier in the session.
56
+ Long engineering sessions degrade. Pi's native compaction calls an LLM to write free-form prose summaries — then compacts those summaries, then compacts *those* summaries. After enough cycles, load-bearing details vanish: why a decision was made, what approaches were already rejected, what the user clarified earlier.
28
57
 
29
58
  The session is still alive. The agent is no longer carrying the real context.
30
59
 
31
- The two upstream projects solve different halves of this:
60
+ The two upstream projects each solve one half:
32
61
 
33
- - **pi-vcc** replaces Pi's LLM-based compaction with a deterministic, zero-cost algorithmic summary. Fast, reproducible, no hallucination risk. But the compacted output is still a summary — repeated compactions still erode detail.
34
- - **pi-observational-memory** captures observations and reflections in a session ledger that survives compactions. But its compaction path still calls an LLM.
62
+ - **pi-vcc** replaces Pi's LLM-based compaction with a deterministic, zero-cost algorithmic summary. Fast, reproducible, no hallucination risk. But repeated compactions still erode detail — it's still a summary.
63
+ - **pi-observational-memory** captures timestamped observations and durable reflections in a session ledger that survives across compactions. But its compaction path still calls an LLM — costing money and risking drift on every compact.
35
64
 
36
65
  **pi-blackhole** puts vcc in the compaction slot and OM in the memory layer, where each does what it's designed for.
37
66
 
67
+
38
68
  ---
39
69
 
40
- ## Feature Comparison
70
+ ## Feature comparison
41
71
 
42
72
  | | pi-blackhole | pi-vcc | pi-obs-memory | Pi default |
43
73
  |---|---|---|---|---|
44
- | Algorithmic compaction (no LLM) | ✓ | ✓ | — | — |
74
+ | Algorithmic compaction (no LLM cost) | ✓ | ✓ | — | — |
45
75
  | Deterministic output | ✓ | ✓ | — | — |
46
76
  | Structured summary sections | ✓ | ✓ | — | — |
47
77
  | Observations + reflections | ✓ | — | ✓ | — |
48
78
  | Context survives across compactions | ✓ | — | ✓ | — |
49
- | Background workers | ✓ | — | ✓ | — |
79
+ | Background memory workers | ✓ | — | ✓ | — |
50
80
  | Searchable history after compaction | ✓ | ✓ | partial | — |
51
81
  | Per-worker model config | ✓ | — | — | — |
52
82
  | Fallback model chains + persisted cooldowns | ✓ | — | — | — |
53
83
  | Manual flush mode (`noAutoCompact`) | ✓ | — | — | — |
54
84
  | Memory toggle (`/blackhole om-off`) | ✓ | — | — | — |
55
85
  | Unified single-file config | ✓ | — | — | — |
86
+ | Per-session pending state | ✓ | — | — | — |
87
+
56
88
 
57
89
  ---
58
90
 
59
91
  ## How it works
60
92
 
61
- On `/blackhole`, the vcc pipeline analyzes the transcript tail and produces a structured summary: session goal, file changes, commits, outstanding blockers, user preferences, and a rolling brief transcript. Accumulated observations and reflections from the session ledger are rendered and appended below it. The agent receives a deterministic recap of recent work and durable facts from the full session history in a single replacement block.
93
+ When you run `/blackhole` (or when auto-compaction fires), two things happen in one shot:
94
+
95
+ 1. **The vcc pipeline** analyzes the transcript tail and produces a structured summary: session goal, file changes, commits, outstanding blockers, user preferences, and a rolling brief transcript.
96
+ 2. **Observational memory injection** renders accumulated observations and reflections from the session ledger and appends them below the summary.
62
97
 
63
- Three background workers from `pi-observational-memory` run during the session:
98
+ The agent receives a deterministic recap of recent work *plus* durable facts from the full session history — in a single replacement block. No LLM was called for the compaction itself.
64
99
 
65
- - **Observer** — reads conversation since the last observation marker and extracts timestamped facts: events, decisions, preferences. Input is capped to `observerChunkMaxTokens` newest-first to prevent context blowup on long sessions.
66
- - **Reflector** — distills new observations into durable reflections: stable facts, patterns, and constraints that survive future compactions.
100
+ ### The three memory workers
101
+
102
+ Three background workers run automatically during the session (when `memory: true`, which is the default):
103
+
104
+ - **Observer** — reads conversation since the last observation marker and extracts timestamped facts: events, decisions, preferences. Input is capped to `observerChunkMaxTokens` newest-first to prevent context blowup on long sessions. Runs most frequently.
105
+ - **Reflector** — distills new observations into durable reflections: stable facts, patterns, and constraints that survive future compactions. Runs less often.
67
106
  - **Dropper** — prunes low-value observations from active memory when the pool exceeds `observationsPoolMaxTokens`, while keeping reflections and other long-term elements safely in the session ledger.
68
107
 
69
108
  ```
70
- [Conversation turn] ──> observeAfterTokens threshold
109
+ [Conversation turn] ──> (accumulated tokens >= observeAfterTokens)
71
110
  │
72
111
  v
73
112
  1. OBSERVER
74
- (extracts observations)
113
+ (extracts timestamped observations via agent loop)
75
114
  │
76
115
  v
77
116
  2. REFLECTOR
78
- (distills reflections)
117
+ (synthesizes durable reflections via agent loop)
79
118
  │
80
119
  v
81
120
  3. DROPPER
82
- (prunes observations, keeps reflections)
121
+ (prunes low-value observations, keeps reflections)
83
122
  ```
84
123
 
85
- The pipeline runs in order: observer -> reflector -> dropper. If any stage fails, remaining stages are skipped and the pipeline retries on the next `agent_start` or `turn_end`. A minimum 30-second interval is enforced between failed runs.
124
+ Each worker uses an `agentLoop` with tool-calling capabilities — they don't just make a single LLM call. The observer, for example, can call `record_observations` multiple times per run to work through a chunk incrementally.
125
+
126
+ ### Graceful degradation
127
+
128
+ If any stage fails (model error, rate limit, timeout), remaining stages are skipped and the full pipeline retries on the next `agent_start` or `turn_end`. A 30-second retry gate prevents hammering failing APIs. Within each stage, the runtime tries all configured fallback models before giving up — each failed model is cooled down and skipped in subsequent attempts.
86
129
 
87
130
  ### What the agent sees after compaction
88
131
 
132
+ After compaction, the agent sees something like this (sections appear only when relevant — a session with no git commits won't show `[Commits]`):
133
+
89
134
  ```
90
135
  [Session Goal]
91
136
  - Fix the authentication bug in login flow
@@ -115,12 +160,28 @@ Root cause is a missing token refresh...
115
160
 
116
161
  ---
117
162
 
163
+ ## Reflections
164
+ [c3d4e5f6a1b2] User is building Acme Dashboard on Next.js 15 with Supabase auth.
165
+
166
+ ## Observations
118
167
  [a1b2c3d4e5f6] 2026-05-23 [high] User decided to switch from REST to GraphQL; motivation was reducing over-fetching.
119
168
  [b2c3d4e5f6a1] 2026-05-23 [medium] GraphQL migration completed; user confirmed working.
120
- [c3d4e5f6a1b2] Reflection: User is building Acme Dashboard on Next.js 15 with Supabase auth.
169
+
170
+ ----
171
+ Use `recall` to search for prior work, decisions, and context from before this summary. Do not redo work already completed.
172
+
173
+ These are condensed memories from earlier in this session.
174
+
175
+ - Reflections: stable, long-lived facts about the user, project, decisions, and constraints.
176
+ - Observations: timestamped events from the conversation history, in chronological order.
177
+
178
+ Treat these as past records. When entries conflict, the most recent observation reflects the latest known state. Work that prior observations describe as completed should not be redone unless the user explicitly asks to revisit it.
179
+
180
+ When exact source context is needed for precision or traceability, use the `recall` tool with the relevant observation or reflection id.
181
+ ----
121
182
  ```
122
183
 
123
- Sections appear only when relevant — a session with no git commits won't have `[Commits]`.
184
+ > **Note:** The OM injection format uses `## Reflections` and `## Observations` Markdown headers with a preamble and footer. Each observation and reflection has a 12-char hex identifier you can use with the `recall` tool to recover source evidence.
124
185
 
125
186
  ---
126
187
 
@@ -129,31 +190,37 @@ Sections appear only when relevant — a session with no git commits won't have
129
190
  Two modes, one shared goal: keep your agent's context sharp without manual housekeeping.
130
191
 
131
192
  - **Auto mode (default):** install and forget. Workers run, observations are appended as invisible conversation markers, compaction fires automatically when tokens exceed threshold.
132
- - **Manual mode (`noAutoCompact: true` — the maintainer's setup):** same workers, same pipeline. But observations go to a disk buffer and compaction only happens when you run `/blackhole`. Cleaner conversation, manual schedule.
193
+ - **Manual mode (`noAutoCompact: true` — the maintainer's daily driver):** same workers, same pipeline. But observations go to per-session disk buffers and compaction only happens when you run `/blackhole`. Cleaner conversation, manual schedule.
133
194
 
134
195
  The tradeoff is simplicity vs cleanliness:
135
196
 
136
197
  | | Auto (default) | Manual (`noAutoCompact: true`) |
137
198
  |---|---|---|
138
199
  | Workers run? | Yes | Yes |
139
- | Observations go to | Conversation markers (invisible in TUI) | Disk (`pending.json`) |
140
- | Observations accumulate across runs | Branch markers (replaced each cycle) | Pending batches — `/memory` shows pending counts |
200
+ | Observations go to | Conversation markers (invisible in TUI) | Disk (`<sessionId>-pending.json`) |
201
+ | Observations accumulate across runs | Branch markers (replaced each cycle) | Pending batches accumulated — `/blackhole-memory` shows pending counts |
141
202
  | Auto-compact on `agent_end` | Yes | No |
142
203
  | `/blackhole` | Optional — use it whenever you want | Required to flush + compact |
143
- | Conversation history | OM marker entries between turns (they exist but do not clutter the display) | Clean — nothing between turns |
204
+ | Conversation history | OM marker entries between turns (they exist but don't clutter the display) | Clean — nothing between turns |
144
205
  | Use case | "I don't want to think about it" | "I want to control when context gets compressed" |
145
206
 
146
207
  **Does `/blackhole` work like a single `/compact` that Just Works?**
147
208
 
148
- Yes, that's exactly the idea, especially in manual mode. When you feel context is getting full or accuracy is slipping, type `/blackhole`. It flushes any accumulated observations from disk, runs algorithmic vcc compaction (zero LLM cost), and injects your durable reflections into the replacement block. One command, everything gets compressed while keeping your long-term memory alive.
209
+ Yes, that's exactly the idea, especially in manual mode. When you feel context is getting full or accuracy is slipping, type `/blackhole`. It flushes any accumulated observations from disk, runs algorithmic vcc compaction (zero LLM cost), and injects your durable reflections into the replacement block. One command, everything gets compressed while keeping your session memory alive.
149
210
 
150
211
  The difference from Pi's built-in `/compact`:
151
212
  - `/compact` calls an LLM to write a free-form summary — costly, lossy, no memory layer.
152
213
  - `/blackhole` uses algorithmic section extraction (goals, files, commits, preferences...) plus injects observations/reflections from the session ledger. No LLM involved in the compaction itself. Fast, deterministic, memory-preserving.
153
214
 
154
- ### Without observational memory
215
+ ### Passive mode
216
+
217
+ Set `passive: true` (or the environment variable `PI_BLACKHOLE_PASSIVE=true`) to completely disable all background workers AND auto-compaction. The extension becomes a no-op until you explicitly run `/blackhole`. This is useful for debugging or if you want zero background activity.
218
+
219
+ ### Without observational memory (vcc-only)
220
+
221
+ Set `memory: false` or run `/blackhole om-off` for pure vcc compaction — no background workers, no memory injection. The compaction still uses the algorithmic vcc pipeline (not Pi's LLM-based compaction). Re-enable with `/blackhole om-on` or setting `memory: true`.
155
222
 
156
- Set `memory: false` or run `/blackhole om-off` for pure pi-vcc compaction — no background workers, no memory injection. Re-enable with `/blackhole om-on` or `memory: true`.
223
+ This is a lighter alternative to `passive`: workers are off but auto-compaction still runs.
157
224
 
158
225
  ---
159
226
 
@@ -161,25 +228,35 @@ Set `memory: false` or run `/blackhole om-off` for pure pi-vcc compaction — no
161
228
 
162
229
  | Command | What it does |
163
230
  |---|---|
164
- | `/blackhole` | Compact. `om-off` / `om-on` toggle observational memory. |
165
- | `/blackhole-memory` | Pipeline status: token progress, worker counts, last errors |
166
- | `/blackhole-memory view` | Visible observations and reflections, copied to clipboard |
167
- | `/blackhole-memory full` | Complete recorded memory, copied to clipboard |
231
+ | `/blackhole` | Compact the conversation. Subcommands: `om-off` / `om-on` toggle observational memory. |
232
+ | `/blackhole-memory` (or `status`) | Pipeline status: token progress, observation/reflection counts, pending data, last errors |
233
+ | `/blackhole-memory view` | Show visible observations and reflections (after compaction trimming), copied to clipboard |
234
+ | `/blackhole-memory full` | Show ALL recorded memory (including dropped observations), copied to clipboard |
168
235
  | `/blackhole-recall <query>` | Search session history. Supports `page:N`, `scope:all`, `mode:file|transcript|touched` |
169
236
 
170
237
  ## Tools
171
238
 
172
- | Tool | Input | Returns |
173
- |---|---|---|
174
- | `recall` | `[12char hex]` — source evidence; `#N` — expand entry; `#N:path` — drill-down file content; free text — BM25+regex ranked search; `mode:file|transcript|touched` — filter or aggregate | Source message(s) with timestamps; file content on drill-down; aggregated file list on touched mode |
239
+ The agent gets a unified `recall` tool that handles three types of input:
240
+
241
+ | Input | What it does |
242
+ |---|---|
243
+ | `[12-char hex]` | Recover source evidence for a specific observation or reflection ID from the session ledger |
244
+ | `#N` | Expand a session entry by index (show full content) |
245
+ | `#N:path` | Drill-down into file content from a tool call (e.g. `#42:auth.ts` shows first 30 lines; `#42:auth.ts:30` shows next 30; `#42:auth.ts:full` shows everything) |
246
+ | Free text | BM25-ranked OR search across transcript and/or file content. Rare terms weighted higher. |
247
+ | `mode:file` | Search only write/edit file content |
248
+ | `mode:transcript` | Search only conversation text |
249
+ | `mode:touched` | Aggregate all files written/edited, grouped by path with entry indices |
250
+ | Regex | Pattern search (e.g. `fork.*pi-vcc`, `hook|inject`) |
251
+ | `scope:all` | Search across all session lineages, not just the active one |
175
252
 
176
253
  ---
177
254
 
178
255
  ## Configuration
179
256
 
180
- All settings in **`~/.pi/agent/pi-blackhole/pi-blackhole-config.json`** — auto-created with defaults on first startup. See [`CONFIG.md`](CONFIG.md) for the full reference and tuning guidance. An annotated example config with explanations is at [`example-config.json`](example-config.json).
257
+ All settings in a single JSON file: **`~/.pi/agent/pi-blackhole/pi-blackhole-config.json`** — auto-created with defaults on first startup. See [`CONFIG.md`](CONFIG.md) for the full reference with detailed explanations for every knob. An annotated example config is at [`example-config.json`](example-config.json).
181
258
 
182
- Quick start with custom models:
259
+ Quick start — just set custom models (if you want):
183
260
 
184
261
  ```json
185
262
  {
@@ -189,36 +266,41 @@ Quick start with custom models:
189
266
  }
190
267
  ```
191
268
 
269
+ That's it. Everything else has sensible defaults.
270
+
192
271
  ### Settings at a glance
193
272
 
194
- | Setting | Default (medium) | What it controls |
273
+ | Setting | Default | What it controls |
195
274
  |---|---|---|
196
- | `overrideDefaultCompaction` | `false` | Route all Pi compactions through blackhole, not just explicit `/blackhole` |
197
- | `noAutoCompact` | `false` | Manual mode: save to disk, disable auto-compaction |
198
- | `memory` | `true` | `false` = pi-vcc only, no workers, no OM injection |
199
- | `passive` | `false` | Disable all workers and auto-compaction entirely |
200
- | `model` | — | Base fallback model for all workers |
201
- | `observerModel` / `observerFallbackModels` | — / `[]` | Primary + fallbacks for observer |
202
- | `reflectorModel` / `reflectorFallbackModels` | — / `[]` | Primary + fallbacks for reflector |
203
- | `dropperModel` / `dropperFallbackModels` | — / `[]` | Primary + fallbacks for dropper |
204
- | `observeAfterTokens` | `10000` | Token cadence between observer runs |
205
- | `reflectAfterTokens` | `20000` | Token cadence for reflector and dropper |
206
- | `compactAfterTokens` | `81000` | Auto-compaction threshold |
275
+ | `overrideDefaultCompaction` | `false` | Route ALL Pi compactions through blackhole, not just explicit `/blackhole` |
276
+ | `noAutoCompact` | `false` | Manual mode: save observations to disk, disable auto-compaction |
277
+ | `memory` | `true` | `false` = vcc only, no workers, no OM injection |
278
+ | `passive` | `false` | Disable all workers AND auto-compaction entirely |
279
+ | `model` | — | Base fallback model for all workers (last resort before session model) |
280
+ | `observerModel` / `observerFallbackModels` | — / `[]` | Primary + fallback models for observer (extracts facts) |
281
+ | `reflectorModel` / `reflectorFallbackModels` | — / `[]` | Primary + fallback models for reflector (synthesizes reflections) |
282
+ | `dropperModel` / `dropperFallbackModels` | — / `[]` | Primary + fallback models for dropper (prunes observations) |
283
+ | *(per model)* `thinking` | `"low"` | Thinking/reasoning level: `off`, `minimal`, `low`, `medium`, `high`, `xhigh` |
284
+ | *(per model)* `cooldownHours` | `1` | How long to skip this model after a retryable error |
285
+ | `observeAfterTokens` | `15000` | Min accumulated tokens before observer runs |
286
+ | `reflectAfterTokens` | `25000` | Min accumulated tokens before reflector + dropper run |
287
+ | `compactAfterTokens` | `81000` | Auto-compaction threshold (when `noAutoCompact` is `false`) |
207
288
  | `observerChunkMaxTokens` | `40000` | Max observer input per run (newest-first) |
208
- | `observerPreambleMaxTokens` | `0` (auto) | Max preamble tokens in observer prompt for `noAutoCompact` mode (auto = 30% of chunk) |
289
+ | `observerPreambleMaxTokens` | `0` (auto) | Preamble cap for observer in `noAutoCompact` mode (auto = 30% of chunk) |
209
290
  | `observationsPoolMaxTokens` | `20000` | Max active observation pool before dropper prunes |
291
+ | `observationsPoolTargetTokens` | `10000` | Target size dropper aims for after pruning (derived: half of pool max) |
210
292
  | `reflectorInputMaxTokens` | `80000` | Max reflector input budget |
211
293
  | `dropperInputMaxTokens` | `80000` | Max dropper input budget |
212
- | `agentMaxTurns` | `16` | Max turns per worker agent loop |
294
+ | `agentMaxTurns` | `16` | Max agent-loop turns per worker per run |
213
295
  | `debug` | `false` | Pre-compaction snapshot to `/tmp/pi-blackhole-debug.json` |
214
- | `debugLog` | `false` | Continuous JSONL log to `~/.pi/agent/pi-blackhole/debug.ndjson` |
296
+ | `debugLog` | `false` | Continuous JSONL debug log to `~/.pi/agent/pi-blackhole/debug.ndjson` |
215
297
 
216
- Env override: `PI_BLACKHOLE_PASSIVE=true` disables all workers without touching the config file.
298
+ **Environment override:** `PI_BLACKHOLE_PASSIVE=true` disables all workers without touching the config file.
217
299
 
218
300
  ### Configuration presets
219
301
 
220
302
  The defaults above target a **medium-context** setup (~128k context window, e.g. GPT-4o, Claude Sonnet).
221
- Paste the appropriate block into your config to match your model's context size.
303
+ Paste the appropriate block into your config to match your main session model's context size.
222
304
 
223
305
  #### Low context (~32k-64k — older models, fast budget models)
224
306
 
@@ -237,12 +319,12 @@ Paste the appropriate block into your config to match your model's context size.
237
319
 
238
320
  #### Medium context (~128k — GPT-4o, Claude Sonnet, Gemini Pro; this is the default)
239
321
 
240
- Our built-in defaults already target this tier. If you reset your config, these are what you get:
322
+ These are the built-in defaults. If you reset your config, these are what you get:
241
323
 
242
324
  ```json
243
325
  {
244
- "observeAfterTokens": 10000,
245
- "reflectAfterTokens": 20000,
326
+ "observeAfterTokens": 15000,
327
+ "reflectAfterTokens": 25000,
246
328
  "compactAfterTokens": 81000,
247
329
  "observerChunkMaxTokens": 40000,
248
330
  "observerPreambleMaxTokens": 0,
@@ -267,19 +349,31 @@ Our built-in defaults already target this tier. If you reset your config, these
267
349
  }
268
350
  ```
269
351
 
270
- **What to tune first:** `compactAfterTokens` should be significantly below your model's total context window — aim for ~60-70%. If you find the agent loses context before compaction fires, lower it. If compaction fires too often and breaks flow, raise it. The other thresholds scale proportionally with this value.
352
+ **What to tune first:** `compactAfterTokens` should be significantly below your model's total context window — aim for ~60-70%. If the agent loses context before compaction fires, lower it. If compaction fires too often and breaks flow, raise it. The other thresholds scale proportionally.
353
+
354
+ ### Tip: comments in config
355
+
356
+ The config preserves unknown keys, so you can add `_comment` or `_notes` fields to document your choices inline. They're ignored by the parser.
357
+
358
+ ```json
359
+ {
360
+ "_comment": "Tuned for my Cerebras + OpenRouter free model setup",
361
+ "observerModel": { "provider": "openrouter", "id": "qwen/qwen3-next-80b-a3b-instruct:free", "thinking": "low" }
362
+ }
363
+ ```
271
364
 
272
365
  ---
273
366
 
274
367
  ## Model fallback chains
275
368
 
276
- Each worker has a primary model and an ordered fallback list. On any error — rate limit, timeout, API failure — the failed model is cooled down and the next candidate is tried. If all candidates are exhausted, the pipeline aborts and retries on the next trigger event. The session model is always the last resort and is never cooled down.
369
+ Each worker has a primary model and an ordered fallback list. On any error — rate limit, timeout, API failure, 5xx — the failed model is cooled down and the next candidate is tried. If all candidates are exhausted, the pipeline aborts and retries on the next trigger event. The session model is always the last resort and is never cooled down.
277
370
 
278
371
  ```
279
- [Worker fails: 429 / timeout / 5xx]
372
+ [Worker fails: 429 / timeout / 5xx / connection error]
280
373
  │
281
374
  v
282
375
  Add model to cooldown list
376
+ (persisted to pi-blackhole-cooldown.json)
283
377
  │
284
378
  v
285
379
  Try next fallback candidate
@@ -293,16 +387,54 @@ Each worker has a primary model and an ordered fallback list. On any error — r
293
387
  Fall back to session model (never cooled down)
294
388
  ```
295
389
 
296
- Cooldowns are persisted to `~/.pi/agent/pi-blackhole/pi-blackhole-cooldown.json` and survive Pi restarts. Each entry records the model key, the triggering error, the responsible stage, and the expiry timestamp.
390
+ Cooldowns survive Pi restarts — they're persisted to `~/.pi/agent/pi-blackhole/pi-blackhole-cooldown.json`. Each entry records the model identifier, the triggering error, which stage failed, and the expiry timestamp.
391
+
392
+ ### Resolution order
393
+
394
+ For each stage, the runtime builds a candidate list from:
395
+
396
+ 1. **Primary stage model** (`observerModel`, `reflectorModel`, `dropperModel`)
397
+ 2. **Stage fallback models** (`observerFallbackModels`, etc.) — tried in order
398
+ 3. **Base model** (`model` — shared across all workers)
399
+ 4. **Session model** (the model used for your main conversation — always the last resort)
400
+
401
+ Models with active cooldowns are transparently skipped. The runtime tries up to 10 model resolutions per stage before giving up entirely.
297
402
 
298
- With a full config, resolution order looks like:
403
+ ### Example: full resolution chain
404
+
405
+ With a fully configured setup:
299
406
 
300
407
  ```
301
- Observer: qwen3-next-80b → gemma4:31b-cloud → gemma-4-31b-it:free → base model → session model
302
- Reflector: gpt-oss-120b (cerebras) → glm-4.7 (z.ai) → gpt-oss-120b:free → base model → session model
303
- Dropper: gpt-oss-120b (cerebras) → glm-4.7 (z.ai) → gpt-oss-120b:free → base model → session model
408
+ Observer: qwen3-next-80b (openrouter) → gemma4:31b-cloud (ollama) → gemma-4-31b-it:free (openrouter) → base model → session model
409
+ Reflector: gpt-oss-120b (cerebras) → glm-4.7 (z.ai) → gpt-oss-120b:free (openrouter) → base model → session model
410
+ Dropper: gpt-oss-120b (cerebras) → glm-4.7 (z.ai) → gpt-oss-120b:free (openrouter) → base model → session model
304
411
  ```
305
412
 
413
+ ### Per-model thinking levels
414
+
415
+ Each model config supports a `thinking` field that controls reasoning effort:
416
+
417
+ ```json
418
+ {
419
+ "observerModel": {
420
+ "provider": "openrouter",
421
+ "id": "qwen/qwen3-next-80b-a3b-instruct:free",
422
+ "thinking": "low", ← reasoning effort for this specific model
423
+ "cooldownHours": 12 ← custom cooldown duration
424
+ }
425
+ }
426
+ ```
427
+
428
+ Valid values: `off`, `minimal`, `low`, `medium`, `high`, `xhigh`. Not all models support every level.
429
+
430
+ ### Retryable error detection
431
+
432
+ The runtime uses a regex to detect retryable errors — it looks for patterns like `rate limit`, `429`, `5xx`, `timeout`, `service unavailable`, `connection error`, `websocket closed`, etc. Non-retryable errors (auth failures, invalid model IDs) immediately skip that candidate and move to the next.
433
+
434
+ ### 30-second retry gate
435
+
436
+ After any stage fails completely, the pipeline waits 30 seconds before attempting another consolidation run. This prevents rapid retry loops that would waste API calls on the same failing models.
437
+
306
438
  ---
307
439
 
308
440
  ## Recall
@@ -311,64 +443,42 @@ Pi's default compaction discards old messages permanently — after compaction,
311
443
 
312
444
  ### `recall` tool (agent-facing)
313
445
 
314
- Searches the raw session JSONL directly, bypassing compaction. Default scope is the active conversation lineage.
446
+ The agent gets one unified tool that searches session history, expands entries, drills into file content, and looks up observational memory. Searches read the raw session file directly, bypassing compaction.
315
447
 
316
448
  | Input | What it does |
317
449
  |---|---|
318
- | `[12char hex]` | Recover source evidence for an observation or reflection |
319
- | `#N` | Expand a transcript entry by index |
320
- | `#N:path` | Drill-down into file content from a tool call (e.g. `#42:auth.ts`) |
321
- | `#N:path:full` | Show all lines of a file (vs. default preview of first 30) |
322
- | `#N:path:offset:limit` | Paginated file content (e.g. `#42:auth.ts:30` = next 30 lines) |
323
- | Free text | BM25-ranked OR search, rare terms weighted higher |
324
- | `mode:file` | Search only write/edit file content (not transcript text) |
325
- | `mode:transcript` | Search only conversation text (not file content) |
326
- | `mode:touched` | Aggregate view of all files written/edited, grouped by path with entry indices |
450
+ | `[12-char hex]` | Recover source evidence for an observation or reflection ID from the session ledger |
451
+ | `#N` | Expand a session entry by index (show full content, not truncated) |
452
+ | `#N:path` | Drill-down into file content from a tool call (e.g. `#42:auth.ts` shows first 30 lines; `#42:auth.ts:30` shows next 30; `#42:auth.ts:full` shows everything) |
453
+ | Free text | BM25-ranked OR search across transcript + file indicators. Rare terms weighted higher. |
454
+ | `mode:file` | Search only write/edit file content |
455
+ | `mode:transcript` | Search only conversation text |
456
+ | `mode:touched` | Aggregate all files written/edited across the session, grouped by path with entry indices |
327
457
  | Regex | Pattern search (e.g. `fork.*pi-vcc`, `hook\|inject`) |
458
+ | `scope:all` | Search across all session lineages (default: active lineage only) |
459
+
460
+ **OM coupling:** When expanding session entries (`#N`), the tool automatically looks up related observations and reflections from the session ledger. If any of your expanded entries are referenced as source evidence by an observation, those observations are shown alongside the expanded content.
328
461
 
329
462
  ### `/blackhole-recall` command (user-facing)
330
463
 
331
- Results are shown as a collapsible message and auto-fed to the agent as context.
464
+ Results are shown as a collapsible message and auto-fed to the agent as context. Same engine as the `recall` tool.
332
465
 
333
466
  ```
334
- /blackhole-recall auth token # active-lineage search, ranked
335
- /blackhole-recall auth token page:2 # paginated (5 results/page)
336
- /blackhole-recall hook|inject # regex
337
- /blackhole-recall fail.*build scope:all # regex across all lineages
338
- /blackhole-recall mode:file # search only write/edit file content
339
- /blackhole-recall mode:transcript # search only conversation
340
- /blackhole-recall mode:touched # aggregate view of all files touched
341
- /blackhole-recall # recent 25 entries
467
+ /blackhole-recall auth token # active-lineage search, ranked
468
+ /blackhole-recall auth token page:2 # paginated (5 results/page)
469
+ /blackhole-recall hook|inject # regex
470
+ /blackhole-recall fail.*build scope:all # regex across all lineages
471
+ /blackhole-recall mode:file # search only write/edit file content
472
+ /blackhole-recall mode:transcript # search only conversation
473
+ /blackhole-recall mode:touched # aggregate view of all files touched
474
+ /blackhole-recall # recent 25 entries
342
475
  ```
343
476
 
344
- ---
345
-
346
- ## Install
477
+ ### Details
347
478
 
348
- ```bash
349
- # From npm (recommended)
350
- pi install npm:pi-blackhole
351
-
352
- # Or directly from GitHub
353
- pi install git:github.com/k0valik/pi-blackhole
354
- ```
479
+ **File drill-down** reads the raw session JSONL to extract file content from tool call operations. Supports offset/limit paging so you can browse long files. Note: edit diffs are not indexed for text search — drill-down reads them from the raw session as original full-file writes.
355
480
 
356
- If you have standalone pi-vcc or pi-observational-memory installed, remove them first — they will conflict:
357
-
358
- ```bash
359
- pi uninstall https://github.com/sting8k/pi-vcc
360
- pi uninstall https://github.com/elpapi42/pi-observational-memory
361
- ```
362
-
363
- Then `/reload` or restart Pi.
364
-
365
- ### Standalone install (without Pi)
366
-
367
- ```bash
368
- npm install pi-blackhole
369
- # or
370
- pnpm add pi-blackhole
371
- ```
481
+ **Touched mode** (`mode:touched`) aggregates all files written, edited, or read across the session, grouped by path. Each entry shows which tool operation touched the file and the line count. Useful for getting a lay of the land after a long session.
372
482
 
373
483
  ## Uninstall
374
484
 
@@ -381,12 +491,24 @@ rm -rf ~/.pi/agent/pi-blackhole
381
491
 
382
492
  ## Credits
383
493
 
384
- The bulk of this codebase comes from two upstream projects:
494
+ pi-blackhole started as a merge of two upstream projects, but has since diverged significantly. The codebase still carries DNA from both:
495
+
496
+ - **[pi-vcc](https://github.com/sting8k/pi-vcc)** by @sting8k — algorithmic conversation compaction (the `compile()` pipeline, section extraction, recall core)
497
+ - **[pi-observational-memory](https://github.com/elpapi42/pi-observational-memory)** by @elpapi42 — session-ledger-based observation/reflection capture, memory agents, ledger folding
385
498
 
386
- - **[pi-vcc](https://github.com/sting8k/pi-vcc)** by @sting8k — algorithmic conversation compaction
387
- - **[pi-observational-memory](https://github.com/elpapi42/pi-observational-memory)** by @elpapi42 — session-ledger-based observation and reflection capture
499
+ What blackhole adds and reworks on top:
388
500
 
389
- pi-blackhole combines, configures, and extends them into a single extension and adds unified configuration, per-worker model fallback chains, persisted cooldowns, manual flush mode, and conflict resolution between the two, and better error detection.
501
+ - **Unified configuration** — one JSON file, not two
502
+ - **Per-worker model fallback chains** with persisted cooldowns that survive Pi restarts
503
+ - **Manual flush mode** — `noAutoCompact` saves observations to per-session disk buffers
504
+ - **Conflict resolution** — OM hooks into vcc's compaction, not Pi's default
505
+ - **Memory toggle** (`/blackhole om-off` / `/blackhole om-on`) — disable the memory layer without uninstalling
506
+ - **Per-session pending state** — isolated per-session JSON files, no cross-session contamination
507
+ - **Custom provider bridge** — consolidation agents loaded via jiti can still use provider stream functions registered by other extensions
508
+ - **Retryable error detection with per-model cooldowns** — models that fail get cooled down, fallbacks tried automatically, 30-second retry gate prevents spam
509
+ - **Improved observer/reflector/dropper prompts** — each heavily customized with detailed extraction rules, relevance guidance, and error handling
510
+ - **OM-recall coupling** — when expanding session entries via `recall`, related observations and reflections are automatically shown
511
+ - **Thinking level support** — per-model `thinking` field for reasoning effort control
390
512
 
391
513
  ## License
392
514
 
package/example-config.json CHANGED
@@ -1,5 +1,5 @@
1
1
  {
2
- "_comment": "blackhole unified configuration — ~/.pi/agent/pi-blackhole-config.json",
2
+ "_comment": "blackhole unified configuration — ~/.pi/agent/pi-blackhole/pi-blackhole-config.json",
3
3
  "_all_settings_in_one_file": true,
4
4
 
5
5
  "overrideDefaultCompaction": false,
@@ -90,12 +90,14 @@
90
90
  "reflectAfterTokens": 25000,
91
91
  "compactAfterTokens": 81000,
92
92
  "observationsPoolMaxTokens": 20000,
93
+ "observationsPoolTargetTokens": 10000,
93
94
  "reflectorInputMaxTokens": 80000,
94
95
  "dropperInputMaxTokens": 80000,
95
96
  "observerChunkMaxTokens": 40000,
96
97
  "observerPreambleMaxTokens": 0,
97
98
  "agentMaxTurns": 16,
98
99
 
100
+ "memory": true,
99
101
  "passive": false,
100
102
  "noAutoCompact": false,
101
103
  "debugLog": false,
@@ -114,7 +116,9 @@
114
116
  "Env override for passive mode: PI_BLACKHOLE_PASSIVE=true.",
115
117
  "",
116
118
  "noAutoCompact mode accumulates observationBatches/reflectionBatches in",
117
- "pending.json across pipeline runs. The observer preamble (CURRENT OBSERVATIONS)",
119
+ "<sessionId>-pending.json (per-session) across pipeline runs.",
120
+ "On /blackhole flush, all accumulated batches are written to the branch and the file is cleared.",
121
+ "The observer preamble (CURRENT OBSERVATIONS)",
118
122
  "is capped to observerPreambleMaxTokens (default 0 = auto 30% of observerChunkMaxTokens).",
119
123
  "This is a soft safety valve — it only trims when accumulated observations exceed",
120
124
  "the budget. High-relevance observations are always kept. Reflections are never trimmed.",
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-blackhole",
3
- "version": "0.3.0",
3
+ "version": "0.3.1",
4
4
  "packageManager": "pnpm@11.2.2",
5
5
  "description": "Unified compaction + observational memory extension for Pi — compresses conversation context while preserving durable observations and reflections",
6
6
  "license": "MIT",
package/src/hooks/before-compact.ts CHANGED
@@ -14,6 +14,7 @@ import { compile } from "../core/summarize";
14
14
  import type { PiVccCompactionDetails } from "../details";
15
15
  import { buildCompactionProjection, renderSummary } from "../om/ledger/index.js";
16
16
  import type { Runtime } from "../om/runtime.js";
17
+ import { debugLog } from "../om/debug-log.js";
17
18
 
18
19
  export const PI_VCC_COMPACT_INSTRUCTION = "__pi_vcc__";
19
20
 
@@ -141,14 +142,28 @@ export const registerBeforeCompactHook = (pi: ExtensionAPI, omRuntime: Runtime)
141
142
  pi.on("session_before_compact", (event, ctx) => {
142
143
  const { preparation, branchEntries, customInstructions } = event;
143
144
  omRuntime.ensureConfig(ctx.cwd ?? process.cwd());
145
+ const trace = (ev: string, d?: Record<string, unknown>) => debugLog(ev, d, omRuntime.config.debugLog === true);
146
+
147
+ trace("before_compact.enter", {
148
+ customInstructions,
149
+ isPiVcc: customInstructions === PI_VCC_COMPACT_INSTRUCTION,
150
+ overrideDefaultCompaction: omRuntime.config.overrideDefaultCompaction,
151
+ noAutoCompact: omRuntime.config.noAutoCompact,
152
+ branchLength: branchEntries.length,
153
+ hasPreviousSummary: !!preparation.previousSummary,
154
+ });
144
155
 
145
156
  // Always handle explicit /blackhole marker.
146
157
  // Otherwise, only handle when user opted in via settings.
147
158
  const isPiVcc = customInstructions === PI_VCC_COMPACT_INSTRUCTION;
148
- if (!isPiVcc && !omRuntime.config.overrideDefaultCompaction) return;
159
+ if (!isPiVcc && !omRuntime.config.overrideDefaultCompaction) {
160
+ trace("before_compact.return_early", { reason: "overrideDefaultCompaction=false and not /blackhole" });
161
+ return;
162
+ }
149
163
 
150
164
  // When noAutoCompact is active, only /blackhole can trigger compaction
151
165
  if (omRuntime.config.noAutoCompact && !isPiVcc) {
166
+ trace("before_compact.cancel", { reason: "noAutoCompact and not /blackhole" });
152
167
  return { cancel: true };
153
168
  }
154
169
 
@@ -212,12 +227,20 @@ export const registerBeforeCompactHook = (pi: ExtensionAPI, omRuntime: Runtime)
212
227
  })),
213
228
  });
214
229
 
230
+ trace("before_compact.cancel", { reason: ownCut.reason, isPiVcc });
215
231
  try {
216
232
  ctx?.ui?.notify?.(REASON_MESSAGES[ownCut.reason], "warning");
217
233
  } catch {}
218
234
  return { cancel: true };
219
235
  }
220
236
 
237
+ trace("before_compact.proceeding", {
238
+ messageCount: ownCut.messages.length,
239
+ firstKeptEntryId: ownCut.firstKeptEntryId,
240
+ compactAll: ownCut.compactAll,
241
+ isPiVcc,
242
+ });
243
+
221
244
  const agentMessages = ownCut.messages;
222
245
  const firstKeptEntryId = ownCut.firstKeptEntryId;
223
246
  const messages = convertToLlm(agentMessages);
@@ -278,6 +301,8 @@ export const registerBeforeCompactHook = (pi: ExtensionAPI, omRuntime: Runtime)
278
301
  sections: [...summary.matchAll(/^\[(.+?)\]/gm)].map((m) => m[1]),
279
302
  });
280
303
 
304
+ trace("before_compact.summary_generated", { summaryLength: summary.length, messageCount: agentMessages.length });
305
+
281
306
  const details: PiVccCompactionDetails = {
282
307
  compactor: "blackhole",
283
308
  version: 1,
@@ -291,6 +316,7 @@ export const registerBeforeCompactHook = (pi: ExtensionAPI, omRuntime: Runtime)
291
316
  // ── Inject observational-memory content ───────────────────────────
292
317
  let omContent = "";
293
318
  let omDetails: Record<string, unknown> | undefined;
319
+ trace("before_compact.om_injection", { memoryEnabled: omRuntime.config.memory !== false });
294
320
  if (omRuntime.config.memory !== false) {
295
321
  const projection = buildCompactionProjection(
296
322
  branchEntries as any[],
package/src/om/compaction-trigger.ts CHANGED
@@ -1,6 +1,7 @@
1
1
  import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
2
2
  import { rawTokensSinceLastCompaction, type Entry } from "./ledger/index.js";
3
3
  import type { Runtime } from "./runtime.js";
4
+ import { debugLog } from "./debug-log.js";
4
5
 
5
6
  /**
6
7
  * Regex matching Pi's internal retryable error detection.
@@ -13,10 +14,36 @@ const RETRYABLE_ERROR_RE =
13
14
  export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): void {
14
15
  pi.on("agent_end", (event: any, ctx: any) => {
15
16
  runtime.ensureConfig(ctx.cwd);
16
- if (runtime.config.passive === true) return;
17
- if (runtime.config.memory === false) return;
18
- if (runtime.config.noAutoCompact === true) return;
19
- if (runtime.compactInFlight) return;
17
+
18
+ // Pass the config flag explicitly — this handler runs outside ALS context
19
+ // (agent_end events don't flow through consolidation's withDebugLogContext),
20
+ // and the setTimeout callback would lose ALS context anyway.
21
+ const dbg = (ev: string, d?: Record<string, unknown>) => debugLog(ev, d, runtime.config.debugLog === true);
22
+
23
+ dbg("compaction_trigger.agent_end", {
24
+ passive: runtime.config.passive,
25
+ memory: runtime.config.memory,
26
+ noAutoCompact: runtime.config.noAutoCompact,
27
+ compactInFlight: runtime.compactInFlight,
28
+ compactAfterTokens: runtime.config.compactAfterTokens,
29
+ });
30
+
31
+ if (runtime.config.passive === true) {
32
+ dbg("compaction_trigger.skip", { reason: "passive" });
33
+ return;
34
+ }
35
+ if (runtime.config.memory === false) {
36
+ dbg("compaction_trigger.skip", { reason: "memory" });
37
+ return;
38
+ }
39
+ if (runtime.config.noAutoCompact === true) {
40
+ dbg("compaction_trigger.skip", { reason: "noAutoCompact" });
41
+ return;
42
+ }
43
+ if (runtime.compactInFlight) {
44
+ dbg("compaction_trigger.skip", { reason: "compactInFlight" });
45
+ return;
46
+ }
20
47
 
21
48
  // Don't trigger compaction if Pi will auto-retry — the agent hasn't truly finished.
22
49
  // Pi emits agent_end before its own retry check, so we must detect this ourselves.
@@ -34,8 +61,14 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
34
61
  }
35
62
 
36
63
  const entries = ctx.sessionManager.getBranch() as Entry[];
64
+ dbg("compaction_trigger.branch_check", { branchLength: entries.length, hasLastEntry: entries.length > 0, lastEntryType: entries.length > 0 ? entries[entries.length - 1].type : "none" });
65
+
37
66
  const tokens = rawTokensSinceLastCompaction(entries);
38
- if (tokens < runtime.config.compactAfterTokens) return;
67
+ dbg("compaction_trigger.tokens", { tokens, compactAfterTokens: runtime.config.compactAfterTokens, branchLength: entries.length });
68
+ if (tokens < runtime.config.compactAfterTokens) {
69
+ dbg("compaction_trigger.skip", { reason: "below_threshold", tokens, threshold: runtime.config.compactAfterTokens });
70
+ return;
71
+ }
39
72
 
40
73
  // Capture ctx properties synchronously — the deferred callback below
41
74
  // may outlive the extension ctx (stale after session replacement/reload).
@@ -43,18 +76,25 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
43
76
  const ui = ctx.ui;
44
77
  const sessionId = ctx.sessionManager.getSessionId();
45
78
 
79
+ dbg("compaction_trigger.threshold_reached", { tokens, sessionId, hasUI });
80
+
46
81
  if (hasUI) ui?.notify(
47
82
  `Observational memory: compaction threshold reached (~${tokens.toLocaleString()} tokens); triggering compaction`,
48
83
  "info",
49
84
  );
50
85
 
51
86
  runtime.compactInFlight = true;
52
- queueMicrotask(() => {
87
+ dbg("compaction_trigger.scheduled", { compactInFlight: runtime.compactInFlight });
88
+
89
+ setTimeout(() => {
90
+ dbg("compaction_trigger.microtask.enter", {});
53
91
  try {
54
92
  // Validate session identity — bail if the session was replaced/reloaded.
55
93
  const currentSessionId = ctx.sessionManager.getSessionId();
94
+ dbg("compaction_trigger.microtask.session_check", { currentSessionId, expectedSessionId: sessionId, match: currentSessionId === sessionId });
56
95
  if (currentSessionId !== sessionId) {
57
96
  runtime.compactInFlight = false;
97
+ dbg("compaction_trigger.microtask.bail", { reason: "session_changed" });
58
98
  if (hasUI) ui?.notify(
59
99
  "Observational memory: compaction cancelled — session changed before compaction",
60
100
  "info",
@@ -62,8 +102,11 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
62
102
  return;
63
103
  }
64
104
 
65
- if (!ctx.isIdle()) {
105
+ const isIdle = ctx.isIdle();
106
+ dbg("compaction_trigger.microtask.idle_check", { isIdle });
107
+ if (!isIdle) {
66
108
  runtime.compactInFlight = false;
109
+ dbg("compaction_trigger.microtask.bail", { reason: "not_idle" });
67
110
  if (hasUI) ui?.notify(
68
111
  "Observational memory: compaction deferred — agent became busy before compaction",
69
112
  "info",
@@ -72,21 +115,27 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
72
115
  }
73
116
  const currentEntries = ctx.sessionManager.getBranch() as Entry[];
74
117
  const currentTokens = rawTokensSinceLastCompaction(currentEntries);
118
+ dbg("compaction_trigger.microtask.recheck_tokens", { currentTokens, threshold: runtime.config.compactAfterTokens, ok: currentTokens >= runtime.config.compactAfterTokens });
75
119
  if (currentTokens < runtime.config.compactAfterTokens) {
76
120
  runtime.compactInFlight = false;
121
+ dbg("compaction_trigger.microtask.bail", { reason: "pressure_relieved", currentTokens, threshold: runtime.config.compactAfterTokens });
77
122
  if (hasUI) ui?.notify(
78
123
  "Observational memory: compaction skipped — another compaction already ran before deferred compaction",
79
124
  "info",
80
125
  );
81
126
  return;
82
127
  }
128
+
129
+ dbg("compaction_trigger.microtask.calling_compact", {});
83
130
  ctx.compact({
84
- onComplete: () => {
131
+ onComplete: (result: any) => {
85
132
  runtime.compactInFlight = false;
133
+ dbg("compaction_trigger.onComplete", { result: !!result });
86
134
  if (hasUI) ui?.notify("Observational memory: compaction complete", "info");
87
135
  },
88
136
  onError: (error: { message: string }) => {
89
137
  runtime.compactInFlight = false;
138
+ dbg("compaction_trigger.onError", { message: error?.message ?? String(error) });
90
139
  if (error.message === "Compaction cancelled") {
91
140
  // We already notified the user with the real reason before returning { cancel: true }.
92
141
  return;
@@ -97,8 +146,9 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
97
146
  } catch (error) {
98
147
  runtime.compactInFlight = false;
99
148
  const msg = error instanceof Error ? error.message : String(error);
149
+ dbg("compaction_trigger.microtask.error", { message: msg });
100
150
  if (hasUI) ui?.notify(`Observational memory: compact threw: ${msg}`, "error");
101
151
  }
102
- });
152
+ }, 0);
103
153
  });
104
154
  }
package/src/om/debug-log.ts CHANGED
@@ -25,9 +25,10 @@ export function withDebugLogContext<T>(context: DebugLogContext, fn: () => T): T
25
25
  return storage.run({ ...parent, ...context }, fn);
26
26
  }
27
27
 
28
- export function debugLog(event: string, data: Record<string, unknown> = {}): void {
28
+ export function debugLog(event: string, data: Record<string, unknown> = {}, forceEnabled?: boolean): void {
29
29
  const context = storage.getStore();
30
- if (context?.enabled !== true) return;
30
+ const enabled = forceEnabled ?? context?.enabled ?? false;
31
+ if (enabled !== true) return;
31
32
 
32
33
  try {
33
34
  const path = join(getAgentDir(), DEBUG_LOG_RELATIVE_PATH);
@@ -36,8 +37,8 @@ export function debugLog(event: string, data: Record<string, unknown> = {}): voi
36
37
  const payload = {
37
38
  ts: new Date().toISOString(),
38
39
  event,
39
- cwd: context.cwd,
40
- runId: context.runId,
40
+ cwd: context?.cwd,
41
+ runId: context?.runId,
41
42
  data,
42
43
  };
43
44
  appendFileSync(path, `${JSON.stringify(payload)}\n`, "utf-8");