pi-blackhole 0.3.5 → 0.3.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -21
- package/example-config.json +2 -0
- package/package.json +1 -1
- package/src/commands/memory.ts +1 -1
- package/src/commands/vcc-recall.ts +16 -3
- package/src/core/recall-scope.ts +3 -3
- package/src/core/search-entries.ts +3 -6
- package/src/core/unified-config.ts +16 -0
- package/src/om/compaction-trigger.ts +54 -11
- package/src/om/consolidation.ts +262 -29
- package/src/om/pending.ts +25 -1
- package/src/om/runtime.ts +70 -0
- package/src/tools/recall.ts +4 -7
package/README.md
CHANGED
|
@@ -1,20 +1,6 @@
|
|
|
1
1
|
# pi-blackhole
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
> **Blackhole is the default compaction engine** (`compactionEngine: "blackhole"`, `compaction: "auto"`). Auto-compaction fires at the configured threshold using blackhole's pipeline — both auto-trigger and Pi's `/compact` command use it. No additional setup needed. To work automatically, blackhole needs to register its hook so it overrides /compact with the `blackhole` compaction engine. Opting out or opting for manual compaction with blackhole can be done as below:
|
|
5
|
-
>
|
|
6
|
-
> | Setting | Auto-trigger after threshold | `/compact` (Pi built-in) |
|
|
7
|
-
> |---|---|---|
|
|
8
|
-
> | `"auto"` + `"blackhole"` (default) | blackhole handles ✓ | blackhole handles |
|
|
9
|
-
> | `"auto"` + `"pi-default"` | Pi handles | Pi handles |
|
|
10
|
-
> | `"manual"` + any | skipped | Pi handles ✓ |
|
|
11
|
-
> | `"off"` + any | skipped (Pi handles) | Pi handles ✓ |
|
|
12
|
-
>
|
|
13
|
-
> The `/blackhole` command always uses blackhole's pipeline regardless of settings.
|
|
14
|
-
>
|
|
15
|
-
> **Upgrading from an older version?** This version replaces legacy keys (`passive`, `noAutoCompact`, `overrideDefaultCompaction`) with `compaction`, `compactionEngine`, and `tailBehavior`. Automatic migration runs at startup — old configs continue to work. Best effort was made to preserve existing behavior, but review [`MIGRATION-GUIDE.md`](MIGRATION-GUIDE.md) if something behaves differently.
|
|
16
|
-
>
|
|
17
|
-
> See [`CONFIG.md`](CONFIG.md) for the full reference.
|
|
3
|
+
|
|
18
4
|
|
|
19
5
|
**Algorithmic compaction + session-aware observational memory for [Pi](https://github.com/badlogic/pi-mono) — in one unified extension.**
|
|
20
6
|
|
|
@@ -244,7 +230,6 @@ The agent gets a unified `recall` tool that handles three types of input:
|
|
|
244
230
|
| `#N:path` | Drill-down into file content from a tool call (e.g. `#42:auth.ts` shows first 30 lines; `#42:auth.ts:30` shows next 30; `#42:auth.ts:full` shows everything) |
|
|
245
231
|
| Free text | BM25-ranked OR search across transcript and/or file content. Rare terms weighted higher. |
|
|
246
232
|
| `mode:file` | Search only write/edit file content |
|
|
247
|
-
| `mode:transcript` | Search only conversation text |
|
|
248
233
|
| `mode:touched` | Aggregate all files written/edited, grouped by path with entry indices |
|
|
249
234
|
| Regex | Pattern search (e.g. `fork.*pi-vcc`, `hook|inject`) |
|
|
250
235
|
| `scope:all` | Search across all session lineages, not just the active one |
|
|
@@ -292,6 +277,7 @@ Everything else has sensible defaults.
|
|
|
292
277
|
| `observationsPoolTargetTokens` | `10000` | Target size dropper aims for after pruning (derived: half of pool max) |
|
|
293
278
|
| `reflectorInputMaxTokens` | `80000` | Max reflector input budget |
|
|
294
279
|
| `dropperInputMaxTokens` | `80000` | Max dropper input budget |
|
|
280
|
+
| `dropperPressureThreshold` | `0.70` | Fraction of `reflectorInputMaxTokens` at which dropper runs even without new data (pressure relief valve) |
|
|
295
281
|
| `agentMaxTurns` | `16` | Max agent-loop turns per worker per run |
|
|
296
282
|
| `debug` | `false` | Pre-compaction snapshot to `/tmp/pi-blackhole-debug.json` |
|
|
297
283
|
| `debugLog` | `false` | Continuous JSONL debug log to `~/.pi/agent/pi-blackhole/debug.ndjson` |
|
|
@@ -314,7 +300,8 @@ Paste the appropriate block into your config to match your main session model's
|
|
|
314
300
|
"observerPreambleMaxTokens": 0,
|
|
315
301
|
"observationsPoolMaxTokens": 8000,
|
|
316
302
|
"reflectorInputMaxTokens": 30000,
|
|
317
|
-
"dropperInputMaxTokens": 30000
|
|
303
|
+
"dropperInputMaxTokens": 30000,
|
|
304
|
+
"dropperPressureThreshold": 0.70
|
|
318
305
|
}
|
|
319
306
|
```
|
|
320
307
|
|
|
@@ -331,7 +318,8 @@ These are the built-in defaults. If you reset your config, these are what you ge
|
|
|
331
318
|
"observerPreambleMaxTokens": 0,
|
|
332
319
|
"observationsPoolMaxTokens": 20000,
|
|
333
320
|
"reflectorInputMaxTokens": 80000,
|
|
334
|
-
"dropperInputMaxTokens": 80000
|
|
321
|
+
"dropperInputMaxTokens": 80000,
|
|
322
|
+
"dropperPressureThreshold": 0.70
|
|
335
323
|
}
|
|
336
324
|
```
|
|
337
325
|
|
|
@@ -346,7 +334,8 @@ These are the built-in defaults. If you reset your config, these are what you ge
|
|
|
346
334
|
"observerPreambleMaxTokens": 0,
|
|
347
335
|
"observationsPoolMaxTokens": 40000,
|
|
348
336
|
"reflectorInputMaxTokens": 160000,
|
|
349
|
-
"dropperInputMaxTokens": 160000
|
|
337
|
+
"dropperInputMaxTokens": 160000,
|
|
338
|
+
"dropperPressureThreshold": 0.70
|
|
350
339
|
}
|
|
351
340
|
```
|
|
352
341
|
|
|
@@ -453,7 +442,6 @@ The agent gets one unified tool that searches session history, expands entries,
|
|
|
453
442
|
| `#N:path` | Drill-down into file content from a tool call (e.g. `#42:auth.ts` shows first 30 lines; `#42:auth.ts:30` shows next 30; `#42:auth.ts:full` shows everything) |
|
|
454
443
|
| Free text | BM25-ranked OR search across transcript + file indicators. Rare terms weighted higher. |
|
|
455
444
|
| `mode:file` | Search only write/edit file content |
|
|
456
|
-
| `mode:transcript` | Search only conversation text |
|
|
457
445
|
| `mode:touched` | Aggregate all files written/edited across the session, grouped by path with entry indices |
|
|
458
446
|
| Regex | Pattern search (e.g. `fork.*pi-vcc`, `hook\|inject`) |
|
|
459
447
|
| `scope:all` | Search across all session lineages (default: active lineage only) |
|
|
@@ -470,7 +458,6 @@ Results are shown as a collapsible message and auto-fed to the agent as context.
|
|
|
470
458
|
/blackhole-recall hook|inject # regex
|
|
471
459
|
/blackhole-recall fail.*build scope:all # regex across all lineages
|
|
472
460
|
/blackhole-recall mode:file # search only write/edit file content
|
|
473
|
-
/blackhole-recall mode:transcript # search only conversation
|
|
474
461
|
/blackhole-recall mode:touched # aggregate view of all files touched
|
|
475
462
|
/blackhole-recall # recent 25 entries
|
|
476
463
|
```
|
package/example-config.json
CHANGED
|
@@ -95,6 +95,7 @@
|
|
|
95
95
|
"observationsPoolTargetTokens": 10000,
|
|
96
96
|
"reflectorInputMaxTokens": 80000,
|
|
97
97
|
"dropperInputMaxTokens": 80000,
|
|
98
|
+
"dropperPressureThreshold": 0.70,
|
|
98
99
|
"observerChunkMaxTokens": 40000,
|
|
99
100
|
"observerPreambleMaxTokens": 0,
|
|
100
101
|
"agentMaxTurns": 16,
|
|
@@ -110,6 +111,7 @@
|
|
|
110
111
|
" compactionEngine (blackhole|pi-default) — who handles it",
|
|
111
112
|
" tailBehavior (pi-default|minimal) — how much stays visible",
|
|
112
113
|
" sessionFallback (true|false) — if false, skip session model as last-resort fallback",
|
|
114
|
+
" dropperPressureThreshold (0-1, default 0.70) — pressure relief valve: fraction of reflectorInputMaxTokens where dropper fires even without new data",
|
|
113
115
|
"",
|
|
114
116
|
"Backward compat: old keys (overrideDefaultCompaction, noAutoCompact, passive)",
|
|
115
117
|
"are migrated to new keys at load time.",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-blackhole",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.8",
|
|
4
4
|
"packageManager": "pnpm@11.2.2",
|
|
5
5
|
"description": "Unified compaction + observational memory extension for Pi — compresses conversation context while preserving durable observations and reflections",
|
|
6
6
|
"license": "MIT",
|
package/src/commands/memory.ts
CHANGED
|
@@ -36,7 +36,7 @@ function firstArg(args: unknown): string | undefined {
|
|
|
36
36
|
}
|
|
37
37
|
|
|
38
38
|
function pct(current: number, total: number): number {
|
|
39
|
-
return total > 0 ? Math.
|
|
39
|
+
return total > 0 ? Math.round((current / total) * 100) : 0;
|
|
40
40
|
}
|
|
41
41
|
|
|
42
42
|
function tokenSum(items: { tokenCount: number }[]): number {
|
|
@@ -6,8 +6,8 @@
|
|
|
6
6
|
*/
|
|
7
7
|
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
|
|
8
8
|
import { loadAllMessages } from "../core/load-messages.js";
|
|
9
|
-
import { searchEntries } from "../core/search-entries.js";
|
|
10
|
-
import { formatRecallOutput } from "../core/format-recall.js";
|
|
9
|
+
import { searchEntries, getTouchedFiles } from "../core/search-entries.js";
|
|
10
|
+
import { formatRecallOutput, formatTouchedOutput } from "../core/format-recall.js";
|
|
11
11
|
import { getActiveLineageEntryIds } from "../core/lineage.js";
|
|
12
12
|
import { parseRecallScope } from "../core/recall-scope.js";
|
|
13
13
|
import {
|
|
@@ -41,7 +41,7 @@ async function augmentWithObservations(
|
|
|
41
41
|
export const registerVccRecallCommand = (pi: ExtensionAPI) => {
|
|
42
42
|
pi.registerCommand("blackhole-recall", {
|
|
43
43
|
description:
|
|
44
|
-
"Search session history. Defaults to active lineage. Usage: /blackhole-recall <query> [page:N] [scope:all] [mode:file|
|
|
44
|
+
"Search session history. Defaults to active lineage. Usage: /blackhole-recall <query> [page:N] [scope:all] [mode:file|touched]",
|
|
45
45
|
handler: async (args: string, ctx) => {
|
|
46
46
|
const sessionFile = ctx.sessionManager.getSessionFile();
|
|
47
47
|
if (!sessionFile) {
|
|
@@ -57,6 +57,19 @@ export const registerVccRecallCommand = (pi: ExtensionAPI) => {
|
|
|
57
57
|
: undefined;
|
|
58
58
|
const mode = parsed.mode;
|
|
59
59
|
|
|
60
|
+
if (mode === "touched") {
|
|
61
|
+
const pageMatch = raw.match(/\bpage:(\d+)\b/i);
|
|
62
|
+
const page = pageMatch ? Math.max(1, parseInt(pageMatch[1], 10)) : 1;
|
|
63
|
+
const { rendered, rawMessages } = loadAllMessages(sessionFile, false, lineageEntryIds);
|
|
64
|
+
const touched = getTouchedFiles(rawMessages, rendered);
|
|
65
|
+
const text = formatTouchedOutput(touched, page);
|
|
66
|
+
pi.sendMessage(
|
|
67
|
+
{ customType: "blackhole-recall", content: text, display: true },
|
|
68
|
+
{ triggerTurn: true },
|
|
69
|
+
);
|
|
70
|
+
return;
|
|
71
|
+
}
|
|
72
|
+
|
|
60
73
|
if (!parsed.text) {
|
|
61
74
|
// No query: show recent entries
|
|
62
75
|
const { rendered } = loadAllMessages(sessionFile, false, lineageEntryIds);
|
package/src/core/recall-scope.ts
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
export type RecallScope = "lineage" | "all";
|
|
2
|
-
export type RecallMode = "hybrid" | "file" | "
|
|
2
|
+
export type RecallMode = "hybrid" | "file" | "touched";
|
|
3
3
|
|
|
4
4
|
const SCOPE_RE = /\bscope:(lineage|all)\b/i;
|
|
5
|
-
const MODE_RE = /\bmode:(hybrid|file|
|
|
5
|
+
const MODE_RE = /\bmode:(hybrid|file|touched)\b/i;
|
|
6
6
|
|
|
7
|
-
const VALID_MODES = new Set(["hybrid", "file", "
|
|
7
|
+
const VALID_MODES = new Set(["hybrid", "file", "touched"]);
|
|
8
8
|
|
|
9
9
|
export const normalizeRecallScope = (scope?: unknown): RecallScope =>
|
|
10
10
|
typeof scope === "string" && scope.toLowerCase() === "all" ? "all" : "lineage";
|
|
@@ -196,10 +196,7 @@ const fullText = (msg: Message, mode?: RecallMode): string => {
|
|
|
196
196
|
if (mode === "file") {
|
|
197
197
|
return toolCallArgsText(msg.content);
|
|
198
198
|
}
|
|
199
|
-
|
|
200
|
-
return textOf(msg.content);
|
|
201
|
-
}
|
|
202
|
-
// hybrid (default): both
|
|
199
|
+
// hybrid (default): both transcript text + tool call args
|
|
203
200
|
const text = textOf(msg.content);
|
|
204
201
|
const toolArgs = toolCallArgsText(msg.content);
|
|
205
202
|
return toolArgs ? `${text}\n${toolArgs}` : text;
|
|
@@ -330,7 +327,7 @@ export const searchEntries = (
|
|
|
330
327
|
const hay = `${e.role} ${text} ${filePart}`;
|
|
331
328
|
if (regex.test(hay)) {
|
|
332
329
|
const snip = lineSnippet(text, regex);
|
|
333
|
-
const fileMatches =
|
|
330
|
+
const fileMatches = computeFileMatches(msg, rawQuery);
|
|
334
331
|
const extra = fileMatches.length > 0 ? { fileMatches } : {};
|
|
335
332
|
hits.push({ ...e, snippet: snip, matchCount: 1, ...extra });
|
|
336
333
|
}
|
|
@@ -366,7 +363,7 @@ export const searchEntries = (
|
|
|
366
363
|
const score = bm25Score(hay, terms, ctx);
|
|
367
364
|
const text = fullTextCache[i];
|
|
368
365
|
const snip = lineSnippet(text, snipRe);
|
|
369
|
-
const fileMatches =
|
|
366
|
+
const fileMatches = computeFileMatches(messages[i], rawQuery);
|
|
370
367
|
const extra = fileMatches.length > 0 ? { fileMatches } : {};
|
|
371
368
|
scored.push({
|
|
372
369
|
hit: { ...e, snippet: snip, matchCount: mc, ...extra },
|
|
@@ -90,6 +90,11 @@ export interface UnifiedConfig {
|
|
|
90
90
|
reflectorInputMaxTokens: number;
|
|
91
91
|
/** Max prompt tokens for dropper model input (rolling window cap). */
|
|
92
92
|
dropperInputMaxTokens: number;
|
|
93
|
+
/** Pressure threshold for dropper. When active observation pool tokens exceed
|
|
94
|
+
* this fraction of reflectorInputMaxTokens, the dropper runs even without new
|
|
95
|
+
* observations/reflections (to keep the pool pruned).
|
|
96
|
+
* Default 0.70 (70%). Must be in range (0, 1]. */
|
|
97
|
+
dropperPressureThreshold: number;
|
|
93
98
|
/** Max source entries tokens sent to observer per chunk. */
|
|
94
99
|
observerChunkMaxTokens: number;
|
|
95
100
|
/** Max preamble tokens (CURRENT REFLECTIONS / OBSERVATIONS) in the observer prompt.
|
|
@@ -149,6 +154,7 @@ export const DEFAULTS: UnifiedConfig = {
|
|
|
149
154
|
observationsPoolTargetTokens: 10_000,
|
|
150
155
|
reflectorInputMaxTokens: 80_000,
|
|
151
156
|
dropperInputMaxTokens: 80_000,
|
|
157
|
+
dropperPressureThreshold: 0.70,
|
|
152
158
|
observerChunkMaxTokens: 40_000,
|
|
153
159
|
observerPreambleMaxTokens: 0,
|
|
154
160
|
agentMaxTurns: 16,
|
|
@@ -238,6 +244,11 @@ function parseConfig(raw: Record<string, unknown>): Partial<UnifiedConfig> {
|
|
|
238
244
|
|
|
239
245
|
// Numeric fields — use nonNegativeInt for observerPreambleMaxTokens (0 = auto)
|
|
240
246
|
const numKeys = ["observeAfterTokens", "reflectAfterTokens", "compactAfterTokens", "observationsPoolMaxTokens", "observationsPoolTargetTokens", "reflectorInputMaxTokens", "dropperInputMaxTokens", "observerChunkMaxTokens", "observerPreambleMaxTokens", "agentMaxTurns"] as const;
|
|
247
|
+
|
|
248
|
+
// dropperPressureThreshold: fractional, must be in (0, 1]
|
|
249
|
+
if (typeof raw.dropperPressureThreshold === "number" && Number.isFinite(raw.dropperPressureThreshold) && raw.dropperPressureThreshold > 0 && raw.dropperPressureThreshold <= 1) {
|
|
250
|
+
c.dropperPressureThreshold = raw.dropperPressureThreshold;
|
|
251
|
+
}
|
|
241
252
|
for (const k of numKeys) {
|
|
242
253
|
// observerPreambleMaxTokens accepts 0 (auto-compute); everything else must be > 0
|
|
243
254
|
const validator = k === "observerPreambleMaxTokens" ? nonNegativeInt : positiveInt;
|
|
@@ -414,6 +425,11 @@ export function loadUnifiedConfig(cwd: string): UnifiedConfig {
|
|
|
414
425
|
|
|
415
426
|
|
|
416
427
|
|
|
428
|
+
// Validate dropperPressureThreshold — must be in (0, 1]
|
|
429
|
+
if (typeof merged.dropperPressureThreshold !== "number" || !Number.isFinite(merged.dropperPressureThreshold) || merged.dropperPressureThreshold <= 0 || merged.dropperPressureThreshold > 1) {
|
|
430
|
+
merged.dropperPressureThreshold = DEFAULTS.dropperPressureThreshold;
|
|
431
|
+
}
|
|
432
|
+
|
|
417
433
|
// Derive observationsPoolTargetTokens if still unset or invalid (must be < max)
|
|
418
434
|
if (
|
|
419
435
|
merged.observationsPoolTargetTokens === undefined ||
|
|
@@ -4,9 +4,41 @@ import type { Runtime } from "./runtime.js";
|
|
|
4
4
|
import { debugLog } from "./debug-log.js";
|
|
5
5
|
import { RETRYABLE_ERROR_RE } from "./retryable-error.js";
|
|
6
6
|
|
|
7
|
+
function getErrorMessage(error: unknown): string {
|
|
8
|
+
if (error instanceof Error) return error.message;
|
|
9
|
+
if (error && typeof error === "object" && "message" in error) {
|
|
10
|
+
return String((error as { message: unknown }).message);
|
|
11
|
+
}
|
|
12
|
+
return String(error);
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
function isStaleExtensionContextError(error: unknown): boolean {
|
|
16
|
+
const message = getErrorMessage(error);
|
|
17
|
+
return message.includes("extension ctx is stale") || message.includes("ctx is stale");
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
function notifySafely(hasUI: boolean, ui: any, message: string, level: "info" | "warning" | "error"): void {
|
|
21
|
+
if (!hasUI) return;
|
|
22
|
+
try {
|
|
23
|
+
ui?.notify(message, level);
|
|
24
|
+
} catch (error) {
|
|
25
|
+
if (!isStaleExtensionContextError(error)) throw error;
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
|
|
7
29
|
export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): void {
|
|
8
30
|
pi.on("agent_end", (event: any, ctx: any) => {
|
|
9
|
-
|
|
31
|
+
try {
|
|
32
|
+
handleAgentEnd(event, ctx, runtime);
|
|
33
|
+
} catch (error) {
|
|
34
|
+
if (isStaleExtensionContextError(error)) return;
|
|
35
|
+
throw error;
|
|
36
|
+
}
|
|
37
|
+
});
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
function handleAgentEnd(event: any, ctx: any, runtime: Runtime): void {
|
|
41
|
+
runtime.ensureConfig(ctx.cwd);
|
|
10
42
|
|
|
11
43
|
// Pass the config flag explicitly — this handler runs outside ALS context
|
|
12
44
|
// (agent_end events don't flow through consolidation's withDebugLogContext),
|
|
@@ -93,7 +125,9 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
|
|
|
93
125
|
|
|
94
126
|
dbg("compaction_trigger.threshold_reached", { tokens, sessionId, hasUI });
|
|
95
127
|
|
|
96
|
-
|
|
128
|
+
notifySafely(
|
|
129
|
+
hasUI,
|
|
130
|
+
ui,
|
|
97
131
|
`Observational memory: compaction threshold reached (~${tokens.toLocaleString()} tokens); triggering compaction`,
|
|
98
132
|
"info",
|
|
99
133
|
);
|
|
@@ -110,7 +144,9 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
|
|
|
110
144
|
if (currentSessionId !== sessionId) {
|
|
111
145
|
runtime.compactInFlight = false;
|
|
112
146
|
dbg("compaction_trigger.microtask.bail", { reason: "session_changed" });
|
|
113
|
-
|
|
147
|
+
notifySafely(
|
|
148
|
+
hasUI,
|
|
149
|
+
ui,
|
|
114
150
|
"Observational memory: compaction cancelled — session changed before compaction",
|
|
115
151
|
"info",
|
|
116
152
|
);
|
|
@@ -122,7 +158,9 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
|
|
|
122
158
|
if (!isIdle) {
|
|
123
159
|
runtime.compactInFlight = false;
|
|
124
160
|
dbg("compaction_trigger.microtask.bail", { reason: "not_idle" });
|
|
125
|
-
|
|
161
|
+
notifySafely(
|
|
162
|
+
hasUI,
|
|
163
|
+
ui,
|
|
126
164
|
"Observational memory: compaction deferred — agent became busy before compaction",
|
|
127
165
|
"info",
|
|
128
166
|
);
|
|
@@ -134,7 +172,9 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
|
|
|
134
172
|
if (currentTokens < runtime.config.compactAfterTokens) {
|
|
135
173
|
runtime.compactInFlight = false;
|
|
136
174
|
dbg("compaction_trigger.microtask.bail", { reason: "pressure_relieved", currentTokens, threshold: runtime.config.compactAfterTokens });
|
|
137
|
-
|
|
175
|
+
notifySafely(
|
|
176
|
+
hasUI,
|
|
177
|
+
ui,
|
|
138
178
|
"Observational memory: compaction skipped — another compaction already ran before deferred compaction",
|
|
139
179
|
"info",
|
|
140
180
|
);
|
|
@@ -146,7 +186,7 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
|
|
|
146
186
|
onComplete: (result: any) => {
|
|
147
187
|
runtime.compactInFlight = false;
|
|
148
188
|
dbg("compaction_trigger.onComplete", { result: !!result });
|
|
149
|
-
|
|
189
|
+
notifySafely(hasUI, ui, "Observational memory: compaction complete", "info");
|
|
150
190
|
},
|
|
151
191
|
onError: (error: { message: string }) => {
|
|
152
192
|
runtime.compactInFlight = false;
|
|
@@ -155,15 +195,18 @@ export function registerCompactionTrigger(pi: ExtensionAPI, runtime: Runtime): v
|
|
|
155
195
|
// We already notified the user with the real reason before returning { cancel: true }.
|
|
156
196
|
return;
|
|
157
197
|
}
|
|
158
|
-
|
|
198
|
+
notifySafely(hasUI, ui, `Observational memory: ${error.message}`, "error");
|
|
159
199
|
},
|
|
160
200
|
});
|
|
161
201
|
} catch (error) {
|
|
162
202
|
runtime.compactInFlight = false;
|
|
163
|
-
const msg =
|
|
203
|
+
const msg = getErrorMessage(error);
|
|
204
|
+
if (isStaleExtensionContextError(error)) {
|
|
205
|
+
dbg("compaction_trigger.microtask.bail", { reason: "stale_ctx", message: msg });
|
|
206
|
+
return;
|
|
207
|
+
}
|
|
164
208
|
dbg("compaction_trigger.microtask.error", { message: msg });
|
|
165
|
-
|
|
209
|
+
notifySafely(hasUI, ui, `Observational memory: compact threw: ${msg}`, "error");
|
|
166
210
|
}
|
|
167
|
-
|
|
168
|
-
});
|
|
211
|
+
}, 0);
|
|
169
212
|
}
|
package/src/om/consolidation.ts
CHANGED
|
@@ -178,10 +178,148 @@ function pendingObservationsCreatedAfter(
|
|
|
178
178
|
return newObs;
|
|
179
179
|
}
|
|
180
180
|
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
181
|
+
/** Cursor-aware stage-due check. Uses cursors when available; falls back to
|
|
182
|
+
* legacy coverage markers when cursors are absent (cold start, fork recovery).
|
|
183
|
+
*
|
|
184
|
+
* In compaction: "manual" mode, the branch has no OM markers — observations
|
|
185
|
+
* live in the per‑session pending file. `pending` provides the pool fullness
|
|
186
|
+
* and new‑data visibility that the reflector/dropper checks need. */
|
|
187
|
+
export function anyStageDue(entries: Entry[], runtime: Runtime, pending?: PendingOMState): boolean {
|
|
188
|
+
const config = runtime.config;
|
|
189
|
+
const cursors = runtime.cursors ?? {};
|
|
190
|
+
|
|
191
|
+
// ── Observer ──────────────────────────────────────────────────────────
|
|
192
|
+
const observerDue = (() => {
|
|
193
|
+
const cursor = cursors.observer;
|
|
194
|
+
if (!cursor) {
|
|
195
|
+
return rawTokensSinceObservationCoverage(entries) >= config.observeAfterTokens;
|
|
196
|
+
}
|
|
197
|
+
const idx = entryIndexForId(entries, cursor.entryId);
|
|
198
|
+
const tokensSince = idx >= 0
|
|
199
|
+
? rawTokensAfterIndex(entries, idx)
|
|
200
|
+
: rawTokensSinceObservationCoverage(entries);
|
|
201
|
+
return tokensSince >= config.observeAfterTokens;
|
|
202
|
+
})();
|
|
203
|
+
|
|
204
|
+
// ── Reflector ─────────────────────────────────────────────────────────
|
|
205
|
+
const reflectorDue = (() => {
|
|
206
|
+
const cursor = cursors.reflector;
|
|
207
|
+
if (!cursor) {
|
|
208
|
+
return rawTokensSinceReflectionCoverage(entries) >= config.reflectAfterTokens;
|
|
209
|
+
}
|
|
210
|
+
const idx = entryIndexForId(entries, cursor.entryId);
|
|
211
|
+
if (idx < 0) {
|
|
212
|
+
return rawTokensSinceReflectionCoverage(entries) >= config.reflectAfterTokens;
|
|
213
|
+
}
|
|
214
|
+
// Must have enough accumulated tokens before considering reflector
|
|
215
|
+
const tokensSince = rawTokensAfterIndex(entries, idx);
|
|
216
|
+
if (tokensSince < config.reflectAfterTokens) {
|
|
217
|
+
return false;
|
|
218
|
+
}
|
|
219
|
+
// Check for new observation batches after the cursor
|
|
220
|
+
for (let i = idx + 1; i < entries.length; i++) {
|
|
221
|
+
const e = entries[i];
|
|
222
|
+
if (e.type === "custom" && e.customType === OM_OBSERVATIONS_RECORDED) {
|
|
223
|
+
// Skip if this marker's coversUpToId is at or before the cursor
|
|
224
|
+
// — data it covers was already processed.
|
|
225
|
+
const markerCoversUpTo: string | undefined = (e as any).data?.coversUpToId;
|
|
226
|
+
if (markerCoversUpTo) {
|
|
227
|
+
const markerCoversIdx = entryIndexForId(entries, markerCoversUpTo);
|
|
228
|
+
if (markerCoversIdx >= 0 && markerCoversIdx <= idx) continue;
|
|
229
|
+
}
|
|
230
|
+
return true;
|
|
231
|
+
}
|
|
232
|
+
}
|
|
233
|
+
// In manual mode, also check pending observation batches that arrived
|
|
234
|
+
// after the cursor (since branch has no OM markers).
|
|
235
|
+
if (pending) {
|
|
236
|
+
const pendingBatches = pending.observationBatches ?? [];
|
|
237
|
+
for (const batch of pendingBatches) {
|
|
238
|
+
if (batch.coversUpToId) {
|
|
239
|
+
const batchIdx = entryIndexForId(entries, batch.coversUpToId);
|
|
240
|
+
if (batchIdx >= 0 && batchIdx > idx) return true;
|
|
241
|
+
}
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
return false;
|
|
245
|
+
})();
|
|
246
|
+
|
|
247
|
+
// ── Dropper ───────────────────────────────────────────────────────────
|
|
248
|
+
// Short‑circuit: only compute dropperDue when observer and reflector are
|
|
249
|
+
// both not due — if either is due, the pipeline launches anyway.
|
|
250
|
+
const dropperDue = observerDue || reflectorDue ? false : (() => {
|
|
251
|
+
// Compute active observation pool tokens (branch + pending in manual mode)
|
|
252
|
+
const folded = foldLedger(entries);
|
|
253
|
+
let poolTokens = folded.activeObservations.reduce(
|
|
254
|
+
(s: number, o: Observation) => s + (o.tokenCount ?? 0),
|
|
255
|
+
0,
|
|
256
|
+
);
|
|
257
|
+
// In manual mode, include pending observation batches
|
|
258
|
+
if (pending) {
|
|
259
|
+
const pendingBatches = pending.observationBatches ?? [];
|
|
260
|
+
for (const batch of pendingBatches) {
|
|
261
|
+
poolTokens += ((batch.data as any)?.observations ?? []).reduce(
|
|
262
|
+
(s: number, o: any) => s + (o.tokenCount ?? 0), 0,
|
|
263
|
+
);
|
|
264
|
+
}
|
|
265
|
+
}
|
|
266
|
+
const fullnessVsPool = config.observationsPoolMaxTokens > 0
|
|
267
|
+
? poolTokens / config.observationsPoolMaxTokens
|
|
268
|
+
: 0;
|
|
269
|
+
|
|
270
|
+
// Must have at least 10% fullness to consider dropper
|
|
271
|
+
if (fullnessVsPool < 0.10) return false;
|
|
272
|
+
|
|
273
|
+
// Pressure check: pool ≥ threshold × reflectorInputMaxTokens
|
|
274
|
+
const pressure = poolTokens >= config.dropperPressureThreshold * config.reflectorInputMaxTokens;
|
|
275
|
+
if (pressure) return true;
|
|
276
|
+
|
|
277
|
+
// New data check: new obs or ref batches after dropper cursor
|
|
278
|
+
const cursor = cursors.dropper;
|
|
279
|
+
if (!cursor) {
|
|
280
|
+
// In manual mode, pending batches are the only source of new‑data
|
|
281
|
+
// visibility (branch has no OM markers).
|
|
282
|
+
const hasPendingNewData = pending
|
|
283
|
+
? (pending.observationBatches?.length ?? 0) > 0 || (pending.reflectionBatches?.length ?? 0) > 0
|
|
284
|
+
: false;
|
|
285
|
+
if (hasPendingNewData) return true;
|
|
286
|
+
return rawTokensSinceDropCoverage(entries) >= config.reflectAfterTokens;
|
|
287
|
+
}
|
|
288
|
+
const idx = entryIndexForId(entries, cursor.entryId);
|
|
289
|
+
if (idx < 0) {
|
|
290
|
+
return rawTokensSinceDropCoverage(entries) >= config.reflectAfterTokens;
|
|
291
|
+
}
|
|
292
|
+
// Must have enough accumulated tokens before considering dropper
|
|
293
|
+
const tokensSince = rawTokensAfterIndex(entries, idx);
|
|
294
|
+
if (tokensSince < config.reflectAfterTokens) {
|
|
295
|
+
return false;
|
|
296
|
+
}
|
|
297
|
+
for (let i = idx + 1; i < entries.length; i++) {
|
|
298
|
+
const e = entries[i];
|
|
299
|
+
if (e.type === "custom" && (e.customType === OM_OBSERVATIONS_RECORDED || e.customType === OM_REFLECTIONS_RECORDED)) {
|
|
300
|
+
const markerCoversUpTo: string | undefined = (e as any).data?.coversUpToId;
|
|
301
|
+
if (markerCoversUpTo) {
|
|
302
|
+
const markerCoversIdx = entryIndexForId(entries, markerCoversUpTo);
|
|
303
|
+
if (markerCoversIdx >= 0 && markerCoversIdx <= idx) continue;
|
|
304
|
+
}
|
|
305
|
+
return true;
|
|
306
|
+
}
|
|
307
|
+
}
|
|
308
|
+
// In manual mode, also check pending batches after the cursor
|
|
309
|
+
if (pending) {
|
|
310
|
+
const pendingObs = pending.observationBatches ?? [];
|
|
311
|
+
const pendingRef = pending.reflectionBatches ?? [];
|
|
312
|
+
for (const batch of [...pendingObs, ...pendingRef]) {
|
|
313
|
+
if (batch.coversUpToId) {
|
|
314
|
+
const batchIdx = entryIndexForId(entries, batch.coversUpToId);
|
|
315
|
+
if (batchIdx >= 0 && batchIdx > idx) return true;
|
|
316
|
+
}
|
|
317
|
+
}
|
|
318
|
+
}
|
|
319
|
+
return false;
|
|
320
|
+
})();
|
|
321
|
+
|
|
322
|
+
return observerDue || reflectorDue || dropperDue;
|
|
185
323
|
}
|
|
186
324
|
|
|
187
325
|
function stageModelConfig(runtime: Runtime, stage: "observer" | "reflector" | "dropper"): ConfiguredModel | undefined {
|
|
@@ -245,6 +383,43 @@ export function registerConsolidationTrigger(pi: ExtensionAPI, runtime: Runtime)
|
|
|
245
383
|
pi.on("turn_end", launch);
|
|
246
384
|
}
|
|
247
385
|
|
|
386
|
+
/** Validate cursors against the current branch. If a cursor's entry ID no longer
|
|
387
|
+
* exists in the branch (fork, navigation, compaction), fall back to the best
|
|
388
|
+
* available coverage marker for that stage. */
|
|
389
|
+
function validateCursors(entries: Entry[], runtime: Runtime): void {
|
|
390
|
+
const cursors = runtime.cursors ?? {};
|
|
391
|
+
|
|
392
|
+
// Observer: fall back to latest OM_OBSERVATIONS_RECORDED marker
|
|
393
|
+
if (cursors.observer && entryIndexForId(entries, cursors.observer.entryId) < 0) {
|
|
394
|
+
const markerId = latestCoverageMarkerId(entries, OM_OBSERVATIONS_RECORDED);
|
|
395
|
+
if (markerId) {
|
|
396
|
+
cursors.observer = { entryId: markerId, state: "initial" };
|
|
397
|
+
} else {
|
|
398
|
+
delete cursors.observer;
|
|
399
|
+
}
|
|
400
|
+
}
|
|
401
|
+
|
|
402
|
+
// Reflector: fall back to latest OM_REFLECTIONS_RECORDED marker
|
|
403
|
+
if (cursors.reflector && entryIndexForId(entries, cursors.reflector.entryId) < 0) {
|
|
404
|
+
const markerId = latestCoverageMarkerId(entries, OM_REFLECTIONS_RECORDED);
|
|
405
|
+
if (markerId) {
|
|
406
|
+
cursors.reflector = { entryId: markerId, state: "initial" };
|
|
407
|
+
} else {
|
|
408
|
+
delete cursors.reflector;
|
|
409
|
+
}
|
|
410
|
+
}
|
|
411
|
+
|
|
412
|
+
// Dropper: fall back to latest OM_OBSERVATIONS_DROPPED marker
|
|
413
|
+
if (cursors.dropper && entryIndexForId(entries, cursors.dropper.entryId) < 0) {
|
|
414
|
+
const markerId = latestCoverageMarkerId(entries, OM_OBSERVATIONS_DROPPED);
|
|
415
|
+
if (markerId) {
|
|
416
|
+
cursors.dropper = { entryId: markerId, state: "initial" };
|
|
417
|
+
} else {
|
|
418
|
+
delete cursors.dropper;
|
|
419
|
+
}
|
|
420
|
+
}
|
|
421
|
+
}
|
|
422
|
+
|
|
248
423
|
function maybeLaunchConsolidation(pi: ExtensionAPI, runtime: Runtime, ctx: ConsolidationCtx): void {
|
|
249
424
|
runtime.ensureConfig(ctx.cwd);
|
|
250
425
|
if (runtime.config.memory === false) return;
|
|
@@ -256,8 +431,28 @@ function maybeLaunchConsolidation(pi: ExtensionAPI, runtime: Runtime, ctx: Conso
|
|
|
256
431
|
if (runtime.consolidationInFlight) return;
|
|
257
432
|
if (runtime.isConsolidationRetryGated()) return;
|
|
258
433
|
|
|
434
|
+
// Load and validate cursors from pending file (once per session; re-load on fork)
|
|
435
|
+
const sessionId = ctx.sessionManager.getSessionId();
|
|
436
|
+
if (runtime.cursorsLoadedSessionId !== sessionId) {
|
|
437
|
+
if (typeof runtime.loadCursorsFromPending === "function") {
|
|
438
|
+
runtime.loadCursorsFromPending(sessionId);
|
|
439
|
+
}
|
|
440
|
+
const entries = ctx.sessionManager.getBranch() as Entry[];
|
|
441
|
+
validateCursors(entries, runtime);
|
|
442
|
+
runtime.cursorsLoadedSessionId = sessionId;
|
|
443
|
+
const c = runtime.cursors ?? {};
|
|
444
|
+
debugLog("cursor.loaded", {
|
|
445
|
+
observer: c.observer ?? null,
|
|
446
|
+
reflector: c.reflector ?? null,
|
|
447
|
+
dropper: c.dropper ?? null,
|
|
448
|
+
});
|
|
449
|
+
}
|
|
450
|
+
|
|
259
451
|
const entries = ctx.sessionManager.getBranch() as Entry[];
|
|
260
|
-
|
|
452
|
+
// In manual mode, the branch has no OM markers — pending state provides
|
|
453
|
+
// pool fullness and new‑data visibility for reflector/dropper checks.
|
|
454
|
+
const pending = runtime.config.noAutoCompact ? readPendingState(sessionId) : undefined;
|
|
455
|
+
if (!anyStageDue(entries, runtime, pending)) return;
|
|
261
456
|
|
|
262
457
|
const runId = `consolidation-${Date.now().toString(36)}-${Math.random().toString(16).slice(2, 8)}`;
|
|
263
458
|
const consolidationCtx: ConsolidationCtx = {
|
|
@@ -314,6 +509,16 @@ export async function runConsolidationPipeline(
|
|
|
314
509
|
} catch (error) {
|
|
315
510
|
debugLog("dropper.error", { errorMessage: runtime.recordConsolidationStageError(ctx, "dropper", error) });
|
|
316
511
|
}
|
|
512
|
+
|
|
513
|
+
// Flush cursors to pending file after all stages complete (non‑blocking)
|
|
514
|
+
const sessionId = ctx.sessionManager.getSessionId();
|
|
515
|
+
runtime.scheduleCursorFlush(sessionId);
|
|
516
|
+
const c = runtime.cursors ?? {};
|
|
517
|
+
debugLog("cursor.saved", {
|
|
518
|
+
observer: c.observer ?? null,
|
|
519
|
+
reflector: c.reflector ?? null,
|
|
520
|
+
dropper: c.dropper ?? null,
|
|
521
|
+
});
|
|
317
522
|
}
|
|
318
523
|
|
|
319
524
|
// ── Observer stage (with fallback) ──────────────────────────────────────────
|
|
@@ -325,14 +530,27 @@ async function runObserverStage(
|
|
|
325
530
|
resolveModel: (stage: "observer") => Promise<ResolvedModel | undefined>,
|
|
326
531
|
): Promise<StageOutcome> {
|
|
327
532
|
const entries = ctx.sessionManager.getBranch() as Entry[];
|
|
328
|
-
const
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
const
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
533
|
+
const sessionId = ctx.sessionManager.getSessionId();
|
|
534
|
+
|
|
535
|
+
// Determine start index: cursor takes priority, fall back to coverage markers
|
|
536
|
+
const observerCursor = runtime.getCursor("observer");
|
|
537
|
+
let effectiveStart: number;
|
|
538
|
+
if (observerCursor) {
|
|
539
|
+
const cursorIdx = entryIndexForId(entries, observerCursor.entryId);
|
|
540
|
+
effectiveStart = cursorIdx >= 0 ? cursorIdx : latestCoverageIndex(entries, OM_OBSERVATIONS_RECORDED);
|
|
541
|
+
} else {
|
|
542
|
+
const lastCoverageIdx = latestCoverageIndex(entries, OM_OBSERVATIONS_RECORDED);
|
|
543
|
+
effectiveStart = lastCoverageIdx >= 0 ? lastCoverageIdx : findLastCompactionIndex(entries);
|
|
544
|
+
}
|
|
545
|
+
|
|
546
|
+
const tokens = effectiveStart >= 0 ? rawTokensAfterIndex(entries, effectiveStart) : 0;
|
|
547
|
+
if (tokens < runtime.config.observeAfterTokens) {
|
|
548
|
+
// Not due — advance cursor to last source entry so we don't re-check immediately
|
|
549
|
+
const lastSourceId = [...entries].reverse().find((e: Entry) => isSourceEntry(e))?.id;
|
|
550
|
+
if (lastSourceId) runtime.advanceCursor("observer", lastSourceId, "not_due");
|
|
551
|
+
return "continue";
|
|
552
|
+
}
|
|
553
|
+
|
|
336
554
|
let chunkEntries = sourceEntriesAfter(entries, effectiveStart);
|
|
337
555
|
|
|
338
556
|
// Cap observer input to observerChunkMaxTokens (newest-to-oldest)
|
|
@@ -349,8 +567,6 @@ async function runObserverStage(
|
|
|
349
567
|
if (!chunk.trim() || sourceEntryIds.length === 0) return "continue";
|
|
350
568
|
const chunkTokens = Math.ceil(chunk.length / 4);
|
|
351
569
|
|
|
352
|
-
const sessionId = ctx.sessionManager.getSessionId();
|
|
353
|
-
|
|
354
570
|
const memory = fullProjection(entries);
|
|
355
571
|
let priorReflections = memory.reflections.map(reflectionToSummaryLine);
|
|
356
572
|
let priorObservations = memory.observations.map(observationToSummaryLine);
|
|
@@ -436,7 +652,7 @@ async function runObserverStage(
|
|
|
436
652
|
|
|
437
653
|
if (result.observations && result.observations.length > 0) {
|
|
438
654
|
const data = buildObservationsRecordedData(result.observations, coversUpToId);
|
|
439
|
-
if (!data) return "continue";
|
|
655
|
+
if (!data) { runtime.advanceCursor("observer", coversUpToId, "empty"); return "continue"; }
|
|
440
656
|
debugLog("observer.records", { count: result.observations.length, observationTokens: result.observations.reduce((s: number, o: any) => s + o.tokenCount, 0), coversUpToId });
|
|
441
657
|
if (runtime.config.noAutoCompact) {
|
|
442
658
|
savePendingObservation(sessionId, { coversUpToId, data });
|
|
@@ -445,6 +661,7 @@ async function runObserverStage(
|
|
|
445
661
|
appendEntry(pi, OM_OBSERVATIONS_RECORDED, data);
|
|
446
662
|
debugLog("observer.appended", { count: result.observations.length, coversUpToId });
|
|
447
663
|
}
|
|
664
|
+
runtime.advanceCursor("observer", coversUpToId, "recorded");
|
|
448
665
|
if (ctx.hasUI) ctx.ui?.notify(`Observational memory: ${result.observations.length} observation${result.observations.length === 1 ? "" : "s"} recorded`, "info");
|
|
449
666
|
return "continue";
|
|
450
667
|
}
|
|
@@ -468,6 +685,7 @@ async function runObserverStage(
|
|
|
468
685
|
: "warning"
|
|
469
686
|
: "warning";
|
|
470
687
|
debugLog("observer.empty", { coversUpToId, reason: reason?.kind });
|
|
688
|
+
runtime.advanceCursor("observer", coversUpToId, "empty");
|
|
471
689
|
if (ctx.hasUI) ctx.ui?.notify(`Observational memory: no observations — ${reasonLabel}`, reasonLevel);
|
|
472
690
|
return "continue";
|
|
473
691
|
} catch (error) {
|
|
@@ -502,15 +720,15 @@ async function runReflectorStage(
|
|
|
502
720
|
const pending = readPendingState(sessionId);
|
|
503
721
|
// Check any accumulated batch for unprocessed observations, not just the latest
|
|
504
722
|
const hasPendingObs = (pending.observationBatches ?? []).some((b: any) => (b.data as any)?.observations?.length);
|
|
505
|
-
if (!hasPendingObs) return { outcome: "continue", sameRunReflections: [] };
|
|
723
|
+
if (!hasPendingObs) { runtime.advanceCursor("reflector", entries.at(-1)?.id ?? "unknown", "skipped"); return { outcome: "continue", sameRunReflections: [] }; }
|
|
506
724
|
observationCoverageId = pending.observation?.coversUpToId;
|
|
507
725
|
if (pending.reflection?.coversUpToId) {
|
|
508
726
|
const obsIdx = entryIndexForId(entries, pending.observation?.coversUpToId ?? "");
|
|
509
727
|
const refIdx = entryIndexForId(entries, pending.reflection.coversUpToId);
|
|
510
|
-
if (obsIdx >= 0 && refIdx >= 0 && obsIdx <= refIdx) return { outcome: "continue", sameRunReflections: [] };
|
|
728
|
+
if (obsIdx >= 0 && refIdx >= 0 && obsIdx <= refIdx) { runtime.advanceCursor("reflector", pending.reflection.coversUpToId, "skipped"); return { outcome: "continue", sameRunReflections: [] }; }
|
|
511
729
|
if (refIdx >= 0) {
|
|
512
730
|
reflectionTokens = rawTokensAfterIndex(entries, refIdx);
|
|
513
|
-
if (reflectionTokens < runtime.config.reflectAfterTokens) return { outcome: "continue", sameRunReflections: [] };
|
|
731
|
+
if (reflectionTokens < runtime.config.reflectAfterTokens) { runtime.advanceCursor("reflector", pending.reflection.coversUpToId, "not_due"); return { outcome: "continue", sameRunReflections: [] }; }
|
|
514
732
|
} else {
|
|
515
733
|
reflectionTokens = rawTokensSinceObservationCoverage(entries);
|
|
516
734
|
}
|
|
@@ -519,9 +737,9 @@ async function runReflectorStage(
|
|
|
519
737
|
}
|
|
520
738
|
} else {
|
|
521
739
|
reflectionTokens = rawTokensSinceReflectionCoverage(entries);
|
|
522
|
-
if (reflectionTokens < runtime.config.reflectAfterTokens) return { outcome: "continue", sameRunReflections: [] };
|
|
740
|
+
if (reflectionTokens < runtime.config.reflectAfterTokens) { runtime.advanceCursor("reflector", entries.at(-1)?.id ?? "unknown", "not_due"); return { outcome: "continue", sameRunReflections: [] }; }
|
|
523
741
|
observationCoverageId = latestCoverageMarkerId(entries, OM_OBSERVATIONS_RECORDED);
|
|
524
|
-
if (!observationCoverageId) return { outcome: "continue", sameRunReflections: [] };
|
|
742
|
+
if (!observationCoverageId) { runtime.advanceCursor("reflector", entries.at(-1)?.id ?? "unknown", "skipped"); return { outcome: "continue", sameRunReflections: [] }; }
|
|
525
743
|
}
|
|
526
744
|
|
|
527
745
|
for (let attempt = 0; attempt < MAX_STAGE_ATTEMPTS; attempt++) {
|
|
@@ -550,6 +768,7 @@ async function runReflectorStage(
|
|
|
550
768
|
if (idx >= 0) effectiveReflectionTokens = rawTokensAfterIndex(entries, idx);
|
|
551
769
|
}
|
|
552
770
|
}
|
|
771
|
+
debugLog("reflector.start", { tokens: effectiveReflectionTokens, inputTokens: reflectorInputTokens, newObsCount: newObservations.length, newRefCount: newReflections.length });
|
|
553
772
|
if (ctx.hasUI) ctx.ui?.notify(`Observational memory: reflector running (~${effectiveReflectionTokens.toLocaleString()} tokens accumulated, ~${reflectorInputTokens.toLocaleString()}-token input)`, "info");
|
|
554
773
|
|
|
555
774
|
// Resolve thinking level for the specific model (fallbacks may have their own thinking config)
|
|
@@ -597,16 +816,26 @@ async function runReflectorStage(
|
|
|
597
816
|
thinkingLevel: stageThinkingLevel(runtime, "reflector", stageModelForThinking),
|
|
598
817
|
});
|
|
599
818
|
|
|
600
|
-
if (!reflections || reflections.length === 0)
|
|
601
|
-
|
|
819
|
+
if (!reflections || reflections.length === 0) {
|
|
820
|
+
runtime.advanceCursor("reflector", observationCoverageId ?? entries.at(-1)?.id ?? "unknown", "empty");
|
|
821
|
+
return { outcome: "continue", sameRunReflections: [] };
|
|
822
|
+
}
|
|
823
|
+
if (!observationCoverageId) {
|
|
824
|
+
runtime.advanceCursor("reflector", entries.at(-1)?.id ?? "unknown", "empty");
|
|
825
|
+
return { outcome: "continue", sameRunReflections: [] };
|
|
826
|
+
}
|
|
602
827
|
|
|
603
828
|
const data = buildReflectionsRecordedData(reflections, observationCoverageId);
|
|
604
|
-
if (!data)
|
|
829
|
+
if (!data) {
|
|
830
|
+
runtime.advanceCursor("reflector", observationCoverageId, "empty");
|
|
831
|
+
return { outcome: "continue", sameRunReflections: [] };
|
|
832
|
+
}
|
|
605
833
|
if (runtime.config.noAutoCompact) {
|
|
606
834
|
savePendingReflection(sessionId, { coversUpToId: data.coversUpToId, data });
|
|
607
835
|
} else {
|
|
608
836
|
appendEntry(pi, OM_REFLECTIONS_RECORDED, data);
|
|
609
837
|
}
|
|
838
|
+
runtime.advanceCursor("reflector", data.coversUpToId, "recorded");
|
|
610
839
|
return {
|
|
611
840
|
outcome: "continue",
|
|
612
841
|
sameRunReflections: reflections,
|
|
@@ -642,15 +871,15 @@ async function runDropperStage(
|
|
|
642
871
|
const pending = readPendingState(sessionId);
|
|
643
872
|
// Check any accumulated batch for unprocessed observations, not just the latest
|
|
644
873
|
const hasPendingObs = (pending.observationBatches ?? []).some((b: any) => (b.data as any)?.observations?.length);
|
|
645
|
-
if (!hasPendingObs) return "continue";
|
|
874
|
+
if (!hasPendingObs) { runtime.advanceCursor("dropper", entries.at(-1)?.id ?? "unknown", "skipped"); return "continue"; }
|
|
646
875
|
observationCoverageId = pending.observation?.coversUpToId;
|
|
647
876
|
if (pending.dropped?.coversUpToId) {
|
|
648
877
|
const obsIdx = entryIndexForId(entries, pending.observation?.coversUpToId ?? "");
|
|
649
878
|
const dropIdx = entryIndexForId(entries, pending.dropped.coversUpToId);
|
|
650
|
-
if (obsIdx >= 0 && dropIdx >= 0 && obsIdx <= dropIdx) return "continue";
|
|
879
|
+
if (obsIdx >= 0 && dropIdx >= 0 && obsIdx <= dropIdx) { runtime.advanceCursor("dropper", pending.dropped.coversUpToId, "skipped"); return "continue"; }
|
|
651
880
|
if (dropIdx >= 0) {
|
|
652
881
|
dropTokens = rawTokensAfterIndex(entries, dropIdx);
|
|
653
|
-
if (dropTokens < runtime.config.reflectAfterTokens) return "continue";
|
|
882
|
+
if (dropTokens < runtime.config.reflectAfterTokens) { runtime.advanceCursor("dropper", pending.dropped.coversUpToId, "not_due"); return "continue"; }
|
|
654
883
|
} else {
|
|
655
884
|
dropTokens = rawTokensSinceDropCoverage(entries);
|
|
656
885
|
}
|
|
@@ -659,9 +888,9 @@ async function runDropperStage(
|
|
|
659
888
|
}
|
|
660
889
|
} else {
|
|
661
890
|
dropTokens = rawTokensSinceDropCoverage(entries);
|
|
662
|
-
if (dropTokens < runtime.config.reflectAfterTokens) return "continue";
|
|
891
|
+
if (dropTokens < runtime.config.reflectAfterTokens) { runtime.advanceCursor("dropper", entries.at(-1)?.id ?? "unknown", "not_due"); return "continue"; }
|
|
663
892
|
observationCoverageId = latestCoverageMarkerId(entries, OM_OBSERVATIONS_RECORDED);
|
|
664
|
-
if (!observationCoverageId) return "continue";
|
|
893
|
+
if (!observationCoverageId) { runtime.advanceCursor("dropper", entries.at(-1)?.id ?? "unknown", "skipped"); return "continue"; }
|
|
665
894
|
}
|
|
666
895
|
|
|
667
896
|
for (let attempt = 0; attempt < MAX_STAGE_ATTEMPTS; attempt++) {
|
|
@@ -746,6 +975,10 @@ async function runDropperStage(
|
|
|
746
975
|
} else {
|
|
747
976
|
appendEntry(pi, OM_OBSERVATIONS_DROPPED, data);
|
|
748
977
|
}
|
|
978
|
+
runtime.advanceCursor("dropper", coversUpToId, "recorded");
|
|
979
|
+
} else {
|
|
980
|
+
// No drops selected (maxDropsAllowed=0 or LLM returned no candidates)
|
|
981
|
+
runtime.advanceCursor("dropper", coversUpToId ?? observationCoverageId ?? entries.at(-1)?.id ?? "unknown", "empty");
|
|
749
982
|
}
|
|
750
983
|
return "continue";
|
|
751
984
|
} catch (error) {
|
package/src/om/pending.ts
CHANGED
|
@@ -59,6 +59,12 @@ export interface PendingOMState {
|
|
|
59
59
|
* /blackhole flush.
|
|
60
60
|
*/
|
|
61
61
|
droppedBatches?: PendingDropped[];
|
|
62
|
+
/** Pipeline progress cursors — persist across restarts and fork recovery. */
|
|
63
|
+
cursors?: {
|
|
64
|
+
observer?: { entryId: string; state: string };
|
|
65
|
+
reflector?: { entryId: string; state: string };
|
|
66
|
+
dropper?: { entryId: string; state: string };
|
|
67
|
+
};
|
|
62
68
|
}
|
|
63
69
|
|
|
64
70
|
// ── Persistence ─────────────────────────────────────────────────────────────
|
|
@@ -88,6 +94,8 @@ function defaultState(): PendingOMState {
|
|
|
88
94
|
}
|
|
89
95
|
|
|
90
96
|
function isEmptyState(s: PendingOMState): boolean {
|
|
97
|
+
const hasCursors = s.cursors && (s.cursors.observer || s.cursors.reflector || s.cursors.dropper);
|
|
98
|
+
if (hasCursors) return false;
|
|
91
99
|
return !s.observation && !s.reflection && !s.dropped
|
|
92
100
|
&& (!s.observationBatches || s.observationBatches.length === 0)
|
|
93
101
|
&& (!s.reflectionBatches || s.reflectionBatches.length === 0)
|
|
@@ -159,7 +167,9 @@ function isPendingOMState(value: unknown): value is PendingOMState {
|
|
|
159
167
|
const hasDrop = !!(v.dropped && typeof v.dropped === "object" && typeof (v.dropped as any).coversUpToId === "string");
|
|
160
168
|
// Also accept states with only batch arrays (no singular fields)
|
|
161
169
|
const hasBatches = Array.isArray(v.observationBatches) || Array.isArray(v.reflectionBatches) || Array.isArray(v.droppedBatches);
|
|
162
|
-
|
|
170
|
+
// Accept cursor-only states (no observations/reflections yet, but persisted)
|
|
171
|
+
const hasCursors = !!(v.cursors && typeof v.cursors === "object");
|
|
172
|
+
return hasObs || hasRef || hasDrop || hasBatches || hasCursors;
|
|
163
173
|
}
|
|
164
174
|
|
|
165
175
|
/**
|
|
@@ -268,6 +278,20 @@ export function hasPendingData(sessionId: string): boolean {
|
|
|
268
278
|
return !isEmptyState(readSessionState(sessionId));
|
|
269
279
|
}
|
|
270
280
|
|
|
281
|
+
/** Read cursors from pending state for a session. */
|
|
282
|
+
export function readPendingCursors(sessionId: string): PendingOMState["cursors"] {
|
|
283
|
+
const state = readSessionState(sessionId);
|
|
284
|
+
return state.cursors;
|
|
285
|
+
}
|
|
286
|
+
|
|
287
|
+
/** Write cursors to pending state for a session (replaces existing cursors).
|
|
288
|
+
* Uses assignment (not merge) so deletions from validateCursors persist. */
|
|
289
|
+
export function writePendingCursors(sessionId: string, cursors: PendingOMState["cursors"]): void {
|
|
290
|
+
const state = readSessionState(sessionId);
|
|
291
|
+
state.cursors = { ...cursors };
|
|
292
|
+
writeSessionState(sessionId, state);
|
|
293
|
+
}
|
|
294
|
+
|
|
271
295
|
/**
|
|
272
296
|
* List all session IDs that have pending data by scanning the pending directory
|
|
273
297
|
* for *-pending.json files.
|
package/src/om/runtime.ts
CHANGED
|
@@ -11,6 +11,8 @@
|
|
|
11
11
|
*/
|
|
12
12
|
import { type Config, type ConfiguredModel, DEFAULTS, loadConfig } from "./config.js";
|
|
13
13
|
import { isCooldownActive, getCooldownEntry, recordCooldown, expireCooldowns, modelKey } from "./cooldown.js";
|
|
14
|
+
import { readPendingCursors, writePendingCursors } from "./pending.js";
|
|
15
|
+
import type { PendingOMState } from "./pending.js";
|
|
14
16
|
|
|
15
17
|
export type ResolveResult =
|
|
16
18
|
| { ok: true; model: any; apiKey: string; headers?: Record<string, string>; cooldownApplied?: boolean }
|
|
@@ -20,6 +22,19 @@ type NotifyLevel = "warning" | "info" | "error";
|
|
|
20
22
|
type Notify = (message: string, type?: NotifyLevel) => void;
|
|
21
23
|
export type ConsolidationPhase = "observer" | "reflector" | "dropper";
|
|
22
24
|
|
|
25
|
+
export type CursorState = "initial" | "recorded" | "empty" | "error" | "skipped" | "not_due";
|
|
26
|
+
|
|
27
|
+
export interface PipelineCursor {
|
|
28
|
+
entryId: string;
|
|
29
|
+
state: CursorState;
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
export interface PipelineCursors {
|
|
33
|
+
observer?: PipelineCursor;
|
|
34
|
+
reflector?: PipelineCursor;
|
|
35
|
+
dropper?: PipelineCursor;
|
|
36
|
+
}
|
|
37
|
+
|
|
23
38
|
export interface ResolveCtx {
|
|
24
39
|
model: unknown;
|
|
25
40
|
modelRegistry: any;
|
|
@@ -64,6 +79,10 @@ export class Runtime {
|
|
|
64
79
|
compactionStats: { summarized: number; kept: number; keptTokensEst: number } | null = null;
|
|
65
80
|
/** Whether the most recent compaction was triggered by /blackhole (vs auto-compact). */
|
|
66
81
|
compactWasPiVcc = false;
|
|
82
|
+
/** In‑memory pipeline cursors — authoritative copy for gating decisions. */
|
|
83
|
+
cursors: PipelineCursors = {};
|
|
84
|
+
/** Session ID for which cursors have been loaded/validated. Undefined until first load. */
|
|
85
|
+
cursorsLoadedSessionId: string | undefined = undefined;
|
|
67
86
|
|
|
68
87
|
ensureConfig(cwd: string): void {
|
|
69
88
|
if (this.configLoaded) return;
|
|
@@ -251,6 +270,57 @@ export class Runtime {
|
|
|
251
270
|
return Date.now() - this.lastConsolidationErrorAt < CONSOLIDATION_RETRY_COOLDOWN_MS;
|
|
252
271
|
}
|
|
253
272
|
|
|
273
|
+
/** Get the current cursor for a pipeline stage. */
|
|
274
|
+
getCursor(stage: ConsolidationPhase): PipelineCursor | undefined {
|
|
275
|
+
return this.cursors[stage];
|
|
276
|
+
}
|
|
277
|
+
|
|
278
|
+
/** Advance a stage's cursor to a new entry ID with the given state. */
|
|
279
|
+
advanceCursor(stage: ConsolidationPhase, entryId: string, state: CursorState): void {
|
|
280
|
+
this.cursors[stage] = { entryId, state };
|
|
281
|
+
}
|
|
282
|
+
|
|
283
|
+
/** Load cursors from the per‑session pending file into the in‑memory map. */
|
|
284
|
+
loadCursorsFromPending(sessionId: string): void {
|
|
285
|
+
try {
|
|
286
|
+
const stored = readPendingCursors(sessionId);
|
|
287
|
+
if (!stored) return;
|
|
288
|
+
if (stored.observer?.entryId && stored.observer?.state) {
|
|
289
|
+
this.cursors.observer = { entryId: stored.observer.entryId, state: stored.observer.state as CursorState };
|
|
290
|
+
}
|
|
291
|
+
if (stored.reflector?.entryId && stored.reflector?.state) {
|
|
292
|
+
this.cursors.reflector = { entryId: stored.reflector.entryId, state: stored.reflector.state as CursorState };
|
|
293
|
+
}
|
|
294
|
+
if (stored.dropper?.entryId && stored.dropper?.state) {
|
|
295
|
+
this.cursors.dropper = { entryId: stored.dropper.entryId, state: stored.dropper.state as CursorState };
|
|
296
|
+
}
|
|
297
|
+
} catch {
|
|
298
|
+
// Best‑effort: missing or corrupt files are harmless.
|
|
299
|
+
}
|
|
300
|
+
}
|
|
301
|
+
|
|
302
|
+
/** Save in‑memory cursors to the per‑session pending file (synchronous, for tests). */
|
|
303
|
+
saveCursorsToPending(sessionId: string): void {
|
|
304
|
+
try {
|
|
305
|
+
writePendingCursors(sessionId, this.cursors as PendingOMState["cursors"]);
|
|
306
|
+
} catch {
|
|
307
|
+
// Best‑effort: graceful degradation on read‑only filesystems.
|
|
308
|
+
}
|
|
309
|
+
}
|
|
310
|
+
|
|
311
|
+
/** Schedule an async flush of cursors to the pending file.
|
|
312
|
+
* Uses a micro‑task to avoid blocking the pipeline. */
|
|
313
|
+
scheduleCursorFlush(sessionId: string): void {
|
|
314
|
+
const cursors = { ...this.cursors };
|
|
315
|
+
queueMicrotask(() => {
|
|
316
|
+
try {
|
|
317
|
+
writePendingCursors(sessionId, cursors as PendingOMState["cursors"]);
|
|
318
|
+
} catch {
|
|
319
|
+
// Best‑effort: graceful degradation.
|
|
320
|
+
}
|
|
321
|
+
});
|
|
322
|
+
}
|
|
323
|
+
|
|
254
324
|
launchConsolidationTask(ctx: LaunchCtx, work: () => Promise<void>): Promise<void> {
|
|
255
325
|
this.consolidationInFlight = true;
|
|
256
326
|
this.consolidationPhase = undefined;
|
package/src/tools/recall.ts
CHANGED
|
@@ -238,13 +238,10 @@ export function registerRecallTool(pi: ExtensionAPI): void {
|
|
|
238
238
|
"Search session history and earlier lines omitted, file write/edit content by text/regex. " +
|
|
239
239
|
"Expand entries (#N), drill-down file content (#N:path) with paging, or aggregate touched files (mode:touched).",
|
|
240
240
|
promptSnippet:
|
|
241
|
-
"recall: Search session history + file write/edit content by text/regex. #N expand, #N:path drill-down with offset
|
|
241
|
+
"recall: Search session history + file write/edit content by text/regex. #N expand, #N:path drill-down with optional :offset:limit or :full, mode:file/touched.",
|
|
242
242
|
promptGuidelines: [
|
|
243
|
-
"Use recall —
|
|
244
|
-
"Use recall —
|
|
245
|
-
"Use recall — drill-down supports paging: #42:auth.ts shows first 30 lines, #42:auth.ts:30 shows next 30, #42:auth.ts:full shows everything. Note: edit diffs are not indexed for text search — drill-down reads them from raw JSONL.",
|
|
246
|
-
"Use recall — when a drill-down path matches multiple files, options are listed. Narrow with a more specific path substring.",
|
|
247
|
-
"Use recall — hex observation/reflection ids (12-char hex) link memory evidence to session entries with cross-references for navigation.",
|
|
243
|
+
"Use recall — literal text/regex search across session history and file write/edit content. #N expands an entry; #N:path with optional :offset:limit or :full drills down into file content; 12-char hex ids recover observation/reflection sources. mode:file for file-content-only, mode:touched for aggregated files-by-path. scope:'all' to search the full session. If no results, try fewer terms or a regex pattern.",
|
|
244
|
+
"Use recall — when a drill-down path matches multiple files, options are listed. Narrow with a more specific path substring. Only full-file writes are indexed for text search (edit diffs are not).",
|
|
248
245
|
],
|
|
249
246
|
parameters: Type.Object({
|
|
250
247
|
query: Type.Optional(
|
|
@@ -260,7 +257,7 @@ export function registerRecallTool(pi: ExtensionAPI): void {
|
|
|
260
257
|
StringEnum(["lineage", "all"] as const, { description: "Search scope. lineage = active lineage (default), all = entire session." }),
|
|
261
258
|
),
|
|
262
259
|
mode: Type.Optional(
|
|
263
|
-
StringEnum(["hybrid", "file", "
|
|
260
|
+
StringEnum(["hybrid", "file", "touched"] as const, { description: "What content to search. hybrid (default) = all session content. file = file content only. touched = files-by-path summary with entry indices." }),
|
|
264
261
|
),
|
|
265
262
|
}),
|
|
266
263
|
async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
|