pi-observational-memory-extension 0.1.1 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -0
- package/README.md +23 -12
- package/extensions/index.ts +433 -68
- package/package.json +32 -5
- package/scripts/test.mjs +70 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [0.1.3] - 2026-06-23
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- **settings**: Added the unified `/om` command with `status`, `set`, `reset`, `enable`, `disable`, `observe`, `reflect`, and `memory` actions.
|
|
12
|
+
- **settings**: Persisted model overrides, thresholds, caveman compression mode, attachment observation toggle, and scope configuration in OM state.
|
|
13
|
+
- **tests**: Added a focused regression test suite for redaction, prompt limiting, stale lock recovery, settings parsing, v1→v2 migration, and atomic writes.
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
- **durability**: State files now use atomic temp-file writes with `.bak` recovery support.
|
|
17
|
+
- **stability**: Stale operation locks are recovered automatically instead of blocking future OM runs forever.
|
|
18
|
+
- **security**: State, debug logs, recall/observer serialization, and model outputs now redact common API keys, npm tokens, GitHub tokens, bearer tokens, passwords, and secret fields.
|
|
19
|
+
- **context**: Observer prompts now cap previous observations to a bounded tail to prevent observation-context bloat.
|
|
20
|
+
|
|
21
|
+
## [0.1.2] - 2026-06-22
|
|
22
|
+
|
|
23
|
+
### Fixed
|
|
24
|
+
- **om**: Resolved "stale context" errors during asynchronous `session_shutdown` handlers and background buffer tasks.
|
|
25
|
+
- **om**: Safely caught uncaught promise rejections within internal `bufferObservation` tasks when contexts are deactivated by Pi.
|
|
26
|
+
|
|
27
|
+
## [0.1.1] - 2026-06-22
|
|
28
|
+
|
|
29
|
+
### Added
|
|
30
|
+
- **om**: Added capability to force immediate observational compaction on `session_shutdown` if pending message tokens exist. This guarantees subagent and short-lived loopflow sessions persist their memories before exiting.
|
|
31
|
+
|
|
32
|
+
## [0.1.0] - 2026-06-22
|
|
33
|
+
|
|
34
|
+
### Added
|
|
35
|
+
- **om**: First release of the Mastra-style **Observational Memory (OM)** extension.
|
|
36
|
+
- **om**: Implemented the 3-Agent psychological memory model: Actor (Main Agent), Observer (Extraction Agent), and Reflector (Consolidation/Compression Agent).
|
|
37
|
+
- **om**: Automatically intercepts `/compact` and `session_before_compact` events.
|
|
38
|
+
- **om**: Real-time TUI status bar panel underneath the input showing memory, message, and token thresholds.
|
|
39
|
+
- **om**: Color-coded, responsive status indicators for high (🔴), medium (🟡), and low (🟢) priority observations.
|
|
40
|
+
- **om**: Dedicated interactive, fullscreen overlay `/om-memory` with interactive tab switching (`Memory`, `Status`, `Debug`) and ANSI word wrapping.
|
package/README.md
CHANGED
|
@@ -27,27 +27,36 @@ Legacy compaction compresses raw history into a single monolithic block of text,
|
|
|
27
27
|
- **Recall-driven Retrieval:** Exposes an `om_recall` tool to the Actor so it can retrieve full, raw message payloads from observed history when exact code, quotes, or numbers are needed.
|
|
28
28
|
- **Custom Compaction Hook:** Plugs directly into Pi's `session_before_compact` lifecycle event. Typing `/compact` or triggering auto-compaction launches the Observer/Reflector memory consolidation flow instead of Pi's legacy summary compaction.
|
|
29
29
|
- **TUI Overlay Panel:** Fully custom, responsive, multi-tab overlay (`/om-memory`) in interactive mode for inspecting memory. Features tabbed navigation, smooth scroll, and precise border framing.
|
|
30
|
-
- **Durable Persistence:** Serializes and loads state files safely under `.pi/om/<session-id>.json`. Outputs JSON-formatted diagnostic logs to `.pi/om/debug/` for every operation.
|
|
30
|
+
- **Durable Persistence:** Serializes and loads state files safely under `.pi/om/<session-id>.json` using atomic writes and `.bak` recovery. Outputs JSON-formatted diagnostic logs to `.pi/om/debug/` for every operation.
|
|
31
|
+
- **Secret Redaction:** Redacts common API keys, npm/GitHub tokens, bearer tokens, passwords, and secret fields before writing state/debug artifacts or observer text.
|
|
32
|
+
- **Stale Lock Recovery:** Recovers old interrupted OM operation locks automatically so a crashed or killed process does not block future memory runs.
|
|
33
|
+
- **Bounded Observer Context:** Sends only a safe tail of previous observations to the Observer, preventing recursive prompt bloat while preserving full memory in the main runtime.
|
|
31
34
|
|
|
32
35
|
---
|
|
33
36
|
|
|
34
37
|
## 📋 Configuration & Settings
|
|
35
38
|
|
|
36
|
-
|
|
39
|
+
Use `/om` in Pi to inspect and update runtime settings. Settings are persisted in the session OM state file under `.pi/om/<session-id>.json`.
|
|
37
40
|
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
41
|
+
```bash
|
|
42
|
+
/om
|
|
43
|
+
/om set observation-threshold 30000
|
|
44
|
+
/om set reflection-threshold 40000
|
|
45
|
+
/om set block-after 36000
|
|
46
|
+
/om set buffer-tokens 6000
|
|
47
|
+
/om set buffer-activation 80%
|
|
48
|
+
/om set observation-model google/gemini-2.5-flash
|
|
49
|
+
/om set reflection-model google/gemini-2.5-flash
|
|
50
|
+
/om set caveman on
|
|
51
|
+
/om set attachments off
|
|
52
|
+
/om reset
|
|
46
53
|
```
|
|
47
54
|
|
|
48
|
-
- **Observation Trigger Threshold:**
|
|
49
|
-
- **Reflection Trigger Threshold:**
|
|
55
|
+
- **Observation Trigger Threshold:** default `30,000` raw message tokens.
|
|
56
|
+
- **Reflection Trigger Threshold:** default `40,000` observation tokens.
|
|
50
57
|
- **Default Models:** `google/gemini-2.5-flash` with 0.3 temperature for Observer, and 0.0 temperature for Reflector.
|
|
58
|
+
- **Caveman Mode:** optional terse compression style for denser memory.
|
|
59
|
+
- **Attachment Observation Toggle:** controls whether image/attachment placeholders are exposed to observation text.
|
|
51
60
|
|
|
52
61
|
---
|
|
53
62
|
|
|
@@ -55,6 +64,7 @@ You can customize Observational Memory behaviors by specifying fields inside you
|
|
|
55
64
|
|
|
56
65
|
This extension registers the following slash commands in Pi:
|
|
57
66
|
|
|
67
|
+
- `/om` — Unified settings/status command. Supports `set`, `reset`, `enable`, `disable`, `observe`, `reflect`, and `memory`.
|
|
58
68
|
- `/om-status` — Shows a detailed breakdown of pending/observation tokens, active locks, thresholds, and last operation results.
|
|
59
69
|
- `/om-memory` — Opens the interactive multi-tab overlay panel (observations, status stats, and background debug details).
|
|
60
70
|
- `/om-observe` — Forces an immediate Observation pass on all pending raw message history.
|
|
@@ -88,6 +98,7 @@ Run the typechecker and validation scripts before packaging or releasing:
|
|
|
88
98
|
```bash
|
|
89
99
|
npm run typecheck
|
|
90
100
|
npm run validate
|
|
101
|
+
npm test
|
|
91
102
|
```
|
|
92
103
|
|
|
93
104
|
### Commit Guidelines
|
package/extensions/index.ts
CHANGED
|
@@ -7,12 +7,12 @@ import {
|
|
|
7
7
|
} from "@earendil-works/pi-coding-agent";
|
|
8
8
|
import { Key, matchesKey, truncateToWidth, wrapTextWithAnsi } from "@earendil-works/pi-tui";
|
|
9
9
|
import { Type } from "typebox";
|
|
10
|
-
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
|
10
|
+
import { copyFile, mkdir, readFile, rename, unlink, writeFile } from "node:fs/promises";
|
|
11
11
|
import { existsSync } from "node:fs";
|
|
12
12
|
import { basename, dirname, join } from "node:path";
|
|
13
13
|
|
|
14
14
|
const EXTENSION_ID = "pi-observational-memory";
|
|
15
|
-
const STATE_VERSION =
|
|
15
|
+
const STATE_VERSION = 2;
|
|
16
16
|
const DEFAULT_OBSERVATION_MODEL = "google/gemini-2.5-flash";
|
|
17
17
|
const DEFAULT_REFLECTION_MODEL = "google/gemini-2.5-flash";
|
|
18
18
|
const OBSERVATION_THRESHOLD = 30_000;
|
|
@@ -25,6 +25,9 @@ const TOOL_RESULT_MAX_CHARS = 8_000;
|
|
|
25
25
|
const MESSAGE_PART_MAX_CHARS = 20_000;
|
|
26
26
|
const MAX_OBSERVATION_LINE_CHARS = 10_000;
|
|
27
27
|
const MAX_RESTART_ERRORS = 3;
|
|
28
|
+
const STALE_OPERATION_LOCK_MS = 15 * 60 * 1000;
|
|
29
|
+
const PREVIOUS_OBSERVATIONS_MAX_TOKENS = 2_000;
|
|
30
|
+
const ATOMIC_BACKUP_SUFFIX = ".bak";
|
|
28
31
|
|
|
29
32
|
const OBSERVATION_CONTEXT_PROMPT = `The following observations block contains your memory of past conversations with this user.`;
|
|
30
33
|
const OBSERVATION_CONTEXT_INSTRUCTIONS = `IMPORTANT: When responding, reference specific details from these observations. Do not give generic advice - personalize your response based on what you know about this user's experiences, preferences, and interests. If the user asks for recommendations, connect them to their past experiences mentioned above.
|
|
@@ -165,7 +168,8 @@ const OBSERVER_GUIDELINES = `- Be specific enough for the assistant to act on
|
|
|
165
168
|
- Observe WHAT the agent did and WHAT it means
|
|
166
169
|
- If the user provides detailed messages or code snippets, observe all important details`;
|
|
167
170
|
|
|
168
|
-
function buildObserverSystemPrompt(): string {
|
|
171
|
+
function buildObserverSystemPrompt(caveman = false): string {
|
|
172
|
+
const cavemanInstruction = caveman ? `\n\nCAVEMAN MODE: Write brutally short, dense observations. Remove filler. Preserve facts, decisions, dates, paths, and errors only.` : "";
|
|
169
173
|
return `You are the memory consciousness of an AI assistant. Your observations will be the ONLY information the assistant has about past interactions with this user.
|
|
170
174
|
|
|
171
175
|
Extract observations that will help the assistant remember:
|
|
@@ -188,20 +192,22 @@ Do NOT add thread identifiers, thread IDs, or <thread> tags to your observations
|
|
|
188
192
|
|
|
189
193
|
Remember: These observations are the assistant's ONLY memory. Make them count.
|
|
190
194
|
|
|
191
|
-
User messages are extremely important. If the user asks a question or gives a new task, make it clear in <current-task> that this is the priority. If the assistant needs to respond to the user, indicate in <suggested-response> that it should pause for user reply before continuing other tasks
|
|
195
|
+
User messages are extremely important. If the user asks a question or gives a new task, make it clear in <current-task> that this is the priority. If the assistant needs to respond to the user, indicate in <suggested-response> that it should pause for user reply before continuing other tasks.${cavemanInstruction}`;
|
|
192
196
|
}
|
|
193
197
|
|
|
194
198
|
function buildObserverTaskPrompt(existingObservations: string | undefined, opts: { priorCurrentTask?: string; priorSuggestedResponse?: string; wasTruncated?: boolean } = {}): string {
|
|
195
199
|
let prompt = "";
|
|
196
|
-
|
|
197
|
-
|
|
200
|
+
const limitedExisting = limitTextByTokens(existingObservations, PREVIOUS_OBSERVATIONS_MAX_TOKENS);
|
|
201
|
+
const previousWasTruncated = Boolean(existingObservations?.trim()) && limitedExisting !== existingObservations;
|
|
202
|
+
if (limitedExisting?.trim()) {
|
|
203
|
+
prompt += `## Previous Observations\n\n${limitedExisting}\n\n---\n\nDo not repeat these existing observations. Your new observations will be appended to the existing observations. Previous observations may be truncated to the most recent/relevant tail for context budget safety.\n\n`;
|
|
198
204
|
}
|
|
199
205
|
const metadata: string[] = [];
|
|
200
206
|
if (opts.priorCurrentTask) metadata.push(`- prior current-task: ${opts.priorCurrentTask}`);
|
|
201
207
|
if (opts.priorSuggestedResponse) metadata.push(`- prior suggested-response: ${opts.priorSuggestedResponse}`);
|
|
202
208
|
if (metadata.length) {
|
|
203
209
|
prompt += `## Prior Thread Metadata\n\n${metadata.join("\n")}\n\n`;
|
|
204
|
-
if (opts.wasTruncated) {
|
|
210
|
+
if (opts.wasTruncated || previousWasTruncated) {
|
|
205
211
|
prompt += `Previous observations were truncated for context budget reasons. The main agent still has full memory context outside this observer window.\n`;
|
|
206
212
|
}
|
|
207
213
|
prompt += `Use prior current-task and suggested-response as continuity hints, then update them based on the new messages.\n\n---\n\n`;
|
|
@@ -210,7 +216,8 @@ function buildObserverTaskPrompt(existingObservations: string | undefined, opts:
|
|
|
210
216
|
return prompt;
|
|
211
217
|
}
|
|
212
218
|
|
|
213
|
-
function buildReflectorSystemPrompt(): string {
|
|
219
|
+
function buildReflectorSystemPrompt(caveman = false): string {
|
|
220
|
+
const cavemanInstruction = caveman ? `\n\nCAVEMAN MODE: Compress aggressively. Prefer terse facts over prose. Preserve only actionable, durable memory.` : "";
|
|
214
221
|
return `You are the memory consciousness of an AI assistant. Your memory observation reflections will be the ONLY information the assistant has about past interactions with this user.
|
|
215
222
|
|
|
216
223
|
The following instructions were given to another part of your psyche (the observer) to create memories.
|
|
@@ -258,7 +265,7 @@ State current task(s) explicitly: primary and secondary pending tasks. Mark wait
|
|
|
258
265
|
Hint for the agent's immediate next message.
|
|
259
266
|
</suggested-response>
|
|
260
267
|
|
|
261
|
-
User messages are extremely important. If the user asks a question or gives a new task, make it clear in <current-task> that this is the priority. If the assistant needs to respond, indicate in <suggested-response> that it should pause for user reply before continuing other tasks
|
|
268
|
+
User messages are extremely important. If the user asks a question or gives a new task, make it clear in <current-task> that this is the priority. If the assistant needs to respond, indicate in <suggested-response> that it should pause for user reply before continuing other tasks.${cavemanInstruction}`;
|
|
262
269
|
}
|
|
263
270
|
|
|
264
271
|
type CompressionLevel = 0 | 1 | 2 | 3 | 4;
|
|
@@ -292,6 +299,13 @@ type BufferedChunk = {
|
|
|
292
299
|
createdAt: string;
|
|
293
300
|
};
|
|
294
301
|
|
|
302
|
+
type PiOMSettings = {
|
|
303
|
+
observationModel: string;
|
|
304
|
+
reflectionModel: string;
|
|
305
|
+
caveman: boolean;
|
|
306
|
+
observeAttachments: boolean;
|
|
307
|
+
};
|
|
308
|
+
|
|
295
309
|
type PiOMRecord = {
|
|
296
310
|
version: number;
|
|
297
311
|
enabled: boolean;
|
|
@@ -307,6 +321,7 @@ type PiOMRecord = {
|
|
|
307
321
|
pendingMessageTokens: number;
|
|
308
322
|
observationTokens: number;
|
|
309
323
|
thresholds: { observation: number; reflection: number; blockAfter: number; bufferTokens: number; bufferActivation: number };
|
|
324
|
+
settings: PiOMSettings;
|
|
310
325
|
buffered: { observations: BufferedChunk[]; reflection?: BufferedChunk };
|
|
311
326
|
operationLock?: { type: OperationType; startedAt: string };
|
|
312
327
|
lastOperation?: { type: OperationType; startedAt: string; endedAt?: string; inputTokens: number; outputTokens?: number; error?: string; model?: string; compressionLevel?: number };
|
|
@@ -380,7 +395,13 @@ export default function (pi: ExtensionAPI) {
|
|
|
380
395
|
runtime.currentOperation?.abort();
|
|
381
396
|
if (runtime.state) {
|
|
382
397
|
try {
|
|
383
|
-
|
|
398
|
+
let isStale = false;
|
|
399
|
+
try {
|
|
400
|
+
const _ = ctx.cwd;
|
|
401
|
+
} catch {
|
|
402
|
+
isStale = true;
|
|
403
|
+
}
|
|
404
|
+
if (!isStale && runtime.state.enabled && runtime.state.pendingMessageTokens > 0) {
|
|
384
405
|
await observeNow(ctx, { force: true, reason: "session_shutdown" });
|
|
385
406
|
}
|
|
386
407
|
} catch (error) {
|
|
@@ -388,8 +409,12 @@ export default function (pi: ExtensionAPI) {
|
|
|
388
409
|
}
|
|
389
410
|
await saveState(runtime.state);
|
|
390
411
|
}
|
|
391
|
-
|
|
392
|
-
|
|
412
|
+
try {
|
|
413
|
+
ctx.ui.setStatus("om", undefined);
|
|
414
|
+
ctx.ui.setWidget("om", undefined);
|
|
415
|
+
} catch (e) {
|
|
416
|
+
// Ignore if UI or context is already stale
|
|
417
|
+
}
|
|
393
418
|
});
|
|
394
419
|
|
|
395
420
|
pi.on("model_select", async (event, ctx) => {
|
|
@@ -555,6 +580,71 @@ export default function (pi: ExtensionAPI) {
|
|
|
555
580
|
},
|
|
556
581
|
});
|
|
557
582
|
|
|
583
|
+
pi.registerCommand("om", {
|
|
584
|
+
description: "Manage Observational Memory settings. Usage: /om, /om set <key> <value>, /om enable|disable|observe|reflect|memory",
|
|
585
|
+
handler: async (args, ctx) => {
|
|
586
|
+
const state = await ensureState(ctx);
|
|
587
|
+
const input = (args || "").trim();
|
|
588
|
+
if (!input || input === "status") {
|
|
589
|
+
await refreshCounts(ctx);
|
|
590
|
+
ctx.ui.notify(formatSettingsText(state), state.status === "failed" ? "error" : "info");
|
|
591
|
+
updateStatus(ctx);
|
|
592
|
+
return;
|
|
593
|
+
}
|
|
594
|
+
const [cmd, ...rest] = input.split(/\s+/);
|
|
595
|
+
if (cmd === "enable") {
|
|
596
|
+
state.enabled = true;
|
|
597
|
+
state.status = "idle";
|
|
598
|
+
await saveState(state);
|
|
599
|
+
updateStatus(ctx);
|
|
600
|
+
ctx.ui.notify("Observational Memory enabled", "info");
|
|
601
|
+
return;
|
|
602
|
+
}
|
|
603
|
+
if (cmd === "disable") {
|
|
604
|
+
state.enabled = false;
|
|
605
|
+
state.status = "disabled";
|
|
606
|
+
await saveState(state);
|
|
607
|
+
updateStatus(ctx);
|
|
608
|
+
ctx.ui.notify("Observational Memory disabled", "warning");
|
|
609
|
+
return;
|
|
610
|
+
}
|
|
611
|
+
if (cmd === "observe") {
|
|
612
|
+
await observeNow(ctx, { force: true, reason: "manual" });
|
|
613
|
+
updateStatus(ctx);
|
|
614
|
+
ctx.ui.notify("OM observation complete", "info");
|
|
615
|
+
return;
|
|
616
|
+
}
|
|
617
|
+
if (cmd === "reflect") {
|
|
618
|
+
await reflectNow(ctx, { reason: "manual", manualPrompt: rest.join(" ") || undefined });
|
|
619
|
+
updateStatus(ctx);
|
|
620
|
+
ctx.ui.notify("OM reflection complete", "info");
|
|
621
|
+
return;
|
|
622
|
+
}
|
|
623
|
+
if (cmd === "memory") {
|
|
624
|
+
ctx.ui.notify(formatMemoryText(state), "info");
|
|
625
|
+
return;
|
|
626
|
+
}
|
|
627
|
+
if (cmd === "set") {
|
|
628
|
+
const key = rest[0];
|
|
629
|
+
const value = rest.slice(1).join(" ");
|
|
630
|
+
applySetting(state, key, value);
|
|
631
|
+
await saveState(state);
|
|
632
|
+
updateStatus(ctx);
|
|
633
|
+
ctx.ui.notify(`OM setting updated: ${key}=${value}`, "info");
|
|
634
|
+
return;
|
|
635
|
+
}
|
|
636
|
+
if (cmd === "reset") {
|
|
637
|
+
state.thresholds = defaultThresholds();
|
|
638
|
+
state.settings = defaultSettings();
|
|
639
|
+
await saveState(state);
|
|
640
|
+
updateStatus(ctx);
|
|
641
|
+
ctx.ui.notify("OM settings reset to defaults", "info");
|
|
642
|
+
return;
|
|
643
|
+
}
|
|
644
|
+
throw new Error(`Unknown /om command: ${cmd}. Use /om, /om set <key> <value>, /om enable|disable|observe|reflect|memory|reset`);
|
|
645
|
+
},
|
|
646
|
+
});
|
|
647
|
+
|
|
558
648
|
pi.registerCommand("om-compact", {
|
|
559
649
|
description: "Run Pi compaction; pi-observational-memory will replace the summary with OM.",
|
|
560
650
|
handler: async (args, ctx) => {
|
|
@@ -568,6 +658,14 @@ export default function (pi: ExtensionAPI) {
|
|
|
568
658
|
}
|
|
569
659
|
|
|
570
660
|
async function safeBoundary(ctx: any, boundary: string): Promise<void> {
|
|
661
|
+
let isStale = false;
|
|
662
|
+
try {
|
|
663
|
+
const _ = ctx.cwd;
|
|
664
|
+
} catch {
|
|
665
|
+
isStale = true;
|
|
666
|
+
}
|
|
667
|
+
if (isStale) return;
|
|
668
|
+
|
|
571
669
|
const state = await ensureState(ctx);
|
|
572
670
|
if (!state.enabled || state.status === "failed") return;
|
|
573
671
|
await refreshCounts(ctx);
|
|
@@ -586,17 +684,40 @@ async function safeBoundary(ctx: any, boundary: string): Promise<void> {
|
|
|
586
684
|
}
|
|
587
685
|
if (shouldBuffer(state)) {
|
|
588
686
|
void bufferObservation(ctx, `${boundary}:buffer`).catch(async (error) => {
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
687
|
+
try {
|
|
688
|
+
let innerStale = false;
|
|
689
|
+
try {
|
|
690
|
+
const _ = ctx.cwd;
|
|
691
|
+
} catch {
|
|
692
|
+
innerStale = true;
|
|
693
|
+
}
|
|
694
|
+
if (innerStale) return;
|
|
695
|
+
|
|
696
|
+
const s = await ensureState(ctx);
|
|
697
|
+
s.status = "failed";
|
|
698
|
+
s.lastError = `OM buffer failed: ${errorMessage(error)}`;
|
|
699
|
+
await saveState(s);
|
|
700
|
+
updateStatus(ctx);
|
|
701
|
+
ctx.ui.notify(s.lastError, "error");
|
|
702
|
+
} catch (innerError) {
|
|
703
|
+
// Prevent any uncaught exceptions from background handler
|
|
704
|
+
}
|
|
595
705
|
});
|
|
596
706
|
}
|
|
597
707
|
}
|
|
598
708
|
|
|
599
709
|
async function ensureState(ctx: any): Promise<PiOMRecord> {
|
|
710
|
+
let isStale = false;
|
|
711
|
+
try {
|
|
712
|
+
const _ = ctx.cwd;
|
|
713
|
+
} catch {
|
|
714
|
+
isStale = true;
|
|
715
|
+
}
|
|
716
|
+
if (isStale) {
|
|
717
|
+
if (runtime.state) return runtime.state;
|
|
718
|
+
throw new Error("Cannot ensure state because the extension context is stale.");
|
|
719
|
+
}
|
|
720
|
+
|
|
600
721
|
if (runtime.state && runtime.state.cwd === ctx.cwd && runtime.state.sessionFile === ctx.sessionManager.getSessionFile()) return runtime.state;
|
|
601
722
|
const sessionFile = ctx.sessionManager.getSessionFile?.();
|
|
602
723
|
const sessionId = ctx.sessionManager.getSessionId?.() || (sessionFile ? basename(sessionFile, ".jsonl") : "in-memory");
|
|
@@ -609,48 +730,141 @@ async function ensureState(ctx: any): Promise<PiOMRecord> {
|
|
|
609
730
|
let state: PiOMRecord | undefined;
|
|
610
731
|
if (existsSync(statePath)) {
|
|
611
732
|
try {
|
|
612
|
-
state = JSON.parse(await readFile(statePath, "utf8"))
|
|
733
|
+
state = normalizeState(JSON.parse(await readFile(statePath, "utf8")), { sessionId, sessionFile, cwd: ctx.cwd });
|
|
613
734
|
} catch (error) {
|
|
614
|
-
|
|
735
|
+
const backupPath = `${statePath}${ATOMIC_BACKUP_SUFFIX}`;
|
|
736
|
+
if (existsSync(backupPath)) {
|
|
737
|
+
try {
|
|
738
|
+
state = normalizeState(JSON.parse(await readFile(backupPath, "utf8")), { sessionId, sessionFile, cwd: ctx.cwd });
|
|
739
|
+
} catch {
|
|
740
|
+
throw new Error(`Failed to load OM state ${statePath} and backup ${backupPath}: ${errorMessage(error)}`);
|
|
741
|
+
}
|
|
742
|
+
} else {
|
|
743
|
+
throw new Error(`Failed to load OM state ${statePath}: ${errorMessage(error)}`);
|
|
744
|
+
}
|
|
615
745
|
}
|
|
616
746
|
}
|
|
617
|
-
if (!state
|
|
618
|
-
state = {
|
|
619
|
-
version: STATE_VERSION,
|
|
620
|
-
enabled: true,
|
|
621
|
-
sessionId,
|
|
622
|
-
sessionFile,
|
|
623
|
-
cwd: ctx.cwd,
|
|
624
|
-
scope: "session",
|
|
625
|
-
status: "idle",
|
|
626
|
-
observations: "",
|
|
627
|
-
pendingMessageTokens: 0,
|
|
628
|
-
observationTokens: 0,
|
|
629
|
-
thresholds: {
|
|
630
|
-
observation: OBSERVATION_THRESHOLD,
|
|
631
|
-
reflection: REFLECTION_THRESHOLD,
|
|
632
|
-
blockAfter: Math.round(OBSERVATION_THRESHOLD * DEFAULT_BLOCK_AFTER_MULTIPLIER),
|
|
633
|
-
bufferTokens: Math.round(OBSERVATION_THRESHOLD * BUFFER_TOKENS_RATIO),
|
|
634
|
-
bufferActivation: BUFFER_ACTIVATION_RATIO,
|
|
635
|
-
},
|
|
636
|
-
buffered: { observations: [] },
|
|
637
|
-
updatedAt: new Date().toISOString(),
|
|
638
|
-
};
|
|
747
|
+
if (!state) {
|
|
748
|
+
state = createDefaultState({ sessionId, sessionFile, cwd: ctx.cwd });
|
|
639
749
|
await saveState(state);
|
|
640
750
|
}
|
|
751
|
+
const recovered = recoverStaleOperationLock(state);
|
|
641
752
|
runtime.state = state;
|
|
753
|
+
if (recovered || state.version !== STATE_VERSION) await saveState(state);
|
|
642
754
|
return state;
|
|
643
755
|
}
|
|
644
756
|
|
|
757
|
+
function defaultSettings(): PiOMSettings {
|
|
758
|
+
return { observationModel: DEFAULT_OBSERVATION_MODEL, reflectionModel: DEFAULT_REFLECTION_MODEL, caveman: false, observeAttachments: true };
|
|
759
|
+
}
|
|
760
|
+
|
|
761
|
+
function defaultThresholds(): PiOMRecord["thresholds"] {
|
|
762
|
+
return {
|
|
763
|
+
observation: OBSERVATION_THRESHOLD,
|
|
764
|
+
reflection: REFLECTION_THRESHOLD,
|
|
765
|
+
blockAfter: Math.round(OBSERVATION_THRESHOLD * DEFAULT_BLOCK_AFTER_MULTIPLIER),
|
|
766
|
+
bufferTokens: Math.round(OBSERVATION_THRESHOLD * BUFFER_TOKENS_RATIO),
|
|
767
|
+
bufferActivation: BUFFER_ACTIVATION_RATIO,
|
|
768
|
+
};
|
|
769
|
+
}
|
|
770
|
+
|
|
771
|
+
function createDefaultState(identity: { sessionId: string; sessionFile?: string; cwd: string }): PiOMRecord {
|
|
772
|
+
return {
|
|
773
|
+
version: STATE_VERSION,
|
|
774
|
+
enabled: true,
|
|
775
|
+
sessionId: identity.sessionId,
|
|
776
|
+
sessionFile: identity.sessionFile,
|
|
777
|
+
cwd: identity.cwd,
|
|
778
|
+
scope: "session",
|
|
779
|
+
status: "idle",
|
|
780
|
+
observations: "",
|
|
781
|
+
pendingMessageTokens: 0,
|
|
782
|
+
observationTokens: 0,
|
|
783
|
+
thresholds: defaultThresholds(),
|
|
784
|
+
settings: defaultSettings(),
|
|
785
|
+
buffered: { observations: [] },
|
|
786
|
+
updatedAt: new Date().toISOString(),
|
|
787
|
+
};
|
|
788
|
+
}
|
|
789
|
+
|
|
790
|
+
function normalizeState(raw: any, identity: { sessionId: string; sessionFile?: string; cwd: string }): PiOMRecord {
|
|
791
|
+
const defaults = createDefaultState(identity);
|
|
792
|
+
const state = {
|
|
793
|
+
...defaults,
|
|
794
|
+
...raw,
|
|
795
|
+
version: STATE_VERSION,
|
|
796
|
+
sessionId: raw?.sessionId || identity.sessionId,
|
|
797
|
+
sessionFile: raw?.sessionFile || identity.sessionFile,
|
|
798
|
+
cwd: raw?.cwd || identity.cwd,
|
|
799
|
+
thresholds: { ...defaults.thresholds, ...(raw?.thresholds ?? {}) },
|
|
800
|
+
settings: { ...defaults.settings, ...(raw?.settings ?? {}) },
|
|
801
|
+
buffered: { observations: [], ...(raw?.buffered ?? {}) },
|
|
802
|
+
} as PiOMRecord;
|
|
803
|
+
state.observations = redactSecrets(String(state.observations ?? ""));
|
|
804
|
+
state.currentTask = state.currentTask ? redactSecrets(state.currentTask) : undefined;
|
|
805
|
+
state.suggestedResponse = state.suggestedResponse ? redactSecrets(state.suggestedResponse) : undefined;
|
|
806
|
+
state.lastError = state.lastError ? redactSecrets(state.lastError) : undefined;
|
|
807
|
+
state.observationTokens = estimateTokens(state.observations);
|
|
808
|
+
return state;
|
|
809
|
+
}
|
|
810
|
+
|
|
811
|
+
function isStaleOperationLock(lock: PiOMRecord["operationLock"], now = Date.now()): boolean {
|
|
812
|
+
if (!lock) return false;
|
|
813
|
+
const started = Date.parse(lock.startedAt);
|
|
814
|
+
return Number.isFinite(started) && now - started > STALE_OPERATION_LOCK_MS;
|
|
815
|
+
}
|
|
816
|
+
|
|
817
|
+
function recoverStaleOperationLock(state: PiOMRecord, now = Date.now()): boolean {
|
|
818
|
+
if (!isStaleOperationLock(state.operationLock, now)) return false;
|
|
819
|
+
const lock = state.operationLock!;
|
|
820
|
+
state.lastOperation = { type: lock.type, startedAt: lock.startedAt, endedAt: new Date(now).toISOString(), inputTokens: 0, error: `Recovered stale OM operation lock after ${Math.round((now - Date.parse(lock.startedAt)) / 1000)}s` };
|
|
821
|
+
state.lastError = state.lastOperation.error;
|
|
822
|
+
state.operationLock = undefined;
|
|
823
|
+
state.status = state.enabled ? "idle" : "disabled";
|
|
824
|
+
return true;
|
|
825
|
+
}
|
|
826
|
+
|
|
645
827
|
async function saveState(state: PiOMRecord): Promise<void> {
|
|
646
828
|
if (!runtime.statePath) return;
|
|
647
829
|
state.updatedAt = new Date().toISOString();
|
|
648
|
-
|
|
649
|
-
|
|
830
|
+
state.observations = redactSecrets(state.observations);
|
|
831
|
+
if (state.currentTask) state.currentTask = redactSecrets(state.currentTask);
|
|
832
|
+
if (state.suggestedResponse) state.suggestedResponse = redactSecrets(state.suggestedResponse);
|
|
833
|
+
if (state.lastError) state.lastError = redactSecrets(state.lastError);
|
|
834
|
+
await atomicWriteJson(runtime.statePath, state);
|
|
650
835
|
runtime.overlayHandle?.requestRender();
|
|
651
836
|
}
|
|
652
837
|
|
|
838
|
+
async function atomicWriteJson(filePath: string, value: unknown): Promise<void> {
|
|
839
|
+
await mkdir(dirname(filePath), { recursive: true });
|
|
840
|
+
const tmpPath = `${filePath}.${process.pid}.${Date.now()}.tmp`;
|
|
841
|
+
const backupPath = `${filePath}${ATOMIC_BACKUP_SUFFIX}`;
|
|
842
|
+
const json = redactSecrets(JSON.stringify(value, null, 2)) + "\n";
|
|
843
|
+
await writeFile(tmpPath, json, "utf8");
|
|
844
|
+
if (existsSync(filePath)) {
|
|
845
|
+
try {
|
|
846
|
+
await copyFile(filePath, backupPath);
|
|
847
|
+
} catch {
|
|
848
|
+
// Backup is best-effort; rename below is the durability boundary.
|
|
849
|
+
}
|
|
850
|
+
}
|
|
851
|
+
try {
|
|
852
|
+
await rename(tmpPath, filePath);
|
|
853
|
+
} catch (error) {
|
|
854
|
+
try { await unlink(tmpPath); } catch {}
|
|
855
|
+
throw error;
|
|
856
|
+
}
|
|
857
|
+
}
|
|
858
|
+
|
|
653
859
|
async function refreshCounts(ctx: any): Promise<void> {
|
|
860
|
+
let isStale = false;
|
|
861
|
+
try {
|
|
862
|
+
const _ = ctx.cwd;
|
|
863
|
+
} catch {
|
|
864
|
+
isStale = true;
|
|
865
|
+
}
|
|
866
|
+
if (isStale) return;
|
|
867
|
+
|
|
654
868
|
const state = await ensureState(ctx);
|
|
655
869
|
const branch = ctx.sessionManager.getBranch() as SessionEntry[];
|
|
656
870
|
const pending = entriesAfter(branch, state.lastObservedEntryId).filter(isMessageLikeEntry);
|
|
@@ -662,11 +876,23 @@ async function refreshCounts(ctx: any): Promise<void> {
|
|
|
662
876
|
function updateStatus(ctx: any): void {
|
|
663
877
|
const state = runtime.state;
|
|
664
878
|
if (!state) return;
|
|
665
|
-
|
|
879
|
+
try {
|
|
880
|
+
ctx.ui.setStatus("om", formatShortStatusColored(state));
|
|
881
|
+
} catch (e) {
|
|
882
|
+
// Context might be stale or UI disposed
|
|
883
|
+
}
|
|
666
884
|
runtime.overlayHandle?.requestRender();
|
|
667
885
|
}
|
|
668
886
|
|
|
669
887
|
async function observeNow(ctx: any, opts: { force: boolean; reason: string; signal?: AbortSignal; manualText?: string }): Promise<void> {
|
|
888
|
+
let isStale = false;
|
|
889
|
+
try {
|
|
890
|
+
const _ = ctx.cwd;
|
|
891
|
+
} catch {
|
|
892
|
+
isStale = true;
|
|
893
|
+
}
|
|
894
|
+
if (isStale) return;
|
|
895
|
+
|
|
670
896
|
const state = await ensureState(ctx);
|
|
671
897
|
assertNoOperation(state);
|
|
672
898
|
const branch = ctx.sessionManager.getBranch() as SessionEntry[];
|
|
@@ -742,7 +968,7 @@ async function bufferObservation(ctx: any, reason: string): Promise<void> {
|
|
|
742
968
|
});
|
|
743
969
|
state.operationLock = undefined;
|
|
744
970
|
state.status = "idle";
|
|
745
|
-
state.lastOperation = { type: "buffer", startedAt, endedAt: new Date().toISOString(), inputTokens: estimateTokens(inputText), outputTokens: estimateTokens(observations), model: DEFAULT_OBSERVATION_MODEL };
|
|
971
|
+
state.lastOperation = { type: "buffer", startedAt, endedAt: new Date().toISOString(), inputTokens: estimateTokens(inputText), outputTokens: estimateTokens(observations), model: state.settings.observationModel || DEFAULT_OBSERVATION_MODEL };
|
|
746
972
|
await writeDebug(ctx, "buffer", { startedAt, reason, inputText, rawOutput: result.rawOutput, parsed: result });
|
|
747
973
|
await saveState(state);
|
|
748
974
|
} catch (error) {
|
|
@@ -758,6 +984,14 @@ async function bufferObservation(ctx: any, reason: string): Promise<void> {
|
|
|
758
984
|
}
|
|
759
985
|
|
|
760
986
|
async function activateBuffered(ctx: any, reason: string): Promise<void> {
|
|
987
|
+
let isStale = false;
|
|
988
|
+
try {
|
|
989
|
+
const _ = ctx.cwd;
|
|
990
|
+
} catch {
|
|
991
|
+
isStale = true;
|
|
992
|
+
}
|
|
993
|
+
if (isStale) return;
|
|
994
|
+
|
|
761
995
|
const state = await ensureState(ctx);
|
|
762
996
|
if (state.buffered.observations.length === 0) return;
|
|
763
997
|
assertNoOperation(state);
|
|
@@ -794,6 +1028,14 @@ async function activateBuffered(ctx: any, reason: string): Promise<void> {
|
|
|
794
1028
|
}
|
|
795
1029
|
|
|
796
1030
|
async function reflectNow(ctx: any, opts: { reason: string; signal?: AbortSignal; manualPrompt?: string }): Promise<void> {
|
|
1031
|
+
let isStale = false;
|
|
1032
|
+
try {
|
|
1033
|
+
const _ = ctx.cwd;
|
|
1034
|
+
} catch {
|
|
1035
|
+
isStale = true;
|
|
1036
|
+
}
|
|
1037
|
+
if (isStale) return;
|
|
1038
|
+
|
|
797
1039
|
const state = await ensureState(ctx);
|
|
798
1040
|
assertNoOperation(state);
|
|
799
1041
|
if (!state.observations.trim()) throw new Error("No observations to reflect");
|
|
@@ -826,7 +1068,7 @@ async function reflectNow(ctx: any, opts: { reason: string; signal?: AbortSignal
|
|
|
826
1068
|
state.operationLock = undefined;
|
|
827
1069
|
state.status = "idle";
|
|
828
1070
|
state.lastError = undefined;
|
|
829
|
-
state.lastOperation = { type: "reflection", startedAt, endedAt: new Date().toISOString(), inputTokens: originalTokens, outputTokens: reflectedTokens, model: DEFAULT_REFLECTION_MODEL, compressionLevel: level };
|
|
1071
|
+
state.lastOperation = { type: "reflection", startedAt, endedAt: new Date().toISOString(), inputTokens: originalTokens, outputTokens: reflectedTokens, model: state.settings.reflectionModel || DEFAULT_REFLECTION_MODEL, compressionLevel: level };
|
|
830
1072
|
await saveState(state);
|
|
831
1073
|
return;
|
|
832
1074
|
}
|
|
@@ -846,9 +1088,11 @@ async function reflectNow(ctx: any, opts: { reason: string; signal?: AbortSignal
|
|
|
846
1088
|
}
|
|
847
1089
|
|
|
848
1090
|
async function runObserver(ctx: any, historyText: string, signal: AbortSignal | undefined, opts: { source: string; existingObservations?: string }): Promise<ObserverResult> {
|
|
849
|
-
const
|
|
1091
|
+
const state = await ensureState(ctx);
|
|
1092
|
+
const modelId = state.settings.observationModel || DEFAULT_OBSERVATION_MODEL;
|
|
1093
|
+
const model = resolveModel(ctx, modelId);
|
|
850
1094
|
const response = await runModel(ctx, model, [
|
|
851
|
-
{ role: "user", content: [{ type: "text", text: buildObserverSystemPrompt() }] },
|
|
1095
|
+
{ role: "user", content: [{ type: "text", text: buildObserverSystemPrompt(state.settings.caveman) }] },
|
|
852
1096
|
{ role: "user", content: [{ type: "text", text: `## New Message History to Observe\n\n${historyText}\n\n---\n\n${buildObserverTaskPrompt(opts.existingObservations, { priorCurrentTask: runtime.state?.currentTask, priorSuggestedResponse: runtime.state?.suggestedResponse })}` }] },
|
|
853
1097
|
], { temperature: 0.3, maxTokens: 100_000, signal });
|
|
854
1098
|
const text = responseText(response);
|
|
@@ -859,9 +1103,11 @@ async function runObserver(ctx: any, historyText: string, signal: AbortSignal |
|
|
|
859
1103
|
}
|
|
860
1104
|
|
|
861
1105
|
async function runReflector(ctx: any, observations: string, level: CompressionLevel, signal: AbortSignal | undefined, manualPrompt?: string): Promise<ReflectorResult> {
|
|
862
|
-
const
|
|
1106
|
+
const state = await ensureState(ctx);
|
|
1107
|
+
const modelId = state.settings.reflectionModel || DEFAULT_REFLECTION_MODEL;
|
|
1108
|
+
const model = resolveModel(ctx, modelId);
|
|
863
1109
|
const response = await runModel(ctx, model, [
|
|
864
|
-
{ role: "user", content: [{ type: "text", text: buildReflectorSystemPrompt() }] },
|
|
1110
|
+
{ role: "user", content: [{ type: "text", text: buildReflectorSystemPrompt(state.settings.caveman) }] },
|
|
865
1111
|
{ role: "user", content: [{ type: "text", text: buildReflectorPrompt(observations, level, manualPrompt) }] },
|
|
866
1112
|
], { temperature: 0, maxTokens: 100_000, signal });
|
|
867
1113
|
const text = responseText(response);
|
|
@@ -909,7 +1155,7 @@ function extractListItemsOnly(content: string): string {
|
|
|
909
1155
|
}
|
|
910
1156
|
|
|
911
1157
|
function sanitizeObservationLines(observations: string): string {
|
|
912
|
-
return observations.split("\n").map(line => line.length > MAX_OBSERVATION_LINE_CHARS ? `${line.slice(0, MAX_OBSERVATION_LINE_CHARS)} … [truncated]` : line).join("\n").trim();
|
|
1158
|
+
return redactSecrets(observations).split("\n").map(line => line.length > MAX_OBSERVATION_LINE_CHARS ? `${line.slice(0, MAX_OBSERVATION_LINE_CHARS)} … [truncated]` : line).join("\n").trim();
|
|
913
1159
|
}
|
|
914
1160
|
|
|
915
1161
|
function detectDegenerateRepetition(text: string): boolean {
|
|
@@ -937,7 +1183,7 @@ function appendObservations(state: PiOMRecord, result: ObserverResult, inputToke
|
|
|
937
1183
|
state.currentTask = result.currentTask ?? state.currentTask;
|
|
938
1184
|
state.suggestedResponse = result.suggestedContinuation ?? state.suggestedResponse;
|
|
939
1185
|
state.observationTokens = estimateTokens(state.observations);
|
|
940
|
-
state.lastOperation = { type: "observation", startedAt: state.operationLock?.startedAt ?? new Date().toISOString(), endedAt: new Date().toISOString(), inputTokens, outputTokens: estimateTokens(observations), model: DEFAULT_OBSERVATION_MODEL };
|
|
1186
|
+
state.lastOperation = { type: "observation", startedAt: state.operationLock?.startedAt ?? new Date().toISOString(), endedAt: new Date().toISOString(), inputTokens, outputTokens: estimateTokens(observations), model: state.settings.observationModel || DEFAULT_OBSERVATION_MODEL };
|
|
941
1187
|
}
|
|
942
1188
|
|
|
943
1189
|
function buildOMContextMessage(state: PiOMRecord): AgentMessage {
|
|
@@ -1059,15 +1305,50 @@ function formatAgentMessage(msg: any, mode: "observer" | "recall" = "observer",
|
|
|
1059
1305
|
}
|
|
1060
1306
|
}
|
|
1061
1307
|
|
|
1308
|
+
|
|
1309
|
+
function limitTextByTokens(text: string | undefined, maxTokens: number): string | undefined {
|
|
1310
|
+
if (!text) return text;
|
|
1311
|
+
if (estimateTokens(text) <= maxTokens) return text;
|
|
1312
|
+
const maxChars = Math.max(0, maxTokens * 4);
|
|
1313
|
+
const tail = text.slice(Math.max(0, text.length - maxChars));
|
|
1314
|
+
const lineBoundary = tail.indexOf("\n");
|
|
1315
|
+
const trimmedTail = lineBoundary >= 0 ? tail.slice(lineBoundary + 1) : tail;
|
|
1316
|
+
return `[Earlier observations truncated for observer prompt safety: kept last ~${maxTokens} tokens.]\n${trimmedTail}`;
|
|
1317
|
+
}
|
|
1318
|
+
|
|
1319
|
+
function redactSecrets(input: string): string {
|
|
1320
|
+
if (!input) return input;
|
|
1321
|
+
return input
|
|
1322
|
+
.replace(/npm_[A-Za-z0-9]{16,}/g, "[REDACTED_NPM_TOKEN]")
|
|
1323
|
+
.replace(/github_pat_[A-Za-z0-9_]{20,}/g, "[REDACTED_GITHUB_TOKEN]")
|
|
1324
|
+
.replace(/gh[pousr]_[A-Za-z0-9_]{20,}/g, "[REDACTED_GITHUB_TOKEN]")
|
|
1325
|
+
.replace(/sk-[A-Za-z0-9_-]{20,}/g, "[REDACTED_API_KEY]")
|
|
1326
|
+
.replace(/(?<=(?:\b|["'])(?:api[_-]?key|token|secret|password)(?:\b|["'])\s*[:=]\s*["']?)[^"'\s,;}]{8,}/gi, "[REDACTED_SECRET]")
|
|
1327
|
+
.replace(/\bBearer\s+[A-Za-z0-9._~+/=-]{16,}/gi, "Bearer [REDACTED_TOKEN]");
|
|
1328
|
+
}
|
|
1329
|
+
|
|
1330
|
+
function redactDeep<T>(value: T): T {
|
|
1331
|
+
if (typeof value === "string") return redactSecrets(value) as T;
|
|
1332
|
+
if (Array.isArray(value)) return value.map(item => redactDeep(item)) as T;
|
|
1333
|
+
if (value && typeof value === "object") {
|
|
1334
|
+
const out: Record<string, unknown> = {};
|
|
1335
|
+
for (const [key, child] of Object.entries(value as Record<string, unknown>)) {
|
|
1336
|
+
out[key] = /apiKey|authorization|token|secret|password/i.test(key) ? "[REDACTED_SECRET]" : redactDeep(child);
|
|
1337
|
+
}
|
|
1338
|
+
return out as T;
|
|
1339
|
+
}
|
|
1340
|
+
return value;
|
|
1341
|
+
}
|
|
1342
|
+
|
|
1062
1343
|
function formatContent(content: any, cap: number): string {
|
|
1063
|
-
if (typeof content === "string") return truncateText(content, cap);
|
|
1064
|
-
if (!Array.isArray(content)) return truncateText(JSON.stringify(content), cap);
|
|
1344
|
+
if (typeof content === "string") return truncateText(redactSecrets(content), cap);
|
|
1345
|
+
if (!Array.isArray(content)) return truncateText(redactSecrets(JSON.stringify(redactDeep(content))), cap);
|
|
1065
1346
|
return content.map(part => {
|
|
1066
|
-
if (part.type === "text") return part.text;
|
|
1067
|
-
if (part.type === "thinking") return `[Thinking]: ${part.thinking}`;
|
|
1068
|
-
if (part.type === "image") return `[Image: ${part.mimeType ?? "unknown"}]`;
|
|
1069
|
-
if (part.type === "toolCall") return `[Tool Call ${part.name}: ${truncateText(JSON.stringify(part.arguments ?? {}), 2_000)}]`;
|
|
1070
|
-
return `[${part.type}: ${truncateText(JSON.stringify(part), 2_000)}]`;
|
|
1347
|
+
if (part.type === "text") return redactSecrets(part.text);
|
|
1348
|
+
if (part.type === "thinking") return `[Thinking]: ${redactSecrets(part.thinking ?? "")}`;
|
|
1349
|
+
if (part.type === "image") return runtime.state?.settings.observeAttachments === false ? "[Image omitted: attachment observation disabled]" : `[Image: ${part.mimeType ?? "unknown"}]`;
|
|
1350
|
+
if (part.type === "toolCall") return `[Tool Call ${part.name}: ${truncateText(redactSecrets(JSON.stringify(redactDeep(part.arguments ?? {}))), 2_000)}]`;
|
|
1351
|
+
return `[${part.type}: ${truncateText(redactSecrets(JSON.stringify(redactDeep(part))), 2_000)}]`;
|
|
1071
1352
|
}).join("\n").slice(0, cap);
|
|
1072
1353
|
}
|
|
1073
1354
|
|
|
@@ -1102,9 +1383,9 @@ function truncateText(text: string, maxChars: number): string {
|
|
|
1102
1383
|
}
|
|
1103
1384
|
|
|
1104
1385
|
function responseText(response: any): string {
|
|
1105
|
-
if (typeof response?.text === "string") return response.text;
|
|
1106
|
-
if (Array.isArray(response?.content)) return response.content.filter((c: any) => c?.type === "text").map((c: any) => c.text).join("\n");
|
|
1107
|
-
if (typeof response === "string") return response;
|
|
1386
|
+
if (typeof response?.text === "string") return redactSecrets(response.text);
|
|
1387
|
+
if (Array.isArray(response?.content)) return redactSecrets(response.content.filter((c: any) => c?.type === "text").map((c: any) => c.text).join("\n"));
|
|
1388
|
+
if (typeof response === "string") return redactSecrets(response);
|
|
1108
1389
|
return "";
|
|
1109
1390
|
}
|
|
1110
1391
|
|
|
@@ -1142,6 +1423,70 @@ function formatShortStatusColored(state: PiOMRecord): string {
|
|
|
1142
1423
|
return `om: ${statusColor}${state.status}\x1b[0m \x1b[2;37m|\x1b[0m msg \x1b[1;33m${formatTokens(state.pendingMessageTokens)}\x1b[0m/\x1b[2;37m${formatTokens(effectiveObservationThreshold(state))}\x1b[0m [${msgPctStr}] \x1b[2;37m|\x1b[0m mem \x1b[1;36m${formatTokens(state.observationTokens)}\x1b[0m/\x1b[2;37m${formatTokens(state.thresholds.reflection)}\x1b[0m [${memPctStr}]${buffer}${err}`;
|
|
1143
1424
|
}
|
|
1144
1425
|
|
|
1426
|
+
function parsePositiveIntSetting(key: string, value: string): number {
|
|
1427
|
+
const n = Number(value);
|
|
1428
|
+
if (!Number.isFinite(n) || n <= 0) throw new Error(`OM setting ${key} must be a positive number`);
|
|
1429
|
+
return Math.round(n);
|
|
1430
|
+
}
|
|
1431
|
+
|
|
1432
|
+
function parseRatioSetting(key: string, value: string): number {
|
|
1433
|
+
const raw = value.trim();
|
|
1434
|
+
const n = raw.endsWith("%") ? Number(raw.slice(0, -1)) / 100 : Number(raw);
|
|
1435
|
+
if (!Number.isFinite(n) || n <= 0 || n > 1) throw new Error(`OM setting ${key} must be a ratio between 0 and 1, or percent like 80%`);
|
|
1436
|
+
return n;
|
|
1437
|
+
}
|
|
1438
|
+
|
|
1439
|
+
function parseBooleanSetting(key: string, value: string): boolean {
|
|
1440
|
+
const normalized = value.trim().toLowerCase();
|
|
1441
|
+
if (["1", "true", "yes", "on", "enabled"].includes(normalized)) return true;
|
|
1442
|
+
if (["0", "false", "no", "off", "disabled"].includes(normalized)) return false;
|
|
1443
|
+
throw new Error(`OM setting ${key} must be on/off or true/false`);
|
|
1444
|
+
}
|
|
1445
|
+
|
|
1446
|
+
function applySetting(state: PiOMRecord, key: string | undefined, value: string): void {
|
|
1447
|
+
if (!key) throw new Error("Missing OM setting key");
|
|
1448
|
+
if (!value) throw new Error(`Missing OM setting value for ${key}`);
|
|
1449
|
+
const normalized = key.toLowerCase().replace(/_/g, "-");
|
|
1450
|
+
if (["observation", "observation-threshold", "observe-threshold"].includes(normalized)) state.thresholds.observation = parsePositiveIntSetting(key, value);
|
|
1451
|
+
else if (["reflection", "reflection-threshold", "reflect-threshold"].includes(normalized)) state.thresholds.reflection = parsePositiveIntSetting(key, value);
|
|
1452
|
+
else if (["block-after", "blockafter"].includes(normalized)) state.thresholds.blockAfter = parsePositiveIntSetting(key, value);
|
|
1453
|
+
else if (["buffer", "buffer-tokens"].includes(normalized)) state.thresholds.bufferTokens = parsePositiveIntSetting(key, value);
|
|
1454
|
+
else if (["buffer-activation", "activation-ratio"].includes(normalized)) state.thresholds.bufferActivation = parseRatioSetting(key, value);
|
|
1455
|
+
else if (["observation-model", "observer-model", "observe-model"].includes(normalized)) state.settings.observationModel = value.trim();
|
|
1456
|
+
else if (["reflection-model", "reflector-model", "reflect-model"].includes(normalized)) state.settings.reflectionModel = value.trim();
|
|
1457
|
+
else if (normalized === "caveman") state.settings.caveman = parseBooleanSetting(key, value);
|
|
1458
|
+
else if (["attachments", "observe-attachments", "attachment-observation"].includes(normalized)) state.settings.observeAttachments = parseBooleanSetting(key, value);
|
|
1459
|
+
else if (normalized === "scope") {
|
|
1460
|
+
const scope = value.trim().toLowerCase();
|
|
1461
|
+
if (scope !== "session" && scope !== "project") throw new Error("OM scope must be session or project");
|
|
1462
|
+
state.scope = scope;
|
|
1463
|
+
} else {
|
|
1464
|
+
throw new Error(`Unknown OM setting: ${key}`);
|
|
1465
|
+
}
|
|
1466
|
+
if (state.thresholds.blockAfter < state.thresholds.observation) state.thresholds.blockAfter = Math.round(state.thresholds.observation * DEFAULT_BLOCK_AFTER_MULTIPLIER);
|
|
1467
|
+
state.observationTokens = estimateTokens(state.observations);
|
|
1468
|
+
}
|
|
1469
|
+
|
|
1470
|
+
function formatSettingsText(state: PiOMRecord): string {
|
|
1471
|
+
return [
|
|
1472
|
+
formatDetailedStatus(state),
|
|
1473
|
+
"",
|
|
1474
|
+
"settings:",
|
|
1475
|
+
` observation-model: ${state.settings.observationModel}`,
|
|
1476
|
+
` reflection-model: ${state.settings.reflectionModel}`,
|
|
1477
|
+
` caveman: ${state.settings.caveman ? "on" : "off"}`,
|
|
1478
|
+
` attachments: ${state.settings.observeAttachments ? "on" : "off"}`,
|
|
1479
|
+
` scope: ${state.scope}`,
|
|
1480
|
+
"",
|
|
1481
|
+
"usage:",
|
|
1482
|
+
" /om set observation-threshold 30000",
|
|
1483
|
+
" /om set reflection-threshold 40000",
|
|
1484
|
+
" /om set observation-model google/gemini-2.5-flash",
|
|
1485
|
+
" /om set caveman on",
|
|
1486
|
+
" /om reset",
|
|
1487
|
+
].join("\n");
|
|
1488
|
+
}
|
|
1489
|
+
|
|
1145
1490
|
function formatDetailedStatus(state: PiOMRecord): string {
|
|
1146
1491
|
return `${formatShortStatus(state)}\nlastObservedEntryId: ${state.lastObservedEntryId ?? "none"}\ncurrentTask: ${state.currentTask ?? "none"}\nsuggestedResponse: ${state.suggestedResponse ?? "none"}\nlastOperation: ${state.lastOperation ? JSON.stringify(state.lastOperation, null, 2) : "none"}\nstatePath: ${runtime.statePath ?? "unknown"}`;
|
|
1147
1492
|
}
|
|
@@ -1320,11 +1665,15 @@ function formatTokens(tokens: number): string {
|
|
|
1320
1665
|
}
|
|
1321
1666
|
|
|
1322
1667
|
async function writeDebug(ctx: any, name: string, payload: unknown): Promise<void> {
|
|
1323
|
-
|
|
1324
|
-
|
|
1325
|
-
|
|
1326
|
-
|
|
1327
|
-
|
|
1668
|
+
try {
|
|
1669
|
+
const state = await ensureState(ctx);
|
|
1670
|
+
const dir = runtime.debugDir || join(ctx.cwd, CONFIG_DIR_NAME, "om", "debug");
|
|
1671
|
+
await mkdir(dir, { recursive: true });
|
|
1672
|
+
const file = join(dir, `${new Date().toISOString().replace(/[:.]/g, "-")}-${sanitizeFileName(name)}.json`);
|
|
1673
|
+
await writeFile(file, redactSecrets(JSON.stringify(redactDeep({ extension: EXTENSION_ID, sessionId: state.sessionId, ...payload as any }), null, 2)) + "\n", "utf8");
|
|
1674
|
+
} catch (error) {
|
|
1675
|
+
// Ignore debug write errors if context is stale
|
|
1676
|
+
}
|
|
1328
1677
|
}
|
|
1329
1678
|
|
|
1330
1679
|
function sanitizeFileName(name: string): string {
|
|
@@ -1346,6 +1695,22 @@ function mergeAbortSignals(a?: AbortSignal, b?: AbortSignal): AbortSignal | unde
|
|
|
1346
1695
|
return controller.signal;
|
|
1347
1696
|
}
|
|
1348
1697
|
|
|
1698
|
+
export const __test = {
|
|
1699
|
+
redactSecrets,
|
|
1700
|
+
redactDeep,
|
|
1701
|
+
limitTextByTokens,
|
|
1702
|
+
buildObserverTaskPrompt,
|
|
1703
|
+
defaultSettings,
|
|
1704
|
+
defaultThresholds,
|
|
1705
|
+
createDefaultState,
|
|
1706
|
+
normalizeState,
|
|
1707
|
+
isStaleOperationLock,
|
|
1708
|
+
recoverStaleOperationLock,
|
|
1709
|
+
atomicWriteJson,
|
|
1710
|
+
applySetting,
|
|
1711
|
+
formatSettingsText,
|
|
1712
|
+
};
|
|
1713
|
+
|
|
1349
1714
|
class OMOverlay {
|
|
1350
1715
|
private scroll = 0;
|
|
1351
1716
|
private tab: "memory" | "status" | "debug" = "memory";
|
package/package.json
CHANGED
|
@@ -1,13 +1,39 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-observational-memory-extension",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.3",
|
|
4
4
|
"description": "Mastra-style Observational Memory extension for Pi compaction and runtime context.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Nikita Nosov <20nik.nosov21@gmail.com>",
|
|
7
7
|
"type": "module",
|
|
8
|
-
"keywords": [
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
"keywords": [
|
|
9
|
+
"pi-package",
|
|
10
|
+
"pi",
|
|
11
|
+
"pi-coding-agent",
|
|
12
|
+
"observational-memory",
|
|
13
|
+
"memory",
|
|
14
|
+
"compaction"
|
|
15
|
+
],
|
|
16
|
+
"homepage": "https://github.com/nik1t7n/pi-observational-memory-extension#readme",
|
|
17
|
+
"repository": {
|
|
18
|
+
"type": "git",
|
|
19
|
+
"url": "git+https://github.com/nik1t7n/pi-observational-memory-extension.git"
|
|
20
|
+
},
|
|
21
|
+
"bugs": {
|
|
22
|
+
"url": "https://github.com/nik1t7n/pi-observational-memory-extension/issues"
|
|
23
|
+
},
|
|
24
|
+
"files": [
|
|
25
|
+
"extensions",
|
|
26
|
+
"docs",
|
|
27
|
+
"scripts",
|
|
28
|
+
"README.md",
|
|
29
|
+
"LICENSE",
|
|
30
|
+
"CHANGELOG.md"
|
|
31
|
+
],
|
|
32
|
+
"pi": {
|
|
33
|
+
"extensions": [
|
|
34
|
+
"./extensions"
|
|
35
|
+
]
|
|
36
|
+
},
|
|
11
37
|
"peerDependencies": {
|
|
12
38
|
"@earendil-works/pi-coding-agent": "*",
|
|
13
39
|
"@earendil-works/pi-ai": "*",
|
|
@@ -24,6 +50,7 @@
|
|
|
24
50
|
"typecheck": "tsc --noEmit",
|
|
25
51
|
"validate": "node scripts/validate.mjs",
|
|
26
52
|
"pack:check": "npm pack --dry-run",
|
|
27
|
-
"prepublishOnly": "npm run validate && npm run typecheck && npm run pack:check"
|
|
53
|
+
"prepublishOnly": "npm run validate && npm run typecheck && npm test && npm run pack:check",
|
|
54
|
+
"test": "node scripts/test.mjs"
|
|
28
55
|
}
|
|
29
56
|
}
|
package/scripts/test.mjs
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
import assert from "node:assert/strict";
|
|
3
|
+
import { execFileSync } from "node:child_process";
|
|
4
|
+
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
|
5
|
+
import { existsSync } from "node:fs";
|
|
6
|
+
import { join } from "node:path";
|
|
7
|
+
import { tmpdir } from "node:os";
|
|
8
|
+
|
|
9
|
+
const dist = ".tmp-test-dist";
|
|
10
|
+
await rm(dist, { recursive: true, force: true });
|
|
11
|
+
execFileSync("npx", ["tsc", "--outDir", dist, "--declaration", "false", "--noEmit", "false", "--rootDir", "."], { stdio: "inherit" });
|
|
12
|
+
const { __test } = await import(`../${dist}/extensions/index.js?${Date.now()}`);
|
|
13
|
+
|
|
14
|
+
// Secret redaction: state/debug/observer text must not persist common tokens.
|
|
15
|
+
const redacted = __test.redactSecrets("npm_abcdefghijklmnopqrstuvwxyz sk-abcdefghijklmnopqrstuvwxyz Bearer abcdefghijklmnopqrstuvwxyz token=supersecrettoken");
|
|
16
|
+
assert(!redacted.includes("npm_abcdefghijklmnopqrstuvwxyz"));
|
|
17
|
+
assert(!redacted.includes("sk-abcdefghijklmnopqrstuvwxyz"));
|
|
18
|
+
assert(!redacted.includes("Bearer abcdefghijklmnopqrstuvwxyz"));
|
|
19
|
+
assert(!redacted.includes("supersecrettoken"));
|
|
20
|
+
assert(redacted.includes("[REDACTED_NPM_TOKEN]"));
|
|
21
|
+
|
|
22
|
+
// Observer prompt limiting: previous observations are capped to a safe tail.
|
|
23
|
+
const huge = Array.from({ length: 5000 }, (_, i) => `* 🔴 old observation ${i}`).join("\n");
|
|
24
|
+
const prompt = __test.buildObserverTaskPrompt(huge, { priorCurrentTask: "continue tests" });
|
|
25
|
+
assert(prompt.length < huge.length / 2, "observer prompt should not include full previous observations");
|
|
26
|
+
assert(prompt.includes("truncated"));
|
|
27
|
+
assert(prompt.includes("continue tests"));
|
|
28
|
+
|
|
29
|
+
// Stale lock recovery: old operation locks are cleared and recorded as errors.
|
|
30
|
+
const state = __test.createDefaultState({ sessionId: "s", cwd: process.cwd() });
|
|
31
|
+
state.operationLock = { type: "observation", startedAt: new Date(Date.now() - 30 * 60 * 1000).toISOString() };
|
|
32
|
+
state.status = "observing";
|
|
33
|
+
assert.equal(__test.recoverStaleOperationLock(state), true);
|
|
34
|
+
assert.equal(state.operationLock, undefined);
|
|
35
|
+
assert.equal(state.status, "idle");
|
|
36
|
+
assert.match(state.lastError, /Recovered stale OM operation lock/);
|
|
37
|
+
|
|
38
|
+
// Settings parser: full /om set surface mutates persisted config safely.
|
|
39
|
+
__test.applySetting(state, "observation-model", "google/gemini-2.5-flash");
|
|
40
|
+
__test.applySetting(state, "reflection-threshold", "12345");
|
|
41
|
+
__test.applySetting(state, "caveman", "on");
|
|
42
|
+
__test.applySetting(state, "attachments", "off");
|
|
43
|
+
assert.equal(state.settings.observationModel, "google/gemini-2.5-flash");
|
|
44
|
+
assert.equal(state.thresholds.reflection, 12345);
|
|
45
|
+
assert.equal(state.settings.caveman, true);
|
|
46
|
+
assert.equal(state.settings.observeAttachments, false);
|
|
47
|
+
assert.throws(() => __test.applySetting(state, "buffer-activation", "2"), /ratio/);
|
|
48
|
+
|
|
49
|
+
// Migration/normalization: v1 state keeps observations and receives v2 settings.
|
|
50
|
+
const migrated = __test.normalizeState({ version: 1, observations: "token=do-not-keep-this-secret", thresholds: { observation: 10 } }, { sessionId: "m", cwd: process.cwd() });
|
|
51
|
+
assert.equal(migrated.version, 2);
|
|
52
|
+
assert.equal(migrated.thresholds.observation, 10);
|
|
53
|
+
assert.equal(migrated.settings.caveman, false);
|
|
54
|
+
assert(!migrated.observations.includes("do-not-keep-this-secret"));
|
|
55
|
+
|
|
56
|
+
// Atomic write: writes valid JSON, creates backup on subsequent write, removes temp files.
|
|
57
|
+
const dir = await mkdtemp(join(tmpdir(), "pi-om-test-"));
|
|
58
|
+
try {
|
|
59
|
+
const file = join(dir, "state.json");
|
|
60
|
+
await __test.atomicWriteJson(file, { a: 1, token: "supersecrettoken" });
|
|
61
|
+
assert.equal(JSON.parse(await readFile(file, "utf8")).token, "[REDACTED_SECRET]");
|
|
62
|
+
await __test.atomicWriteJson(file, { a: 2 });
|
|
63
|
+
assert.equal(JSON.parse(await readFile(file, "utf8")).a, 2);
|
|
64
|
+
assert.equal(existsSync(`${file}.bak`), true);
|
|
65
|
+
} finally {
|
|
66
|
+
await rm(dir, { recursive: true, force: true });
|
|
67
|
+
await rm(dist, { recursive: true, force: true });
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
console.log("pi-observational-memory-extension tests passed");
|