pi-observational-memory-extension 0.1.2 → 0.1.3

package/CHANGELOG.md

@@ -0,0 +1,40 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.3] - 2026-06-23
+
+### Added
+- **settings**: Added the unified `/om` command with `status`, `set`, `reset`, `enable`, `disable`, `observe`, `reflect`, and `memory` actions.
+- **settings**: Persisted model overrides, thresholds, caveman compression mode, attachment observation toggle, and scope configuration in OM state.
+- **tests**: Added a focused regression test suite for redaction, prompt limiting, stale lock recovery, settings parsing, v1→v2 migration, and atomic writes.
+
+### Fixed
+- **durability**: State files now use atomic temp-file writes with `.bak` recovery support.
+- **stability**: Stale operation locks are recovered automatically instead of blocking future OM runs forever.
+- **security**: State, debug logs, recall/observer serialization, and model outputs now redact common API keys, npm tokens, GitHub tokens, bearer tokens, passwords, and secret fields.
+- **context**: Observer prompts now cap previous observations to a bounded tail to prevent observation-context bloat.
+
+## [0.1.2] - 2026-06-22
+
+### Fixed
+- **om**: Resolved "stale context" errors during asynchronous `session_shutdown` handlers and background buffer tasks.
+- **om**: Safely caught uncaught promise rejections within internal `bufferObservation` tasks when contexts are deactivated by Pi.
+
+## [0.1.1] - 2026-06-22
+
+### Added
+- **om**: Added capability to force immediate observational compaction on `session_shutdown` if pending message tokens exist. This guarantees subagent and short-lived loopflow sessions persist their memories before exiting.
+
+## [0.1.0] - 2026-06-22
+
+### Added
+- **om**: First release of the Mastra-style **Observational Memory (OM)** extension.
+- **om**: Implemented the 3-Agent psychological memory model: Actor (Main Agent), Observer (Extraction Agent), and Reflector (Consolidation/Compression Agent).
+- **om**: Automatically intercepts `/compact` and `session_before_compact` events.
+- **om**: Real-time TUI status bar panel underneath the input showing memory, message, and token thresholds.
+- **om**: Color-coded, responsive status indicators for high (🔴), medium (🟡), and low (🟢) priority observations.
+- **om**: Dedicated interactive, fullscreen overlay `/om-memory` with interactive tab switching (`Memory`, `Status`, `Debug`) and ANSI word wrapping.

package/README.md

@@ -27,27 +27,36 @@ Legacy compaction compresses raw history into a single monolithic block of text,
 - **Recall-driven Retrieval:** Exposes an `om_recall` tool to the Actor so it can retrieve full, raw message payloads from observed history when exact code, quotes, or numbers are needed.
 - **Custom Compaction Hook:** Plugs directly into Pi's `session_before_compact` lifecycle event. Typing `/compact` or triggering auto-compaction launches the Observer/Reflector memory consolidation flow instead of Pi's legacy summary compaction.
 - **TUI Overlay Panel:** Fully custom, responsive, multi-tab overlay (`/om-memory`) in interactive mode for inspecting memory. Features tabbed navigation, smooth scroll, and precise border framing.
-- **Durable Persistence:** Serializes and loads state files safely under `.pi/om/<session-id>.json`. Outputs JSON-formatted diagnostic logs to `.pi/om/debug/` for every operation.
+- **Durable Persistence:** Serializes and loads state files safely under `.pi/om/<session-id>.json` using atomic writes and `.bak` recovery. Outputs JSON-formatted diagnostic logs to `.pi/om/debug/` for every operation.
+- **Secret Redaction:** Redacts common API keys, npm/GitHub tokens, bearer tokens, passwords, and secret fields before writing state/debug artifacts or observer text.
+- **Stale Lock Recovery:** Recovers old interrupted OM operation locks automatically so a crashed or killed process does not block future memory runs.
+- **Bounded Observer Context:** Sends only a safe tail of previous observations to the Observer, preventing recursive prompt bloat while preserving full memory in the main runtime.
 
 ---
 
 ## 📋 Configuration & Settings
 
-You can customize Observational Memory behaviors by specifying fields inside your project's `.pi/settings.json` or globally inside `~/.pi/agent/settings.json`.
+Use `/om` in Pi to inspect and update runtime settings. Settings are persisted in the session OM state file under `.pi/om/<session-id>.json`.
 
-```json
-{
-  "compaction": {
-    "enabled": true,
-    "reserveTokens": 16384,
-    "keepRecentTokens": 20000
-  }
-}
+```bash
+/om
+/om set observation-threshold 30000
+/om set reflection-threshold 40000
+/om set block-after 36000
+/om set buffer-tokens 6000
+/om set buffer-activation 80%
+/om set observation-model google/gemini-2.5-flash
+/om set reflection-model google/gemini-2.5-flash
+/om set caveman on
+/om set attachments off
+/om reset
 ```
 
-- **Observation Trigger Threshold:** ~`30,000` raw message tokens.
-- **Reflection Trigger Threshold:** ~`40,000` observation tokens.
+- **Observation Trigger Threshold:** default `30,000` raw message tokens.
+- **Reflection Trigger Threshold:** default `40,000` observation tokens.
 - **Default Models:** `google/gemini-2.5-flash` with 0.3 temperature for Observer, and 0.0 temperature for Reflector.
+- **Caveman Mode:** optional terse compression style for denser memory.
+- **Attachment Observation Toggle:** controls whether image/attachment placeholders are exposed to observation text.
 
 ---
 
@@ -55,6 +64,7 @@ You can customize Observational Memory behaviors by specifying fields inside you
 
 This extension registers the following slash commands in Pi:
 
+- `/om` — Unified settings/status command. Supports `set`, `reset`, `enable`, `disable`, `observe`, `reflect`, and `memory`.
 - `/om-status` — Shows a detailed breakdown of pending/observation tokens, active locks, thresholds, and last operation results.
 - `/om-memory` — Opens the interactive multi-tab overlay panel (observations, status stats, and background debug details).
 - `/om-observe` — Forces an immediate Observation pass on all pending raw message history.
@@ -88,6 +98,7 @@ Run the typechecker and validation scripts before packaging or releasing:
 ```bash
 npm run typecheck
 npm run validate
+npm test
 ```
 
 ### Commit Guidelines

package/extensions/index.ts

@@ -7,12 +7,12 @@ import {
 } from "@earendil-works/pi-coding-agent";
 import { Key, matchesKey, truncateToWidth, wrapTextWithAnsi } from "@earendil-works/pi-tui";
 import { Type } from "typebox";
-import { mkdir, readFile, writeFile } from "node:fs/promises";
+import { copyFile, mkdir, readFile, rename, unlink, writeFile } from "node:fs/promises";
 import { existsSync } from "node:fs";
 import { basename, dirname, join } from "node:path";
 
 const EXTENSION_ID = "pi-observational-memory";
-const STATE_VERSION = 1;
+const STATE_VERSION = 2;
 const DEFAULT_OBSERVATION_MODEL = "google/gemini-2.5-flash";
 const DEFAULT_REFLECTION_MODEL = "google/gemini-2.5-flash";
 const OBSERVATION_THRESHOLD = 30_000;
@@ -25,6 +25,9 @@ const TOOL_RESULT_MAX_CHARS = 8_000;
 const MESSAGE_PART_MAX_CHARS = 20_000;
 const MAX_OBSERVATION_LINE_CHARS = 10_000;
 const MAX_RESTART_ERRORS = 3;
+const STALE_OPERATION_LOCK_MS = 15 * 60 * 1000;
+const PREVIOUS_OBSERVATIONS_MAX_TOKENS = 2_000;
+const ATOMIC_BACKUP_SUFFIX = ".bak";
 
 const OBSERVATION_CONTEXT_PROMPT = `The following observations block contains your memory of past conversations with this user.`;
 const OBSERVATION_CONTEXT_INSTRUCTIONS = `IMPORTANT: When responding, reference specific details from these observations. Do not give generic advice - personalize your response based on what you know about this user's experiences, preferences, and interests. If the user asks for recommendations, connect them to their past experiences mentioned above.
@@ -165,7 +168,8 @@ const OBSERVER_GUIDELINES = `- Be specific enough for the assistant to act on
 - Observe WHAT the agent did and WHAT it means
 - If the user provides detailed messages or code snippets, observe all important details`;
 
-function buildObserverSystemPrompt(): string {
+function buildObserverSystemPrompt(caveman = false): string {
+  const cavemanInstruction = caveman ? `\n\nCAVEMAN MODE: Write brutally short, dense observations. Remove filler. Preserve facts, decisions, dates, paths, and errors only.` : "";
   return `You are the memory consciousness of an AI assistant. Your observations will be the ONLY information the assistant has about past interactions with this user.
 
 Extract observations that will help the assistant remember:
@@ -188,20 +192,22 @@ Do NOT add thread identifiers, thread IDs, or <thread> tags to your observations
 
 Remember: These observations are the assistant's ONLY memory. Make them count.
 
-User messages are extremely important. If the user asks a question or gives a new task, make it clear in <current-task> that this is the priority. If the assistant needs to respond to the user, indicate in <suggested-response> that it should pause for user reply before continuing other tasks.`;
+User messages are extremely important. If the user asks a question or gives a new task, make it clear in <current-task> that this is the priority. If the assistant needs to respond to the user, indicate in <suggested-response> that it should pause for user reply before continuing other tasks.${cavemanInstruction}`;
 }
 
 function buildObserverTaskPrompt(existingObservations: string | undefined, opts: { priorCurrentTask?: string; priorSuggestedResponse?: string; wasTruncated?: boolean } = {}): string {
   let prompt = "";
-  if (existingObservations?.trim()) {
-    prompt += `## Previous Observations\n\n${existingObservations}\n\n---\n\nDo not repeat these existing observations. Your new observations will be appended to the existing observations.\n\n`;
+  const limitedExisting = limitTextByTokens(existingObservations, PREVIOUS_OBSERVATIONS_MAX_TOKENS);
+  const previousWasTruncated = Boolean(existingObservations?.trim()) && limitedExisting !== existingObservations;
+  if (limitedExisting?.trim()) {
+    prompt += `## Previous Observations\n\n${limitedExisting}\n\n---\n\nDo not repeat these existing observations. Your new observations will be appended to the existing observations. Previous observations may be truncated to the most recent/relevant tail for context budget safety.\n\n`;
   }
   const metadata: string[] = [];
   if (opts.priorCurrentTask) metadata.push(`- prior current-task: ${opts.priorCurrentTask}`);
   if (opts.priorSuggestedResponse) metadata.push(`- prior suggested-response: ${opts.priorSuggestedResponse}`);
   if (metadata.length) {
     prompt += `## Prior Thread Metadata\n\n${metadata.join("\n")}\n\n`;
-    if (opts.wasTruncated) {
+    if (opts.wasTruncated || previousWasTruncated) {
       prompt += `Previous observations were truncated for context budget reasons. The main agent still has full memory context outside this observer window.\n`;
     }
     prompt += `Use prior current-task and suggested-response as continuity hints, then update them based on the new messages.\n\n---\n\n`;
@@ -210,7 +216,8 @@ function buildObserverTaskPrompt(existingObservations: string | undefined, opts:
   return prompt;
 }
 
-function buildReflectorSystemPrompt(): string {
+function buildReflectorSystemPrompt(caveman = false): string {
+  const cavemanInstruction = caveman ? `\n\nCAVEMAN MODE: Compress aggressively. Prefer terse facts over prose. Preserve only actionable, durable memory.` : "";
   return `You are the memory consciousness of an AI assistant. Your memory observation reflections will be the ONLY information the assistant has about past interactions with this user.
 
 The following instructions were given to another part of your psyche (the observer) to create memories.
@@ -258,7 +265,7 @@ State current task(s) explicitly: primary and secondary pending tasks. Mark wait
 Hint for the agent's immediate next message.
 </suggested-response>
 
-User messages are extremely important. If the user asks a question or gives a new task, make it clear in <current-task> that this is the priority. If the assistant needs to respond, indicate in <suggested-response> that it should pause for user reply before continuing other tasks.`;
+User messages are extremely important. If the user asks a question or gives a new task, make it clear in <current-task> that this is the priority. If the assistant needs to respond, indicate in <suggested-response> that it should pause for user reply before continuing other tasks.${cavemanInstruction}`;
 }
 
 type CompressionLevel = 0 | 1 | 2 | 3 | 4;
@@ -292,6 +299,13 @@ type BufferedChunk = {
   createdAt: string;
 };
 
+type PiOMSettings = {
+  observationModel: string;
+  reflectionModel: string;
+  caveman: boolean;
+  observeAttachments: boolean;
+};
+
 type PiOMRecord = {
   version: number;
   enabled: boolean;
@@ -307,6 +321,7 @@ type PiOMRecord = {
   pendingMessageTokens: number;
   observationTokens: number;
   thresholds: { observation: number; reflection: number; blockAfter: number; bufferTokens: number; bufferActivation: number };
+  settings: PiOMSettings;
   buffered: { observations: BufferedChunk[]; reflection?: BufferedChunk };
   operationLock?: { type: OperationType; startedAt: string };
   lastOperation?: { type: OperationType; startedAt: string; endedAt?: string; inputTokens: number; outputTokens?: number; error?: string; model?: string; compressionLevel?: number };
@@ -565,6 +580,71 @@ export default function (pi: ExtensionAPI) {
     },
   });
 
+  pi.registerCommand("om", {
+    description: "Manage Observational Memory settings. Usage: /om, /om set <key> <value>, /om enable|disable|observe|reflect|memory",
+    handler: async (args, ctx) => {
+      const state = await ensureState(ctx);
+      const input = (args || "").trim();
+      if (!input || input === "status") {
+        await refreshCounts(ctx);
+        ctx.ui.notify(formatSettingsText(state), state.status === "failed" ? "error" : "info");
+        updateStatus(ctx);
+        return;
+      }
+      const [cmd, ...rest] = input.split(/\s+/);
+      if (cmd === "enable") {
+        state.enabled = true;
+        state.status = "idle";
+        await saveState(state);
+        updateStatus(ctx);
+        ctx.ui.notify("Observational Memory enabled", "info");
+        return;
+      }
+      if (cmd === "disable") {
+        state.enabled = false;
+        state.status = "disabled";
+        await saveState(state);
+        updateStatus(ctx);
+        ctx.ui.notify("Observational Memory disabled", "warning");
+        return;
+      }
+      if (cmd === "observe") {
+        await observeNow(ctx, { force: true, reason: "manual" });
+        updateStatus(ctx);
+        ctx.ui.notify("OM observation complete", "info");
+        return;
+      }
+      if (cmd === "reflect") {
+        await reflectNow(ctx, { reason: "manual", manualPrompt: rest.join(" ") || undefined });
+        updateStatus(ctx);
+        ctx.ui.notify("OM reflection complete", "info");
+        return;
+      }
+      if (cmd === "memory") {
+        ctx.ui.notify(formatMemoryText(state), "info");
+        return;
+      }
+      if (cmd === "set") {
+        const key = rest[0];
+        const value = rest.slice(1).join(" ");
+        applySetting(state, key, value);
+        await saveState(state);
+        updateStatus(ctx);
+        ctx.ui.notify(`OM setting updated: ${key}=${value}`, "info");
+        return;
+      }
+      if (cmd === "reset") {
+        state.thresholds = defaultThresholds();
+        state.settings = defaultSettings();
+        await saveState(state);
+        updateStatus(ctx);
+        ctx.ui.notify("OM settings reset to defaults", "info");
+        return;
+      }
+      throw new Error(`Unknown /om command: ${cmd}. Use /om, /om set <key> <value>, /om enable|disable|observe|reflect|memory|reset`);
+    },
+  });
+
   pi.registerCommand("om-compact", {
     description: "Run Pi compaction; pi-observational-memory will replace the summary with OM.",
     handler: async (args, ctx) => {
@@ -650,47 +730,132 @@ async function ensureState(ctx: any): Promise<PiOMRecord> {
   let state: PiOMRecord | undefined;
   if (existsSync(statePath)) {
     try {
-      state = JSON.parse(await readFile(statePath, "utf8")) as PiOMRecord;
+      state = normalizeState(JSON.parse(await readFile(statePath, "utf8")), { sessionId, sessionFile, cwd: ctx.cwd });
     } catch (error) {
-      throw new Error(`Failed to load OM state ${statePath}: ${errorMessage(error)}`);
+      const backupPath = `${statePath}${ATOMIC_BACKUP_SUFFIX}`;
+      if (existsSync(backupPath)) {
+        try {
+          state = normalizeState(JSON.parse(await readFile(backupPath, "utf8")), { sessionId, sessionFile, cwd: ctx.cwd });
+        } catch {
+          throw new Error(`Failed to load OM state ${statePath} and backup ${backupPath}: ${errorMessage(error)}`);
+        }
+      } else {
+        throw new Error(`Failed to load OM state ${statePath}: ${errorMessage(error)}`);
+      }
     }
   }
-  if (!state || state.version !== STATE_VERSION) {
-    state = {
-      version: STATE_VERSION,
-      enabled: true,
-      sessionId,
-      sessionFile,
-      cwd: ctx.cwd,
-      scope: "session",
-      status: "idle",
-      observations: "",
-      pendingMessageTokens: 0,
-      observationTokens: 0,
-      thresholds: {
-        observation: OBSERVATION_THRESHOLD,
-        reflection: REFLECTION_THRESHOLD,
-        blockAfter: Math.round(OBSERVATION_THRESHOLD * DEFAULT_BLOCK_AFTER_MULTIPLIER),
-        bufferTokens: Math.round(OBSERVATION_THRESHOLD * BUFFER_TOKENS_RATIO),
-        bufferActivation: BUFFER_ACTIVATION_RATIO,
-      },
-      buffered: { observations: [] },
-      updatedAt: new Date().toISOString(),
-    };
+  if (!state) {
+    state = createDefaultState({ sessionId, sessionFile, cwd: ctx.cwd });
     await saveState(state);
   }
+  const recovered = recoverStaleOperationLock(state);
   runtime.state = state;
+  if (recovered || state.version !== STATE_VERSION) await saveState(state);
   return state;
 }
 
+function defaultSettings(): PiOMSettings {
+  return { observationModel: DEFAULT_OBSERVATION_MODEL, reflectionModel: DEFAULT_REFLECTION_MODEL, caveman: false, observeAttachments: true };
+}
+
+function defaultThresholds(): PiOMRecord["thresholds"] {
+  return {
+    observation: OBSERVATION_THRESHOLD,
+    reflection: REFLECTION_THRESHOLD,
+    blockAfter: Math.round(OBSERVATION_THRESHOLD * DEFAULT_BLOCK_AFTER_MULTIPLIER),
+    bufferTokens: Math.round(OBSERVATION_THRESHOLD * BUFFER_TOKENS_RATIO),
+    bufferActivation: BUFFER_ACTIVATION_RATIO,
+  };
+}
+
+function createDefaultState(identity: { sessionId: string; sessionFile?: string; cwd: string }): PiOMRecord {
+  return {
+    version: STATE_VERSION,
+    enabled: true,
+    sessionId: identity.sessionId,
+    sessionFile: identity.sessionFile,
+    cwd: identity.cwd,
+    scope: "session",
+    status: "idle",
+    observations: "",
+    pendingMessageTokens: 0,
+    observationTokens: 0,
+    thresholds: defaultThresholds(),
+    settings: defaultSettings(),
+    buffered: { observations: [] },
+    updatedAt: new Date().toISOString(),
+  };
+}
+
+function normalizeState(raw: any, identity: { sessionId: string; sessionFile?: string; cwd: string }): PiOMRecord {
+  const defaults = createDefaultState(identity);
+  const state = {
+    ...defaults,
+    ...raw,
+    version: STATE_VERSION,
+    sessionId: raw?.sessionId || identity.sessionId,
+    sessionFile: raw?.sessionFile || identity.sessionFile,
+    cwd: raw?.cwd || identity.cwd,
+    thresholds: { ...defaults.thresholds, ...(raw?.thresholds ?? {}) },
+    settings: { ...defaults.settings, ...(raw?.settings ?? {}) },
+    buffered: { observations: [], ...(raw?.buffered ?? {}) },
+  } as PiOMRecord;
+  state.observations = redactSecrets(String(state.observations ?? ""));
+  state.currentTask = state.currentTask ? redactSecrets(state.currentTask) : undefined;
+  state.suggestedResponse = state.suggestedResponse ? redactSecrets(state.suggestedResponse) : undefined;
+  state.lastError = state.lastError ? redactSecrets(state.lastError) : undefined;
+  state.observationTokens = estimateTokens(state.observations);
+  return state;
+}
+
+function isStaleOperationLock(lock: PiOMRecord["operationLock"], now = Date.now()): boolean {
+  if (!lock) return false;
+  const started = Date.parse(lock.startedAt);
+  return Number.isFinite(started) && now - started > STALE_OPERATION_LOCK_MS;
+}
+
+function recoverStaleOperationLock(state: PiOMRecord, now = Date.now()): boolean {
+  if (!isStaleOperationLock(state.operationLock, now)) return false;
+  const lock = state.operationLock!;
+  state.lastOperation = { type: lock.type, startedAt: lock.startedAt, endedAt: new Date(now).toISOString(), inputTokens: 0, error: `Recovered stale OM operation lock after ${Math.round((now - Date.parse(lock.startedAt)) / 1000)}s` };
+  state.lastError = state.lastOperation.error;
+  state.operationLock = undefined;
+  state.status = state.enabled ? "idle" : "disabled";
+  return true;
+}
+
 async function saveState(state: PiOMRecord): Promise<void> {
   if (!runtime.statePath) return;
   state.updatedAt = new Date().toISOString();
-  await mkdir(dirname(runtime.statePath), { recursive: true });
-  await writeFile(runtime.statePath, JSON.stringify(state, null, 2) + "\n", "utf8");
+  state.observations = redactSecrets(state.observations);
+  if (state.currentTask) state.currentTask = redactSecrets(state.currentTask);
+  if (state.suggestedResponse) state.suggestedResponse = redactSecrets(state.suggestedResponse);
+  if (state.lastError) state.lastError = redactSecrets(state.lastError);
+  await atomicWriteJson(runtime.statePath, state);
   runtime.overlayHandle?.requestRender();
 }
 
+async function atomicWriteJson(filePath: string, value: unknown): Promise<void> {
+  await mkdir(dirname(filePath), { recursive: true });
+  const tmpPath = `${filePath}.${process.pid}.${Date.now()}.tmp`;
+  const backupPath = `${filePath}${ATOMIC_BACKUP_SUFFIX}`;
+  const json = redactSecrets(JSON.stringify(value, null, 2)) + "\n";
+  await writeFile(tmpPath, json, "utf8");
+  if (existsSync(filePath)) {
+    try {
+      await copyFile(filePath, backupPath);
+    } catch {
+      // Backup is best-effort; rename below is the durability boundary.
+    }
+  }
+  try {
+    await rename(tmpPath, filePath);
+  } catch (error) {
+    try { await unlink(tmpPath); } catch {}
+    throw error;
+  }
+}
+
 async function refreshCounts(ctx: any): Promise<void> {
   let isStale = false;
   try {
@@ -803,7 +968,7 @@ async function bufferObservation(ctx: any, reason: string): Promise<void> {
     });
     state.operationLock = undefined;
     state.status = "idle";
-    state.lastOperation = { type: "buffer", startedAt, endedAt: new Date().toISOString(), inputTokens: estimateTokens(inputText), outputTokens: estimateTokens(observations), model: DEFAULT_OBSERVATION_MODEL };
+    state.lastOperation = { type: "buffer", startedAt, endedAt: new Date().toISOString(), inputTokens: estimateTokens(inputText), outputTokens: estimateTokens(observations), model: state.settings.observationModel || DEFAULT_OBSERVATION_MODEL };
     await writeDebug(ctx, "buffer", { startedAt, reason, inputText, rawOutput: result.rawOutput, parsed: result });
     await saveState(state);
   } catch (error) {
@@ -903,7 +1068,7 @@ async function reflectNow(ctx: any, opts: { reason: string; signal?: AbortSignal
         state.operationLock = undefined;
         state.status = "idle";
         state.lastError = undefined;
-        state.lastOperation = { type: "reflection", startedAt, endedAt: new Date().toISOString(), inputTokens: originalTokens, outputTokens: reflectedTokens, model: DEFAULT_REFLECTION_MODEL, compressionLevel: level };
+        state.lastOperation = { type: "reflection", startedAt, endedAt: new Date().toISOString(), inputTokens: originalTokens, outputTokens: reflectedTokens, model: state.settings.reflectionModel || DEFAULT_REFLECTION_MODEL, compressionLevel: level };
         await saveState(state);
         return;
       }
@@ -923,9 +1088,11 @@ async function reflectNow(ctx: any, opts: { reason: string; signal?: AbortSignal
 }
 
 async function runObserver(ctx: any, historyText: string, signal: AbortSignal | undefined, opts: { source: string; existingObservations?: string }): Promise<ObserverResult> {
-  const model = resolveModel(ctx, DEFAULT_OBSERVATION_MODEL);
+  const state = await ensureState(ctx);
+  const modelId = state.settings.observationModel || DEFAULT_OBSERVATION_MODEL;
+  const model = resolveModel(ctx, modelId);
   const response = await runModel(ctx, model, [
-    { role: "user", content: [{ type: "text", text: buildObserverSystemPrompt() }] },
+    { role: "user", content: [{ type: "text", text: buildObserverSystemPrompt(state.settings.caveman) }] },
     { role: "user", content: [{ type: "text", text: `## New Message History to Observe\n\n${historyText}\n\n---\n\n${buildObserverTaskPrompt(opts.existingObservations, { priorCurrentTask: runtime.state?.currentTask, priorSuggestedResponse: runtime.state?.suggestedResponse })}` }] },
   ], { temperature: 0.3, maxTokens: 100_000, signal });
   const text = responseText(response);
@@ -936,9 +1103,11 @@ async function runObserver(ctx: any, historyText: string, signal: AbortSignal |
 }
 
 async function runReflector(ctx: any, observations: string, level: CompressionLevel, signal: AbortSignal | undefined, manualPrompt?: string): Promise<ReflectorResult> {
-  const model = resolveModel(ctx, DEFAULT_REFLECTION_MODEL);
+  const state = await ensureState(ctx);
+  const modelId = state.settings.reflectionModel || DEFAULT_REFLECTION_MODEL;
+  const model = resolveModel(ctx, modelId);
   const response = await runModel(ctx, model, [
-    { role: "user", content: [{ type: "text", text: buildReflectorSystemPrompt() }] },
+    { role: "user", content: [{ type: "text", text: buildReflectorSystemPrompt(state.settings.caveman) }] },
     { role: "user", content: [{ type: "text", text: buildReflectorPrompt(observations, level, manualPrompt) }] },
   ], { temperature: 0, maxTokens: 100_000, signal });
   const text = responseText(response);
@@ -986,7 +1155,7 @@ function extractListItemsOnly(content: string): string {
 }
 
 function sanitizeObservationLines(observations: string): string {
-  return observations.split("\n").map(line => line.length > MAX_OBSERVATION_LINE_CHARS ? `${line.slice(0, MAX_OBSERVATION_LINE_CHARS)} … [truncated]` : line).join("\n").trim();
+  return redactSecrets(observations).split("\n").map(line => line.length > MAX_OBSERVATION_LINE_CHARS ? `${line.slice(0, MAX_OBSERVATION_LINE_CHARS)} … [truncated]` : line).join("\n").trim();
 }
 
 function detectDegenerateRepetition(text: string): boolean {
@@ -1014,7 +1183,7 @@ function appendObservations(state: PiOMRecord, result: ObserverResult, inputToke
   state.currentTask = result.currentTask ?? state.currentTask;
   state.suggestedResponse = result.suggestedContinuation ?? state.suggestedResponse;
   state.observationTokens = estimateTokens(state.observations);
-  state.lastOperation = { type: "observation", startedAt: state.operationLock?.startedAt ?? new Date().toISOString(), endedAt: new Date().toISOString(), inputTokens, outputTokens: estimateTokens(observations), model: DEFAULT_OBSERVATION_MODEL };
+  state.lastOperation = { type: "observation", startedAt: state.operationLock?.startedAt ?? new Date().toISOString(), endedAt: new Date().toISOString(), inputTokens, outputTokens: estimateTokens(observations), model: state.settings.observationModel || DEFAULT_OBSERVATION_MODEL };
 }
 
 function buildOMContextMessage(state: PiOMRecord): AgentMessage {
@@ -1136,15 +1305,50 @@ function formatAgentMessage(msg: any, mode: "observer" | "recall" = "observer",
   }
 }
 
+
+function limitTextByTokens(text: string | undefined, maxTokens: number): string | undefined {
+  if (!text) return text;
+  if (estimateTokens(text) <= maxTokens) return text;
+  const maxChars = Math.max(0, maxTokens * 4);
+  const tail = text.slice(Math.max(0, text.length - maxChars));
+  const lineBoundary = tail.indexOf("\n");
+  const trimmedTail = lineBoundary >= 0 ? tail.slice(lineBoundary + 1) : tail;
+  return `[Earlier observations truncated for observer prompt safety: kept last ~${maxTokens} tokens.]\n${trimmedTail}`;
+}
+
+function redactSecrets(input: string): string {
+  if (!input) return input;
+  return input
+    .replace(/npm_[A-Za-z0-9]{16,}/g, "[REDACTED_NPM_TOKEN]")
+    .replace(/github_pat_[A-Za-z0-9_]{20,}/g, "[REDACTED_GITHUB_TOKEN]")
+    .replace(/gh[pousr]_[A-Za-z0-9_]{20,}/g, "[REDACTED_GITHUB_TOKEN]")
+    .replace(/sk-[A-Za-z0-9_-]{20,}/g, "[REDACTED_API_KEY]")
+    .replace(/(?<=(?:\b|["'])(?:api[_-]?key|token|secret|password)(?:\b|["'])\s*[:=]\s*["']?)[^"'\s,;}]{8,}/gi, "[REDACTED_SECRET]")
+    .replace(/\bBearer\s+[A-Za-z0-9._~+/=-]{16,}/gi, "Bearer [REDACTED_TOKEN]");
+}
+
+function redactDeep<T>(value: T): T {
+  if (typeof value === "string") return redactSecrets(value) as T;
+  if (Array.isArray(value)) return value.map(item => redactDeep(item)) as T;
+  if (value && typeof value === "object") {
+    const out: Record<string, unknown> = {};
+    for (const [key, child] of Object.entries(value as Record<string, unknown>)) {
+      out[key] = /apiKey|authorization|token|secret|password/i.test(key) ? "[REDACTED_SECRET]" : redactDeep(child);
+    }
+    return out as T;
+  }
+  return value;
+}
+
 function formatContent(content: any, cap: number): string {
-  if (typeof content === "string") return truncateText(content, cap);
-  if (!Array.isArray(content)) return truncateText(JSON.stringify(content), cap);
+  if (typeof content === "string") return truncateText(redactSecrets(content), cap);
+  if (!Array.isArray(content)) return truncateText(redactSecrets(JSON.stringify(redactDeep(content))), cap);
   return content.map(part => {
-    if (part.type === "text") return part.text;
-    if (part.type === "thinking") return `[Thinking]: ${part.thinking}`;
-    if (part.type === "image") return `[Image: ${part.mimeType ?? "unknown"}]`;
-    if (part.type === "toolCall") return `[Tool Call ${part.name}: ${truncateText(JSON.stringify(part.arguments ?? {}), 2_000)}]`;
-    return `[${part.type}: ${truncateText(JSON.stringify(part), 2_000)}]`;
+    if (part.type === "text") return redactSecrets(part.text);
+    if (part.type === "thinking") return `[Thinking]: ${redactSecrets(part.thinking ?? "")}`;
+    if (part.type === "image") return runtime.state?.settings.observeAttachments === false ? "[Image omitted: attachment observation disabled]" : `[Image: ${part.mimeType ?? "unknown"}]`;
+    if (part.type === "toolCall") return `[Tool Call ${part.name}: ${truncateText(redactSecrets(JSON.stringify(redactDeep(part.arguments ?? {}))), 2_000)}]`;
+    return `[${part.type}: ${truncateText(redactSecrets(JSON.stringify(redactDeep(part))), 2_000)}]`;
   }).join("\n").slice(0, cap);
 }
 
@@ -1179,9 +1383,9 @@ function truncateText(text: string, maxChars: number): string {
 }
 
 function responseText(response: any): string {
-  if (typeof response?.text === "string") return response.text;
-  if (Array.isArray(response?.content)) return response.content.filter((c: any) => c?.type === "text").map((c: any) => c.text).join("\n");
-  if (typeof response === "string") return response;
+  if (typeof response?.text === "string") return redactSecrets(response.text);
+  if (Array.isArray(response?.content)) return redactSecrets(response.content.filter((c: any) => c?.type === "text").map((c: any) => c.text).join("\n"));
+  if (typeof response === "string") return redactSecrets(response);
   return "";
 }
 
@@ -1219,6 +1423,70 @@ function formatShortStatusColored(state: PiOMRecord): string {
   return `om: ${statusColor}${state.status}\x1b[0m \x1b[2;37m|\x1b[0m msg \x1b[1;33m${formatTokens(state.pendingMessageTokens)}\x1b[0m/\x1b[2;37m${formatTokens(effectiveObservationThreshold(state))}\x1b[0m [${msgPctStr}] \x1b[2;37m|\x1b[0m mem \x1b[1;36m${formatTokens(state.observationTokens)}\x1b[0m/\x1b[2;37m${formatTokens(state.thresholds.reflection)}\x1b[0m [${memPctStr}]${buffer}${err}`;
 }
 
+function parsePositiveIntSetting(key: string, value: string): number {
+  const n = Number(value);
+  if (!Number.isFinite(n) || n <= 0) throw new Error(`OM setting ${key} must be a positive number`);
+  return Math.round(n);
+}
+
+function parseRatioSetting(key: string, value: string): number {
+  const raw = value.trim();
+  const n = raw.endsWith("%") ? Number(raw.slice(0, -1)) / 100 : Number(raw);
+  if (!Number.isFinite(n) || n <= 0 || n > 1) throw new Error(`OM setting ${key} must be a ratio between 0 and 1, or percent like 80%`);
+  return n;
+}
+
+function parseBooleanSetting(key: string, value: string): boolean {
+  const normalized = value.trim().toLowerCase();
+  if (["1", "true", "yes", "on", "enabled"].includes(normalized)) return true;
+  if (["0", "false", "no", "off", "disabled"].includes(normalized)) return false;
+  throw new Error(`OM setting ${key} must be on/off or true/false`);
+}
+
+function applySetting(state: PiOMRecord, key: string | undefined, value: string): void {
+  if (!key) throw new Error("Missing OM setting key");
+  if (!value) throw new Error(`Missing OM setting value for ${key}`);
+  const normalized = key.toLowerCase().replace(/_/g, "-");
+  if (["observation", "observation-threshold", "observe-threshold"].includes(normalized)) state.thresholds.observation = parsePositiveIntSetting(key, value);
+  else if (["reflection", "reflection-threshold", "reflect-threshold"].includes(normalized)) state.thresholds.reflection = parsePositiveIntSetting(key, value);
+  else if (["block-after", "blockafter"].includes(normalized)) state.thresholds.blockAfter = parsePositiveIntSetting(key, value);
+  else if (["buffer", "buffer-tokens"].includes(normalized)) state.thresholds.bufferTokens = parsePositiveIntSetting(key, value);
+  else if (["buffer-activation", "activation-ratio"].includes(normalized)) state.thresholds.bufferActivation = parseRatioSetting(key, value);
+  else if (["observation-model", "observer-model", "observe-model"].includes(normalized)) state.settings.observationModel = value.trim();
+  else if (["reflection-model", "reflector-model", "reflect-model"].includes(normalized)) state.settings.reflectionModel = value.trim();
+  else if (normalized === "caveman") state.settings.caveman = parseBooleanSetting(key, value);
+  else if (["attachments", "observe-attachments", "attachment-observation"].includes(normalized)) state.settings.observeAttachments = parseBooleanSetting(key, value);
+  else if (normalized === "scope") {
+    const scope = value.trim().toLowerCase();
+    if (scope !== "session" && scope !== "project") throw new Error("OM scope must be session or project");
+    state.scope = scope;
+  } else {
+    throw new Error(`Unknown OM setting: ${key}`);
+  }
+  if (state.thresholds.blockAfter < state.thresholds.observation) state.thresholds.blockAfter = Math.round(state.thresholds.observation * DEFAULT_BLOCK_AFTER_MULTIPLIER);
+  state.observationTokens = estimateTokens(state.observations);
+}
+
+function formatSettingsText(state: PiOMRecord): string {
+  return [
+    formatDetailedStatus(state),
+    "",
+    "settings:",
+    `  observation-model: ${state.settings.observationModel}`,
+    `  reflection-model: ${state.settings.reflectionModel}`,
+    `  caveman: ${state.settings.caveman ? "on" : "off"}`,
+    `  attachments: ${state.settings.observeAttachments ? "on" : "off"}`,
+    `  scope: ${state.scope}`,
+    "",
+    "usage:",
+    "  /om set observation-threshold 30000",
+    "  /om set reflection-threshold 40000",
+    "  /om set observation-model google/gemini-2.5-flash",
+    "  /om set caveman on",
+    "  /om reset",
+  ].join("\n");
+}
+
 function formatDetailedStatus(state: PiOMRecord): string {
   return `${formatShortStatus(state)}\nlastObservedEntryId: ${state.lastObservedEntryId ?? "none"}\ncurrentTask: ${state.currentTask ?? "none"}\nsuggestedResponse: ${state.suggestedResponse ?? "none"}\nlastOperation: ${state.lastOperation ? JSON.stringify(state.lastOperation, null, 2) : "none"}\nstatePath: ${runtime.statePath ?? "unknown"}`;
 }
@@ -1402,7 +1670,7 @@ async function writeDebug(ctx: any, name: string, payload: unknown): Promise<voi
     const dir = runtime.debugDir || join(ctx.cwd, CONFIG_DIR_NAME, "om", "debug");
     await mkdir(dir, { recursive: true });
     const file = join(dir, `${new Date().toISOString().replace(/[:.]/g, "-")}-${sanitizeFileName(name)}.json`);
-    await writeFile(file, JSON.stringify({ extension: EXTENSION_ID, sessionId: state.sessionId, ...payload as any }, null, 2) + "\n", "utf8");
+    await writeFile(file, redactSecrets(JSON.stringify(redactDeep({ extension: EXTENSION_ID, sessionId: state.sessionId, ...payload as any }), null, 2)) + "\n", "utf8");
   } catch (error) {
     // Ignore debug write errors if context is stale
   }
@@ -1427,6 +1695,22 @@ function mergeAbortSignals(a?: AbortSignal, b?: AbortSignal): AbortSignal | unde
   return controller.signal;
 }
 
+export const __test = {
+  redactSecrets,
+  redactDeep,
+  limitTextByTokens,
+  buildObserverTaskPrompt,
+  defaultSettings,
+  defaultThresholds,
+  createDefaultState,
+  normalizeState,
+  isStaleOperationLock,
+  recoverStaleOperationLock,
+  atomicWriteJson,
+  applySetting,
+  formatSettingsText,
+};
+
 class OMOverlay {
   private scroll = 0;
   private tab: "memory" | "status" | "debug" = "memory";

package/package.json

@@ -1,13 +1,39 @@
 {
   "name": "pi-observational-memory-extension",
-  "version": "0.1.2",
+  "version": "0.1.3",
   "description": "Mastra-style Observational Memory extension for Pi compaction and runtime context.",
   "license": "MIT",
   "author": "Nikita Nosov <20nik.nosov21@gmail.com>",
   "type": "module",
-  "keywords": ["pi-package", "pi", "pi-coding-agent", "observational-memory", "memory", "compaction"],
-  "files": ["extensions", "docs", "scripts", "README.md", "LICENSE", "CHANGELOG.md"],
-  "pi": { "extensions": ["./extensions"] },
+  "keywords": [
+    "pi-package",
+    "pi",
+    "pi-coding-agent",
+    "observational-memory",
+    "memory",
+    "compaction"
+  ],
+  "homepage": "https://github.com/nik1t7n/pi-observational-memory-extension#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/nik1t7n/pi-observational-memory-extension.git"
+  },
+  "bugs": {
+    "url": "https://github.com/nik1t7n/pi-observational-memory-extension/issues"
+  },
+  "files": [
+    "extensions",
+    "docs",
+    "scripts",
+    "README.md",
+    "LICENSE",
+    "CHANGELOG.md"
+  ],
+  "pi": {
+    "extensions": [
+      "./extensions"
+    ]
+  },
   "peerDependencies": {
     "@earendil-works/pi-coding-agent": "*",
     "@earendil-works/pi-ai": "*",
@@ -24,6 +50,7 @@
     "typecheck": "tsc --noEmit",
     "validate": "node scripts/validate.mjs",
     "pack:check": "npm pack --dry-run",
-    "prepublishOnly": "npm run validate && npm run typecheck && npm run pack:check"
+    "prepublishOnly": "npm run validate && npm run typecheck && npm test && npm run pack:check",
+    "test": "node scripts/test.mjs"
   }
 }

package/scripts/test.mjs

@@ -0,0 +1,70 @@
+#!/usr/bin/env node
+import assert from "node:assert/strict";
+import { execFileSync } from "node:child_process";
+import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import { existsSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+
+const dist = ".tmp-test-dist";
+await rm(dist, { recursive: true, force: true });
+execFileSync("npx", ["tsc", "--outDir", dist, "--declaration", "false", "--noEmit", "false", "--rootDir", "."], { stdio: "inherit" });
+const { __test } = await import(`../${dist}/extensions/index.js?${Date.now()}`);
+
+// Secret redaction: state/debug/observer text must not persist common tokens.
+const redacted = __test.redactSecrets("npm_abcdefghijklmnopqrstuvwxyz sk-abcdefghijklmnopqrstuvwxyz Bearer abcdefghijklmnopqrstuvwxyz token=supersecrettoken");
+assert(!redacted.includes("npm_abcdefghijklmnopqrstuvwxyz"));
+assert(!redacted.includes("sk-abcdefghijklmnopqrstuvwxyz"));
+assert(!redacted.includes("Bearer abcdefghijklmnopqrstuvwxyz"));
+assert(!redacted.includes("supersecrettoken"));
+assert(redacted.includes("[REDACTED_NPM_TOKEN]"));
+
+// Observer prompt limiting: previous observations are capped to a safe tail.
+const huge = Array.from({ length: 5000 }, (_, i) => `* 🔴 old observation ${i}`).join("\n");
+const prompt = __test.buildObserverTaskPrompt(huge, { priorCurrentTask: "continue tests" });
+assert(prompt.length < huge.length / 2, "observer prompt should not include full previous observations");
+assert(prompt.includes("truncated"));
+assert(prompt.includes("continue tests"));
+
+// Stale lock recovery: old operation locks are cleared and recorded as errors.
+const state = __test.createDefaultState({ sessionId: "s", cwd: process.cwd() });
+state.operationLock = { type: "observation", startedAt: new Date(Date.now() - 30 * 60 * 1000).toISOString() };
+state.status = "observing";
+assert.equal(__test.recoverStaleOperationLock(state), true);
+assert.equal(state.operationLock, undefined);
+assert.equal(state.status, "idle");
+assert.match(state.lastError, /Recovered stale OM operation lock/);
+
+// Settings parser: full /om set surface mutates persisted config safely.
+__test.applySetting(state, "observation-model", "google/gemini-2.5-flash");
+__test.applySetting(state, "reflection-threshold", "12345");
+__test.applySetting(state, "caveman", "on");
+__test.applySetting(state, "attachments", "off");
+assert.equal(state.settings.observationModel, "google/gemini-2.5-flash");
+assert.equal(state.thresholds.reflection, 12345);
+assert.equal(state.settings.caveman, true);
+assert.equal(state.settings.observeAttachments, false);
+assert.throws(() => __test.applySetting(state, "buffer-activation", "2"), /ratio/);
+
+// Migration/normalization: v1 state keeps observations and receives v2 settings.
+const migrated = __test.normalizeState({ version: 1, observations: "token=do-not-keep-this-secret", thresholds: { observation: 10 } }, { sessionId: "m", cwd: process.cwd() });
+assert.equal(migrated.version, 2);
+assert.equal(migrated.thresholds.observation, 10);
+assert.equal(migrated.settings.caveman, false);
+assert(!migrated.observations.includes("do-not-keep-this-secret"));
+
+// Atomic write: writes valid JSON, creates backup on subsequent write, removes temp files.
+const dir = await mkdtemp(join(tmpdir(), "pi-om-test-"));
+try {
+  const file = join(dir, "state.json");
+  await __test.atomicWriteJson(file, { a: 1, token: "supersecrettoken" });
+  assert.equal(JSON.parse(await readFile(file, "utf8")).token, "[REDACTED_SECRET]");
+  await __test.atomicWriteJson(file, { a: 2 });
+  assert.equal(JSON.parse(await readFile(file, "utf8")).a, 2);
+  assert.equal(existsSync(`${file}.bak`), true);
+} finally {
+  await rm(dir, { recursive: true, force: true });
+  await rm(dist, { recursive: true, force: true });
+}
+
+console.log("pi-observational-memory-extension tests passed");