pi-observational-memory-extension 0.1.2 → 0.2.0

package/CHANGELOG.md

@@ -0,0 +1,83 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+### 📖 How to Work with This Changelog
+
+When preparing a new release:
+1. **Always use Semantic Versioning** (`[Major.Minor.Patch]`).
+2. **Group changes under standard subheadings**:
+   - `Added` for new features.
+   - `Changed` for changes in existing functionality.
+   - `Deprecated` for soon-to-be removed features.
+   - `Removed` for now removed features.
+   - `Fixed` for any bug fixes.
+   - `Security` in case of vulnerabilities or security updates.
+3. **Keep descriptions concise, technical, and objective** (lowcase, founder-builder voice).
+4. **Link references or commit highlights** if appropriate.
+
+---
+
+## [0.2.0] - 2026-06-23
+
+### Added
+- **retrieval**: Added vector retrieval for observations before context injection.
+- **retrieval**: Added local hash/BOW retrieval as the default offline provider.
+- **retrieval**: Added configurable Gemini embedding retrieval with `gemini-embedding-001`.
+- **scope**: Added `session` and `project` memory scopes with separate durable storage paths.
+- **attachments**: Added `auto`, `on`, and `off` attachment observation modes with image and size safeguards.
+- **caveman**: Replaced the simplified terse mode with Mastra-style caveman compression instructions.
+- **dedupe**: Added exact-hash and similarity-based duplicate suppression before appending observations.
+
+### Changed
+- **state**: Migrated OM state schema to v3 for retrieval, vectors, attachment modes, and scoped storage.
+- **settings**: Extended `/om set` with `scope`, `retrieval`, `retrieval-model`, `retrieval-top-k`, and `retrieval-threshold`.
+
+### Fixed
+- **state**: Resolved JSON state file corruption caused by running regex redactions after serialization.
+- **redaction**: Refined token/secret sanitization to avoid over-redacting non-sensitive metrics (e.g., `observationTokens`).
+- **scope**: Added durable session pointer resolution, ensuring project scope selection persists across separate run processes.
+- **durability**: Added automatic `.corrupted` rename recovery for unparseable state files instead of crashing.
+
+### Tested
+- **regression**: Added tests for scope settings, attachment gates, duplicate suppression, and local retrieval.
+- **provider**: Smoke-tested Gemini embeddings against `gemini-embedding-001` with valid 3072-dimensional normalized vectors.
+
+## [0.1.3] - 2026-06-23
+
+### Added
+- **settings**: Added the unified `/om` command with `status`, `set`, `reset`, `enable`, `disable`, `observe`, `reflect`, and `memory` actions.
+- **settings**: Persisted model overrides, thresholds, caveman compression mode, attachment observation toggle, and scope configuration in OM state.
+- **tests**: Added a focused regression test suite for redaction, prompt limiting, stale lock recovery, settings parsing, v1→v2 migration, and atomic writes.
+
+### Fixed
+- **durability**: State files now use atomic temp-file writes with `.bak` recovery support.
+- **stability**: Stale operation locks are recovered automatically instead of blocking future OM runs forever.
+- **security**: State, debug logs, recall/observer serialization, and model outputs now redact common API keys, npm tokens, GitHub tokens, bearer tokens, passwords, and secret fields.
+- **context**: Observer prompts now cap previous observations to a bounded tail to prevent observation-context bloat.
+
+## [0.1.2] - 2026-06-22
+
+### Fixed
+- **om**: Resolved "stale context" errors during asynchronous `session_shutdown` handlers and background buffer tasks.
+- **om**: Safely caught uncaught promise rejections within internal `bufferObservation` tasks when contexts are deactivated by Pi.
+
+## [0.1.1] - 2026-06-22
+
+### Added
+- **om**: Added capability to force immediate observational compaction on `session_shutdown` if pending message tokens exist. This guarantees subagent and short-lived loopflow sessions persist their memories before exiting.
+
+## [0.1.0] - 2026-06-22
+
+### Added
+- **om**: First release of the Mastra-style **Observational Memory (OM)** extension.
+- **om**: Implemented the 3-Agent psychological memory model: Actor (Main Agent), Observer (Extraction Agent), and Reflector (Consolidation/Compression Agent).
+- **om**: Automatically intercepts `/compact` and `session_before_compact` events.
+- **om**: Real-time TUI status bar panel underneath the input showing memory, message, and token thresholds.
+- **om**: Color-coded, responsive status indicators for high (🔴), medium (🟡), and low (🟢) priority observations.
+- **om**: Dedicated interactive, fullscreen overlay `/om-memory` with interactive tab switching (`Memory`, `Status`, `Debug`) and ANSI word wrapping.

package/README.md

@@ -25,29 +25,50 @@ Legacy compaction compresses raw history into a single monolithic block of text,
 - **Async Buffering:** Extracts observations in the background at regular intervals (every 20% of the threshold) so that when the main threshold is hit, the accumulated memory is swapped/activated instantly with zero LLM-latency overhead.
 - **Adaptive Thresholds:** The active message-observation threshold expands dynamically based on remaining memory capacity in the reflection pool, maximizing raw context usage safely.
 - **Recall-driven Retrieval:** Exposes an `om_recall` tool to the Actor so it can retrieve full, raw message payloads from observed history when exact code, quotes, or numbers are needed.
+- **Vector Retrieval:** Ranks relevant observations before context injection. Local hash/BOW retrieval is enabled by default; Gemini embeddings (`gemini-embedding-001`) are available through `/om set retrieval gemini`.
+- **Session or Project Scope:** Stores memory either per session head or shared project memory via `/om set scope session|project`.
+- **Attachment Observation Modes:** Supports `auto`, `on`, and `off` with image-only auto mode and size limits to avoid unsafe prompt bloat.
+- **Duplicate Suppression:** Skips exact and near-duplicate observations before appending new memory.
 - **Custom Compaction Hook:** Plugs directly into Pi's `session_before_compact` lifecycle event. Typing `/compact` or triggering auto-compaction launches the Observer/Reflector memory consolidation flow instead of Pi's legacy summary compaction.
 - **TUI Overlay Panel:** Fully custom, responsive, multi-tab overlay (`/om-memory`) in interactive mode for inspecting memory. Features tabbed navigation, smooth scroll, and precise border framing.
-- **Durable Persistence:** Serializes and loads state files safely under `.pi/om/<session-id>.json`. Outputs JSON-formatted diagnostic logs to `.pi/om/debug/` for every operation.
+- **Durable Persistence:** Serializes and loads state files safely under `.pi/om/sessions/<session-id>.json` or `.pi/om/projects/project.json` using atomic writes and `.bak` recovery. Outputs JSON-formatted diagnostic logs to `.pi/om/debug/` for every operation.
+- **Secret Redaction:** Redacts common API keys, npm/GitHub tokens, bearer tokens, passwords, and secret fields before writing state/debug artifacts or observer text.
+- **Stale Lock Recovery:** Recovers old interrupted OM operation locks automatically so a crashed or killed process does not block future memory runs.
+- **Bounded Observer Context:** Sends only a safe tail of previous observations to the Observer, preventing recursive prompt bloat while preserving full memory in the main runtime.
 
 ---
 
 ## 📋 Configuration & Settings
 
-You can customize Observational Memory behaviors by specifying fields inside your project's `.pi/settings.json` or globally inside `~/.pi/agent/settings.json`.
+Use `/om` in Pi to inspect and update runtime settings. Settings are persisted in the OM state file under `.pi/om/sessions/<session-id>.json` or `.pi/om/projects/project.json`.
 
-```json
-{
-  "compaction": {
-    "enabled": true,
-    "reserveTokens": 16384,
-    "keepRecentTokens": 20000
-  }
-}
+```bash
+/om
+/om set observation-threshold 30000
+/om set reflection-threshold 40000
+/om set block-after 36000
+/om set buffer-tokens 6000
+/om set buffer-activation 80%
+/om set observation-model google/gemini-2.5-flash
+/om set reflection-model google/gemini-2.5-flash
+/om set caveman on
+/om set attachments auto
+/om set scope project
+/om set retrieval local
+/om set retrieval gemini
+/om set retrieval-model gemini-embedding-001
+/om set retrieval-top-k 6
+/om set retrieval-threshold 12%
+/om reset
 ```
 
-- **Observation Trigger Threshold:** ~`30,000` raw message tokens.
-- **Reflection Trigger Threshold:** ~`40,000` observation tokens.
+- **Observation Trigger Threshold:** default `30,000` raw message tokens.
+- **Reflection Trigger Threshold:** default `40,000` observation tokens.
 - **Default Models:** `google/gemini-2.5-flash` with 0.3 temperature for Observer, and 0.0 temperature for Reflector.
+- **Caveman Mode:** optional terse compression style for denser memory.
+- **Attachment Observation Mode:** `auto` observes small images only, `on` allows supported attachments, and `off` omits them.
+- **Scope:** `session` keeps each session isolated; `project` shares one memory file for the current project.
+- **Retrieval:** `local` is the default and works offline; `gemini` uses Gemini embeddings when `GEMINI_API_KEY` or `GOOGLE_GENERATIVE_AI_API_KEY` is available; `off` disables vector retrieval.
 
 ---
 
@@ -55,6 +76,7 @@ You can customize Observational Memory behaviors by specifying fields inside you
 
 This extension registers the following slash commands in Pi:
 
+- `/om` — Unified settings/status command. Supports `set`, `reset`, `enable`, `disable`, `observe`, `reflect`, and `memory`.
 - `/om-status` — Shows a detailed breakdown of pending/observation tokens, active locks, thresholds, and last operation results.
 - `/om-memory` — Opens the interactive multi-tab overlay panel (observations, status stats, and background debug details).
 - `/om-observe` — Forces an immediate Observation pass on all pending raw message history.
@@ -88,6 +110,7 @@ Run the typechecker and validation scripts before packaging or releasing:
 ```bash
 npm run typecheck
 npm run validate
+npm test
 ```
 
 ### Commit Guidelines