@bcmyguest/openmem 0.6.3 → 0.9.0

package/{skills → .agents/skills}/openmem/SKILL.md

@@ -15,12 +15,17 @@ For project work, pass the project bank explicitly so memories do not land in th
 
 - For past decisions or project context, call `memory_recall` with a focused query and the project bank. `memory_search` is also available for direct search.
 - Use `memory_reflect` when a question needs a synthesized answer across several memories. Check the returned evidence before treating it as current truth.
+- Before fixing a bug or changing a convention, call `memory_pages_search` (then `memory_pages_read`; `memory_pages_list` shows what exists) for curated decision and convention pages. Hindsight only.
+- When an `[openmem: continuing session <sid>; …]` pointer says earlier messages are not in your context, call `smart_compact_search` with that `session_id` and a query to recover the detail. Pass `pattern` (a JavaScript regex, `ignore_case` optional) instead of `query` for exact strings, and `context` (0–5) for neighbouring rows; `role` defaults to user and tool rows (`"all"` adds assistant text) and `in_context: false` keeps only trimmed rows; `line_matches` is an exact count of matching lines across the whole session (use it for "how many" questions). For counts, groupings or joins, call `smart_compact_sql` with the `session_id` and one read-only `SELECT` over the session database (`messages`, `lines`, `messages_fts`, `lines_fts`; the tool description lists the schema). For a compressed tool result, call `smart_compact_get` with its `id` (page with `offset`). All three read local files and work with every provider.
+- To check whether context management is working, call `context_status` (or run `openmem context status`), not docker logs. It reports trims, estimated tokens saved and fail-opens. A fail-open means the request still succeeded but went to the model untrimmed; `proxy_stale: true` means the proxy runs older code than the checkout and needs a restart.
 - If the tools are unavailable, say that memory was unavailable. Do not claim to have searched it.
 
 ## Store and correct
 
 - Session hooks capture supported conversations. When the user explicitly asks to remember a durable fact, external note, or decision now, call `memory_add` with a concise, self-contained record and the project bank. On Hindsight, `memory_add` defaults to **async** ingest (omit `async` or pass `async: true`) so MCP clients do not time out; use `async: false` only when you must wait for extraction.
 - When verified code or current documentation contradicts a memory, call `memory_add` with a correction that states the old claim, the current fact, and the evidence. Do not silently leave a known false record to resurface.
+- After the user approves a plan for a new feature, call `memory_capture_initiative` before writing code; skip bug fixes and small tweaks.
+- Call `memory_status` before relying on recall of just-stored content; it reports whether async ingest has drained.
 - `memory_forget` can delete an entire Hindsight bank. Confirm the exact scope before using it for bulk deletion.
 
 Use `memory_provider_status` to identify the active provider when that matters. The provider can change without changing this skill or the tool names.

package/README.md

@@ -2,18 +2,45 @@
 
 [![npm version](https://img.shields.io/npm/v/@bcmyguest/openmem.svg)](https://www.npmjs.com/package/@bcmyguest/openmem)
 [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
-[![openmem-mcp](https://glama.ai/mcp/servers/@bcmyguest/openmem-mcp/badges/score.svg)](https://glama.ai/mcp/servers/@bcmyguest/openmem)
+[![openmem-mcp](https://glama.ai/mcp/servers/bcmyguest/openmem/badges/score.svg)](https://glama.ai/mcp/servers/bcmyguest/openmem/score)
 [![GitHub](https://img.shields.io/badge/github-bcmyguest%2Fopenmem-blue)](https://github.com/bcmyguest/openmem)
 
-[![openmem-mcp](https://glama.ai/mcp/servers/@bcmyguest/openmem-mcp/badges/card.svg)](https://glama.ai/mcp/servers/@bcmyguest/openmem)
+[![openmem-mcp](https://glama.ai/mcp/servers/bcmyguest/openmem/badges/card.svg)](https://glama.ai/mcp/servers/bcmyguest/openmem)
 
-**Docs:** [bcmyguest.github.io/openmem/](https://bcmyguest.github.io/openmem/) · **npm:** [@bcmyguest/openmem](https://www.npmjs.com/package/@bcmyguest/openmem) · **MCP:** `npx -y -p @bcmyguest/openmem openmem-mcp` · **Repo:** [github.com/bcmyguest/openmem](https://github.com/bcmyguest/openmem)
+**OpenMem.** Memory and context management for coding agents: smart compaction that keeps every turn searchable, pluggable [memory providers](https://bcmyguest.github.io/openmem/docs/provider-status/) (Hindsight, mem0, Cognee), one setup wizard.
 
-**OpenMem** is a **memory interface for coding agents**: a stable `memory_*` API and ops surface over pluggable backends (Hindsight, mem0, Cognee, …), with shared wizard, indexer, and import/export formats (PMM first).
+**A. Memory**: your sources become memory any agent can query.
 
-> Formerly **personal-memory**. Use the `openmem` CLI and MCP server name; installers drop the old `personal-memory` MCP entry.
+<p align="center">
+<picture>
+<source media="(prefers-color-scheme: dark)" srcset="https://bcmyguest.github.io/openmem/media/overview-memory-wide-dark.png" />
+<img src="https://bcmyguest.github.io/openmem/media/overview-memory-wide.png" alt="A, memory: sources (Git repos, Obsidian vaults, notes, papers, bookmarks, feeds), OpenMem indexer, one active provider (Hindsight default, mem0, Cognee), OpenMem MCP tools, coding agents with session hooks" />
+</picture>
+</p>
+
+**B. Context management**: long sessions stay under the limit; nothing is summarized away.
+
+<p align="center">
+<picture>
+<source media="(prefers-color-scheme: dark)" srcset="https://bcmyguest.github.io/openmem/media/overview-context-wide-dark.png" />
+<img src="https://bcmyguest.github.io/openmem/media/overview-context-wide.png" alt="B, context management: coding agent, context proxy trimming at 125k tokens (Haiku 95k) with no LLM call, model API, SQLite session store read back with smart_compact_search and smart_compact_sql; OpenCode 2 uses a native plugin" />
+</picture>
+</p>
+
+**Benchmarks:** In seed-matched [ContextBench](https://github.com/RulinShao/ContextBench) log tasks on Haiku 5.5, smart compaction scored 0.469 higher than keep-everything (95% bootstrap interval 0.339–0.578; eight pairs). All arms scored 1.0 on kv R4000. Costs fell at R4000, L6400, and L12800; the L1600 saving remains uncertain. Claude Code's default native arm made no compaction in these tasks. [Methods and results](docs/context-management.md#benchmarks) · [paired intervals and pending work](docs/evidence-plan.md#completed-seed-paired-comparisons). Compact-Bench real-coding results are pending.
+
+<p align="center">
+<picture>
+<source media="(prefers-color-scheme: dark)" srcset="https://bcmyguest.github.io/openmem/media/contextbench-quality-compact-dark.png" />
+<img src="https://bcmyguest.github.io/openmem/media/contextbench-quality-compact.png" alt="ContextBench scores: kv R4000 scores 1.00 in all arms; on log tasks, OpenMem scores 1.00 at L1600, 0.88 at L6400 and 1.00 at L12800, above both baselines" width="480" />
+</picture>
+</p>
 
-**Start here:** [Concepts](https://bcmyguest.github.io/openmem/docs/concepts/) · [Quickstart](https://bcmyguest.github.io/openmem/docs/getting-started/)
+**Docs:** [bcmyguest.github.io/openmem/](https://bcmyguest.github.io/openmem/) 
+
+**MCP:** `npx -y -p @bcmyguest/openmem openmem-mcp` (after the wizard — it needs a running provider)
+
+**Start here:** [Concepts](https://bcmyguest.github.io/openmem/docs/concepts/) · [Quickstart](https://bcmyguest.github.io/openmem/docs/getting-started/) · [Compatibility](https://bcmyguest.github.io/openmem/docs/compatibility/)
 
 ## Quickstart
 
@@ -27,91 +54,61 @@
 
 ```sh
 git clone https://github.com/bcmyguest/openmem.git && cd openmem
-npm install && npx . setup
+npm install && npm run setup
 ```
 
-**Needs:** Docker Compose v2, Node ≥ 20.12, and an LLM route if you use Hindsight (lemond, Ollama, or OpenAI-compatible — set in the wizard).
+Other commands from a checkout: `node cli/openmem.mjs <command>`.
 
-### Day-2
+**Needs:** Docker Compose v2, Node ≥ 24 (built-in `node:sqlite` with FTS5), and an LLM route if you use Hindsight (lemond, Ollama, OpenAI-compatible, or a Claude subscription through Meridian — set in the wizard).
 
-```sh
-npx @bcmyguest/openmem up
-npx @bcmyguest/openmem down
-npx @bcmyguest/openmem switch mem0 --yes
-npx @bcmyguest/openmem doctor
-```
+Later: `npx @bcmyguest/openmem up` / `down` / `switch <provider> --yes` / `doctor`.
 
 Wizard dry-run (what first setup plans — no writes):
 
-![Setup dry-run: npx @bcmyguest/openmem setup --dry-run](https://bcmyguest.github.io/openmem/setup-demo.gif)
+<p align="center">
+<img src="https://bcmyguest.github.io/openmem/setup-demo.gif" alt="Terminal recording of the setup wizard dry run: preflight checks, memory provider, model backend, secrets, sources and stack steps, each planned write printed as a [dry-run] line" width="720" />
+</p>
 
 Full guide: [Quickstart](https://bcmyguest.github.io/openmem/docs/getting-started/) · [Concepts](https://bcmyguest.github.io/openmem/docs/concepts/) · [Architecture](https://bcmyguest.github.io/openmem/docs/architecture/)
 
 ## Architecture
 
+The two diagrams at the top show memory (A) and context management (B).
+
+<details>
+<summary>Same diagrams as Mermaid</summary>
+
 ```mermaid
-flowchart TB
-  subgraph disk["Sources on disk"]
-    GIT[Git repos]
-    OBS[Obsidian vaults]
-    MD[notes · papers · bookmarks]
-    FEED[Feed cache]
-  end
-
-  subgraph compose["Docker Compose — one active provider"]
-    IDX[indexer]
-    PROV[(Provider API)]
-    IDX -->|retain / update| PROV
-    LIT[LiteLLM]
-    NLM[NotebookLM sync]
-    LIT -.->|Hindsight stack + optional reflect synth| PROV
-    NLM -.->|opt-in profile| PROV
-  end
-
-  LIT --> LLM[(lemond · Ollama · OpenAI-compatible)]
-
-  subgraph host["Host — coding agents"]
-    AG[Claude Code · Codex · Cursor Agent · OpenCode]
-    UMCP["OpenMem MCP\n(memory_*)"]
-    SHIM["Session hooks"]
-    PLG["Hindsight plugin\n(optional)"]
-    SKILL["openmem skill"]
-    PMCP[Provider MCP optional]
-    NLMCP[NotebookLM MCP optional]
-  end
-
-  GIT --> IDX
-  OBS --> IDX
-  MD --> IDX
-  FEED --> IDX
-  AG <-->|stdio| UMCP
-  AG <-->|hooks| SHIM
-  AG --> SKILL
-  AG <-->|optional| PLG
-  SHIM --> UMCP
-  AG <-->|stdio| PMCP
-  AG <-->|stdio| NLMCP
-  UMCP -->|loopback HTTP| PROV
-  PLG -->|loopback HTTP| PROV
-  PMCP -->|loopback HTTP| PROV
-  NLMCP -->|loopback HTTP| PROV
-
-  W["npx @bcmyguest/openmem\nsetup · install-mcp"] -.->|configure| compose
-  W -.->|mcp.json| UMCP
+flowchart LR
+  S["<b>Sources</b><br/>Git repos: docs, code outlines<br/>Obsidian vaults · notes · papers · bookmarks<br/>RSS · OpenAlex feeds"] -->|index| I["<b>OpenMem indexer</b><br/>polls repos, watches folders"]
+  I -->|retain| P[("<b>Memory provider</b><br/>one active, swappable<br/>Hindsight (default) · mem0 · Cognee · …")]
+  P <--> M["<b>OpenMem MCP</b><br/>memory_recall · memory_add<br/>memory_search · memory_reflect<br/>memory_pages_* + export/import, status…"]
+  M <-->|tools| A["<b>Coding agents</b><br/>Claude Code · Codex · Cursor · OpenCode"]
+  A -.->|"session hooks: recall at start · retain at stop"| M
+  classDef om fill:#e8eaf9,stroke:#4651a6,color:#1b1d2a
+  class I,M om
 ```
 
-| Connection | What it is |
-| --- | --- |
-| Sources → indexer → provider | Shared watcher; sink from active manifest (`MEMORY_SINK`) |
-| Agents → MCP / hooks → provider | `install-coding-agent` installs the shared skill, unified `memory_*` MCP, and session hooks for Claude Code, Codex, Cursor Agent, and OpenCode |
-| Wizard (dashed) | Writes env, starts compose, registers MCP — not on the hot indexing path |
+```mermaid
+flowchart LR
+  A["<b>Coding agent</b><br/>Claude Code · Codex · OpenCode 1"] -->|requests| X["<b>Context proxy</b><br/>LiteLLM pre-call hook<br/>trims at 125k tokens (Haiku 95k) · no LLM call"]
+  X -->|trimmed request| API["<b>Model API</b><br/>Anthropic · ChatGPT (Codex)"]
+  X -->|every turn| S[("<b>Session store</b><br/>SQLite · every turn, verbatim")]
+  S -->|"MCP: smart_compact_search · smart_compact_sql"| A
+  O["<b>OpenCode 2</b><br/>native plugin, no proxy"] -.->|same store| S
+  classDef om fill:#e8eaf9,stroke:#4651a6,color:#1b1d2a
+  class X,S om
+```
 
-On Hindsight stacks, `memory_reflect` uses Hindsight's native reflect API and returns its answer with source evidence. On mem0 and Cognee, it searches first and uses the configured LiteLLM route for synthesis when available; otherwise it returns the search hits. See [Architecture](https://bcmyguest.github.io/openmem/docs/architecture/) for MCP and hooks limits.
+</details>
+
+Native compaction replaces history with an LLM summary, so exact values can be lost; OpenMem keeps every turn verbatim and searchable. The optional [context proxy](https://bcmyguest.github.io/openmem/docs/context-proxy/) trims long agent sessions without an LLM summary. See the [full architecture](https://bcmyguest.github.io/openmem/docs/architecture/) for setup, optional integrations, and provider-specific behavior.
 
 ## Providers and portability
 
-- **Providers** — manifests under `providers/` ([Hindsight](https://bcmyguest.github.io/openmem/docs/hindsight/), [mem0](https://bcmyguest.github.io/openmem/docs/mem0/), [Cognee](https://bcmyguest.github.io/openmem/docs/cognee/)). Add another with a compose fragment, indexer sink, and wizard hooks.
+- **Providers** — [Provider status](https://bcmyguest.github.io/openmem/docs/provider-status/) lists every provider and its support level from the manifests. Contributor details: [adding a provider](https://github.com/bcmyguest/openmem/blob/main/docs/memory-providers.md).
 - **MCP** — unified `memory_*` plus optional native/NotebookLM servers: [MCP setup](https://bcmyguest.github.io/openmem/docs/mcp-setup/).
+- **Context management** — smart compaction on/off, per-agent support, tools and config: [Context management](https://bcmyguest.github.io/openmem/docs/context-management/).
 - **Portable memory** — `memory_export` / `memory_import` speak **PMM**: [Portable memory](https://bcmyguest.github.io/openmem/docs/memory-portability/).
 - **Switch** — `npx @bcmyguest/openmem switch <provider>` does not migrate data by itself; export/import (or re-index) moves content.
 
@@ -125,12 +122,20 @@ OpenMem is not affiliated with these projects; they shape the problem or the pat
 | --- | --- |
 | **[OpenFeature](https://openfeature.dev/)** | Inspiration for a vendor-agnostic interface + pluggable providers (there: feature flags; here: coding-agent memory). |
 | **[mise](https://mise.jdx.dev/) / [asdf](https://asdf-vm.com/)** | One CLI over many backends; declarative config and tasks without replacing each tool’s own settings. |
-| **[Hindsight](https://github.com/vectorize-io/hindsight)** | Default memory provider in this repo. |
-| **[mem0](https://github.com/mem0ai/mem0)** | Supported memory provider. |
-| **[Cognee](https://github.com/topoteretes/cognee)** | Supported memory provider. |
-| **[LiteLLM](https://github.com/BerriAI/litellm)** | Model proxy used on Hindsight stacks. |
+| **[LiteLLM](https://github.com/BerriAI/litellm)** | Model proxy used on Hindsight stacks; its pre-call hook hosts the context proxy. |
+| **[Hindsight coding agents](https://www.npmjs.com/package/@vectorize-io/hindsight-coding-agents)** | Reference implementation for knowledge pages, initiative capture and ingest status; OpenMem's hooks replace it in `openmem` mode. |
+| **[Mainplane](https://github.com/mainplane-ai/mainplane)** | Inspiration for smart compaction: continue in a fresh context with a pointer to the saved transcript and retrieve as you go, instead of an LLM summary. |
+| **[Letta](https://www.letta.com/) / [MemGPT](https://arxiv.org/abs/2310.08560)** | Inspiration for smart compaction: evicted turns kept verbatim in searchable recall storage the agent pages back with tools, and [file-style search over the transcript](https://www.letta.com/blog/benchmarking-ai-agent-memory/) (74.0% on LoCoMo). OpenMem does this in a proxy and an MCP server, so no Letta server is needed. |
+| **[rtk](https://github.com/rtk-ai/rtk)** | Compresses shell command output before the agent sees it; `install-coding-agent` runs `rtk init` when it is installed. Inspired the context proxy's compression of large tool results (reimplemented, not bundled). |
+| **[opencode-clm](https://github.com/bcmyguest/opencode-clm)** | OpenCode plugin for [Context Language Models](https://arxiv.org/abs/2609.37725): the model edits a file mirror of its own context. Same goal as smart compaction, with the model doing the trimming ([comparison](https://bcmyguest.github.io/openmem/docs/context-research/)). |
+| **[Meridian](https://github.com/rynfar/meridian)** | Bridges a Claude subscription to an Anthropic-compatible API; optional Hindsight model backend ([guide](https://bcmyguest.github.io/openmem/docs/model-configuration/)). |
+| **[OpenCode](https://opencode.ai/)** / **[Claude Code hooks](https://code.claude.com/docs/en/hooks)** / **[Codex hooks](https://developers.openai.com/codex/hooks)** | The agent hook and plugin APIs OpenMem translates between ([compatibility](https://bcmyguest.github.io/openmem/docs/compatibility/)). |
+
+Research notes and benchmark results for the context proxy: [Context research](https://bcmyguest.github.io/openmem/docs/context-research/).
 
-More setup analogues (Chezmoi, Home Assistant integrations, …): [Provider status](https://bcmyguest.github.io/openmem/docs/provider-status/).
+More setup analogues: [Pluggable memory providers](https://github.com/bcmyguest/openmem/blob/main/docs/memory-providers.md).
+
+> Formerly **personal-memory**. Use the `openmem` CLI and MCP server name; installers drop the old `personal-memory` MCP entry.
 
 ## License
 

package/cli/coding-agent/capabilities.mjs

@@ -5,12 +5,23 @@ import { CODING_AGENTS, DEFAULT_AGENT_IDS } from "./agents.mjs";
 export const AGENT_IDS = Object.freeze(CODING_AGENTS.map((a) => a.id));
 export { DEFAULT_AGENT_IDS };
 
-/** @typedef {"retain" | "recall" | "reflect" | "consolidate" | "bank_scope"} Feature */
+/**
+ * @typedef {"retain" | "recall" | "reflect" | "consolidate" | "bank_scope" | "knowledge_pages" | "initiatives" | "ingest_status"} Feature
+ */
 
 /** @type {Record<string, Partial<Record<Feature, boolean>>>} */
 export const PROVIDER_CAPABILITIES = {
-  hindsight: { retain: true, recall: true, reflect: true, consolidate: true, bank_scope: true },
-  mem0: { retain: true, recall: true, reflect: true, consolidate: false, bank_scope: true },
+  hindsight: {
+    retain: true,
+    recall: true,
+    reflect: true,
+    consolidate: true,
+    bank_scope: true,
+    knowledge_pages: true,
+    initiatives: true,
+    ingest_status: true,
+  },
+  mem0: { retain: true, recall: true, reflect: true, consolidate: false, bank_scope: true, ingest_status: true },
   cognee: { retain: true, recall: true, reflect: true, consolidate: true, bank_scope: true },
 };
 

package/cli/coding-agent/context-store.mjs

@@ -0,0 +1,81 @@
+/** Hook side of the context store contract (docs/context-store.md). */
+import fs from "node:fs";
+import path from "node:path";
+
+/** Marker in the installed smart-compact command body; survives command expansion. */
+export const SMART_COMPACT_MARKER = "openmem:smart-compact";
+/** Prompt that `/smart-compact` sends in every agent (Claude Code command file, OpenCode 2 command). */
+export const SMART_COMPACT_BODY = `<!-- ${SMART_COMPACT_MARKER} -->
+OpenMem will trim this session's earlier context on the next request. Reply with one short line acknowledging this; do not summarize the conversation. After the trim, use \`smart_compact_search\` with this session's id to look up anything earlier.`;
+/** Claude Code command file (`~/.claude/commands/smart-compact.md`): frontmatter plus SMART_COMPACT_BODY. */
+export const SMART_COMPACT_COMMAND = `---
+description: Trim this session's context on the next request (OpenMem)
+---
+${SMART_COMPACT_BODY}
+`;
+const SMART_COMPACT_PREFIX = /^\s*\/smart-compact(?:\s|$)/;
+const SMART_COMPACT_TAG = "<command-name>/smart-compact</command-name>";
+/** ASCII whitespace only, so the hooks and the proxy trim the same characters (session ids, OPENMEM_STATE_DIR). */
+const ASCII_TRIM_RE = /^[ \t\n\v\f\r]+|[ \t\n\v\f\r]+$/g;
+const SID_UNSAFE_RE = /[^A-Za-z0-9._-]/gu;
+
+/** `$OPENMEM_STATE_DIR/context` (value trimmed), default `~/.local/share/openmem/context`. */
+export function contextRoot(home, env = process.env) {
+  const state = String(env.OPENMEM_STATE_DIR || "").replace(ASCII_TRIM_RE, "") || path.join(home, ".local", "share", "openmem");
+  return path.join(state, "context");
+}
+
+/**
+ * Contract sanitizer (docs/context-store.md): trim surrounding ASCII whitespace, replace every character
+ * outside `[A-Za-z0-9._-]` with `_`, throw on an empty result.
+ */
+export function sanitizeSessionId(sessionId) {
+  const sid = String(sessionId ?? "").replace(ASCII_TRIM_RE, "").replace(SID_UNSAFE_RE, "_");
+  if (!sid) throw new Error("empty session id");
+  return sid;
+}
+
+function sessionsDir(home, env) {
+  const root = contextRoot(home, env);
+  const dir = path.join(root, "sessions");
+  fs.mkdirSync(dir, { recursive: true, mode: 0o700 });
+  fs.chmodSync(root, 0o700);
+  fs.chmodSync(dir, 0o700);
+  return dir;
+}
+
+function writePrivate(file, text) {
+  const temporary = `${file}.${process.pid}.tmp`;
+  fs.writeFileSync(temporary, text, { mode: 0o600 });
+  fs.renameSync(temporary, file);
+}
+
+/** Write `sessions/<sid>.json`. Returns the path, or null without a session id. */
+export function registerSession({ home, env, sessionId, agent, cwd, bank, now = new Date() }) {
+  if (!sessionId) return null;
+  const sid = sanitizeSessionId(sessionId);
+  const file = path.join(sessionsDir(home, env), `${sid}.json`);
+  const registration = {
+    session_id: sid,
+    agent,
+    cwd,
+    bank,
+    registered_at: now.toISOString().replace(/\.\d{3}Z$/, "Z"),
+  };
+  writePrivate(file, `${JSON.stringify(registration)}\n`);
+  return file;
+}
+
+/** Write the empty `sessions/<sid>.force` flag. Returns the path, or null without a session id. */
+export function writeForceFlag({ home, env, sessionId }) {
+  if (!sessionId) return null;
+  const file = path.join(sessionsDir(home, env), `${sanitizeSessionId(sessionId)}.force`);
+  writePrivate(file, "");
+  return file;
+}
+
+/** True for `/smart-compact …`, Claude Code's command tag, or the expanded command body. */
+export function isSmartCompactPrompt(prompt) {
+  const text = String(prompt || "");
+  return SMART_COMPACT_PREFIX.test(text) || text.includes(SMART_COMPACT_TAG) || text.includes(SMART_COMPACT_MARKER);
+}