@bcmyguest/openmem 0.0.0-stage → 0.6.3

package/docs/cognee.md

@@ -0,0 +1,37 @@
+# Cognee (loopback container)
+
+Self-hosted graph/RAG memory via a pinned `cognee/cognee` image on a loopback port (`COGNEE_PORT`, default **8766**). Auth is off in this stack. Cognify defaults to local GLiNER + FastEmbed (no API key); optional LLM keys live in env files under `~/.config/personal-memory/`, not in stack `.env` or `mcp.json`.
+
+## Coding agents
+
+No plugin package. Agents use REST (`/docs`, `/health`, `/api/v1/add`, `/cognify`, `/search`) or the **read-only** MCP in this repo.
+
+### Cursor MCP
+
+When `MEMORY_PROVIDER=cognee`:
+
+```sh
+npx @bcmyguest/openmem install-mcp --yes --provider-only
+```
+
+Registers `cli/cognee-mcp.mjs` with `COGNEE_API_URL` only — **never** `COGNEE_API_KEY` in `mcp.json`. If you enable auth, export the key in the agent environment.
+
+| Tool | Purpose |
+| --- | --- |
+| `cognee_health` | `GET /health` |
+| `cognee_search` | `POST /api/v1/search` |
+
+Ingest and cognify stay with the indexer or direct API calls; the MCP is search/health only.
+
+Restart Cursor Agent after install.
+
+## Switch provider
+
+```sh
+npx @bcmyguest/openmem switch cognee --yes
+# later: npx @bcmyguest/openmem switch hindsight --yes
+```
+
+Updates active provider config, restarts the stack (unless `--no-stack`), runs MCP install (unless `--no-mcp`). Does **not** migrate memories.
+
+See also [Architecture](architecture.md), [Hindsight](hindsight.md), and [Provider status](provider-status.md).

package/docs/concepts.md

@@ -0,0 +1,112 @@
+# Concepts
+
+OpenMem is a **memory interface for coding agents**: a stable API and operator surface, with pluggable backends underneath.
+
+## What’s coding-agent memory?
+
+Coding agents forget between sessions. A memory system stores durable facts, decisions, and project context so the next turn (or agent) can recall them without re-deriving everything from the repo.
+
+In practice that means:
+
+- **Write** — retain notes, session transcripts, indexed sources
+- **Read** — search, recall, or synthesize an answer from stored memories
+- **Scope** — keep personal notes, vaults, and per-repo banks from colliding
+- **Operate** — run a backend locally (or point at one), keep secrets off the wire you did not choose
+
+Full use of memory needs a **memory system**: an API (or library) plus storage, extraction, and usually an admin or ops path. Hindsight, mem0, Cognee, and in-house stores are examples. Each has its own API, deploy story, and agent wiring.
+
+## What’s OpenMem?
+
+OpenMem is **not** a memory engine. It is the layer that sits between coding agents and whichever engine you run:
+
+| Layer | Role |
+| --- | --- |
+| **Agents** | Claude Code, Codex, Cursor Agent, OpenCode — call stable `memory_*` tools and share one skill |
+| **OpenMem** | Wizard, Compose, indexer, unified MCP, session hooks, import/export formats |
+| **Provider** | Hindsight, mem0, Cognee, or the next manifest — owns semantics, storage, and native APIs |
+
+One active provider at a time. `openmem switch` rewires the stack; it does not pretend to migrate your data. Portability is explicit through the format layer (PMM today).
+
+```mermaid
+flowchart TB
+  AG[Coding agents]
+  OM["OpenMem surface<br/>memory_* · skill · session hooks · wizard · indexer"]
+  AG --> OM
+  OM --> HS[Hindsight]
+  OM --> M0[mem0]
+  OM --> CG[Cognee]
+  OM --> NX["…next provider manifest"]
+```
+
+## Why an interface layer?
+
+Without a shared front door, every agent and indexer hard-codes one vendor’s SDK or HTTP shape. Switching backends means rewriting hooks, MCP entries, and indexing. Trying a second system means a second install path.
+
+Standardizing the **agent and ops surface** means:
+
+- Keep using your current memory product
+- Try another backend without rewriting agent prompts and MCP tool names
+- Share indexer, wizard, and install plumbing across backends
+- Grow a community of providers behind one contract
+
+OpenMem owns machine plumbing. Providers own memory quality.
+
+## Core pieces
+
+### Providers
+
+A **provider** is the translation layer between OpenMem and a memory backend. It maps manifests, Compose fragments, wizard hooks, and an indexer sink onto that product’s API so agents and the indexer stay backend-agnostic.
+
+Ship a provider as `providers/<id>/` (manifest + hooks + compose). Status (`supported` / `experimental` / `planned`) comes from the manifest. See [Provider status](provider-status.md) and [Pluggable memory providers](memory-providers.md).
+
+Only one provider is registered as active (`MEMORY_PROVIDER`). Switching replaces the active backend for Compose, the indexer sink, and unified MCP.
+
+### Evaluation API (`memory_*`)
+
+Agents do not call Hindsight or mem0 tool names by default. They call a **stable tool set** on the `openmem` MCP server:
+
+| Tool | Intent |
+| --- | --- |
+| `memory_add` | Store a durable record |
+| `memory_search` / `memory_recall` | Retrieve |
+| `memory_reflect` / `memory_consolidate` | Synthesize (provider-dependent) |
+| `memory_forget` | Delete (scoped; confirm before bulk) |
+| `memory_export` / `memory_import` | Portability (PMM) |
+| `memory_provider_status` / `memory_project_scope` | Ops / bank resolution |
+
+The facade (`cli/mcp/memory-facade.mjs`) maps each call onto the active provider’s HTTP API. Native provider MCP (`install-mcp --provider-only`) remains available when you want vendor tools.
+
+### Context and banks
+
+**Bank** (or provider-equivalent `user_id` / dataset) is the scope key for a write or read. Default MCP scope is `openmem-mcp`. Project work should pass the resolved project bank so session noise does not land in the default.
+
+Resolution order: longest matching `index.yml` bank prefix → `~/.config/personal-memory/coding-agent.json` `mapPathToBank` → legacy Hindsight mapping → `coding-agent::<repo-directory-name>`. Details: [Coding-agent scopes](coding-agent-scopes.md).
+
+### Formats
+
+Import/export goes through a **format layer**, not a provider-specific dump. **PMM** (Personal Memory Manifest) is the first format; more formats plug into the same orchestrator. See [Portable memory (PMM)](memory-portability.md).
+
+### Session hooks and skill
+
+OpenMem installs one **skill** and **session hooks** (SessionStart / prompt / Stop) so agents recall and retain without remembering a vendor SDK. Optional: upstream Hindsight plugin via `--hindsight-plugin`.
+
+### Indexer
+
+A shared **indexer** watches git repos, Obsidian vaults, notes trees, and feed caches, then writes through the active provider’s sink. Sources are provider-agnostic; only the sink changes with `MEMORY_PROVIDER`.
+
+## What OpenMem is not
+
+- A replacement for Hindsight, mem0, or Cognee product UIs
+- A guarantee that memories move losslessly between backends (export/import is explicit)
+- A multi-provider fan-out in one process (one active provider; switch to change)
+
+## Next
+
+| Doc | Use when |
+| --- | --- |
+| [Quickstart](quickstart.md) | First install |
+| [MCP setup](mcp-setup.md) | Unified `memory_*` MCP, native mem0/Cognee, NotebookLM |
+| [Portable memory (PMM)](memory-portability.md) | Export/import between providers |
+| [Architecture](architecture.md) | Runtime data paths |
+| [Provider status](provider-status.md) | Which backends ship |
+| [mem0](mem0.md) · [Hindsight](hindsight.md) · [Cognee](cognee.md) | Provider operator guides |

package/docs/hindsight.md

@@ -0,0 +1,74 @@
+# Hindsight (default stack)
+
+**Hindsight** is the default memory provider: Docker Compose runs **LiteLLM**, the **Hindsight API**, and the shared **indexer** on loopback. Retain, recall, reflect, and consolidate go through Hindsight’s HTTP API; LiteLLM routes model calls to lemond, Ollama, or another OpenAI-compatible backend.
+
+Upstream project: [vectorize-io/hindsight](https://github.com/vectorize-io/hindsight).
+
+## What openmem runs
+
+| Service | Role |
+| --- | --- |
+| `litellm` | Proxy for Hindsight aliases `memory-retain`, `memory-reflect`, `memory-consolidate` |
+| `hindsight` | Memory API (default host port **8888**, `HINDSIGHT_PORT`) |
+| `indexer` | Git, Obsidian, markdown trees, feed cache → Hindsight banks via `MEMORY_SINK=hindsight` |
+
+Health check after setup:
+
+```sh
+curl -fsS http://127.0.0.1:8888/health
+```
+
+Control plane URL is printed by the wizard (separate from the API port).
+
+## Secrets and config (never in the repo)
+
+| Path | Purpose |
+| --- | --- |
+| `~/.config/hindsight/hindsight.env` | Hindsight API settings |
+| `~/.config/hindsight/litellm.yaml` | Optional full LiteLLM route override |
+| `~/.config/hindsight/index.yml` | Index sources ([index-sources.md](index-sources.md)) |
+| `~/.config/hindsight/wizard.conf` | Active provider and wizard choices |
+| `~/.hindsight/coding-agent.json` | **Real file on disk** — repo maps, opt-in paths, NotebookLM access |
+
+Shipped route templates live in `config/litellm.lemond.yaml` and `config/litellm.ollama.yaml`. Details: [Model configuration](model-configuration.md).
+
+## Coding agents and MCP
+
+**Default (all providers):** unified `memory_*` MCP plus in-repo session hooks:
+
+```sh
+npx @bcmyguest/openmem install-coding-agent --yes
+```
+
+**Optional:** also install the upstream Hindsight plugin hooks:
+
+```sh
+npx @bcmyguest/openmem install-coding-agent --yes --hindsight-plugin
+```
+
+MCP without session hooks:
+
+```sh
+npx @bcmyguest/openmem install-mcp --yes
+```
+
+Restart Cursor Agent after MCP changes.
+
+## Switch provider
+
+```sh
+npx @bcmyguest/openmem switch mem0 --yes
+# return: npx @bcmyguest/openmem switch hindsight --yes
+```
+
+Updates compose profile, env, and MCP wiring. Does **not** migrate memories between backends.
+
+## NotebookLM (opt-in)
+
+NotebookLM sync and MCP are provider-agnostic but were built for Hindsight-first workflows. See [docs/notebooklm.md](notebooklm.md).
+
+## See also
+
+- [Architecture](architecture.md) — indexing and MCP paths
+- [Model configuration](model-configuration.md) — LiteLLM aliases and backends
+- [Provider status](provider-status.md)

package/docs/index-sources-roadmap.md

@@ -0,0 +1,38 @@
+# Index sources — roadmap
+
+Today the indexer **ships** git repos, Obsidian vaults, markdown directory trees (`notes`, `papers`, `bookmarks` in `index.yml`), and a **feed cache** (`feeds` + `feed-sync`). Everything else below is **planned** or **bring-your-own-export** (point a folder at an export until a native connector exists).
+
+## Shipped (filesystem)
+
+| Kind | Config | Typical content |
+| --- | --- | --- |
+| Git repo | `roots`, `repos` | Code + tracked docs |
+| Obsidian vault | `vaults` | Notes with `.obsidian` |
+| Notes | `notes.paths` | Plain markdown trees |
+| Papers | `papers.paths` | Paper notes, Zotero md export, reading notes |
+| Bookmarks | `bookmarks.paths` | Saved links as markdown/HTML exports |
+| Feed cache | `feeds.out_dir` + `feed-sync` | RSS / OpenAlex markdown under a watched dir |
+
+## Planned connectors (API / sync jobs)
+
+| Source | Idea | Likely bank prefix | Notes |
+| --- | --- | --- | --- |
+| **Zotero** | API or periodic export dir | `papers::zotero` | Prefer export-to-`papers.paths` first |
+| **OpenAlex / OpenResearch** | Metadata + abstract pull | `openresearch::` | Read-only citeable snippets, not full PDF hosting |
+| **Semantic Scholar** | Library sync | `papers::s2` | Same pattern as OpenAlex |
+| **Readwise** | Reader / highlight export | `readwise::` | Official export → markdown dir |
+| **Hypothesis** | Annotation export | `hypothesis::` | Web annotation JSON → indexer adapter |
+| **Pinboard / Linkwarden / Karakeep** | Bookmark API or export | `bookmarks::` | Overlap with `bookmarks.paths` |
+| **RSS / arXiv feed** | `npx @bcmyguest/openmem feed-sync` | `feed::*` | Shipped; see [sources-feeds.md](sources-feeds.md) |
+| **Email** | mbox / selective forward | `mail::` | High privacy risk; opt-in only |
+| **Slack / Discord export** | JSON dump dirs | `chat::` | Enterprise exports only |
+| **NotebookLM** | Already optional Hindsight profile | `notebooklm::*` | Not a generic source type |
+
+## Design rules for new sources
+
+1. **One sink** — all sources retain into the active provider via the same indexer process.
+2. **Stable bank ids** — prefix + slug; override with `banks:` in `index.yml`.
+3. **Filesystem first** — if the tool can export to a directory, use `*.paths` before building OAuth sync.
+4. **No secrets in repo** — API keys in `~/.config/personal-memory/`, not index.yml.
+
+Contributing a connector: add a row here, then either a new `*.paths` kind (markdown) or an indexer sidecar script that writes retains through the existing sink.

package/docs/index-sources.md

@@ -0,0 +1,39 @@
+# Index sources
+
+The shared indexer reads `~/.config/hindsight/index.yml` (or `config/index.yml` defaults). One active provider sink receives retains from every source.
+
+| Source kind | Config | Detection | Default bank prefix |
+| --- | --- | --- | --- |
+| **Git repo** | `roots`, `repos` | Poll `git` fingerprint | `coding-agent::<repo>` |
+| **Obsidian vault** | `roots` + `vaults.auto`, `vaults.paths` | inotify after quiet period | `obsidian::<slug>` |
+| **Notes** | `notes.paths` | inotify | `notes::<slug>` |
+| **Papers** | `papers.paths` | inotify | `papers::<slug>` |
+| **Bookmarks** | `bookmarks.paths` | inotify (.md, .html, .txt) | `bookmarks::<slug>` |
+| **Feed cache** | `feeds` + `feed-sync` | inotify on `feeds.out_dir` | `feed::<slug>` |
+
+Markdown trees index `.md` (and Obsidian-style structure) the same way; use separate keys so banks stay readable in Hindsight/mem0 scopes.
+
+Override banks with the `banks:` map (longest prefix wins).
+
+```yaml
+notes:
+  paths:
+    - ~/Documents/plain-markdown
+papers:
+  paths:
+    - ~/Zotero/storage/export-md
+bookmarks:
+  paths:
+    - ~/Exports/linkwarden-markdown
+repos:
+  - ~/src/my-fork
+vaults:
+  paths:
+    - ~/Obsidian/Main
+```
+
+Exclude paths with `exclude:` globs. Tune `poll_seconds`, `vault_quiet_seconds`, and `relist_seconds` for watcher behavior.
+
+**Guides:** [Feed sync](sources-feeds.md) (RSS / OpenAlex → `feed::*`) · [Zotero exports](sources-zotero.md) · [Readwise exports](sources-readwise.md). Native API connectors and more kinds: [index-sources-roadmap.md](index-sources-roadmap.md).
+
+See [indexer sinks](indexer-sinks.md) for how banks map to mem0/Cognee/Hindsight scopes.

package/docs/indexer-sinks.md

@@ -0,0 +1,122 @@
+# Indexer sink drivers
+
+The shared indexer (`bin/hindsight-project-index`, watcher, compose `indexer` service) turns
+git repos, Obsidian vaults, markdown trees (`notes` / `papers` / `bookmarks`), and feed-cache
+folders into documents and writes them through a **sink**: a Python module
+under `indexer/sinks/<name>.py`. Provider manifests declare how compose and the wizard wire that
+sink; see [Pluggable memory providers](memory-providers.md) for manifests and wizard ownership.
+
+## Manifest fields vs runtime ids
+
+| Concept | Where it lives | Role |
+| --- | --- | --- |
+| **`indexer.sink`** | `providers/<id>/manifest.json` | Module name: `indexer/sinks/<sink>.py`. Becomes `MEMORY_SINK` in compose and `--sink` on the CLI. |
+| **`indexer.sinkDriver`** | same | Human-readable wire-dialect label only (for docs and status pages). **Does not** select a module. |
+| **`indexer.apiUrlEnv`** | same | Name of the env var the wizard documents for the provider API base URL on the host. |
+
+Registered providers today:
+
+| Provider | `sink` (module) | `sinkDriver` | `apiUrlEnv` |
+| --- | --- | --- | --- |
+| Hindsight (`supported`) | `hindsight` | `hindsight-rest-v1` | `HINDSIGHT_URL` |
+| mem0 | `mem0` | `mem0-rest` | `MEM0_API_URL` |
+| Cognee | `cognee` | `cognee-rest` | `COGNEE_API_URL` |
+
+Contract tests: `indexer/tests/sink_contract.py`, provider-specific tests under `indexer/tests/`.
+
+## Choosing a sink and API URL
+
+| Input | Meaning |
+| --- | --- |
+| `--sink NAME` | Sink module name. Default: `$MEMORY_SINK`, then `hindsight`. |
+| `MEMORY_SINK` | Set in compose fragments for mem0/Cognee (`providers/*/compose.yaml`). |
+| `--url URL` | Overrides all URL env vars for this run. |
+| `HINDSIGHT_URL` | API base for the **hindsight** sink only (default `http://127.0.0.1:8888`). |
+| `MEMORY_API_URL` | API base for **every non-Hindsight** sink (mem0, Cognee). Required for those sinks when `--url` is omitted. |
+
+Resolution order in the CLI: `--url` → `HINDSIGHT_URL` (hindsight sink) or `MEMORY_API_URL` (other
+sinks) → sink-specific default (Hindsight only). Credentials are read inside each sink’s `create()`
+from provider-specific env vars, never from indexed file paths.
+
+Provider secret files (mode `600`, outside the repo):
+
+| Provider | Typical file | Indexer-relevant keys |
+| --- | --- | --- |
+| mem0 | `~/.config/personal-memory/mem0.env` | `MEM0_API_KEY` → `X-API-Key` on API calls (not on `GET /docs` health) |
+| Cognee | `~/.config/personal-memory/cognee-indexer.env` | `COGNEE_API_KEY` when the server requires auth (default single-user image does not) |
+
+Stack `.env` holds host ports and derived URLs (for example `MEM0_INDEXER_URL` for the indexer
+container’s view of mem0). LLM keys for Cognee cognify live in `cognee.env`, not in the indexer env
+file — see [Cognee](cognee.md).
+
+## Indexer `bank` → provider namespace
+
+The indexer uses **`bank`** for the logical scope of a source (repo, vault, or `index.yml` override).
+Examples: `coding-agent::my-repo`, `obsidian::my-vault`, `notebooklm::work-notes` (when synced into
+Hindsight). **`doc_id`** is a stable per-document id (`project-index:v1:…`). Each sink maps `bank` /
+`doc_id` to its backend’s scoping model.
+
+### Hindsight (`hindsight-rest-v1`)
+
+| Indexer field | Hindsight API |
+| --- | --- |
+| `bank` | Bank id in the path: `/v1/default/banks/{bank}/…` (URL-encoded) |
+| `doc_id` | Carried in retain payload / document identity for updates and deletes |
+
+Behavior:
+
+- `prepare(bank)` ensures the bank exists, sets retain strategy `project-index`, disables auto-consolidation for indexed banks.
+- Retains are **synchronous** (`async: false`); no `flush()` hook.
+- Default URL when unset: `http://127.0.0.1:8888`. Compose sets `HINDSIGHT_URL: http://hindsight:8888` on the indexer service.
+
+NotebookLM sync writes into Hindsight banks `notebooklm::<slug>`; that path is separate from sink
+selection (still the hindsight sink when `MEMORY_PROVIDER=hindsight`).
+
+### mem0 (`mem0-rest`)
+
+| Indexer field | mem0 REST field |
+| --- | --- |
+| `bank` | `user_id` (via `scope_id()` — see below) |
+| `doc_id` | `run_id` (same `scope_id` rules) |
+| (fixed) | `agent_id` = `openmem-indexer` (provenance only; lookups use `user_id` + `run_id`) |
+
+**`scope_id`:** If the id has no whitespace, it is used as-is. Otherwise whitespace runs become `_`,
+and `~` plus the first 8 hex digits of SHA-256(original) are appended so ids that differ only in
+whitespace stay distinct (for example `obsidian::My Vault` → `obsidian::My_Vault~<8 hex>`).
+
+Documents are chunked (max 3000 characters per chunk, `infer: false`). Updates list existing memories
+for `(user_id, run_id)`, write new chunks, then delete old ids. There is no mem0 “bank” to create;
+`prepare` is a no-op.
+
+Compose: `MEMORY_SINK=mem0`, `MEMORY_API_URL=${MEM0_INDEXER_URL}` (wizard derives this from host
+`MEM0_API_URL`). Operator runbook: `providers/mem0/OPERATOR.md`, [mem0](mem0.md).
+
+### Cognee (`cognee-rest`)
+
+| Indexer field | Cognee concept |
+| --- | --- |
+| `bank` | **Dataset name** derived from `bank` (not the raw string) |
+| `doc_id` | Stored in upload `external_metadata` as `doc_id` (plus `bank`); used to replace/delete rows |
+
+**Dataset name:** `<slug>_<8 hex of sha256(bank)>` where `slug` is `bank` lowercased, non-`[a-z0-9]`
+collapsed to `_`, trimmed, max 40 chars (empty → `bank`). Two banks that slug identically still differ
+by hash suffix.
+
+Ingestion is **asynchronous**: `retain`/`delete` call REST add/delete; the sink implements **`flush(bank)`**
+to trigger `POST /api/v1/cognify` for that dataset after each changed batch (and retries pending flushes
+from indexer state). Search lags until cognify completes.
+
+Compose: `MEMORY_SINK=cognee`, `MEMORY_API_URL: http://cognee:8000`, optional
+`cognee-indexer.env` for `COGNEE_API_KEY`. Operator notes: [Cognee](cognee.md).
+
+## Optional `flush()` (Cognee)
+
+Sinks with `capabilities.async_ingest=True` may define `flush(bank)`. The indexer calls it on
+`--apply` after retains/deletes for that bank, and persists `::flush-pending` in state when a flush
+fails. Hindsight has no flush. Details: [Pluggable memory providers — Optional flush](memory-providers.md#optional-flush).
+
+## Related
+
+- [Indexing](indexing.md) — sources, `index.yml`, watcher timings
+- [Architecture](architecture.md) — unified MCP mapping to the same backends
+- [MCP setup](mcp-setup.md) — Cursor servers vs indexer sinks

package/docs/indexing.md

@@ -0,0 +1,57 @@
+# Indexing
+
+The indexer indexes each source once at startup, then keeps it current:
+
+- **Git repositories:** the outermost repositories under each `roots` entry, plus `repos`. The indexer polls each one every `poll_seconds` (default 60). A poll fingerprints `HEAD` and the tracked-file status (`git status --porcelain -uno`). After one poll with an unchanged fingerprint, the repository is re-indexed. Edits to an already-modified tracked file change no fingerprint until `HEAD` or the status changes. Repositories with a persistent error re-run on each poll. Banks default to `coding-agent::<repo>` unless `banks` maps the path.
+- **Obsidian vaults:** any folder containing `.obsidian/` within a root's depth, plus `vaults.paths`. Each uses the bank `obsidian::<slug>` unless `banks` maps it. Changes are watched with inotify. A vault is re-indexed after `vault_quiet_seconds` (default 30) without changes. Dot-prefixed paths are ignored.
+- **Markdown trees:** `notes.paths`, `papers.paths`, and `bookmarks.paths` (plain folders without `.obsidian/`). Watched with inotify; banks default to `notes::`, `papers::`, and `bookmarks::` slugs. Bookmarks also index `.html` and `.txt` exports. Use `papers.paths` for Zotero markdown export dirs; see [sources-zotero.md](sources-zotero.md) and [sources-readwise.md](sources-readwise.md).
+- **Feed cache:** when `feeds.rss` or `feeds.openalex_queries` is set in `index.yml`, run `npx @bcmyguest/openmem feed-sync` to refresh markdown under `feeds.out_dir`; the indexer watches that directory as kind `feed` (`feed::<slug>` banks). See [sources-feeds.md](sources-feeds.md).
+- **Source discovery:** sources are re-listed every `relist_seconds` (default 600), so new repos, vaults, and configured paths are picked up without a restart.
+
+Operator overview: [index-sources.md](index-sources.md).
+- Code files get a tree-sitter outline (signatures, doc comments, imports) instead of a CodeGraph run. Parsing runs in-process; grammars are prefetched into the image at `/opt/tree-sitter-grammars`, so indexing needs no network. Supported languages: Python, JavaScript, TypeScript, TSX, Go, Rust, Java, C, C++, C#, Ruby, Lua, Bash, PHP. Known gaps: Go type declarations and TypeScript type aliases are not captured, and files with syntax errors give partial outlines. Outlines are re-retained once after this change, and the old CodeGraph-based code documents are deleted as stale.
+- `optInPaths` filters sources. A missing file means no filter; an empty list means nothing is indexed.
+- Automatic consolidation is disabled for the whole indexed bank, not only for indexed documents.
+- Document IDs include the absolute source path. The first run after this change re-retains every document, which costs extraction time on the model server.
+- Concurrent `--apply` runs (the watcher and a manual run) serialize through a lock on `state.json.lock`.
+
+Set `INDEXER_DRY_RUN=1` to log the commands the watcher would run without executing them. The environment variables `POLL_SECONDS`, `VAULT_QUIET_SECONDS`, and `RELIST_SECONDS` override the timings in `index.yml`.
+
+### Source config: `index.yml`
+
+`config/index.yml` in the repository is the default. The indexer reads the first of these that exists:
+
+1. `--config PATH` (manual runs and `--list`)
+2. `$HINDSIGHT_INDEX_CONFIG`
+3. `~/.config/hindsight/index.yml` (per machine; the container sees it through `scripts/setup.mjs`'s mount)
+4. `config/index.yml` (baked into the image at `/opt/config/index.yml`; rebuild after changing it)
+
+A file found this way replaces the default entirely. Keys it omits take the built-in values, which match `config/index.yml`. Unknown keys and wrong types fail at startup with the file, key, and reason. `~` expands; a bare `~` in YAML is null, which means home under `roots[].path`.
+
+| Key | Meaning |
+| --- | --- |
+| `roots` | `[{path, depth}]`. Repositories are the outermost git repositories found within `depth` levels (default 3). |
+| `repos` | Extra repositories, outside the roots or not. Each must be a git repository. |
+| `vaults.auto` | `true` (default) finds any folder with `.obsidian/` within a root's depth. |
+| `vaults.paths` | Extra vault folders. |
+| `notes.paths`, `papers.paths`, `bookmarks.paths` | Markdown (and bookmark export) directories; see [index-sources.md](index-sources.md). |
+| `feeds.out_dir`, `feeds.rss`, `feeds.openalex_queries` | Feed sync output; requires `npx @bcmyguest/openmem feed-sync`. |
+| `exclude` | Glob patterns on absolute paths. A matching source is skipped. |
+| `banks` | Absolute path prefix to bank id; the longest prefix wins. Applies to all source kinds. Unmatched repos fall back to `~/.config/personal-memory/coding-agent.json` `mapPathToBank`, then legacy `~/.hindsight/coding-agent.json`, then `coding-agent::<repository-name>`. |
+| `max_file_kb` | Largest indexed file, in kilobytes (1 KB = 1000 bytes). Default 128. |
+| `skip_dirs` | Directory names not descended into. Dot-prefixed directories are always skipped. |
+| `poll_seconds`, `vault_quiet_seconds`, `relist_seconds` | Watcher timings. |
+
+`optInPaths` in `~/.hindsight/coding-agent.json` still filters sources after discovery.
+
+Check the result with `--list` before a first `--apply`. Excluded sources are reported on stderr.
+
+Manual runs. The script defaults to a dry run; `--apply` writes:
+
+```sh
+docker compose run --rm --no-deps indexer hindsight-project-index --list
+docker compose run --rm --no-deps indexer hindsight-project-index --repo PATH --apply
+docker compose run --rm --no-deps indexer hindsight-project-index --vault PATH --apply
+```
+
+`--no-deps` avoids starting the `hindsight` service, which the indexer depends on. `PATH` is the host path, which the container also sees.

package/docs/manual-stack-setup.md

@@ -0,0 +1,74 @@
+# Manual stack setup
+
+Use this when you are not running the [setup wizard](setup-wizard.md). Commands assume your **stack root** is the current directory (git checkout or `PM_STACK_ROOT`).
+
+## Hindsight: minimal path
+
+```sh
+mkdir -p ~/.config/hindsight
+cp config/hindsight.env.example ~/.config/hindsight/hindsight.env
+chmod 600 ~/.config/hindsight/hindsight.env
+node scripts/setup.mjs
+node scripts/setup.mjs --migrate-env   # only if hindsight.env still points at lemond/Ollama directly
+docker compose up -d
+curl -fsS http://127.0.0.1:8888/health
+```
+
+`scripts/setup.mjs` must run before any `docker compose` command (including `logs` and `down`). It writes stack `.env`, `compose.override.yaml`, and related files.
+
+## Compose profiles
+
+| Profile | Services |
+| --- | --- |
+| `hindsight` | LiteLLM + Hindsight API (default after wizard) |
+| `ollama` | In-compose Ollama for LiteLLM |
+| `notebooklm` | NotebookLM → Hindsight sync |
+
+Plain `docker compose up` without profiles starts only the indexer unless `.env` sets `COMPOSE_PROFILES`. The wizard passes every needed `--profile` on each compose invocation.
+
+NotebookLM is started **after** the smoke test so sync does not compete with smoke on the LLM.
+
+## Indexer sinks
+
+| Variable | Meaning |
+| --- | --- |
+| `MEMORY_SINK` / `--sink` | `hindsight` (default), `mem0`, `cognee`, … |
+| `HINDSIGHT_URL` | Hindsight API from indexer container |
+| `MEMORY_API_URL` | Non-Hindsight memory API |
+
+Entrypoint: `bin/hindsight-project-index` via `indexer/sinks/`.
+
+## What setup.mjs generates
+
+| Output | Role |
+| --- | --- |
+| `.env` | `HOST_UID`, `HOST_GID`, `LITELLM_CONFIG`, `COMPOSE_PROFILES`, … |
+| `compose.override.yaml` | Masks `~/.ssh`, `~/.gnupg`, `~/.config/hindsight`, agent credential dirs from the indexer; optional mount for `index.yml` |
+| `~/.config/hindsight/litellm.env` | LiteLLM master key (mode 600) if missing |
+
+The indexer runs as your UID (`user: "${HOST_UID}:${HOST_GID}"`). Writable state is only the named volume `indexer-state` (mounted at `$HOME/.cache/hindsight-project-index` inside the container); `$HOME` is read-only.
+
+Rerun `node scripts/setup.mjs` after UID changes or when new secret directories appear (masks are generated only for paths that exist).
+
+## Seed indexer state volume
+
+When moving from a host `state.json` to the volume:
+
+```bash
+docker compose stop indexer
+docker compose run --rm --no-deps indexer-state-init
+docker run --rm --user "$HOST_UID:$HOST_GID" \
+  -v "$HOME/.cache/hindsight-project-index:/src:ro" \
+  -v personal-memory_indexer-state:/dst openmem-indexer:local \
+  cp /src/state.json /dst/
+```
+
+Adjust the volume name with `docker volume ls` if the compose project name differs.
+
+## Bench overlay (optional)
+
+`compose.bench.yaml` also publishes Hindsight on the Docker bridge gateway for sandboxes (see the file for the bind address). Keep `compose.override.yaml` in the `-f` list when you add `-f compose.bench.yaml`.
+
+```bash
+docker compose -f compose.yaml -f compose.override.yaml -f compose.bench.yaml up -d hindsight
+```

package/docs/mcp-install-guide.md

@@ -0,0 +1,4 @@
+# MCP install guide
+
+This page moved to **[MCP setup](mcp-setup.md)** (published on the docs site).
+

package/docs/mcp-setup.md

@@ -0,0 +1,101 @@
+# MCP setup
+
+OpenMem registers **stdio MCP servers** in `~/.cursor/mcp.json` (and wires session hooks via `install-coding-agent`). Day-to-day you want the **unified** `openmem` server (`memory_*` tools). Provider-native and NotebookLM servers are optional add-ons.
+
+For why this exists, see [Concepts](concepts.md). Install path: [Quickstart § Wire your coding agent](quickstart.md#3-wire-your-coding-agent-optional).
+
+## Quick path (recommended)
+
+```sh
+npx @bcmyguest/openmem install-coding-agent --yes
+```
+
+That installs:
+
+1. Unified MCP (`openmem`) — `memory_add`, `memory_search`, `memory_recall`, `memory_reflect`, `memory_consolidate`, `memory_forget`, `memory_export` / `memory_import`, ops tools
+2. OpenMem skill + session hooks (Claude Code, Codex, OpenCode, Cursor CLI)
+
+MCP only (no session hooks):
+
+```sh
+npx @bcmyguest/openmem install-mcp --yes
+```
+
+Restart **Cursor Agent** (or your agent’s MCP host) after changes.
+
+## Server families
+
+| Server key | How to install | Tools | Backend |
+| --- | --- | --- | --- |
+| **`openmem`** | Default `install-mcp` / `install-coding-agent` | Stable `memory_*` | Active provider (`MEMORY_PROVIDER`): Hindsight, mem0, or Cognee |
+| **Provider-native** (`mem0`, `cognee`) | `install-mcp --provider-only` | Upstream / repo-specific names | Same HTTP API, different tool surface |
+| **`notebooklm`** | When NotebookLM sync is on, or `--notebooklm` | `notebooklm_list_banks`, `notebooklm_search` | Search over synced notebooks on the **active** provider — not a memory engine |
+
+Hindsight’s optional in-editor plugin (`--hindsight-plugin`) is **not** an MCP entry; it complements unified MCP on Hindsight stacks. See [Hindsight](hindsight.md).
+
+## When to use which
+
+### Default: unified `memory_*`
+
+Use when you want one tool contract that survives `npx @bcmyguest/openmem switch <provider>` without rewriting prompts.
+
+```sh
+npx @bcmyguest/openmem install-mcp --yes
+npx @bcmyguest/openmem install-mcp --yes --notebooklm   # also force NotebookLM MCP
+```
+
+### Provider-native (optional)
+
+Use when you already have prompts for upstream tool names:
+
+```sh
+MEMORY_PROVIDER=mem0 npx @bcmyguest/openmem install-mcp --yes --provider-only
+npx @bcmyguest/openmem install-mcp --yes --provider-only   # Cognee when active
+```
+
+- **mem0:** `cli/mem0-mcp.mjs` → `@yeyuan98/mem0-mcp` (URL + path to `mem0.env` in JSON — never the key). Details: [mem0](mem0.md).
+- **Cognee:** `cli/cognee-mcp.mjs` (`cognee_health`, `cognee_search`; `COGNEE_API_URL` only). Details: [Cognee](cognee.md).
+
+`install-mcp --provider-only` still merges unified `openmem` unless you also pass `--no-unified-mcp`. Both can coexist in `mcp.json`.
+
+### NotebookLM only
+
+```sh
+npx @bcmyguest/openmem install-mcp --yes --notebooklm-only
+```
+
+Mutually exclusive with `--provider-only`. Read-only search over synced notebooks — not a substitute for project indexing or `memory_*`.
+
+## Flags
+
+```sh
+npx @bcmyguest/openmem install-mcp --yes
+npx @bcmyguest/openmem install-mcp --yes --notebooklm
+npx @bcmyguest/openmem install-mcp --yes --notebooklm-only
+MEMORY_PROVIDER=mem0 npx @bcmyguest/openmem install-mcp --yes --provider-only
+npx @bcmyguest/openmem install-mcp --yes --no-unified-mcp
+npx @bcmyguest/openmem install-mcp --dry-run
+```
+
+| Flag | Effect |
+| --- | --- |
+| `--yes` | No confirmation prompt |
+| `--dry-run` | Print plan; do not write configs |
+| `--notebooklm` | Enable NotebookLM agent access + merge `notebooklm` MCP |
+| `--notebooklm-only` | Only NotebookLM step |
+| `--provider-only` | Native mem0/Cognee MCP for active provider; skip NotebookLM |
+| `--no-unified-mcp` | Do not merge the `openmem` server |
+
+**Provider resolution:** `MEMORY_PROVIDER` → `~/.config/hindsight/wizard.conf` → `hindsight`.
+
+**Merge behavior:** Existing `hindsight`, `codegraph`, and `notebooklm` entries are preserved; mem0/Cognee entries are updated for the active provider.
+
+## Portability tools
+
+On the unified server, `memory_export` / `memory_import` speak **PMM** (Personal Memory Manifest). See [Portable memory (PMM)](memory-portability.md).
+
+## Related
+
+- [mem0](mem0.md) · [Cognee](cognee.md) · [Hindsight](hindsight.md)
+- [Architecture](architecture.md) — how `memory_*` maps to provider HTTP
+- [Quickstart](quickstart.md)