@bcmyguest/openmem 0.0.0-stage → 0.6.3

package/docs/mem0.md

@@ -0,0 +1,103 @@
+# mem0
+
+**mem0** is a supported OpenMem provider. You run a [self-hosted mem0 REST server](https://docs.mem0.ai/open-source/features/rest-api); OpenMem points the shared indexer and unified `memory_*` MCP at it. One active provider at a time — same agent surface as Hindsight or Cognee.
+
+This is **not** the hosted platform at `https://api.mem0.ai` (different API shape). OpenMem talks to the open-source REST server only (no `/v1/` prefix).
+
+Upstream project: [mem0ai/mem0](https://github.com/mem0ai/mem0). Stack details for contributors: `providers/mem0/OPERATOR.md`.
+
+## What you run vs what OpenMem runs
+
+| You | OpenMem |
+| --- | --- |
+| mem0 REST server (JWT, Postgres/pgvector, embedder keys) | `MEMORY_PROVIDER=mem0`, `MEMORY_SINK=mem0` |
+| API key in `~/.config/personal-memory/mem0.env` (mode **600**) | `MEM0_API_URL` / `MEM0_INDEXER_URL` in stack env |
+| Bind so the indexer container can reach the server | Wizard smoke: docs, retain, search, delete-by-id |
+| | Unified MCP + optional native mem0 MCP |
+
+Compose profile `mem0` adds env + `extra_hosts` on the shared **indexer** only — there is no mem0 container in this repo yet.
+
+## First-time setup
+
+1. Start your mem0 REST server (default OpenMem port expectation: **18888**).
+2. Put the API key in the env file (skip if `AUTH_DISABLED=true` on the server for local dev only):
+
+```text
+~/.config/personal-memory/mem0.env
+MEM0_API_KEY=…
+```
+
+Never put `MEM0_API_KEY` in stack `.env`, `wizard.conf`, `config.toml`, or `mcp.json`.
+
+3. Select mem0 in the wizard, or switch later:
+
+```sh
+npx @bcmyguest/openmem              # choose mem0 when prompted
+# or:
+npx @bcmyguest/openmem switch mem0 --yes
+```
+
+4. Confirm:
+
+```sh
+npx @bcmyguest/openmem doctor
+curl -fsS "$MEM0_API_URL/docs"   # URL from wizard / stack .env
+```
+
+## URLs
+
+| Variable | Meaning |
+| --- | --- |
+| `MEM0_API_URL` | Host view (what you curl), e.g. `http://127.0.0.1:18888` |
+| `MEM0_INDEXER_URL` | Derived for the indexer container — loopback hosts are rewritten to the Docker bridge gateway. Do not set this to `127.0.0.1` by hand inside the container. |
+
+**Linux:** a process bound only to loopback is often unreachable from the indexer container. Bind mem0 to `0.0.0.0` (or a real hostname). Prefer `https://` when the server is not on loopback — `X-API-Key` is cleartext on plain HTTP.
+
+Search fails if the server’s embedder is misconfigured; smoke stores with `infer: false` and looks for that fact.
+
+## Banks / scopes
+
+mem0 scopes map to **`user_id`**. OpenMem’s MCP `bank` argument and indexer banks become that `user_id`. Default unified MCP scope is `openmem-mcp` unless tools pass `bank`. Project work should pass the resolved project bank so session noise does not land in the default. See [Concepts — Context and banks](concepts.md#context-and-banks).
+
+## Coding agents and MCP
+
+**Preferred:** unified OpenMem MCP (`memory_*`) — same tools as Hindsight/Cognee:
+
+```sh
+npx @bcmyguest/openmem install-coding-agent --yes
+# or MCP only:
+npx @bcmyguest/openmem install-mcp --yes
+```
+
+**Optional native mem0 tools** (upstream `@yeyuan98/mem0-mcp` names, not `memory_*`):
+
+```sh
+MEMORY_PROVIDER=mem0 npx @bcmyguest/openmem install-mcp --yes --provider-only
+```
+
+That registers `cli/mem0-mcp.mjs` with `MEM0_API_URL` / `MEM0_BASE_URL` and the **path** to `mem0.env` — never the key value in `mcp.json`. Mem0’s cloud MCP (`https://mcp.mem0.ai`) is **not** for self-hosted REST.
+
+Restart Cursor Agent after MCP changes. Full flag matrix: [MCP setup](mcp-setup.md).
+
+There is no Hindsight-style coding-agent plugin for mem0; session hooks + unified MCP cover day-to-day agent memory.
+
+## Portability
+
+Export/import uses **PMM** via `memory_export` / `memory_import` on the unified MCP. Switching providers does **not** migrate data by itself — export, switch, import (or re-index). See [Portable memory (PMM)](memory-portability.md).
+
+## Switch provider
+
+```sh
+npx @bcmyguest/openmem switch mem0 --yes
+# later:
+npx @bcmyguest/openmem switch hindsight --yes
+```
+
+Updates active provider config, restarts the stack (unless `--no-stack`), refreshes MCP (unless `--no-mcp`). Does **not** migrate memories.
+
+## Related
+
+- [Concepts](concepts.md) — interface layer
+- [MCP setup](mcp-setup.md) — unified vs provider-native
+- [Portable memory (PMM)](memory-portability.md)
+- [Architecture](architecture.md) · [Provider status](provider-status.md)

package/docs/memory-portability.md

@@ -0,0 +1,73 @@
+# Portable memory (PMM)
+
+OpenMem’s **portability layer** moves memories between providers without baking in a vendor dump format. **PMM** (Personal Memory Manifest) is the first interchange format: export what a backend stores, import into another (or the same) provider, and keep a portable copy for backup or “give me my data” requests.
+
+`npx @bcmyguest/openmem switch` rewires the stack only — it does **not** migrate content. Use export → switch → import (or re-index) when you change backends.
+
+Status: **`memory_export` / `memory_import`** on the unified MCP are **beta**. PMM v1 is stable; adapters live under `cli/mcp/export/` (orchestrator: `cli/mcp/portability.mjs`). MCP install: [MCP setup](mcp-setup.md).
+
+## PMM format (version 1)
+
+JSON object, or JSONL with one header line followed by one normalized record per line:
+
+| Field | Type | Meaning |
+| --- | --- | --- |
+| `pmm_version` | `1` | Schema version |
+| `exported_at` | ISO-8601 string | When the export was taken |
+| `source_provider` | string | `hindsight` \| `mem0` \| `cognee` |
+| `scope` | object | Provider-specific filter (bank, `user_id`, datasets, …) |
+| `records` | array | Normalized memory records (see below) |
+
+Each **record**:
+
+| Field | Type | Meaning |
+| --- | --- | --- |
+| `id` | string \| null | Provider id when known |
+| `content` | string | Primary text |
+| `metadata` | object | Optional tags, timestamps, provider payload |
+| `scope_key` | string | Bank name, mem0 `user_id`, Cognee dataset, etc. |
+
+Import accepts PMM with `merge_strategy`:
+
+| Strategy | Behavior |
+| --- | --- |
+| `skip_duplicates` (default) | For mem0, skip records whose id or content fingerprint already exists in the same bank |
+| `overwrite` | Re-import all records (may duplicate on backends without idempotent upsert) |
+
+**Cross-provider import** (e.g. PMM from mem0 into Hindsight) is best-effort: we map `content` + `scope_key` into the active provider’s add API; embeddings and graph structure are **not** preserved.
+
+The MCP export currently collects all records in memory before returning them, including with `format: "jsonl"`. Use a bank filter for large stores; the MCP response is not a file stream. Hindsight export reads the memory-list endpoint and does not include knowledge pages, bank configuration, or source documents. Confirm migrated data before switching providers.
+
+## Implementation (orchestrator + adapters)
+
+openmem stays a **meta orchestrator**: PMM shape and import live in `cli/mcp/portability.mjs`; each backend implements **list/map** in `cli/mcp/export/` (`hindsight.mjs`, `mem0.mjs`, `cognee.mjs`). New providers add an adapter and one branch in `cli/mcp/export/index.mjs` — no changes to the MCP tool names agents call.
+
+## Unified MCP tools
+
+On the `openmem` MCP server ([MCP setup](mcp-setup.md), [Architecture](architecture.md)):
+
+```text
+memory_export   { format?: "json"|"jsonl", bank?: string, user_id?: string }
+memory_import   { data: string, merge_strategy?: "skip_duplicates"|"overwrite" }
+```
+
+Example flow (Cursor Agent / any host with the unified server):
+
+1. `memory_export` with `format: "json"` and optional `bank`
+2. `npx @bcmyguest/openmem switch <other> --yes`
+3. `memory_import` with the exported JSON and `merge_strategy: "skip_duplicates"`
+
+Restart Cursor Agent after `npx @bcmyguest/openmem install-mcp --yes`.
+
+## Provider behavior
+
+| Provider | Export | Import |
+| --- | --- | --- |
+| **mem0** | `GET /memories` with `user_id` from `bank` arg (API key from env file) | `POST /memories` per record (`infer: false`) |
+| **Hindsight** | `GET /v1/default/banks` then paginated `GET …/memories/list` per bank (`bank` arg = one bank; omit = all banks) | `POST …/banks/{bank}/memories` per record |
+| **Cognee** | Resolve indexer dataset name from `bank`, `GET /api/v1/datasets`, paginate `GET …/data` (label/name as content; not full file bytes) | `POST /api/v1/add` per record when dataset in scope |
+
+## Related
+
+- [Architecture — provider interface](architecture.md#how-openmem-interfaces-with-providers)
+- [mem0](mem0.md) · [Cognee](cognee.md)

package/docs/memory-providers.md

@@ -0,0 +1,246 @@
+# Pluggable memory providers
+
+This repo ships **one** supported backend: [Hindsight](https://github.com/vectorize-io/hindsight)
+(API + retain/reflect/consolidate, LiteLLM in front of local models). **mem0** and **Cognee** are
+**experimental** plugins (indexer sink, compose profile, wizard hooks, stub tests). Wave 3 hardened
+operator DX (coding-agent steps, secrets, Ollama wiring); live Cognee cognify+search with a small
+local model is not yet a promotion bar — see `docs/WAVE3-orchestrator.md`. Manifests under
+`providers/` show `experimental` vs `supported`.
+
+## Meta setup wizard (what this is — and is not)
+
+`npm run setup` (checkout), `npx @bcmyguest/openmem setup`, or `./setup` is a **meta** wizard: it
+orchestrates **machine-local plumbing** that is the same idea across providers (Docker project,
+ports, indexer sources, smoke “is the API alive?”, coding-agent config file, compose up/down). It does
+**not** replace each provider’s **native** settings UI, CLI, or config format.
+
+| This wizard owns (cross-cutting) | Provider still owns (native) |
+| --- | --- |
+| Pick provider + stack action (configure / up / down) | API keys, org/project ids, mem0 graph settings |
+| Host ports, `HOST_UID`/`HOST_GID`, compose project name | Hindsight retain/reflect models beyond our LiteLLM routes |
+| Shared `index.yml` (what to index) | Provider-specific memory schema, filters, user ids |
+| Wiring Hindsight → LiteLLM proxy (when provider is Hindsight) | mem0 dashboard, cloud vs self-hosted choice |
+| Health/smoke probes declared in manifest | Official provider docs, upgrades, migrations |
+
+When a provider needs extra fields, the manifest should grow **provider-specific wizard steps** (or
+point to “run `mem0 …` / edit `~/.mem0/…`”) — not cram mem0 into Hindsight’s LiteLLM screens.
+
+### Similar patterns elsewhere
+
+These are the usual analogues people mean by “meta setup”:
+
+- **[mise](https://mise.jdx.dev/) / [asdf](https://asdf-vm.com/)** — one CLI to install and select
+  runtimes; each plugin wraps the tool’s own install rules. mise does not replace `.npmrc` or
+  `Cargo.toml`.
+- **[Chezmoi](https://www.chezmoi.io/)** — applies dotfiles from a repo; Vim, Git, and SSH keep their
+  normal config files on disk.
+- **[Supabase CLI](https://supabase.com/docs/guides/cli)** — `supabase start` brings up Postgres,
+  Auth, Studio; each service still has its own env and tuning.
+- **LiteLLM (in this stack)** — meta proxy in front of lemond/Ollama; model servers keep their own
+  configs. `./setup`’s “model backend” step is really “how LiteLLM reaches your LLM,” not a full
+  lemond replacement.
+- **[Home Assistant](https://www.home-assistant.io/)** — core + integrations; Philips Hue still configured
+  in the Hue app, HA only links it.
+- **`npx skills`-style installers** — discover and install artifacts; each skill or package owns its
+  content afterward.
+
+openmem sits in that category: **one front door** for “memory on this machine for coding agents,”
+not a reimplementation of mem0 or Hindsight settings.
+
+Declarative config is **[mise-like](openmem-config.md)**: `~/.config/personal-memory/config.toml`
+plus `./pm` tasks — same *shape* as mise (`[tasks]`, merged config files), different domain.
+
+Provider-specific setup runs via **[wizard hooks](provider-wizard-hooks.md)** (`providers/*/wizard.mjs`)
+after meta steps — like asdf plugin install scripts, not reimplemented npm/mem0 CLIs.
+
+## Layers (what varies per provider)
+
+| Layer | Hindsight today | Should become |
+| --- | --- | --- |
+| **Compose stack** | `litellm`, `hindsight`, `indexer`, optional `notebooklm` | Provider manifest lists services / profiles; shared `indexer` where possible |
+| **Model routing** | LiteLLM + `config/litellm.*.yaml` | Hindsight-specific; other providers may use their own LLM config or none |
+| **Indexer sink** | `bin/hindsight-project-index` → REST `/v1/default/banks/{bank}/memories` | Python **sink** interface; one implementation per provider |
+| **Setup wizard** | Node wizard: preflight, env, smoke retain/recall | Provider manifest: health URL, smoke steps, env file names |
+| **Coding-agent UX** | `@vectorize-io/hindsight-coding-agents` + `coding-agent.json` | Per-provider plugin package or generic adapter |
+
+NotebookLM sync is **orthogonal** to which provider you picked: enable the `notebooklm` profile and the sync container uses the same `MEMORY_SINK` as the indexer (Hindsight, mem0, or Cognee).
+
+Coding-agent session memory uses the **same** in-repo hooks and unified MCP for every provider; optional `--hindsight-plugin` adds the upstream Hindsight plugin on Hindsight stacks only.
+
+## Provider manifest
+
+Each provider lives under `providers/<id>/manifest.json`. The wizard loads manifests via
+`wizard/lib/providers/registry.mjs`. Fields are intentionally small; expand only when a second
+provider needs them.
+
+- **`status`**: `supported` | `experimental` | `planned`
+- **`compose`**: which services belong to this provider, health check, default host port env var
+- **`indexer`**: env var for API base URL, `sink`, and `sinkDriver` (see the table below)
+- **`codingAgent`**: optional npx package and config template path
+- **`wizard`**: smoke test bank name and HTTP paths (templates may include `{bank}`)
+
+| `indexer` field | Meaning | Required |
+| --- | --- | --- |
+| `apiUrlEnv` | Env var that holds the provider's API base URL for the wizard | Per provider |
+| `sink` | The `MEMORY_SINK` value, which equals the `indexer/sinks/<sink>.py` module name. Must match `^[a-z][a-z0-9_]*$` (the same rule as `indexer/sinks/__init__.py`). | `supported` and `experimental`; optional but validated for `planned` |
+| `sinkDriver` | Human-readable wire-dialect label (for example `hindsight-rest-v1`). It names no module and drives no dispatch. | `supported` and `experimental` |
+
+`scripts/tests/test_provider_manifests.mjs` checks that every `supported` or `experimental` manifest
+with an `indexer.sink` has a matching `indexer/sinks/<sink>.py`.
+
+## Selecting a sink
+
+The indexer picks its provider with `--sink NAME`, falling back to the `MEMORY_SINK` environment
+variable, then `hindsight`. Compose fragments for alternate providers set both `MEMORY_SINK` and
+`MEMORY_API_URL` on the `indexer` service from stack `.env` values.
+
+| Variable | Used by | Meaning |
+| --- | --- | --- |
+| `MEMORY_SINK` | indexer | Sink module name (`indexer.sink` in the manifest). Default `hindsight`. |
+| `MEMORY_API_URL` | indexer, every non-Hindsight sink | Generic API base URL of the provider. |
+| `HINDSIGHT_URL` | indexer, Hindsight sink only | API base URL for Hindsight (default `http://127.0.0.1:8888`). |
+
+`--url` overrides either URL variable. A sink reads its credentials from its own environment
+variables inside `create`, never from the document path.
+
+Provider secrets (for example `MEM0_API_KEY`, `COGNEE_API_KEY`, `LLM_API_KEY`) live in
+`~/.config/personal-memory/<id>.env`, outside the repo, mode 600. A provider's compose fragment loads
+that file through an `env_file` entry with `required: false`, so a stack without the file still
+starts. Secrets do not belong in the stack `.env` or in `wizard.conf`, and the wizard never prints
+them.
+
+### Optional `flush()`
+
+A sink for a provider that ingests asynchronously (`capabilities.async_ingest=True`, for example
+Cognee, where `cognify` builds the graph after `add`) may define `flush(bank: str) -> None`. The
+indexer calls it once per bank after that bank's batch of retains and deletes, so the provider starts
+processing once per batch rather than once per document.
+
+- The call happens only with `--apply`. The indexer calls `flush(bank)` with an explicit bank when the
+  batch retained or deleted at least one document for that bank, **or** when a previous flush for that
+  bank is still pending. A dry run never flushes, and `flush(None)` is never called.
+- A sink must treat `flush(bank)` as "process this bank now", even when its own in-memory bookkeeping
+  says the bank is clean: the indexer may call it from a fresh process in which that bookkeeping is gone.
+- Pending flushes are persisted. Per-document progress is saved to the indexer state file as it goes, so
+  a failed flush (or a run that died after some retains) would otherwise never be retried. For a sink
+  with a `flush`, the bank is recorded under the state key `::flush-pending` (a sorted list of banks)
+  before the first write of a batch and removed after `flush` succeeds. Every `--apply` run flushes the
+  banks it handles that are still pending, even when nothing changed. A bank whose source is no longer
+  indexed is left alone. Sinks without `flush` (Hindsight) never get the key, so their state file is
+  unchanged.
+- `flush` is not part of the `Sink` protocol, so sinks without it (Hindsight among them) still satisfy
+  `isinstance(sink, Sink)` and see no change in behavior. Callers use `sinks.flush_sink(sink, bank)`,
+  which does nothing when the sink has no callable `flush`.
+- A flush error is handled like a retain error: the source is reported as failed, the other sources
+  still run, and the pending marker stays so the next run flushes again.
+- The watcher (`indexer/watch.py`) spawns this same CLI once per source, so flush runs there too. In
+  the indexer container the sink and URL come from `MEMORY_SINK` and `MEMORY_API_URL`.
+
+Default provider id: **`hindsight`**. Change it in **interactive `./setup`** (Clack “Memory provider”,
+defaults to `config.toml` and `wizard.conf`), or set `MEMORY_PROVIDER` for `--yes`. The wizard records
+the choice in `wizard.conf`. Stack **up** / **down** are wizard actions (`--up`, `--down`, or Clack
+first step); default action is full **configure and start**.
+
+### Changing provider later
+
+Prefer **`npx @bcmyguest/openmem switch <id> --yes`** (checkout: `npx . switch <id> --yes`) over a full
+interactive setup re-run when you only need to change backends:
+
+- Updates `MEMORY_PROVIDER` in `~/.config/hindsight/wizard.conf`
+- Updates `[memory].provider` in `~/.config/personal-memory/config.toml` when that file exists
+- Unless `--no-stack`: compose down + up for the new provider (with mem0/Cognee readiness checks)
+- Unless `--no-mcp`: runs `install-mcp --provider-only` so Cursor MCP points at the new provider
+
+Switching does **not** migrate memories between backends. The NotebookLM compose profile and
+`notebooklm` MCP entry are left alone. Flags and examples:
+[README — Switch memory provider](../README.md#switch-memory-provider).
+
+## Suggested implementation phases (separate commits)
+
+### Phase A — Registry and docs (this commitset)
+
+- Manifests for `hindsight`, `mem0`, and `cognee` (`supported`).
+- Wizard reads manifest for health/smoke URLs; behavior unchanged for Hindsight.
+- Document extension points (this file).
+
+### Phase B — Indexer sink interface (implemented)
+
+- HTTP retain/delete/health live in `indexer/sinks/`; default sink is `hindsight`.
+- CLI: `--sink <name>` or env `MEMORY_SINK`; non-Hindsight sinks use `MEMORY_API_URL` for the API base.
+- Contract tests: `indexer/tests/sink_contract.py` and `test_sinks.py`. Per-provider bank mapping and compose env: [indexer-sinks.md](indexer-sinks.md).
+
+### Phase D — Compose profiles (implemented for Hindsight)
+
+- `litellm` and `hindsight` use compose profile `hindsight`; the indexer always starts.
+- `./setup` passes `--profile hindsight` on compose commands; `scripts/setup.mjs` writes `COMPOSE_PROFILES=hindsight` into the stack `.env`.
+- Alternate providers layer `providers/<id>/compose.yaml` with their own profile (`MEMORY_SINK`, `MEMORY_API_URL`, `depends_on`).
+
+### Phase C — mem0 and Cognee (Wave 2 + Wave 3 DX)
+
+- mem0 is **URL-only** in v1: the indexer talks to a mem0 server the operator already runs, and this
+  repo ships no mem0 container yet. A mem0 service built from a pinned upstream ref is a follow-up.
+  Operator runbook: `providers/mem0/OPERATOR.md`. Coding-agent: wizard `setupCodingAgent` prints
+  SDK/MCP steps (no fake `pluginPackage`); register Cursor MCP with
+  `npx @bcmyguest/openmem install-mcp --yes` / `npx @bcmyguest/openmem switch mem0 --yes`.
+- Cognee runs in **docker**: a pinned `cognee/cognee` image on a loopback port with a named volume and
+  an optional env file (`~/.config/personal-memory/cognee.env`, mode 600).
+  Operator notes: `providers/cognee/OPERATOR.md`. Coding-agent: REST on the loopback port plus a
+  read-only stdio MCP (`cli/cognee-mcp.mjs`: `cognee_health` / `cognee_search`) registered by
+  `setupCodingAgent` / `npm run install-mcp` when `MEMORY_PROVIDER=cognee`.
+- **Cognee + Ollama:** default endpoints use `host.docker.internal:11434` (host Ollama on the Docker
+  bridge). When this stack’s compose profile `ollama` publishes **loopback-only**
+  (`127.0.0.1:11434`), that default is refused from the Cognee container. Set
+  `PM_COGNEE_OLLAMA_COMPOSE=1` or `MEMORY_OLLAMA_IN_COMPOSE=1` (wizard Clack also detects
+  `MEMORY_OLLAMA_IN_COMPOSE=1` in `wizard.conf`) so defaults become `http://ollama:11434` and attach
+  Cognee to the same Docker network as the `ollama` service. Cognify still needs a model that can
+  emit Cognee’s structured extraction schemas; small local models (for example `gemma4:e2b`) may
+  pass HTTP add/cognify start but never finish the graph — treat as an operator/LLM tuning issue,
+  not a provider status gate.
+- `hindsight`, `mem0`, and `cognee` manifests are **`supported`**; new backends still land as
+  `experimental` until wizard + indexer + smoke are wired.
+- `indexer/sinks/mem0.py` and `indexer/sinks/cognee.py` implementing the sink protocol (Cognee also
+  implements the optional `flush()`).
+- Wizard branch: provider select (Clack) before model backend when multiple providers are supported.
+- Coding-agent story: mem0’s own SDK/plugin or a thin local proxy—document limitations.
+
+### Phase D — Compose profiles
+
+- `docker compose --profile hindsight` vs `--profile mem0` generated or documented from manifests.
+- Avoid duplicating indexer image; only swap `depends_on` and indexer `environment`.
+
+## mem0 notes (non-goals for Phase A)
+
+mem0’s API and deployment model differ from Hindsight banks and retain/reflect aliases. Expect:
+
+- Different **namespace** model instead of `bank` strings.
+- Possibly **no** LiteLLM sidecar in this repo (mem0 cloud or separate compose).
+- Coding-agent integration likely **not** the Hindsight npx package; manifest should name the real install path when known.
+
+## Developer deep dives (repo only)
+
+These stay under `docs/` and are **not** synced to the Starlight site (see `site/scripts/lib/docs.mjs` —
+only user-facing guides like [Architecture](architecture.md) are listed in `PAGES`).
+
+- [Indexer sink drivers](indexer-sinks.md) — `sink` vs `sinkDriver`, env vars, bank/scope mapping for Hindsight, mem0, Cognee
+- [MCP setup](mcp-setup.md) — unified MCP vs `--provider-only` vs NotebookLM MCP
+
+## Index sources
+
+See [index-sources.md](index-sources.md), [sources-feeds.md](sources-feeds.md), [sources-zotero.md](sources-zotero.md), [sources-readwise.md](sources-readwise.md), and [index-sources-roadmap.md](index-sources-roadmap.md).
+
+## Coding-agent session hooks
+
+See [coding-agent-hooks.md](coding-agent-hooks.md), [coding-agent-scopes.md](coding-agent-scopes.md), and `npx @bcmyguest/openmem install-coding-agent`.
+
+## Site and docs
+
+Published pages sync from `docs/*.md` at build time. If you change mermaid in a synced doc (for example `docs/architecture.md`), run `cd site && pnpm render-diagrams` and commit `site/public/diagrams/*.svg`. Then run `SITE_URL=https://bcmyguest.github.io BASE_PATH=/openmem/ pnpm build` in `site/` — the same check as CI and GitHub Pages.
+
+## Adding a provider (checklist)
+
+1. Add `providers/<id>/manifest.json` with accurate `status`.
+2. Implement indexer sink (Phase B+).
+3. Add compose fragment or profile and document ports/env.
+4. Extend wizard: provider selection, provider-specific steps, smoke test.
+5. Document opt-in / agent config for that provider.
+6. Extend `scripts/tests/test_wizard.sh` or add provider-specific tests.

package/docs/model-configuration.md

@@ -0,0 +1,84 @@
+# Model configuration
+
+Hindsight never names a model server. It calls the LiteLLM proxy at `http://litellm:4000/v1` with three fixed aliases, and the proxy's YAML maps each alias to a model:
+
+| Alias | Hindsight operation |
+| --- | --- |
+| `memory-retain` | retain (extraction on ingest) |
+| `memory-reflect` | reflect, the base `HINDSIGHT_API_LLM_MODEL`, and mental-model refresh. Needs a model that makes tool calls |
+| `memory-consolidate` | consolidation |
+
+## Shipped routes
+
+Edit the YAML (or a per-machine override) to change models.
+
+| File | Backend | Default split |
+| --- | --- | --- |
+| `config/litellm.lemond.yaml` | lemond on the host | Reflect: Gemma 4 E4B GGUF (`unsloth/`); retain and consolidate: E2B |
+| `config/litellm.ollama.yaml` | Ollama on the host | Used when `MEMORY_BACKEND=ollama`; `gemma4:e4b` needs Ollama newer than 0.20.3 |
+
+**Why reflect uses a larger model:** E2B often returns no usable tool call on Hindsight reflect (`finish_reason=length`). The shipped lemond route uses E4B for `memory-reflect` only. Qwen3.5-4B works as an alternative if you change the route; pick any model that passes a direct tool-call check and a smoke reflect.
+
+## Route selection
+
+| Priority | Path |
+| --- | --- |
+| 1 | `~/.config/hindsight/litellm.yaml` — full override, mounted read-only into the proxy |
+| 2 | `config/litellm.$MEMORY_BACKEND.yaml` — `lemond` (default) or `ollama` |
+
+`scripts/setup.mjs` writes `LITELLM_CONFIG` in the stack `.env`. After changing backend, override, or routes:
+
+```sh
+node scripts/setup.mjs
+docker compose up -d litellm
+```
+
+Optional Ollama backend:
+
+```sh
+MEMORY_BACKEND=ollama node scripts/setup.mjs
+docker compose up -d litellm
+```
+
+## Thinking and token limits
+
+Thinking is off on all three aliases in the route YAML (`enable_thinking: false` / `think: false`) so replies are not empty when Hindsight sends low `reasoning_effort`.
+
+| Layer | Role |
+| --- | --- |
+| LiteLLM `litellm_params.max_tokens` | Default when the client omits `max_tokens` (4096 in shipped routes) |
+| `HINDSIGHT_API_*_MAX_COMPLETION_TOKENS` in `hindsight.env` | Hard cap Hindsight enforces (retain 16384; reflect and consolidation 2048) |
+
+Retain cap must exceed each retain strategy's chunk size (coding-agent strategies use 12000) or Hindsight returns `400 Invalid retain strategy`.
+
+## Secrets
+
+`~/.config/hindsight/litellm.env` (mode 600): `LITELLM_MASTER_KEY` (Hindsight uses this as `HINDSIGHT_API_LLM_API_KEY`), plus provider keys referenced as `os.environ/NAME` in the YAML.
+
+Example hosted reflect model:
+
+```yaml
+  - model_name: memory-reflect
+    litellm_params:
+      model: anthropic/claude-haiku-4-5
+      api_key: os.environ/ANTHROPIC_API_KEY
+```
+
+Hindsight always sends the alias names above; you do not edit model ids in `hindsight.env` for route changes.
+
+## Verify and migrate
+
+Health check from the compose network:
+
+```sh
+docker compose exec litellm python3 -c "import urllib.request;print(urllib.request.urlopen('http://127.0.0.1:4000/health/liveliness').read())"
+```
+
+Older `hindsight.env` files that pointed at lemond/Ollama directly:
+
+```sh
+node scripts/setup.mjs --migrate-env
+docker compose up -d
+```
+
+Migration rewrites LLM lines to use `http://litellm:4000/v1` and the three aliases; other lines stay as they are.

package/docs/notebooklm.md

@@ -0,0 +1,83 @@
+# NotebookLM
+
+Opt-in. Syncs notebooks, sources, notes, and (optionally) chats from Google NotebookLM into **`notebooklm::<notebook slug>`** banks on whichever memory provider you run (**Hindsight, mem0, or Cognee** — same `MEMORY_SINK` / indexer sinks as the project indexer). Each notebook gets one bank; duplicate titles get an ID suffix. The service runs only under the `notebooklm` compose profile, so a plain `docker compose up` never starts it unless that profile is enabled.
+
+Risks:
+
+- The default `client` backend uses the unofficial `notebooklm-py` library (pinned to 0.8.5 with the `markdown` extra). Google can change or block it at any time, and the sync then fails until the library is updated.
+- The stored login is a Google session cookie set. Anyone who reads `~/.config/notebooklm` can act on that Google account. The directory is bind-mounted read-write into the sync container only, and masked from the indexer.
+- Source text and notes are copied into your active memory backend (`notebooklm::*` scope). With Hindsight, retains may invoke the configured model server; mem0/Cognee store chunks via the indexer sink (Cognee runs cognify after each sync batch).
+
+Login happens in **`npm run setup`** when you enable NotebookLM: the wizard installs
+`notebooklm-py` (via `uv`) if needed, opens the Google sign-in browser, then writes
+`NOTEBOOKLM_HOME=~/.config/notebooklm` into the process environment and `wizard.conf`.
+Cookies land in `~/.config/notebooklm/profiles/default/storage_state.json` (bind-mounted
+into the sync container). Re-run setup when Google expires the session.
+
+Manual equivalent (only if you skip the wizard step):
+
+```sh
+mkdir -p ~/.config/notebooklm
+uv tool install 'notebooklm-py[browser]==0.8.5'
+uv tool run --from 'notebooklm-py[browser]==0.8.5' python -m playwright install chromium
+NOTEBOOKLM_HOME=~/.config/notebooklm notebooklm login        # or: ... login --browser-cookies chrome
+NOTEBOOKLM_HOME=~/.config/notebooklm notebooklm auth check --test --json
+```
+
+Enable and start:
+
+1. Create `~/.config/hindsight/notebooklm.yml` with only the keys you change, for example:
+
+   ```yaml
+   enabled: true
+   interval_minutes: 60        # default 60
+   include_notebooks: ["Work*"] # fnmatch globs, case-insensitive; empty means all
+   ```
+
+   Compose refuses to start if this file is missing; create it first (`touch` is enough when there are no changes). Keys merge with `config/notebooklm.yml`; lists and scalars replace the shipped value.
+2. Start the service from the repository:
+
+   ```sh
+   docker compose --profile notebooklm up -d notebooklm
+   docker compose --profile notebooklm logs -f notebooklm
+   ```
+
+Without a login, the service logs the login commands and retries every 15 minutes. With `enabled: false` it sleeps and checks again hourly.
+
+### Coding-agent access (`notebooklmAccess`)
+
+Synced notebooks land in Hindsight banks named `notebooklm::<slug>`. The coding-agent plugin still uses per-repo `coding-agent::*` banks only. Agents can **list/search** NotebookLM banks on demand (no auto-inject) via an opt-in MCP.
+
+**Primary paths** (pick one):
+
+1. **Wizard** — when you enable NotebookLM sync in setup (step 6), the wizard asks whether to allow coding agents to query those banks via MCP (`notebooklm_list_banks` / `notebooklm_search`). Default is yes after sync is enabled. On yes it writes `notebooklmAccess.enabled: true` into a **real** `~/.hindsight/coding-agent.json` (if that path was a symlink to the repo template, it is copied to a regular file first so machine settings never land in `config/coding-agent.json`) and merges a `notebooklm` entry into `~/.cursor/mcp.json`.
+2. **`npx @bcmyguest/openmem install-mcp`** — same enable + MCP merge when NotebookLM is already on (`notebooklm.yml` `enabled: true`) or with `--notebooklm` / `--notebooklm-only`. See [Coding agents and MCP](coding-agents.md).
+
+Restart Cursor Agent after either path. Manual JSON remains a fallback if you skipped the prompt or need to change allowlists later:
+
+```json
+"notebooklmAccess": {
+  "enabled": true,
+  "banks": ["*"]
+}
+```
+
+- `enabled: false` (shipped default) — the NotebookLM MCP exits immediately; setup does not register it.
+- `banks: ["*"]` — every `notebooklm::*` bank; or list slugs / full ids (e.g. `"sitrep-analysis"` or `"notebooklm::sitrep-analysis"`).
+
+MCP tools (stdio server `cli/notebooklm-mcp.mjs`):
+
+| Tool | Purpose |
+| --- | --- |
+| `notebooklm_list_banks` | Allowlisted `notebooklm::*` banks (`bank_id`, `fact_count`) |
+| `notebooklm_search` | Fast full-text list on one allowlisted bank (`bank_id` + `query`; optional `mode=recall` for slow LLM synthesis) |
+
+When `enabled` is true, the wizard or `install-mcp` merges a `notebooklm` entry into `~/.cursor/mcp.json` (does not overwrite `hindsight` or `codegraph`). To run the server by hand:
+
+```sh
+# after enabling notebooklmAccess (wizard, install-mcp, or hand-edit)
+node cli/notebooklm-mcp.mjs
+# or: openmem-notebooklm-mcp
+```
+
+Enterprise backend (`backend: enterprise`): the documented NotebookLM Enterprise REST API lists recently viewed notebooks only. It returns no source or note text, so this backend syncs notebook metadata only. It is untested against a live Enterprise project.