@bcmyguest/openmem 0.0.0-stage → 0.8.0

package/docs/provider-wizard-hooks.md

@@ -0,0 +1,222 @@
+# Provider wizard hooks (sub-wizards)
+
+The meta wizard (`./setup`, `./pm`) owns cross-provider steps. Each **provider plugin** ships a
+**`providers/<id>/wizard.mjs`** module, the same relationship as asdf/mise plugins: the core tool
+runs the plugin's hooks; it does not reimplement Node, Rust, mem0, or a model router.
+
+The meta wizard knows nothing about any one provider. `wizard/lib/run-wizard.mjs`,
+`wizard/lib/ports.mjs` and `wizard/lib/validate-setup.mjs` contain no provider names (a test greps
+for this). A provider that exports no hook for a phase gets a clean no-op for that phase: there is
+no default provider and no fallback implementation.
+
+## Hook API version
+
+`helpers.apiVersion` is `2`. It is bumped when a helper is removed or changes meaning; adding a
+helper does not bump it. Version 1 (the `chooseBackend` helper and `fallback:` argument) is gone.
+The v1 phase names still work unchanged.
+
+## Package root vs stack root
+
+| Field | Env | Meaning | Used for |
+| --- | --- | --- | --- |
+| `ctx.repoRoot` | `PM_STACK_ROOT` | Mutable stack root | compose project dir (`cwd` of every `docker compose` call), `.env`, `compose.override.yaml`, `.build-hash`, generated files |
+| `ctx.packageRoot` | `PM_PACKAGE_ROOT` (default: `repoRoot`) | Read-only code root | `providers/` manifests and hook modules, `scripts/setup.mjs`, `scripts/gen-masks.mjs` |
+
+For a git checkout they are the same directory. For an installed package the integrator passes the
+stack root as `repoRoot` (`runWizard(stackRoot, argv)`) and the package directory through
+`PM_PACKAGE_ROOT`. `runWizard` exports `PM_PACKAGE_ROOT` so child code sees it.
+`PM_PROVIDERS_DIR` overrides the providers directory entirely (tests use it for fake providers).
+Provider code must write only under `ctx.repoRoot` and read bundled files only under
+`ctx.packageRoot`.
+
+## Execution order
+
+```
+meta.preflight
+meta.provider                   pick memory provider (Clack / TOML / --yes)
+provider.init                   quiet: seed ctx defaults (also runs before preflight)
+provider.resolvePorts           host ports (also after the provider is chosen)
+provider.afterProvider
+provider.configureModelRouting  models, keys, logins
+meta.wizardConf                 written unless the provider already saved it
+meta.setupEnv                   indexer state dir, then provider.setupEnv or scripts/setup.mjs
+meta.index                      index.yml
+meta.notebooklm                 optional profile
+provider.beforeStack            compose fragments, migrations
+meta.stack                      build, up, health, provider.afterStackHealthy, then smoke
+meta.codingAgent                generic npx plugin, then provider.setupCodingAgent
+meta.validate                   generic checks, then provider.extendValidation
+meta.syncConfig                 config.toml (provider.configToml sections)
+meta.summary                    provider.summary
+```
+
+**Stack-only actions** (`./setup --up`, `./pm up`) skip the configure phases:
+
+```
+meta.preflight -> meta.provider -> hydrate -> provider.resolvePorts -> provider.beforeStack
+  -> meta.stack -> meta.validate -> meta.summary
+```
+
+**Down** runs `meta.preflight -> meta.stackDown -> meta.summary`. There are no down hooks yet.
+
+## Hook module contract
+
+Path: `providers/<id>/wizard.mjs` (ESM), found through the manifest:
+
+```json
+"wizardHooks": { "module": "wizard.mjs" }
+```
+
+The path must stay inside the provider directory. Every export is an optional
+`async function(ctx, helpers)`; implement only what you need. Hooks marked announced print
+`provider <id>: <phase>` when they run; quiet hooks return a value to the meta wizard.
+
+| Export | Kind | Return | When it runs |
+| --- | --- | --- | --- |
+| `init` | quiet | ignored | Seed provider defaults on `ctx` before anything reads them |
+| `hydrate` | quiet | ignored | Restore provider state from the saved `wizard.conf` |
+| `afterProvider` | announced | ignored | Right after the provider is selected |
+| `configureModelRouting` | announced | ignored | Model backend, API keys, logins. Honor `ctx.skipBackendStep` (Clack already asked) |
+| `resolvePorts` | announced | ignored | Choose free host ports (`helpers.choosePort`, `helpers.resolveHostPort`) |
+| `stackEnv` | quiet | `{ description?, env? }` | What the generic `scripts/setup.mjs` run should print and receive |
+| `setupEnv` | announced | ignored | Replace the generic secrets/env step; call `helpers.runStackSetup` to still run `scripts/setup.mjs` |
+| `beforeStack` | announced | ignored | After env and index, before `docker compose up` |
+| `apiBase` | quiet | string | HTTP base URL when it is not `http://127.0.0.1:<compose.hostPortEnv>` |
+| `smoke` | announced | ignored | Replace the manifest HTTP smoke test |
+| `afterStackHealthy` | announced | ignored | After health, before smoke (e.g. pull compose Ollama models) |
+| `setupCodingAgent` | announced | ignored | After the generic coding-agent plugin step |
+| `extendValidation` | announced | ignored | Push to `ctx.validationErrors` / `ctx.validationChecks`; runs after the generic checks |
+| `configToml` | quiet | `{ stack?, memory?, <table>: {...} }` | Sections for `config.toml`: `stack` merges into `[stack]`, `memory` sub-tables (for example `memory.mem0`) merge into `[memory]` and `provider` there is always set by the wizard, every other key is a table |
+| `summary` | quiet | `{ running?, configFiles?, notes?, logServices? }` | Lines for the closing summary |
+| `clackSetup` | none | `{ env }` | Called by the Clack front end as `(ctxStub, ui, helpers)` |
+| `prepareStackEnv` | script | `{ entries, drop, report }` | Called by `scripts/setup.mjs`; `.env` always gets `HOST_UID`/`HOST_GID` |
+| `migrateEnv` | script | n/a | `scripts/setup.mjs --migrate-env` |
+| `setupNotices` | script | `string[]` | Lines `scripts/setup.mjs` prints at the end |
+
+`prepareStackEnv`, `migrateEnv` and `setupNotices` receive `{ repo, home, env }` (`repo` is the
+stack root) and run in a separate Node process, so they may not use `helpers`.
+
+Unrecognised exports are ignored. Providers must not import `wizard/lib/run-wizard.mjs`; it
+imports them. Everything the meta wizard owns reaches a hook through `helpers`.
+
+## Helpers
+
+`createHelpers()` in `wizard/lib/provider-wizard/helpers.mjs` returns a frozen object. Functions
+that take `ctx` honor `--dry-run` (they print instead of acting).
+
+| Helper | Purpose |
+| --- | --- |
+| `apiVersion` | `2` |
+| `step`, `info`, `ok`, `warn`, `die` | Logging; `die` exits 1 |
+| `problem(ctx, msg)` | `die`, or `warn` in a dry run |
+| `ask`, `confirm` | Prompts (answers come from `ctx.yes` and `PM_*` env when non-interactive) |
+| `act(ctx, desc, cmd, args, {env, cwd})`, `actWithEnv` | Run a command, printed only in a dry run |
+| `commandExists(name)` | PATH lookup |
+| `envValue(file, key)`, `expandHome`, `ensureConfigDir(ctx)` | Paths and wizard.conf reads |
+| `curlGet`, `curlJson`, `curlRequest` | HTTP via curl |
+| `portFree`, `choosePort(ctx, label, port, {envVars})`, `resolveHostPort(ctx, label)` | Ports; `resolveHostPort` reads the manifest `compose.hostPortEnv` and `defaultHostPort` |
+| `composeArgs(ctx, ...rest)` | `docker compose` argv: project, optional `-f` files, optional `--profile` |
+| `saveWizardConf(ctx, extra, {pmSections})` | Write `wizard.conf` (and `config.toml` sections); no-op in a dry run. Always includes `MEMORY_PROVIDER` |
+| `runStackSetup(ctx, opts)` | Run `<packageRoot>/scripts/setup.mjs` against the stack root |
+| `apiBase(ctx)` | The provider's resolved HTTP base URL |
+
+## Manifest fields the meta wizard reads
+
+```json
+"compose": {
+  "profile": "mem0",
+  "files": ["compose.mem0.yaml"],
+  "hostPortEnv": "MEM0_PORT",
+  "defaultHostPort": 8000,
+  "healthService": "mem0",
+  "logServices": ["mem0"]
+},
+"wizard": { "recallHint": "check the server ({compose} logs mem0)" }
+```
+
+- `compose.profile` adds `--profile <name>`; `compose.files` adds `-f` flags after `compose.yaml`
+  and after `compose.override.yaml` when that exists. Both apply to every compose call: `up`,
+  `down`, `build`, `ps`, and the indexer run. Files must be inside the stack root (no absolute
+  paths, no `..`). With neither field the command text is exactly what it was before.
+- `compose.services` is documentation only: `up` is scoped by the profile, not by service list.
+- The API base URL comes from `compose.hostPortEnv` and `defaultHostPort`, or from an `apiBase`
+  hook. `ctx.apiUrl` (`--api-url`) wins over both.
+- `wizard.recallHint` is the error hint when the smoke recall fails. `{compose}` expands to the
+  full compose command prefix and `{project}` to the project name. Without it the generic
+  message uses `compose.logServices` or `compose.healthService`.
+- A provider with no model backend sets none of `MEMORY_BACKEND`, `MEMORY_MODEL`, ...; generic
+  validation does not require them.
+
+## Delegated commands (asdf-style)
+
+A manifest may declare commands the provider owns outright:
+
+```json
+"wizardHooks": {
+  "module": "wizard.mjs",
+  "delegates": {
+    "mem0-login": { "description": "Run mem0 CLI login", "run": ["mem0", "login"] }
+  }
+}
+```
+
+`./pm mem0-login` runs `run` with inherited stdio when that provider is active. Prefer delegates
+over reimplementing provider CLIs in Node.
+
+## Hindsight
+
+Everything about LiteLLM routes, lemond, Ollama, custom OpenAI-compatible backends and the Clack
+model prompts lives in `providers/hindsight/`:
+
+| File | Role |
+| --- | --- |
+| `wizard.mjs` | Thin hook exports; re-exports `prepareStackEnv`, `migrateEnv`, `setupNotices` |
+| `lib/state.mjs` | `ensureState`, backend defaults, `wizard.conf` reads |
+| `lib/model-backend.mjs` | Backend choice (lemond / ollama / custom), reflect model, model pulls |
+| `lib/litellm-routes.mjs` | Write the LiteLLM route file and `hindsight.env`; saves `wizard.conf` |
+| `lib/probe.mjs` | Probe lemond/ollama endpoints and list models |
+| `lib/config.mjs` | `defaultsFromConfig`, `applyConfigEnv`, `configToml` (`[stack]` ports, `[model]`) |
+| `lib/ports.mjs` | `resolvePorts` for the API and control-plane ports |
+| `lib/env-setup.mjs` | `stackEnv`, `setupEnv` (secrets, LiteLLM env) |
+| `lib/stack-env.mjs` | `prepareStackEnv`, `migrateEnv`, `setupNotices` for `scripts/setup.mjs` |
+| `lib/validate.mjs` | Hindsight validation (model backend, route file, ports) |
+| `lib/summary.mjs` | Closing summary lines |
+| `lib/coding-agent.mjs` | seed/materialize `~/.hindsight/coding-agent.json` (real file) and opt-in paths check |
+| `lib/clack.mjs` | `clackSetup`: backend and model prompts for the Clack front end |
+
+Its `wizard.conf` keeps `MEMORY_PROVIDER`, `MEMORY_BACKEND`, `MEMORY_MODEL`, `MEMORY_API_BASE` and
+`MEMORY_REFLECT_MODEL`. The manifest sets `compose.profile: "hindsight"`, so a
+non-Hindsight provider never starts its services.
+
+`configureModelRouting` takes `ctx.skipBackendStep` (`WIZARD_SKIP_BACKEND_STEP=1`) to skip the
+prompts when Clack has already collected the answers.
+
+## Loader
+
+`wizard/lib/provider-wizard/run-steps.mjs`:
+
+- `loadProviderWizard(manifest)`: import `providers/<id>/wizard.mjs`, cached per file; `{}` when absent
+- `callProviderHook(ctx, phase, helpers)`: returns `{ ran, value }`; announces the announced phases
+- `runProviderPhase(ctx, phase, helpers)`: the same, returning whether the hook ran
+- `runDelegatedCommand`, `listDelegateCommands`: manifest `delegates`
+
+## Tests
+
+- `scripts/tests/test_provider_wizard_hooks.mjs`: helpers `apiVersion`, frozen helpers, hook loader,
+  stubbed-helpers tests of the Hindsight `configureModelRouting` (`--yes`, `skipBackendStep`, the
+  three backends), static checks (no provider imports `run-wizard.mjs`; no provider names in the
+  meta files)
+- `scripts/tests/test_provider_manifests.mjs`: manifest schema incl. `compose.profile`/`files`
+- `scripts/tests/test_wizard.sh`: dry-run and real-run scenarios with a docker stub, including a
+  provider with no hooks, a provider with every v2 hook (fixtures in
+  `scripts/tests/fixtures/providers/`, selected with `PM_PROVIDERS_DIR`), and a package-root /
+  stack-root split
+
+## Adding hooks (checklist)
+
+1. Add `wizardHooks.module` (and `compose.*`) to `manifest.json`
+2. Create `providers/<id>/wizard.mjs`; implement only the phases you need
+3. Keep provider state on `ctx` through `init`/`hydrate`; return `configToml`/`summary` data instead of writing in the meta files
+4. Add `delegates` only when a native CLI must run on the host
+5. Document native vs meta ownership in manifest `settings`
+6. Extend the tests and the [add-memory-provider](../.agents/skills/add-memory-provider/SKILL.md) skill

package/docs/quickstart.md

@@ -0,0 +1,107 @@
+# Quickstart
+
+Get from zero to a running stack in one wizard run. **Choose your memory provider** in the wizard (Hindsight by default; mem0 or Cognee if you prefer) — same CLI either way. Cloning the repo is optional; **`npx @bcmyguest/openmem`** is the usual path. For why OpenMem exists (interface layer, not a memory engine), see [Concepts](concepts.md).
+
+## Before you start
+
+| Requirement | Why |
+| --- | --- |
+| Docker + Compose v2 | Runs the stack; your user must run `docker` without sudo |
+| Node.js ≥ 24 | CLI, setup wizard, and the session context store (`node:sqlite` with FTS5) |
+| LLM for Hindsight (default provider) | lemond, Ollama, OpenAI-compatible, or a Claude subscription through [Meridian](meridian-litellm.md) — wizard wires LiteLLM |
+| git, curl | Wizard preflight |
+
+Default stack is [Hindsight](hindsight.md). Other providers: wizard or `npx @bcmyguest/openmem providers` ([provider status](provider-status.md)).
+
+## 1. Run the wizard
+
+**From npm (recommended):**
+
+```sh
+npx @bcmyguest/openmem
+```
+
+**From a git checkout** (repo directory is the stack root):
+
+```sh
+git clone https://github.com/bcmyguest/openmem.git
+cd openmem
+npm install
+npm run setup
+```
+
+Run other commands from the checkout with `node cli/openmem.mjs <command>`.
+
+The wizard asks for a memory provider, provider settings, what to index, then starts Docker and runs a smoke test.
+
+Useful variants:
+
+```sh
+npx @bcmyguest/openmem setup --dry-run     # plan only, no writes
+npx @bcmyguest/openmem configure --yes       # defaults + start, no prompts
+```
+
+Wizard dry-run (what first setup plans — no writes):
+
+<p align="center">
+<img src="https://bcmyguest.github.io/openmem/setup-demo.gif" alt="Terminal recording of the setup wizard dry run: preflight checks, memory provider, model backend, secrets, sources and stack steps, each planned write printed as a [dry-run] line" width="720" />
+</p>
+
+## 2. Confirm it is up
+
+```sh
+npx @bcmyguest/openmem doctor
+curl -fsS http://127.0.0.1:8888/health   # Hindsight default; see wizard output for other providers
+```
+
+## 3. Wire your coding agent (optional)
+
+**Session memory** (after step 1 — MCP and hooks need the running stack): `npx @bcmyguest/openmem install-coding-agent --yes` — unified MCP **plus** session hooks for Claude Code, Codex, OpenCode, Cursor CLI (any active provider). On Hindsight, `--provider-plugin` uses the upstream Hindsight plugin hooks instead (one hook set at a time). Not the same as `install-mcp` alone.
+
+**`install-mcp`** merges the **`openmem` unified MCP** (`memory_add`, `memory_search`, `memory_recall`, `memory_reflect`, `memory_consolidate`, `memory_forget` on whichever provider you picked) and, when NotebookLM sync is on, the **NotebookLM MCP**:
+
+```sh
+npx @bcmyguest/openmem install-mcp --yes
+npx @bcmyguest/openmem install-mcp --yes --notebooklm   # force NotebookLM entry
+```
+
+Session hooks use the in-repo shim by default (`[agents] integration = "openmem"`); `--provider-plugin` replaces them with the upstream Hindsight plugin and `--no-provider-plugin` switches back. Optional native mem0/Cognee MCP: `install-mcp --provider-only`. Full guide: [MCP setup](mcp-setup.md).
+
+Restart Cursor Agent after MCP changes. Move memories between providers with [Portable memory (PMM)](memory-portability.md) — `switch` alone does not migrate data.
+
+## Day-2
+
+```sh
+npx @bcmyguest/openmem up
+npx @bcmyguest/openmem down
+npx @bcmyguest/openmem switch mem0 --yes
+npx @bcmyguest/openmem doctor
+```
+
+| Command | Purpose |
+| --- | --- |
+| `up` / `down` | Start or stop the stack (down keeps data volumes) |
+| `switch <id> --yes` | Change provider (no memory migration) |
+| `doctor` | Validate config without the smoke test |
+| `schedule install` | Refresh configured RSS/OpenAlex feeds hourly with the OS scheduler |
+
+In a checkout, use `npx .` instead of `npx @bcmyguest/openmem`.
+
+## Where files live
+
+| Install | Stack root (compose, `.env`) |
+| --- | --- |
+| `npx @bcmyguest/openmem` | `~/.local/share/personal-memory/stack` (override: `PM_STACK_ROOT`) |
+| Git checkout | The clone directory |
+
+Secrets and index config: `~/.config/hindsight/` (and optional `~/.config/personal-memory/config.toml`).
+
+## Next
+
+- [MCP setup](mcp-setup.md) — unified `memory_*`, provider-native, NotebookLM
+- [Portable memory (PMM)](memory-portability.md) — export/import
+- [Architecture](architecture.md) — data flow, indexing, MCP by provider
+- [Index sources](index-sources.md) — git, Obsidian, notes/papers/bookmarks, feed cache
+- [Feed sync](sources-feeds.md) — configure RSS/OpenAlex and schedule refreshes
+- [Install with npx](npx-install.md) — upgrades, troubleshooting
+- [Configuration file](openmem-config.md) — declarative `config.toml`

package/docs/setup-wizard.md

@@ -0,0 +1,117 @@
+# Setup wizard
+
+The wizard is a Node program (`wizard/bin/setup.mjs`). It is exposed as:
+
+| How you run it | When |
+| --- | --- |
+| `npx @bcmyguest/openmem` / `npx @bcmyguest/openmem setup` | Normal install ([Install with npx](npx-install.md)) |
+| `npm run setup` / `./setup` | Git checkout (stack root stays in the repo); other commands: `node cli/openmem.mjs <command>` |
+
+Interactive runs use [@clack/prompts](https://github.com/bombshell-dev/clack). `--yes` and `--dry-run` skip Clack.
+
+## Stack actions
+
+Before provider questions, pick what the run should do (Clack or `PM_WIZARD_ACTION`):
+
+| Action | Flag | Behavior |
+| --- | --- | --- |
+| Configure | default | Full wizard: provider, env, index sources, start stack, optional coding-agent step |
+| Start | `--up` | Requires existing `~/.config/hindsight/wizard.conf`; starts stack and validates |
+| Stop | `--down` | `docker compose down` for the project |
+
+Non-interactive shortcuts:
+
+```sh
+npx @bcmyguest/openmem configure --yes   # setup --yes
+npx @bcmyguest/openmem up                # wizard --up (after first configure)
+npx @bcmyguest/openmem down              # wizard --down
+```
+
+## Wizard phases
+
+Configure runs these steps in order (provider hooks replace or extend some steps):
+
+| Step | Title | What happens |
+| --- | --- | --- |
+| 1 | Preflight | `git`, `curl`, `docker`, Compose v2; host UID/GID; API/control-plane ports |
+| 2 | Memory provider | Hindsight, mem0, or Cognee ([Provider status](provider-status.md)) |
+| 3 | Provider config | See table below |
+| 4 | Secrets and env | Files under `~/.config/hindsight/` (mode 600); runs `scripts/setup.mjs` for stack `.env` and `compose.override.yaml` |
+| 5 | Sources to index | Writes `~/.config/hindsight/index.yml`; lists sources with a dry indexer run |
+| 6 | NotebookLM sync | Opt-in; Google login when enabled |
+| 7 | Start the stack | Build indexer image if inputs changed; health wait; smoke retain/recall (unless `--no-smoke`) |
+| 8 | Context management (smart compaction) | Multi-select of the installed agents (Claude Code, OpenCode 1, OpenCode 2). Defaults: saved `[context_proxy.agents]` keys, else `[context_proxy] enabled`, else OpenCode 2 on and the proxy-routed agents off. Saves all four keys to `config.toml`. Claude Code or OpenCode 1 on: writes the proxy key file and store directories, then starts the `context-proxy` compose service. Neither on: stops a running `context-proxy` after step 9, unless an agent still points at it ([Context management](context-management.md)) |
+| 9 | Coding agents (optional) | For detected CLIs (claude, codex, opencode, cursor-agent), offers `install-coding-agent --yes`: unified MCP, OpenMem skill, session hooks; honours `[agents] integration` (OpenMem hooks or the provider plugin) and `[context_proxy.agents]`. Default answer is no |
+| 10 | Validate memory setup | Provider checks + shared validation |
+
+The last line is `wizard: changed=N` (files and containers this run touched). An unchanged re-run prints `changed=0`.
+
+## Step 3 by provider
+
+| Provider | Step 3 asks for |
+| --- | --- |
+| Hindsight | Model backend: lemond, Ollama (host or compose profile), custom OpenAI-compatible URL, or a Claude subscription through a running Meridian ([guide](meridian-litellm.md#with-the-wizard)); LiteLLM routes for `memory-retain`, `memory-reflect`, `memory-consolidate` |
+| mem0 | Self-hosted REST base URL and API key file path |
+| Cognee | Loopback port and `LLM_API_KEY` for graph build |
+
+Details: [Model configuration](model-configuration.md) (Hindsight), `providers/mem0/OPERATOR.md`, `providers/cognee/OPERATOR.md`.
+
+## Where configuration lives
+
+| Path | Written by | Purpose |
+| --- | --- | --- |
+| Stack root `.env`, `.build-hash`, `compose.override.yaml` | `scripts/setup.mjs` | Compose project; indexer UID; secret-path masks |
+| `~/.config/hindsight/wizard.conf` | Wizard | Provider id, backend choice, ports |
+| `~/.config/hindsight/hindsight.env`, `litellm.env` | Setup / Hindsight hooks | Hindsight + LiteLLM secrets |
+| `~/.config/hindsight/index.yml` | Wizard step 5 | Index roots, repos, vaults; optional `notes` / `papers` / `bookmarks` / `feeds` ([index-sources.md](index-sources.md)) |
+| `~/.config/hindsight/notebooklm.yml` | Wizard step 6 | NotebookLM sync toggles |
+| `~/.config/personal-memory/config.toml` | Optional; synced on configure | Declarative overrides ([Configuration file](openmem-config.md)) |
+| `~/.hindsight/coding-agent.json` | Wizard / install-mcp | **Real file on disk** — do not symlink to the repo template |
+
+### Stack root (npx installs)
+
+When the CLI is not running from a git checkout, mutable compose files live under:
+
+```text
+${PM_STACK_ROOT:-${XDG_DATA_HOME:-~/.local/share}/personal-memory/stack}
+```
+
+Wizard code and provider hooks still run from the npm package. See [Install with npx](npx-install.md#stack-root).
+
+## Wizard flags
+
+| Flag | Effect |
+| --- | --- |
+| `--yes` / `-y` | Accept defaults; no prompts |
+| `--dry-run` | Plan only; read-only checks; no writes or containers |
+| `--no-smoke` | Skip retain/recall smoke test |
+| `--rebuild` | Rebuild indexer image even if sources unchanged |
+| `--up` / `--down` | Start or stop only (see stack actions) |
+
+Environment (common):
+
+| Variable | Meaning |
+| --- | --- |
+| `MEMORY_PROVIDER` | Default provider id |
+| `PM_BACKEND` | Hindsight backend default (`lemond`, `ollama`, `custom`) |
+| `PM_PROJECT` | Docker Compose project name |
+| `PM_STACK_ROOT` / `PM_USE_CHECKOUT` | Stack root location |
+| `HINDSIGHT_PORT`, `HINDSIGHT_CP_PORT` | Host API and control-plane ports |
+
+Run `./setup --help` in a checkout for the full list.
+
+## Indexer image builds
+
+The wizard rebuilds the indexer image only when its inputs change (Dockerfile, `compose.yaml`, copied sources). The hash is stored in `.build-hash` at the stack root. If nothing changed, it uses `up -d --no-build` so containers are not recreated unnecessarily.
+
+## Related docs
+
+| Topic | Doc |
+| --- | --- |
+| npx, stack root, upgrades | [npx-install.md](npx-install.md) |
+| `config.toml` and day-2 tasks | [openmem-config.md](openmem-config.md) |
+| Manual compose without the wizard | [manual-stack-setup.md](manual-stack-setup.md) |
+| LiteLLM routes and models | [model-configuration.md](model-configuration.md) |
+| Index sources and `index.yml` | [indexing.md](indexing.md) |
+| NotebookLM sync and MCP | [notebooklm.md](notebooklm.md) |
+| Plugins and MCP install | [coding-agents.md](coding-agents.md) |

package/docs/sources-feeds.md

@@ -0,0 +1,39 @@
+# RSS and OpenAlex feeds
+
+Configure `feeds` in `index.yml`, run sync, then the indexer watches `feeds.out_dir` as kind **feed** (`feed::*` banks).
+
+```yaml
+feeds:
+  out_dir: ~/.local/share/personal-memory/feeds
+  rss:
+    - url: https://rss.arxiv.org/rss/cs.AI
+      slug: arxiv-cs-ai
+  openalex_queries:
+    - url: https://api.openalex.org/works?search=machine%20learning&per_page=25
+      slug: openalex-ml
+```
+
+```sh
+npx @bcmyguest/openmem feed-sync
+# checkout: npm run feed-sync
+```
+
+Then ensure the stack indexer is running. Each run writes markdown files under `out_dir`; the watcher indexes new/changed files.
+
+Network use: feed-sync calls public RSS/OpenAlex URLs from the host (not from CI by default).
+
+`feed-sync` needs Python 3 with PyYAML on the host. On Windows it uses `py -3`, then `python`; on macOS and Linux it uses `python3`, then `python`. Set `PYTHON` to choose a specific executable.
+
+## Schedule feed sync (optional)
+
+Install an hourly job with the current user's native scheduler. The GitHub npx source works before the next npm release includes `schedule`:
+
+```sh
+npx --yes github:bcmyguest/openmem schedule install --dry-run
+npx --yes github:bcmyguest/openmem schedule install
+npx --yes github:bcmyguest/openmem schedule remove
+```
+
+After that release, `npx @bcmyguest/openmem schedule ...` runs the same command from npm.
+
+OpenMem installs a systemd user timer on Linux, a launchd agent on macOS, or a Task Scheduler task on Windows. The job calls the published npm package through a stable runner in your user config directory. Installation records the Python interpreter with PyYAML, so the job does not depend on your interactive shell's PATH. It runs when your user scheduler is active; the Linux timer catches up after downtime. Re-run `schedule install` if you move or upgrade Node.js or Python.

package/docs/sources-readwise.md

@@ -0,0 +1,23 @@
+# Readwise → indexer
+
+Export highlights from [Readwise Reader](https://readwise.io/read) or the legacy export as **Markdown**.
+
+1. Export to a directory (Readwise → Export → Markdown).
+2. Add to `notes.paths` or `papers.paths` depending on how you split reading vs research:
+
+```yaml
+notes:
+  paths:
+    - ~/Readwise/Full-Sync
+```
+
+3. Re-run indexing; chunks retain with bank `notes::full-sync` (slug from folder name).
+
+Optional stable bank id:
+
+```yaml
+banks:
+  ~/Readwise/Full-Sync: readwise::full-sync
+```
+
+For automatic sync, use Readwise’s scheduled export + the same path (filesystem watcher picks up changes).

package/docs/sources-zotero.md

@@ -0,0 +1,22 @@
+# Zotero → indexer
+
+**Assumption:** use export-to-disk first; no Zotero API key in the stack yet.
+
+1. Install [Better BibTeX](https://retorque.re/zotero-better-bibtex/) (optional) for stable cite keys.
+2. Export markdown or JSON notes to a folder, e.g. `~/Zotero/export-md`.
+3. Point `papers.paths` at that folder in `~/.config/hindsight/index.yml`:
+
+```yaml
+papers:
+  paths:
+    - ~/Zotero/export-md
+```
+
+4. Restart the indexer container or run `bin/hindsight-project-index --apply --papers ~/Zotero/export-md`.
+
+Banks default to `papers::<folder-name>` unless overridden under `banks:`:
+
+```yaml
+banks:
+  ~/Zotero/export-md: papers::zotero
+```

package/glama.json

@@ -0,0 +1,4 @@
+{
+  "$schema": "https://glama.ai/mcp/schemas/server.json",
+  "maintainers": ["bcmyguest"]
+}

package/indexer/Dockerfile

@@ -0,0 +1,38 @@
+# Project indexer: one pass at startup, then re-index each source after it stops changing.
+FROM python:3.12-slim
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+# Tree-sitter grammars are prefetched here, so the runtime never downloads and can run with no network and a
+# read-only $HOME. The directory is world-readable because the container runs as the host UID.
+ENV TREE_SITTER_GRAMMAR_DIR=/opt/tree-sitter-grammars
+RUN pip install --no-cache-dir watchfiles==1.3.0 pyyaml==6.0.3 tree-sitter-language-pack==1.22.1 \
+    && python3 -c "import os; from tree_sitter_language_pack import PackConfig, download, init; \
+init(PackConfig(cache_dir=os.environ['TREE_SITTER_GRAMMAR_DIR'])); \
+download(['python', 'javascript', 'typescript', 'tsx', 'go', 'rust', 'java', 'c', 'cpp', 'csharp', 'ruby', 'lua', 'bash', 'php'])" \
+    && chmod -R a+rX /opt/tree-sitter-grammars
+
+# The compose user is the host's UID/GID, which may have no passwd entry here. Trust every repository owner, since
+# the indexer only reads them and the owner is the host user.
+RUN git config --system safe.directory '*'
+
+COPY bin/hindsight-project-index /usr/local/bin/hindsight-project-index
+COPY indexer/watch.py indexer/index_config.py indexer/code_outline.py indexer/feed_sync.py /opt/indexer/
+# Memory sinks (one module per provider). A new indexer/sinks/<name>.py needs no Dockerfile edit.
+COPY indexer/sinks/ /opt/indexer/sinks/
+# Shipped default config. index_config.py finds it at ../config/index.yml relative to itself.
+COPY config/index.yml /opt/config/index.yml
+RUN chmod 0755 /usr/local/bin/hindsight-project-index /opt/indexer/watch.py \
+    && chmod -R a+rX /opt/indexer
+
+# $HOME is bind-mounted, so ignore the host's ~/.local site-packages (they would shadow the pinned packages).
+# PYTHONPATH makes index_config, code_outline and the sinks package importable from both the indexer script and
+# watch.py.
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONNOUSERSITE=1 \
+    PYTHONPATH=/opt/indexer \
+    HINDSIGHT_URL=http://hindsight:8888
+
+CMD ["python3", "/opt/indexer/watch.py"]