open-memex 0.1.0 → 0.2.0-alpha

package/AGENTS.md

@@ -46,25 +46,51 @@ Layout: `memories/<scope_key>/<id>.md` (YAML frontmatter + body) + `index.db` (S
 
 ## Scope keys
 
-- `user` scope key is literal `"user"`.
+- `personal` scope key is literal `"personal"` (v1 called this `user`; renamed in v2, design §19).
 - `project` scope key: `project__<sanitized-name>__<12-hex-sha256>`, seeded from normalized git origin URL, else lowercased cwd. See `src/scope.ts`. Same repo across machines → same key (intentional; enables future git-commit of memories).
+- `org`, `team`, `public` are reserved — the schema rejects writes. See `docs/SCOPES.md`.
 - The scope key **changes** when a repo gains/loses a git origin. `src/index.ts` logs a one-shot warning on load if it finds files under the legacy cwd-only key (`resolveCwdScope`). Use `cli scopes` to enumerate all scope dirs and `cli migrate --from <old>` to reconcile — see `src/store/migrate.ts`. No auto-migration; two unrelated repos at the same cwd would silently merge.
 - Read-only callers must use `memoriesDirPath` (in `src/paths.ts`), never `memoriesDirFor`. The latter `mkdir -p`s the directory as a side effect and will pollute storage with empty scope dirs.
 
 ## Capture / write path invariants
 
 Every write path (tool, keyword hook, CLI `add`) must:
-1. Call `redact(content, cfg.redactPatterns)`.
+1. Call `redact(content, cfg.redactPatterns)`. Built-in provider patterns live in `src/redact.ts` (always on); config `redactPatterns` is for user extras only.
 2. If `hadSecret` → refuse the write (do not save `[REDACTED]` unless the user wrapped it in `<private>…</private>`).
-3. `writeMemoryFile` first, then `readMemoryFile` + `upsertFromFile` to keep FTS in sync.
+3. Check `findDuplicates` (design §3.4): identical content is idempotent (return existing id); near-duplicates (similarity ≥ 0.8) warn but save — suggest `supersede` when the new content replaces the old.
+4. `writeMemoryFile` first, then `readMemoryFile` + `upsertFromFile` to keep FTS in sync.
+
+## Lifecycle invariants (design §3.3)
+
+- Statuses: `active → superseded | deprecated | retracted | archived`. Retrieval excludes `retracted`/`archived`, ranks `active` above `deprecated`.
+- Never hand-write `status: superseded` or half a chain. Use the `supersede` code path (`src/store/lifecycle.ts`): the old record keeps its file, flips to `superseded`, and both sides get `supersedes`/`superseded_by`. Only `active` memories can be superseded.
+- Chain integrity is self-healing: on read, a missing counterpart is auto-completed with a warning; a dangling pointer warns but is never fabricated. Don't "fix" chains by editing frontmatter directly — let the read path do it.
+- Frontmatter is `schema_version: 2`. The SQLite index schema is versioned separately and rebuilds automatically on version change — never hand-edit `index.db`.
 
 Keyword capture fires from `chat.message` on the assistant's `output.parts` text. Patterns live in `src/capture/keywords.ts` / config `keywordPatterns`; regex group 1 is the memory body.
 
 Context injection happens exactly once per session in `experimental.chat.system.transform`, guarded by an in-memory `Set<sessionID>` in `src/index.ts`. It is not persisted — restarting opencode re-injects on the next first turn.
 
-## Design constraints from PLAN.md worth honoring
+## Design constraints — read the frozen design first
+
+`docs/V2-DESIGN.md` is the frozen protocol v0.2 (zero open questions). Per its §12:
+AGENTS.md answers "how should AI work here"; the design doc answers "why is it
+built this way" (principles, iron rules, D1–D13 decision log). Before changing
+architecture, scope semantics, lifecycle, or the protocol surface (frontmatter
+schema, MCP tools, CLI contract), read the relevant design section — the decision
+log records what was already considered and rejected.
+
+Still hard: no cloud, no silent sync (explicit pull only), Markdown is the source
+of truth, `personal` scope never leaves the machine. Embeddings are an *optional
+capability* per the design — do not add them (or LLM-driven extraction, or a
+knowledge graph) without updating the design doc first. `PLAN.md` tracks the
+build roadmap; the design doc tracks the *why*.
+
+## Branch workflow
 
-MVP is intentionally: no cloud, no embeddings/vector search, no LLM-driven extraction, no knowledge graph. Do not add these without updating `PLAN.md`. Items 1–8 in the v2 roadmap have a rough priority order; prefer extending existing modules over new top-level concepts.
+`main` (stable, mirrors npm) ← `V2` (v2 integration) ← `V2-dev-p<n>`
+(phase work; draft PRs into `V2`). Never create `V2/<anything>` — git can't
+hold `V2` and `V2/…` simultaneously. Full rules: `CONTRIBUTING.md`.
 
 ## Style notes
 

package/CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing to open-memex
+
+## Branch workflow
+
+- `main` — stable. Mirrors the npm release line. Never commit directly;
+  only merge from `V2` when a milestone is tested and ready to release.
+- `V2` — integration branch for the v2 line. Phase work lands here via
+  pull request. Merges to `main` only after the milestone is dogfooded
+  (plugin tested in opencode, `migrate --to-v2 --dry-run` clean on real data).
+- `V2-dev-p<n>` — phase dev branches (e.g. `V2-dev-p1`). Open as **draft**
+  PRs against `V2`; mark ready and merge after local testing passes.
+
+**Naming rule:** never create `V2/<anything>` — git cannot hold a branch
+named `V2` and a branch named `V2/…` at the same time (ref file vs.
+directory conflict). Use the flat `V2-dev-*` form instead.
+
+## Before opening a PR
+
+- `npm run typecheck` is clean
+- `node --experimental-strip-types scripts/smoke-pure.ts` passes
+- If you touched SQLite/FTS: `open-memex reindex` works against a scratch
+  `MY_O_MEMORY_HOME`
+- No secrets in fixtures (redaction tests are the exception)
+
+## Design authority
+
+Protocol decisions live in `docs/V2-DESIGN.md` (frozen v0.2, decision log
+D1–D13). Changing architecture, scope semantics, lifecycle, or the protocol
+surface (frontmatter schema, MCP tools, CLI contract) requires updating the
+design doc first. `AGENTS.md` has the working notes for AI agents; this file
+has the contributor workflow.

package/README.md

@@ -30,8 +30,9 @@ Restart opencode.
 | Tool | What it does |
 |---|---|
 | `memory_add`     | Save a fact, preference, decision, note |
-| `memory_search`  | Keyword search (BM25) across project + user memories |
+| `memory_search`  | Keyword search (BM25) across project + personal memories |
 | `memory_list`    | List memories in a scope, newest first |
+| `memory_supersede` | Replace a memory with a newer version (keeps a supersede chain) |
 | `memory_forget`  | Delete a memory by id |
 
 ## Capture
@@ -43,11 +44,13 @@ Restart opencode.
 ## Scopes
 
 - **project** — scoped to the current repo (keyed off the git origin URL hash, or the cwd if no remote). Default for new memories.
-- **user** — global across all your projects; use for personal preferences.
+- **personal** — global across all your projects, this machine only, never synced. Use for personal preferences. (v1 called this `user`; `migrate --to-v2` renames it.)
+
+See [docs/SCOPES.md](./docs/SCOPES.md) for the full scope model: key derivation, migration, visibility, reserved names.
 
 ## Retrieval
 
-On the first turn of every session, `open-memex` injects a `[OPEN-MEMEX]` block into the system prompt containing top-N recent project memories + top-N user preferences. The agent can also call `memory_search` on demand.
+On the first turn of every session, `open-memex` injects a `[OPEN-MEMEX]` block into the system prompt containing top-N recent project memories + top-N personal preferences. The agent can also call `memory_search` on demand.
 
 ## Storage layout
 
@@ -56,13 +59,17 @@ On the first turn of every session, `open-memex` injects a `[OPEN-MEMEX]` block
 $XDG_DATA_HOME/open-memex/          (Linux/macOS)
 ├── index.db                         # SQLite FTS5 index (rebuildable)
 └── memories/
-    ├── user/
+    ├── personal/
     │   └── <id>.md
     └── project__<name>__<hash12>/
         └── <id>.md
 ```
 
-Each `.md` file has YAML frontmatter (`id, scope_key, type, tags, created_at, ...`) followed by the memory content. You can edit them by hand — the plugin re-syncs on startup by comparing file mtimes.
+Each `.md` file has v2 YAML frontmatter (`id, scope, scope_key, visibility, role, type,
+importance, status, tags, created_at, updated_at, schema_version`, ...) followed by the
+memory content. You can edit them by hand — the plugin re-syncs on startup by comparing
+file mtimes. Markdown is the source of truth; the SQLite index is derived and rebuildable
+(`open-memex reindex`).
 
 ## Config
 
@@ -77,10 +84,13 @@ node --experimental-strip-types src/cli.ts where
 node --experimental-strip-types src/cli.ts list --scope project
 node --experimental-strip-types src/cli.ts search "auth flow"
 node --experimental-strip-types src/cli.ts add "This repo uses better-sqlite3" --type project-config
+node --experimental-strip-types src/cli.ts supersede <id> "Updated content"
+node --experimental-strip-types src/cli.ts status <id> deprecated
 node --experimental-strip-types src/cli.ts forget <id>
 node --experimental-strip-types src/cli.ts reindex
 node --experimental-strip-types src/cli.ts scopes
 node --experimental-strip-types src/cli.ts migrate --from <old-scope-key> [--dry-run]
+node --experimental-strip-types src/cli.ts migrate --to-v2 [--dry-run]
 ```
 
 Or via the npm script: `npm run cli -- list --scope project`.
@@ -88,8 +98,8 @@ Or via the npm script: `npm run cli -- list --scope project`.
 
 ## Status
 
-MVP. See `PLAN.md` for the v2 roadmap (local embeddings, auto-capture, compaction hook, etc).
+v2 alpha (`0.2.0-alpha`): v2 data model + migration, dedup + lifecycle (supersede/status), redaction hardening, CJK bigram retrieval. See `PLAN.md` for the roadmap (local embeddings, auto-capture, compaction hook, etc).
 
 ## License
 
-[MIT](./LICENSE)
+[Apache-2.0](./LICENSE)

package/docs/SCOPES.md

@@ -0,0 +1,81 @@
+# Scopes
+
+**Scope = ownership** (who owns the memory). It answers "whose memory is
+this and where does it live", not "who may read it" — that's `visibility`,
+a separate v2 field (see below).
+
+## The three scopes
+
+| Scope      | Owner            | Lives where                          | Synced?        |
+|------------|------------------|--------------------------------------|----------------|
+| `personal` | you              | this machine only (`memories/personal/`) | **never** |
+| `project`  | repo collaborators | this machine, keyed by repo (`memories/project__<name>__<hash>/`) | via git, only if you opt in |
+| `org`      | org members      | dedicated org memory repo (planned)  | via git (planned) |
+
+**`personal` never leaves the machine.** No sync, no upload, no exceptions.
+Put anything here that should never be shared: credentials-adjacent notes,
+private preferences, personal instructions.
+
+**`project`** is the default for new memories. It is keyed off the repo, so
+the same project resolves to the same scope on every machine.
+
+**`org`** is reserved for a future dedicated org memory repo. The schema
+accepts it; the CLI does not create org scopes yet.
+
+## How the project key is derived
+
+`resolveProjectScope(cwd)` (`src/scope.ts`):
+
+1. Read `git config --get remote.origin.url` in the cwd.
+2. If a remote exists: normalize it (strip `.git`, `git@host:` → `https://host/`,
+   lowercase) and take `sha256(normalized).slice(0, 12)` as the key suffix.
+   The project name comes from the last URL path segment.
+3. If no remote: fall back to the absolute cwd path (lowercased), hashed the
+   same way. This is also how v1 "legacy" scopes are detected during
+   migration when a repo gains a remote later.
+
+Key format: `project__<sanitized-name>__<12-hex-chars>`, e.g.
+`project__open-memex__9e2a8a546c21`. The 12-char hash keeps collisions
+astronomically unlikely while staying readable in `ls`.
+
+## CLI
+
+```
+open-memex add "content" --scope personal      # default is project
+open-memex list --scope personal
+open-memex search "query" --scope both         # project + personal
+open-memex scopes                              # list all known scope dirs
+open-memex migrate --from <old-key> --to <new-key> [--dry-run]
+```
+
+`--scope user` is accepted as a deprecated alias of `--scope personal`.
+
+When a repo gains a git remote after memories were already stored under the
+cwd-based key, `migrate --from <cwd-key> --to <remote-key>` moves them
+(`scopes` shows you the exact keys). Nothing is automatic — you run it
+explicitly, previewing with `--dry-run` first.
+
+## v1 → v2 rename
+
+v1's `user` scope is renamed to `personal` in v2 (design §19 — "user" was
+ambiguous next to "org members are users too"). `open-memex migrate --to-v2`
+moves `memories/user/` → `memories/personal/` and rewrites the frontmatter
+(`scope: user` → `scope: personal`). A dated backup of the pre-migration
+tree is kept. Reads remain backward compatible: a v1 file with `scope: user`
+is interpreted as `personal`.
+
+## Visibility (planned, not yet enforced)
+
+v2 frontmatter carries a separate `visibility` field (`private` | `internal` |
+`shared`). The intended rule: `visibility: private` inside a shared scope is
+**physically isolated** — written to a local-only cache directory, never
+under `.open-memex/` — rather than relying on `.gitignore`. This is not
+implemented yet; today, treat `personal` as the only confidentiality
+boundary and review anything you place under `.open-memex/` before pushing.
+
+## Reserved names
+
+`team` and `public` are reserved scope names: the schema rejects writes to
+them. Rationale: `project` already expresses team sharing; a distinct `team`
+scope needs a clear semantic difference (e.g. cross-repo) before it earns
+existence.