mrplex 0.0.2 → 0.1.0

package/README.md

@@ -2,7 +2,11 @@
 
 *Markdown Repos, plexed.*
 
-**Turn ordinary Markdown folders into queryable, versioned, graph-aware knowledge stores — without giving up files.** Your notes stay plain `.md` on disk; agents and humans share the same repository, every write is versioned, and links survive renames.
+mrplex is not a better notes app. It is a small kernel that leaves your files as files and makes the folder a multi-agent substrate.
+
+You keep working with ordinary `.md` with YAML frontmatter. mrplex adds versions, optimistic concurrency, CEL filters, full-text search, optional embeddings, and a link graph bound to document identity so renames do not smash backlinks. CLI, MCP, and REST are the same model with different sockets.
+
+If you wanted an Obsidian clone, this is the wrong repo. If you're an Obsidian user, now your agents can utilize and manage your vault through mrplex's MCP interface.
 
 ## Install
 
@@ -12,247 +16,165 @@ Node ≥ 20.11:
 npm install -g mrplex
 ```
 
-Prefer not to install globally? `npx mrplex …` works the same way.
+`npx mrplex …` works. From a git checkout: `npm install && npm link`.
 
-Point every command at a database and default repo once (the tour below uses `starship`):
+No config required. With no flags, the CLI uses `./mrplex.db`. Pass `-r <repo>` when a command needs a repo.
 
-```bash
-export MRPLEX_DATABASE=./demo.db
-export MRPLEX_REPO=starship
-```
+## Five minutes on the USS Meridian
 
-Or persist settings in `~/.config/mrplex/config.json`:
-
-```bash
-mrplex config set-database ./demo.db
-mrplex config set-repo starship
-```
-
-Each setting resolves **flag → env → config file → default**, so one-off overrides never require editing anything.
-
-> **From a git checkout?** Run `npm install && npm link` (or `npm run cli -- …`). Every `mrplex …` command below is equivalent to `npx mrplex …`.
-
-## A five-minute tour
-
-This walkthrough uses the **USS Meridian** — a sample starship knowledge base in [`fixtures/starship/`](fixtures/starship/): crew files, mission records, officer logs, and equipment status, all ordinary Markdown with YAML frontmatter.
-
-### 1. Sync the sample corpus
-
-Clone the repo (for the fixture files), install, and sync the folder into a fresh mrplex repo:
+The fixture in `fixtures/starship/` is a starship knowledge base: crew, missions, logs, equipment. Ordinary Markdown. The product is the questions you can ask without opening every file.
 
 ```bash
 git clone https://github.com/usergenic/mrplex
 cd mrplex
-npm install && npm link    # skip if you installed globally above
-
-export MRPLEX_DATABASE=./demo.db
-export MRPLEX_REPO=starship
+npm install && npm link
 
 mrplex repos create starship
 mrplex sync fixtures/starship --once -r starship
 ```
 
-`sync` is the normal way to load a Markdown folder: it pushes local files into the repo, materializes version metadata, and keeps a cursor for two-way updates. Frontmatter values written as repo-root paths (`/crew/foo.md`) and embedded wikilinks/inline links are indexed automatically — no link-config setup step.
+`sync` loads a folder, versions it, and keeps a cursor for two-way updates. Wikilinks, inline links, and frontmatter values written as repo-root paths (`/crew/foo.md`) are indexed automatically.
 
-No bootstrap, no token — local mode is full-trust (whoever can run the binary owns the database).
-
-### 2. Query it like data
-
-Frontmatter is queryable with [CEL](https://github.com/google/cel-spec) filters:
+### Ask the folder like data
 
 ```bash
-# What's broken or offline right now?
-mrplex query --filter 'status == "damaged" || status == "offline"'
-
-# Who reports directly to the captain?
-mrplex query --filter '$has_static("crew/kestrel-vance.md", "reports_to")'
-```
-
-```text
-PATH                              STATUS
-equipment/plasma-manifold-3.md    damaged
-equipment/shuttle-corvid.md       offline
-…
-
-PATH                    TITLE
-crew/aria-okonkwo.md    Commander Aria Okonkwo
-crew/dax-thorne.md      Lieutenant Commander Dax Thorne
-crew/quill-vasquez.md   Doctor Quill Vasquez
-```
+# What's broken?
+mrplex -r starship query --filter 'status == "damaged" || status == "offline"'
 
-Compose with full-text search — filter, text, and semantic (when configured) all AND together:
+# Who reports to the captain?
+mrplex -r starship query --filter '$has("crew/kestrel-vance.md", "reports_to")'
 
-```bash
-mrplex query --text 'manifold coolant'
-mrplex query --filter 'type == "mission"' --text 'Halloway'
+# Filter AND full-text
+mrplex -r starship query --filter 'type == "mission"' --text 'Halloway'
 ```
 
-### 3. Follow the links as a graph
-
-Markdown links, wikilinks, and frontmatter repo-root paths (`/crew/foo.md`) build a derived index bound to document *identity*, so backlinks survive renames. Query membership in CEL:
+`query` returns lean hits. Default projection is paths, not bodies. Hydrate what you need. That is how an agent stays cheap.
 
-```bash
-# Every mission indexed by the mission log MOC
-mrplex query --filter '$in_static("moc/missions.md")'
+Filter, text, and semantic compose with AND. Semantic rank is a shortlist, not an oracle. Inspect the top hits; do not treat cosine as authority.
 
-# What touches the damaged plasma manifold? (maintainer field, body links, logs…)
-mrplex query --filter '$has_static("equipment/plasma-manifold-3.md")'
-```
+### Follow the graph
 
-Explore *how* documents connect — neighborhood expansion from a root set:
+Links live on document identity, not on the string you typed. Move a file and the graph still knows who pointed at it. The written text may go stale; the relation does not.
 
 ```bash
-mrplex graph --roots missions/the-hollow-signal.md --degrees 2 --direction both --render summary
-```
-
-The missing officer, his mission, the encounter, and the logs that mention him appear as a connected neighborhood — the kind of thread a flat file listing can't give you.
+# Everything a map-of-content claims
+mrplex -r starship query --filter '$in("moc/missions.md")'
 
-Mermaid for slides or docs:
+# What touches the damaged manifold?
+mrplex -r starship query --filter '$has("equipment/plasma-manifold-3.md")'
 
-```bash
-mrplex graph --roots crew/kestrel-vance.md --degrees 2 --direction out --render mermaid
+# Neighborhood, not a file listing
+mrplex -r starship graph --roots missions/the-hollow-signal.md --degrees 2 --direction both --render summary
+mrplex -r starship graph --roots crew/kestrel-vance.md --degrees 2 --direction out --render mermaid
 ```
 
-### 4. Change something safely
+### Write without clobbering
 
-Every write inserts a new version; nothing is overwritten. Rename a document and the link graph follows its identity:
+Every write inserts a version. `prev_version_id` is optimistic concurrency. Stale prev loses; the current version comes back. Delete is a move to `:deleted/…`. Restore is a normal put.
 
 ```bash
-# See current version id, then move the manifold note
-V=$(mrplex --json docs get equipment/plasma-manifold-3.md | jq -r .version_id)
-mrplex docs mv equipment/manifold-3.md --prev "$V"
+V=$(mrplex -r starship --json docs get equipment/plasma-manifold-3.md | jq -r .version_id)
+mrplex -r starship docs mv equipment/manifold-3.md --prev "$V"
 
-# Backlinks still resolve — the graph never broke; link text may need repair
-mrplex query --filter '$has_static("equipment/manifold-3.md")'
-mrplex links stale
-```
-
-History and unified diff between any two versions:
+mrplex -r starship query --filter '$has("equipment/manifold-3.md")'
+mrplex -r starship links stale
 
-```bash
-mrplex docs history equipment/manifold-3.md
-mrplex docs diff equipment/manifold-3.md --from v1 --to v2
+mrplex -r starship docs history equipment/manifold-3.md
+mrplex -r starship docs diff equipment/manifold-3.md --from v1 --to v2
 ```
 
-Deletion moves a document to a system-namespace path (`:deleted/…`); restore is a normal `docs put` back to user territory.
-
-### 5. Connect an agent
+## Connect an agent
 
-CLI, MCP, and REST are surfaces over the same model. Point Cursor (or any MCP client) at your local database:
+CLI, MCP, and REST are surfaces over one kernel. For a database only you can touch, local stdio with `--unsafe` is the fastest first attachment:
 
 ```json
 {
   "mcpServers": {
     "mrplex": {
       "command": "mrplex",
-      "args": ["mcp-stdio", "--unsafe", "--database", "./demo.db"],
-      "env": { "MRPLEX_REPO": "starship" }
+      "args": ["mcp-stdio", "--unsafe", "--database", "./mrplex.db"]
     }
   }
 }
 ```
 
-An agent can now ask relational questions — *"What's broken on the ship, who maintains it, and which missions were affected?"* — and recover structured answers through `query` and `graph` instead of reading every file into context.
+`--unsafe` means full-trust kernel: no auth in-process. That is correct for a private local file. It is not a networked default.
 
-Serve HTTP for remote clients or Streamable HTTP MCP:
+The questions that justify the extra process are relational: *what is broken, who maintains it, which missions were hit?* Query and graph first. `docs get` / `docs_get_many` only for the hits you will actually use.
+
+## Policy shell (local or networked)
+
+When you want principals — MCP as maintainer, a separate admin for repo create/delete — scaffold a policy and mint keys. No YAML to copy:
 
 ```bash
-mrplex serve --unsafe --port 8321 &
-mrplex --server http://127.0.0.1:8321 query --filter 'status == "missing"'
+mrplex policy create --principal brendan --author "Brendan Baldwin <brendan@example.com>"
+# → policy.yaml with admin (operator) + brendan (maintainer)
+
+KEY=$(mrplex key mint brendan --policy policy.yaml)   # MCP day-to-day
+# mrplex key mint admin --policy policy.yaml          # when you need repos create/delete
+
+mrplex serve --policy policy.yaml --audit audit.jsonl --port 8321 &
+# or: mrplex mcp-stdio --policy policy.yaml --key "$KEY"
 ```
 
-For shared or networked deployments, run the authenticating shell instead — see [Authentication](#authentication) below. Never expose the raw kernel (`serve --unsafe`) directly to an untrusted network.
+`policy create` writes the minimal file; `policy check` validates it (and can dump a principal's entitlement). Edit the file only when you outgrow the defaults.
+
+Three shapes: **embedded** (`serve --policy`), **launcher** (`mcp-stdio --policy`), **fronting proxy** (`proxy --policy --upstream`). Edit the policy and `kill -HUP` to reload. OIDC device flow via `mrplex login`. Details in [docs/archive/security.md](docs/archive/security.md).
+
+Never expose `serve --unsafe` to an untrusted network. The flag is loud on purpose.
 
 ## How it works
 
-| Concept | What it means |
-|--------|----------------|
-| **Document** | One Markdown file with YAML frontmatter, addressed by repo-relative path |
-| **Version** | Every write appends; `prev_version_id` optimistic concurrency rejects stale writes |
-| **Query** | CEL filters over frontmatter + `$path` / `$body` / `$updated_at` intrinsics; composes with FTS and semantic search |
-| **Link graph** | Derived index over inline links, wikilinks, and frontmatter repo-root paths; `$in`, `$has`, `$backlinks`, `$links` in CEL |
-| **Graph** | BFS neighborhood expansion over the link index — *how* things connect, not just *which* match |
-| **Surfaces** | `mrplex` CLI (local or `--server`), MCP at `/mcp` or `mcp-stdio`, REST at `/repos/{repo}/…` |
-
-Two layers: a **full-trust kernel** (no in-engine auth) and an optional **access-and-identity shell** (API keys, OIDC, per-path write policy, audit log). See [docs/archive/security.md](docs/archive/security.md) for trust boundaries and deployment shapes.
-
-## Features
-
-<details>
-<summary>Full feature list</summary>
-
-- **Versioned Markdown store** — every write inserts; `docs.put` handles in-place update and move; any past state is addressable
-- **Byte-exact frontmatter** — `frontmatter_raw` (verbatim YAML) or `frontmatter` (JSON); exactly one; round-trips are byte-exact via raw
-- **Optimistic concurrency** — stale `prev_version_id` → `stale_prev` with current version returned
-- **Deletion as move** — `:deleted/…` paths; idempotent delete; restore via `docs.put`
-- **Unified diff** — `docs.diff`, REST `/diff`, MCP `docs_diff`, CLI `mrplex docs diff`; `patch(1)`-applicable output
-- **CEL filter queries** — frontmatter fields + `$`-intrinsics; `list()` polymorphism for scalar-or-list fields
-- **Link graph** — inline, wikilink, and frontmatter path extraction (no per-field config); backlinks survive renames; set algebra (`$in("moc/**") && !$in("moc/draft.md")`); `$in_static` for link-only membership; `links stale` / `repair` / `backfill`
-- **Graph exploration** — BFS with direction lens, visibility filter, graph-only `$degrees` intrinsic; CLI `--render summary|yaml|mermaid|json`
-- **Full-text search** — SQLite FTS5 or Postgres `websearch_to_tsquery`; composes with filter via AND
-- **Semantic search** — pluggable `--embedder` hook; chunker + backlog worker; `$semantic_score` in `select`; no hook → `semantic_unavailable`
-- **HTTP surfaces** — MCP (Streamable HTTP + STDIO), REST with `If-Match` / content negotiation / `MOVE`
-- **CLI** — thin MCP client; `--database` local or `--server` remote; identical commands over both
-- **Storage** — SQLite (default) or Postgres+pgvector; same kernel test suite on both
-- **Path policy** — configurable sigils and disallowed chars; NFC + case-insensitive identity, case-preserved storage
-- **Canonical paths** — API responses use slashless repo-relative paths; leading `/` accepted as root alias on input only
-
-Prior design docs in [docs/archive/](docs/archive/) may be out of date where later work supersedes them.
-
-</details>
-
-## Authentication
-
-For anything beyond single-user local use, run the **authenticating shell** — `serve --policy`. It reads declarative YAML (roles, principals, grants, key hashes, OIDC bindings), authenticates each request, and dispatches against a per-principal *guarded* kernel: read visibility narrowed, writes enforced per-path, author derived from the credential, every call audited.
-
-```yaml
-# policy.yaml
-roles:
-  editor:
-    grants:
-      - repo: notes
-        read: "**"
-        write: ["drafts/**", "inbox/**"]
-  operator:
-    grants:
-      - { repo: "*", read: "**", write: "**" }
-    destructive: true
-
-principals:
-  brendan:
-    author: Brendan Baldwin <brendan@example.com>
-    roles: [operator]
-    keys:
-      - sha256:...        # `mrplex key mint brendan --policy policy.yaml`
-  ann:
-    roles: [editor]
-    oidc: { email: ann@example.com }
-```
+| Piece          | Job                                                                                      |
+| -------------- | ---------------------------------------------------------------------------------------- |
+| **Document**   | One Markdown file with YAML frontmatter, addressed by repo-relative path                 |
+| **Version**    | Every write appends. `prev_version_id` rejects stale writers                             |
+| **Query**      | CEL over frontmatter + `$path` / `$body` / `$updated_at`; AND with FTS and semantic      |
+| **Link graph** | Derived from inline links, wikilinks, and frontmatter repo-root paths. Bound to identity |
+| **Graph**      | BFS neighborhood. *How* things connect, not only *which* match                           |
+| **Surfaces**   | CLI (`--database` or `--server`), MCP (`/mcp` or `mcp-stdio`), REST                      |
 
-```bash
-mrplex key mint brendan --policy policy.yaml
-mrplex serve --policy policy.yaml --audit audit.jsonl --port 8321 &
-curl -H "Authorization: Bearer $KEY" http://127.0.0.1:8321/repos
-```
+Two layers: a **full-trust kernel** and an optional **access-and-identity shell** (keys, OIDC, per-path grants, audit). The kernel does not pretend to be a user system. Local CLI against a database file is full-trust — possession of the file is root.
 
-Three deployment shapes — **embedded** (`serve --policy`), **launcher** (`mcp-stdio --policy`), and **fronting proxy** (`proxy --policy --upstream`). Edit the policy and `kill -HUP` to reload grants without restart. OIDC device-flow login via `mrplex login`. Full details in [docs/archive/security.md](docs/archive/security.md).
+What ships today:
+
+- Versioned Markdown store; byte-exact `frontmatter_raw` or structured `frontmatter`
+- Optimistic concurrency; delete-as-move; unified diff
+- CEL + `list()` for scalar-or-list fields; `$in` / `$has` / `$backlinks` / `$links` (membership is a filter, not a separate CLI flag)
+- FTS5 or Postgres `websearch_to_tsquery`
+- Pluggable embedder; no hook → `semantic_unavailable` instead of silent junk vectors
+- SQLite default or Postgres+pgvector; same kernel suite on both
+- NFC + case-insensitive identity, case-preserved storage
 
 ## Embeddings
 
-mrplex never calls an embedding provider itself — wire one with `--embedder` (subprocess command or HTTP URL). For local CPU embeddings:
+mrplex never calls an embedding vendor. Wire a hook:
 
 ```bash
 npm install -g @mrplex/embedder
-export MRPLEX_EMBEDDER=mrplex-embedder   # or: mrplex config set-embedder mrplex-embedder
 
 mrplex serve --unsafe --embedder mrplex-embedder
 mrplex embed backfill -r starship
-mrplex query -r starship --semantic 'distress beacon star map'   # uses env/config
-# or per-invocation in local (--database) mode:
-mrplex query -r starship --embedder mrplex-embedder --semantic 'distress beacon star map'
+mrplex -r starship query --semantic 'distress beacon star map'
+```
+
+With `--server`, configure the embedder on the host. Protocol: [packages/embedder/README.md](packages/embedder/README.md).
+
+Use semantic search to generate candidates. Then filter, hydrate, and read. Rank without inspection is how agents launder a guess into a citation.
+
+## Config & environment
+
+Resolution is always **flag → env → config file → default**. Defaults: database `./mrplex.db`, no default repo (pass `-r`).
+
+Persist defaults when you are tired of typing flags:
+
+```bash
+mrplex config set-database ./mrplex.db
+mrplex config set-repo starship
+mrplex config set-author "Brendan Baldwin <brendan@example.com>"
 ```
 
-Resolution: **flag → `MRPLEX_EMBEDDER` → `mrplex config set-embedder` → unset**. In local mode, `query --embedder` overrides for that call; with `--server`, configure the embedder on the host. See [packages/embedder/README.md](packages/embedder/README.md) for protocol details.
+Or with env vars: `MRPLEX_DATABASE`, `MRPLEX_REPO`, `MRPLEX_AUTHOR`, `MRPLEX_EMBEDDER`.
 
 ## Development
 
@@ -263,22 +185,15 @@ npm test
 npm run typecheck
 npm run lint
 npm run build
-```
 
-Seed the full dev fixture set (notes + starship):
-
-```bash
 npm run seed -- --database ./mrplex.db
 ```
 
-CI runs typecheck + lint + tests on Ubuntu & macOS × Node 20 & 22, plus Postgres+pgvector parity. If tests fail with a `NODE_MODULE_VERSION` error from `better-sqlite3`, run `npm rebuild better-sqlite3`.
-
-### Postgres locally
+CI: typecheck + lint + tests on Ubuntu and macOS × Node 20 and 22, plus Postgres+pgvector parity. If `better-sqlite3` throws `NODE_MODULE_VERSION`, run `npm rebuild better-sqlite3`.
 
 ```bash
 npm run pg:up
-export MRPLEX_DATABASE=postgres://mrplex:mrplex@localhost:5432/mrplex
-mrplex serve --unsafe
+mrplex --database postgres://mrplex:mrplex@localhost:5432/mrplex serve --unsafe
 ```
 
 ## License

package/dist/cli/main.js

@@ -13,7 +13,7 @@
  * is forwarded verbatim as a bearer for a shell fronting the server; mrplex
  * itself ignores it. serve deliberately bypasses the seam — it IS the server.
  */
-import { readFileSync, renameSync, writeFileSync } from "node:fs";
+import { existsSync, readFileSync, renameSync, writeFileSync } from "node:fs";
 import { Command, InvalidArgumentError, Option } from "commander";
 import { parseDocument as parseYamlDocument } from "yaml";
 import { openLocalClient } from "../client/local.js";
@@ -35,7 +35,7 @@ import { fileAuditSink } from "../shell/audit.js";
 import { mintKey } from "../shell/keys.js";
 import { deviceFlowLogin, loadTokenSet, saveTokenSet, } from "../shell/login.js";
 import { createOidcVerifier } from "../shell/oidc.js";
-import { PolicyError, compile, loadPolicyFile } from "../shell/policy.js";
+import { PolicyError, compile, loadPolicyFile, parsePolicy, scaffoldPolicyYaml, } from "../shell/policy.js";
 import { startProxyServer } from "../shell/proxy.js";
 import { startShellServer } from "../shell/serve.js";
 import { startShellStdio } from "../shell/stdio.js";
@@ -359,7 +359,7 @@ function renderEntitlement(principalId, e) {
             .join("");
     return (`principal: ${principalId}\n` +
         `author:    ${e.author}\n` +
-        `destructive: ${e.destructive}   impersonate: ${e.impersonate}\n` +
+        `maintain: ${e.maintain}   destructive: ${e.destructive}   impersonate: ${e.impersonate}\n` +
         `read:\n${claims(e.read)}` +
         `write:\n${claims(e.write)}`);
 }
@@ -745,9 +745,9 @@ function buildProgram() {
         }
     });
     // -------- key (policy tooling) --------
-    // `key mint` and `policy check` read/edit the policy file by definition, so
-    // they take --policy directly and never touch the serve gate (auth-shell §1
-    // "Policy tooling").
+    // `key mint`, `policy create`, and `policy check` read/edit the policy file
+    // by definition, so they take a file path directly and never touch the serve
+    // gate (auth-shell §1 "Policy tooling").
     const key = program.command("key").description("API-key tooling for the auth shell");
     key
         .command("mint <principal>")
@@ -774,6 +774,44 @@ function buildProgram() {
     });
     // -------- policy --------
     const policy = program.command("policy").description("policy-file tooling for the auth shell");
+    policy
+        .command("create [file]")
+        .description("write a starter policy.yaml (admin + maintainer)")
+        .option("--principal <id>", "maintainer principal id (MCP day-to-day)", "local")
+        .option("--force", "overwrite the file if it already exists", false)
+        .action(function (file) {
+        const localOpts = this.opts();
+        const gopts = this.optsWithGlobals();
+        try {
+            const path = file ?? "policy.yaml";
+            if (existsSync(path) && !localOpts.force) {
+                const err = new Error(`policy: ${path} already exists (pass --force to overwrite)`);
+                err.code = "cli_usage";
+                throw err;
+            }
+            const principal = localOpts.principal;
+            // Reuse the global --author / MRPLEX_AUTHOR (same meaning: write stamp).
+            const author = gopts.author ?? `${principal} <${principal}@localhost>`;
+            const text = scaffoldPolicyYaml({ principal, author });
+            // Refuse to write anything that wouldn't load — scaffold is pure, but
+            // this catches template/option bugs before the operator discovers them.
+            parsePolicy(text);
+            const tmp = `${path}.tmp.${process.pid}`;
+            writeFileSync(tmp, text);
+            renameSync(tmp, path);
+            process.stderr.write(`wrote ${path}\n` +
+                `next: mrplex key mint ${principal} --policy ${path}   # MCP\n` +
+                `      mrplex key mint admin --policy ${path}          # repo create/delete\n` +
+                `      mrplex serve --policy ${path} --database <url>\n`);
+        }
+        catch (err) {
+            if (err instanceof PolicyError) {
+                process.stderr.write(`policy: ${err.message}\n`);
+                process.exit(1);
+            }
+            reportError(err);
+        }
+    });
     policy
         .command("check [principal]")
         .description("validate a policy file; with a principal, print its effective entitlement")
@@ -1404,7 +1442,7 @@ function buildProgram() {
         .option("--once", "run startup reconciliation once, then exit (no watcher)", false)
         .option("--interval <ms>", "feed poll interval in ms (daemon; default 5000)", parsePositiveInt)
         .option("--debounce <ms>", "burst debounce in ms (daemon; default 5000)", parsePositiveInt)
-        .option("--settle <ms>", "skip files younger than this many ms (partial saves)", parsePositiveInt)
+        .option("--settle <ms>", "minimum mtime age in ms before sync touches a path (default 0)", parsePositiveInt)
         .option("--include <glob>", "include glob (default **/*.md); repeat to add more", (value, prev) => [...(prev ?? []), value])
         .option("--exclude <glob>", "exclude glob (wins over include); repeat to add more", (value, prev) => [...(prev ?? []), value])
         .option("--dry-run", "report the actions a reconciliation would take, changing nothing", false)