pi-vault-mind 0.14.2 → 0.16.3

package/CHANGELOG.md

@@ -1,5 +1,298 @@
 # Changelog
 
+## 0.16.3 / 0.6.5 — 2026-07-16
+
+### Added
+
+- **`GET /vault-mind/models` endpoint.** Auth-gated route returns pi's
+  provider/model catalog (`{ providers: ProviderInfo[] }`, plus an
+  `error` field when a `models.json` exists but fails to parse), merged
+  from the agent-dir and project-level `models.json` (project overrides
+  agent-dir for same-named providers). `src/models.ts`
+  `readModelProviders()` mirrors the plugin's `pi-config-reader.ts`
+  flattened shape (`$`-stripped `envVarName`, model `name` falls back to
+  `id`, providers sorted by name). New `VaultMindClient.getModels()`
+  (typed `VmModelsResponse`) is the client bridge so the plugin can
+  replace synchronous `models.json` disk reads with a live call. Covered
+  by `test/rest-models.test.ts` (auth, empty, parse, override, malformed).
+
+- **Extension-owned setup and configuration surfaces.** Added masked
+  provider-credential status and atomic extension-owned secret storage,
+  Local/Remote live embedding probes, complete typed setup/configuration
+  client methods, default-vault sibling preservation, and a deterministic
+  configuration E2E smoke. Added the portable seven-step Arrow setup
+  wizard and twelve-category section-save Settings UI, then ported it into
+  the Obsidian plugin behind a thin `RestConfigurationAdapter`. The plugin
+  now uses typed runtime model discovery instead of synchronous
+  `models.json` reads or writes, routes model selection through Pi RPC,
+  mounts Settings through one adapter boundary, renders retryable offline
+  recovery rather than a blank pane, and keeps plugin/extension bridge
+  credentials separate from embedding-provider credentials. Added
+  port-parity canonicalization for type re-exports, CSS/import guards,
+  intrinsic SVG icon bounds, and focused regression coverage across the
+  extension, sandbox, and plugin.
+
+### Fixed
+
+- **Un-imported `modalUrl` in `src/server.ts`** — the
+  `handleVaultMindConfig` function destructured `modalUrl` from
+  `../modal-config.js` without ever importing it. Would have crashed
+  with `TypeError: Cannot read properties of undefined (reading 'space')`
+  if the modal config branch was hit. Caught by
+  `/tmp/pvm-e2e-http-setup-test.mjs`. Import is now in place.
+
+### Documentation
+
+- **`vault-mind-config-index.md` §12 — Modal extraction question.** New
+  section documenting the open architectural question: should
+  `pi-vault-mind`'s Modal-specific code (HTTP client, sync engine,
+  `ModalProvider`, `/vm remote *` slash commands, `embedding.sync`
+  config block, Python `modal/` app) move into a separate package
+  (`pi-vault-mind-remote-modal/`) so the extension is truly
+  provider-agnostic? Three options documented (A: namespace in-tree /
+  B: extract to a package / C: server+client out, CLI stays). Smaller
+  win called out regardless of option: drop `/vm remote *` in favor of
+  `--remoteUrl` / `--remoteApiKey` flags on `/vm setup`. Awaiting
+  user decision; no work committed.
+- **§7.1 scope column.** Every `/vm *` slash command is now tagged
+  `🟢 agnostic` / `🟡 Modal-specific` / `🔵 token` so the reader can
+  see exactly which commands depend on Modal.
+
+## 0.16.2 — 2026-07-07
+
+### Changed
+
+- **`ModalEmbeddingConfig` flattened into `EmbeddingConfig`.** Modal
+  is just an OpenAI-compatible endpoint, so the `embedding.modal: {
+  baseUrl, workspace, apiToken, readToken, writeToken, model, dim,
+  fallback, sync }` wrapper is gone. All fields are now top-level on
+  `EmbeddingConfig` itself: `workspace`, `remoteApiKey`,
+  `remoteReadApiKey`, `remoteWriteApiKey`, `model`, `dim`, `fallback`
+  (now a simplified `{ enabled?: boolean }` — the `provider`
+  discriminator was already removed in 0.16.1+), and `sync`. **`apiKey`
+  is now explicitly the local-Ollama bearer** (defaults to `"ollama"`
+  in `OpenAICompatibleProvider`); the remote-endpoint token is
+  `remoteApiKey` so a single config can carry distinct local-vs-remote
+  credentials. The Modal-only blocks that remain as their own
+  inline objects are `sync` (pull-down watermark behavior) and
+  `fallback` (offline degrade policy). The `MODAL_APP_NAME` URL
+  helper stays in `src/modal-config.ts` (`modalUrl(workspace)`); the
+  Obsidian plugin keeps a small local mirror because it can't import
+  from the extension at runtime. `isModal()` is now simply
+  `!!cfg.embedding.remoteUrl`. The Obsidian plugin's
+  `provider: "ollama" | "modal" | "skip"` UI discriminator was renamed
+  to `"local" | "remote" | "skip"` (display labels stay user-friendly).
+  On save, the Modal wizard removes the old `embedding.modal` block
+  so existing configs upgrade cleanly on next `/vm setup`. The index
+  doc `vault-mind-config-index.md` §2.4 is updated; the per-setting
+  plan `vault-mind-config-plugin-surface.md` is reconciled.
+
+### Removed
+
+- **`ModalEmbeddingConfig` and `ModalFallbackConfig` types removed
+  from `src/types.ts`.** Fields are inlined on `EmbeddingConfig` (see
+  above). `FallbackConfig` (`{ enabled?: boolean }`) is the new
+  simplified fallback shape; `ModalSyncConfig` is unchanged.
+
+### Tests
+
+- 268 tests pass (no change in count — the migration was mechanical).
+  Three test files migrated from `modal: { ... }` fixtures to the
+  flat shape: `test/auth.test.ts`, `test/lance-modal.test.ts`,
+  `test/modal-config.test.ts`. The `ModalFallbackConfig["provider"]`
+  type-level regression test was removed (the type no longer exists).
+  One collateral fixture in `test/sync.test.ts`. One collateral
+  fixture in `test/vault-tools.test.ts` (the `writeConfig` helper
+  needed `model: "testmodel"` added so `resolveModel` returns the
+  right value for the test's `searchFts` assertion). The provider-cache
+  leak in `src/lance.ts`'s `resetConnection()` was fixed (the cache
+  was module-level and survived across tests).
+
+
+## 0.16.1 — 2026-07-07
+
+### Added
+
+- **`src/embedding-providers.ts`** — new store-agnostic `EmbeddingProvider`
+  interface (`embed` / `dim` / `init`) and the three concrete providers
+  (`OpenAICompatibleProvider`, `TransformersProvider`, `ModalProvider`)
+  plus a `createProvider(cfg)` factory. The vector-store layer
+  (`src/lance.ts`) now consumes providers through a 35-line
+  `LanceEmbeddingAdapter` that wraps them into LanceDB's
+  `TextEmbeddingFunction`. **Adding a new vector store (Qdrant, Pinecone,
+  pgvector, …) means writing a new adapter; adding a new embedding
+  provider means implementing the interface.** No downstream code (lance,
+  coalescer, plugin) needs to change for either.
+- **`/vm remote config fallback`** accepts `"ollama"` as a friendlier
+  alias for `"openai-compatible"` (rewritten to the enum value before
+  write). Help text and error messages updated.
+
+### Removed
+
+- **`ModalFallbackConfig.provider` enum reduced to `"openai-compatible"`.**
+  The legacy `"transformers"` value was dead code (no runtime path
+  selected it). The offline transformers path lives separately under
+  `EmbeddingConfig.useTransformers`. `/vm remote config fallback` parser
+  no longer accepts `"transformers"`. Type-level regression test
+  prevents the dead value from being added back.
+
+### Fixed
+
+- **`LanceEmbeddingFunction` now defaults `apiKey` to `"ollama"`** for
+  local endpoints, matching `AgentModelProvider`'s behavior and the
+  JSDoc contract. The previous `string | undefined` field would have
+  sent `Authorization: Bearer undefined` for any vault without an
+  explicit `apiKey`; Ollama happened to ignore the malformed header,
+  but stricter `/v1/embeddings`-compatible servers would have rejected
+  it. Pass `apiKey: null` to opt out (some local servers reject
+  `Bearer ollama` as invalid auth).
+- **`IdentityConfig.role` made optional.** It was required at the type
+  level but `mergeIdentityProfiles` always pulls `role` (and `id`) from
+  the base — config cannot rename an agent. Required type was dead.
+- **Index doc §2.6 `vaultMind.identities` reframed as "wired but
+  unenforced"** — the block is read at engine start
+  (`engine.ts:124-137` calls `mergeIdentityProfiles` and
+  `bridge.ts:100` registers the merged identity), but the registered
+  identity is never read back (`getAgentIdentity` in `bridge.ts:117`
+  has no callers). The block stays typed; future runtime enforcement
+  reuses it. New regression test in `test/bridge-identity.test.ts`
+  pins the merge contract (layer values override, empty array is
+  explicit deny, role/id always from base, undefined layer is a no-op).
+- **Config example drift closed.** `pi-vault-mind.config.example.json`
+  now shows the scaffolded `draft-context` injector instead of an
+  empty array — matches the live `src/scaffold.ts` output so a user
+  hand-rolling a config from the example doesn't silently miss a
+  built-in.
+
+## 0.16.0 — 2026-07-06
+
+### Added
+
+- **New `src/scaffold.ts`** — the single, UI-agnostic implementation of vault
+  config scaffolding (default collections/injectors, `.gitignore`), shared
+  by the CLI `/vm setup` command, the HTTP `POST /vm/setup` route, and any
+  future consumer. Robust to caller order: seeds default collections/
+  injectors whenever they're empty, not only when the config file is brand
+  new, so no future caller can silently end up with `"collections": {}`
+  again by writing embedding/vault fields before calling it.
+
+### Removed
+
+- **Legacy config/token migration dropped entirely.** `migrateLegacyState()`
+  and the `CONFIG_FILES` legacy-filename list are gone; pi-vault-mind no
+  longer rescues config/token from a vault-root `pi-vault-mind.config.json`,
+  `.pi/vault-mind.config.json`, or the global `~/.pi/agent/`. `.vault-mind/`
+  is now the only place pi-vault-mind ever reads its own config/token from —
+  no backward-compat fallback path.
+- **`/vm init` removed as a standalone command.** It only ever duplicated
+  part of `/vm setup`'s work with a deprecation notice; fully consolidated
+  onto `/vm setup`, which now also accepts `--collection <name>` (previously
+  only available via `/vm init --collection`). Every "Run /vm init first"
+  message across `commands.ts`, `settings-ui.ts`, `tools.ts`,
+  `vm-handlers.ts`, and `tools/marksman.ts` now says `/vm setup`.
+- **`POST /vm/init` removed from the HTTP API.** It duplicated a third copy
+  of the collections/injectors scaffold template (alongside the CLI's and
+  the Obsidian plugin's own copies). `POST /vm/setup` now calls the same
+  shared `scaffoldVaultConfig()` after writing embedding/vault fields, so it
+  does everything `/vm/init` did plus its own job, and its response now
+  includes `created`/`updated`/`skipped`/`collectionName`. The Obsidian
+  plugin's `VaultMindClient.init()` (only caller: the chat composer's
+  `/reindex` command) is removed too — it now calls `client.setup({})`
+  directly, since the reindex flow never used `init()`'s return value.
+
+### Fixed
+
+- **`/vm setup` never actually scaffolded default collections.** `handleInit`
+  (the scaffold logic) only populates the default `main`/`pending`/
+  `context_events` collections and the `draft-context` injector when the
+  config file doesn't exist yet — but `setupWizard` always wrote the config
+  file *first*, so `handleInit` permanently saw an existing file and took its
+  merge-safe (no-op-for-collections) branch instead. Every `/vm setup` run,
+  CLI or interactive, ended with `"collections": {}`. Reordered so
+  `handleInit` runs before `setupWizard`; verified end-to-end (fresh vault →
+  `/vm setup` → all three default collections + injector scaffolded with the
+  vault-derived file name, confirmed via direct JSONL read-back).
+- **`loadConfig` returned duplicate injector entries, causing them to fire
+  twice.** `mergeConfigLayer` concatenated `DEFAULT_CONFIG.injectors` with
+  the vault's own config-file injectors with zero deduplication. Since every
+  vault's config now always has its own `draft-context` injector (scaffolded
+  by `/vm setup`), `loadConfig()` returned it twice — and `events.ts`
+  processes every injector's regex match independently, so a matching note
+  would trigger the injector's capture/artifact-sync logic twice per prompt.
+  Fixed by deduplicating by `name` in the merge (layer's entry wins on a
+  collision), mirroring the existing dedupe pattern in `auditConfig`.
+  Verified via `loadConfig` on a freshly-scaffolded vault: exactly one
+  `draft-context` injector, not two.
+
+## 0.15.0 / 0.6.4 — 2026-07-06
+
+### Added
+
+- **`/vm setup` now scaffolds `.vault-mind/.pi/model-router.json`** with tuned
+  primary/fallback model defaults, extracted verbatim from the Obsidian
+  plugin's `scaffoldModelRouterConfig`/`defaultModelRouterChoices` into a new
+  `src/model-router.ts` — previously only the plugin's init flow produced
+  this file, so a CLI-only setup got no model-router config at all.
+- **`scripts/setup-vault-pi.sh` now installs `pi-model-discovery` and
+  `pi-model-router`** alongside `pi-vault-mind`/`pi-context`, matching the
+  plugin's install list (previously missing both, so a CLI-only setup never
+  got model discovery or routing installed).
+
+### Changed
+
+- **Vault isolation** — all pi-vault-mind config, token, and written artifacts now live under `<vault>/.vault-mind/`. The global `~/.pi/agent/vault-mind.config.json` layer is dropped; config is single-layer, vault-local. Legacy config/token is auto-migrated on first session_start (idempotent).
+- **Config consolidation** — `loadConfig` reads only `<vault>/.vault-mind/vault-mind.config.json`; `findConfig` returns a single project path. `getGlobalConfigPath`/`getGlobalConfigDir` removed.
+- **Token env** — `PVM_API_TOKEN` dotenv file moved from `~/.pi/agent/vault-mind.env` to `<vault>/.vault-mind/vault-mind.env`. `auth.ts`, `modal-config.ts`, `commands.ts`, and `settings-ui.ts` all resolve via `getTokenEnvPath(cwd)`.
+- **Framework agent dir** — launch scripts now use `<vault>/.vault-mind/.pi/agent` (relocated from `<vault>/.pi/agent`). `resolveAgentDir(cwd)` honors `PI_CODING_AGENT_DIR` via the SDK.
+- **ACM gate** — auto-ACM now fires only when `isVaultMindReady` is true (config exists + embedding configured), not on any vault with a global config.
+- **Configurable file paths** — `graph.canvasPath`, `files.queue`, `files.serverState`, `files.manualDispatch` in config; `resolveVaultFile` resolves them with defaults under `.vault-mind/`.
+- **Per-vault Modal collection prefix** — `vaults.default.collectionPrefix` (auto-derived from vault slug) isolates server-side Modal tables across vaults sharing a deployment. Local LanceDB tables stay unprefixed.
+- **`/vm setup` consolidation** — now runs `migrateLegacyState` → `setupWizard` → merge-safe scaffold → optional `runPersonalize`. `/vm init` is deprecated (still works, shows notice).
+- **`/vm doctor`** — new command reporting resolved paths, readiness, token source, model-router status, and PASS/WARN for vault isolation.
+- **`GET /vault-mind/paths`** — new auth-gated HTTP endpoint returning absolute + vault-relative paths with `exists` flags.
+- **gitignore** — `.vault-mind/*` + `!.vault-mind/vault-mind.config.json` keeps the hand-tuned config tracked while ignoring local state.
+
+### Fixed
+
+- **Config write-target bugs** — `vm_configure`, `discover_schema`, and the
+  `POST /vm/init` HTTP handler's scaffold-if-missing path all still wrote to
+  the vault-root `pi-vault-mind.config.json` (a path `loadConfig` no longer
+  reads) instead of `.vault-mind/vault-mind.config.json`. Fixed to use
+  `getConfigPath(cwd)` uniformly.
+- **Stale user/dev-facing config paths** — the "no vaults configured" notify
+  string (`commands.ts`), the `discover_schema` tool's prompt guidelines, and
+  a `server.ts` doc-comment all still told the reader to edit the retired
+  `pi-vault-mind.config.json` instead of `.vault-mind/vault-mind.config.json`.
+- **`/vm setup` CLI-mode arg parsing** — when `--vault` was the first flag
+  (exactly the form documented in `CLI_ONLY_WALKTHROUGH.md`), its leading
+  `--` was never stripped by the tokenizer, so `cliArgs.vault` was silently
+  never set and `vaults.default.path` was left empty. Fixed the split to
+  normalize the leading `--` before tokenizing.
+
+### Removed
+
+- Dead `events.ts` hidden `/acm` sendMessage path (never enabled compaction).
+- `handleVaultMindConfig` HTTP response no longer splits `.global`/`.project` (single `config` key).
+
+### Plugin (`packages/obsidian/`)
+
+- **Isolation parity with the extension** — `resolvePluginAgentDir(vaultPath)`
+  now resolves `<vault>/.vault-mind/.pi/agent`, threaded through `main.ts`
+  (init flow, config paths, hasExtensions checks, token/1Password flow),
+  `model-router-config.ts` (models.json + model-router.json paths),
+  `pi-config-reader.ts`, `chat-tab.ts`, `status-tab.ts`, and `panel.ts`.
+- **`.vault-mind/vault-mind.config.json`** — `config.ts`'s `readExtensionConfig`
+  now reads the single vault-scoped config (previously read the removed
+  global `~/.pi/agent/vault-mind.config.json` + vault-root
+  `pi-vault-mind.config.json`).
+- **`.env.1pass` relocated to `.vault-mind/.env.1pass`**, with `--op-env`
+  wired into every pi launch site (`PiConnection` in `chat-tab.ts`, the TUI
+  launcher in `status-tab.ts`, and `scripts/vault-pi.sh`/`setup-vault-pi.sh`) —
+  gated on the file existing AND `pi-1password` being installed, since
+  `--op-env` is a flag `pi-1password` registers; passing it unconditionally
+  would make `pi` reject the whole invocation with "Unknown option" when the
+  extension isn't installed.
+
 ## 0.5.1 — 2026-06-28
 
 ### Fixed

package/README.md

@@ -260,9 +260,9 @@ Edit `pi-vault-mind.config.json` to match your domain:
   "vaultMind": {
     "dataDir": ".lancedb",
     "embedding": {
-      "provider": "transformers",
-      "ollamaModel": "embeddinggemma",
-      "ollamaHost": "http://127.0.0.1:11434"
+      "remoteUrl": "http://127.0.0.1:11434",
+      "model": "embeddinggemma",
+      "dim": 768
     },
     "ftsEnabled": true,
     "graph": { "enabled": true, "canvasSync": false }
@@ -347,7 +347,6 @@ Edit `pi-vault-mind.config.json` to match your domain:
 | ------------------------------ | -------------------------------------------------------- |
 | `/vm help`                   | Show usage help                                          |
 | `/vm setup`                  | **Interactive global config wizard** (vault, embedding) |
-| `/vm init`                   | Scaffold project config + collections                    |
 | `/vm validate`               | Health check LanceDB, config, and all collection paths   |
 | `/vm approve [collection]`   | Batch-review pending entries                              |
 | `/vm settings`               | Open interactive settings dashboard                       |
@@ -358,7 +357,7 @@ Edit `pi-vault-mind.config.json` to match your domain:
 | `/vm injector create`        | Interactive wizard to create a new injector                |
 | `/vm context enable \| disable \| status` | Manage pi-context integration              |
 | `/vm embedding status \| use \| model \| models \| pull` | Manage embedding provider |
-| `/vm modal status \| config \| sync \| jobs \| migrate` | Manage Modal embedding + vector sync |
+| `/vm remote status \| config \| sync \| jobs \| migrate` | Manage remote embedding + vector sync |
 | `/vm watcher start \| stop \| status` | Manage the passive file watcher                  |
 | `/vm server status` | Show HTTP server status, port, and uptime |
 
@@ -375,6 +374,7 @@ Edit `pi-vault-mind.config.json` to match your domain:
 | [docs/getting-started/NEW_VAULT_WALKTHROUGH.md](docs/getting-started/NEW_VAULT_WALKTHROUGH.md) | **Fresh Mac → new vault → Modal → sync → verified**, one linear path (incl. multi-vault storage isolation) |
 | [docs/getting-started/INSTALL.md](docs/getting-started/INSTALL.md) | **Canonical install playbook** — all 5 layers (pi ext, skills, Obsidian, config, external CLIs) |
 | [docs/getting-started/GETTING_STARTED.md](docs/getting-started/GETTING_STARTED.md) | End-to-end workflow + daily "drop & forget" usage |
+| [docs/getting-started/CLI_ONLY_WALKTHROUGH.md](docs/getting-started/CLI_ONLY_WALKTHROUGH.md) | Same setup + daily commands, entirely CLI — no Obsidian plugin required |
 | [docs/getting-started/WALKTHROUGH_PROMPT.md](docs/getting-started/WALKTHROUGH_PROMPT.md) | Paste-into-pi guided setup with checkpoints between phases |
 
 ### Architecture & design

package/agents/broadcaster.agent.md

@@ -2,7 +2,7 @@
 type: BroadcasterAgent
 role: broadcaster
 capabilities: [read, write, edit]
-allowed_tools: [read, write, edit, vm_search, vm_fts_search]
+allowed_tools: [read, write, edit, vm_search]
 write_collections: []
 can_publish: false
 llm_provider: ollama
@@ -16,7 +16,7 @@ and external-facing content from vault knowledge.
 ## Capability Boundary
 
 - **MAY**: read files, write files, edit files
-- **MAY**: search collections via vm_search, vm_fts_search
+- **MAY**: search collections via vm_search
 - **MUST NOT**: run bash commands, grep, or find
 - **MUST NOT**: write to the durable knowledge store
 - **MUST NOT**: publish or spawn sub-agents

package/agents/heavy-lifter.agent.md

@@ -2,7 +2,7 @@
 type: HeavyLifterAgent
 role: heavy-lifter
 capabilities: [read, write, edit, grep, find, ls, bash]
-allowed_tools: [read, write, edit, grep, find, ls, bash, vm_search, vm_fts_search]
+allowed_tools: [read, write, edit, grep, find, ls, bash, vm_search]
 write_collections: []
 can_publish: false
 llm_provider: ollama

package/agents/main.agent.md

@@ -0,0 +1,29 @@
+---
+type: ManagerAgent
+role: main
+capabilities: [read, write, edit, bash, grep, find, ls]
+allowed_tools: [read, write, edit, bash, grep, find, ls, vm_search, vm_query, vm_append, vm_stats, vm_status, vm_describe, vm_configure]
+write_collections: []
+can_publish: false
+llm_provider: ollama
+---
+
+# Main Agent
+
+Primary interactive agent for pi-vault-mind. The default role used by the
+Obsidian plugin's chat interface and the CLI `/vm chat` command.
+
+## Capability Boundary
+
+- **MAY**: all 7 built-in tools (read, write, edit, bash, grep, find, ls)
+- **MAY**: vm_search, vm_query (read paths)
+- **MAY**: vm_append, vm_stats, vm_status, vm_describe, vm_configure (write/config paths)
+- **MAY**: read from the "main" collection
+- **MUST NOT**: write to collections, publish, or access arbitrary vault paths
+
+## Tool Calling Convention
+
+When calling pi-vault-mind tools, always include your role:
+```
+vm_search({ query: "...", role: "main" })
+```

package/agents/manager.agent.md

@@ -2,7 +2,7 @@
 type: ManagerAgent
 role: manager
 capabilities: [read, write, edit, grep, find, ls]
-allowed_tools: [read, write, edit, grep, find, ls, vm_search, vm_fts_search, vm_append, vm_sync, vm_promote, vm_query, vm_configure, vm_describe, vm_stats, vm_export, vm_ingest, vm_status, vm_graph_query]
+allowed_tools: [read, write, edit, grep, find, ls, vm_search, vm_append, vm_sync, vm_promote, vm_query, vm_configure, vm_describe, vm_stats, vm_export, vm_ingest, vm_status]
 write_collections: [main, research, presentations]
 can_publish: true
 llm_provider: ollama

package/agents/miner.agent.md

@@ -2,7 +2,7 @@
 type: MinerAgent
 role: miner
 capabilities: [read, write, edit, grep, find, ls]
-allowed_tools: [read, write, edit, grep, find, ls, vm_search, vm_fts_search, vm_append]
+allowed_tools: [read, write, edit, grep, find, ls, vm_search, vm_append]
 write_collections: [main, research]
 can_publish: false
 llm_provider: ollama