mikser-io 8.3.6 → 8.3.9

package/10.0-PLAN.md

@@ -0,0 +1,182 @@
+# Mikser 10.0 — Planning (stub)
+
+**The shift in one line:** scale moves from "make one mikser hold more" to "compose more miksers." A new coordination layer ships as `party-mikser-io` — separate package, separate ADR series, separate release cadence. `mikser-io` itself stays single-instance-pure.
+
+This is a v10 placeholder — direction-setting, not a finished design. The hard pieces (cross-mikser refs, discovery, partition behavior) get worked out as `party-mikser-io` takes shape. This file gets dropped at release; the surviving decisions become ADRs.
+
+## The gate: v9 has to pay out first
+
+**No v10 work begins until v9 is incorporated into our own daily work and the workflow pays out.** Not "v9 has shipped." Not "v9 has been tested." **Used, daily, for our own real work, until the format-liberation pitch is something we'd defend with our own time.**
+
+This is the strongest gate in the roadmap. It exists because the v10 architecture is more interesting to build than the v9 documentation is to write, and that asymmetry kills projects with strong substrates. The discipline is to refuse v10 design work — even thinking about it as an escape — until v9 has earned the next layer through real use.
+
+What "pays out" means operationally — concrete signals to watch:
+
+| signal | reading |
+|---|---|
+| New work naturally flows into mikser, not into "I should mikser-ize this later" | ✅ working |
+| You stop wanting to switch back to your previous setup | ✅ working |
+| The format-liberation flows (PDF in → CSV out, CSV in → HTML out) are things you actually run, not just demos | ✅ working |
+| The MCP agent integration makes daily work faster, not just possible | ✅ working |
+| Watch-mode + incremental rebuild holds up under your real editing patterns | ✅ working |
+| You stop writing helper scripts that work around mikser instead of through it | ✅ working |
+| Friends/colleagues see what you have running and ask how to use it themselves | ✅ working — the strongest signal |
+
+Failure signals — any of these means v9 hasn't paid out yet, fix before v10:
+
+| signal | reading |
+|---|---|
+| You manually run `mikser --clear` more than weekly | ❌ incremental rebuild has rough edges |
+| You keep editing files outside the mikser folder because watch-mode is slow / breaks | ❌ watch-mode regression somewhere |
+| Specific plugins break in ways that make you avoid them | ❌ plugin needs work, not v10 work |
+| You stop using MCP integration because grep is faster | ❌ MCP surface isn't carrying its weight |
+| You compose pipelines outside mikser (one-off scripts) because composing in mikser is awkward | ❌ recipe documentation or substrate ergonomics gap |
+| You don't find yourself reaching for it for new tasks | ❌ the substrate isn't actually changing how you work |
+
+When the working signals dominate over a sustained period of real use, v10 design starts. Not before.
+
+## Why v10 (not 9.x)
+
+The 9.0 plan deliberately deferred mikser-to-mikser composition to v10. The reasoning is in `9.0-PLAN.md` under "Out of scope":
+
+> Cross-instance concerns (discovery, peer identity, routing, cross-mikser refs, watch propagation, liveness, partition behavior, peer auth) belong to a distributed-systems substrate that iterates at "we hit a production partition bug" speed, not at engine-substrate speed. Folding any of it into `mikser-io` would re-fail ADR-0006 test #5 (release cadence) the moment it ships.
+
+v10 is the version where that substrate ships, alongside the small `mikser-io` additions that make it possible.
+
+Until then: one folder, one mikser, domain-sized catalog. That's the v9 design center, and it earns the v10 work by staying small.
+
+## The split
+
+```
+mikser-io           → single-instance substrate (unchanged from 9.0)
+party-mikser-io     → coordination layer between mikser instances
+```
+
+The repo-split check is one line: **does this code reference anything outside this mikser's `runtime`? If yes → wrong repo.** That makes ADR-0006 test #1 (substrate?) concrete for distributed code, which is otherwise ambiguous.
+
+| concern | lives in | why |
+|---|---|---|
+| Catalog / refs / manifest / journal | `mikser-io` | every mikser needs this |
+| Source plugins, render dispatch, lifecycle | `mikser-io` | same |
+| Discovery (how A finds B) | `party-mikser-io` | mikser-io never needs to know peers exist |
+| Identity (`mikser.name` as a stable handle) | `party-mikser-io` | mikser-io has no notion of "self" beyond `runtime` |
+| Routing (which mikser owns entity X) | `party-mikser-io` | local catalog queries don't ask this |
+| Cross-mikser refs (`$author` in A → entity in B) | `party-mikser-io` | refs.js stays local-graph-pure |
+| Watch propagation across instances | `party-mikser-io` | chokidar covers one folder; cross-mikser is a different transport |
+| Liveness / health / partition behavior | `party-mikser-io` | distributed-systems posture, not file-engine posture |
+| Auth / trust between peers | `party-mikser-io` | engine auth is per-endpoint; peer auth is its own model |
+
+## The metaphor
+
+A "party" — voluntary attendance, each guest brings what they own, the host knows who's there but nobody's centrally in charge.
+
+More accurate than:
+
+- **Cluster** — implies homogeneity (every node same code, same role). A photos mikser and an invoices mikser are different roles.
+- **Mesh** — implies n-to-n by default. Real parties talk to small explicit peer sets.
+- **Federation** — implies central authority or shared schema. Each mikser owns its catalog and exposes whatever shape it wants.
+
+Reads correctly at the call site:
+
+```js
+import { join } from 'party-mikser-io'
+
+await join({
+    as: 'invoices',
+    peers: [
+        { name: 'documents', url: 'http://localhost:8081' },
+        { name: 'photos',    url: 'http://localhost:8082' },
+    ],
+})
+```
+
+## What lands in `mikser-io` to support this
+
+These are the only `mikser-io` 10.0 changes — small, surgical, and they don't introduce peer-awareness into the engine. They expose information party-mikser-io needs in order to coordinate.
+
+### `mikser-io-api` surface additions
+
+Three new endpoints. None of them know about peers; they just expose what a peer would need to read.
+
+| endpoint | purpose | rough LOC |
+|---|---|---|
+| `GET /api/identity` | mikser name (from config) + version + capabilities (plugin list + their versions) | ~20 |
+| `GET /api/checkpoint` | current catalog snapshot id (for partition-recovery sync) | ~30 |
+| `GET /api/refs/:id?direction=in|out` | inbound or outbound refs for an entity | ~40 |
+
+The `subscribe` endpoint also gains a `?since=<checkpoint>` query param for replay-on-reconnect. ~20 LOC.
+
+Total: ~110 LOC of additive endpoints. Nothing existing changes. Each one is independently testable.
+
+### `mikser.config.js` gets a `name` field
+
+```js
+export default {
+    name: 'invoices',   // stable identifier; used by /api/identity and party.join({as})
+    plugins: [...],
+}
+```
+
+Engine reads it, exposes via `runtime.name`. Default falls back to the working folder's basename. That's it — `mikser-io` doesn't act on it; party-mikser-io does.
+
+## What `party-mikser-io` ships in 10.0
+
+Not designing this in detail yet — the design happens as the package gets built. The shape at this stage is:
+
+1. **`join(config)`** — handshake with each declared peer via `/api/identity`. Verifies peer is reachable and reports name/capabilities.
+2. **`runtime.options.party.peers`** — map of peer name → metadata. Available to plugins after join completes.
+3. **`mikser-io-source-party`** — source plugin: mounts a peer's `/api` SSE feed as a local collection. Live-reload via the existing subscribe semantics.
+4. **Cross-mikser ref resolution** — `$author@documents` syntax (entity id + peer name). Resolution proxies through the source-party plugin's mirrored cache. **Design call deferred** to first implementation pass.
+5. **Liveness checks** — periodic `/api/checkpoint` poll per peer. Stale peers logged but don't break the build (their entities go cold; resync on reconnect).
+
+## Open questions (working proposals at v10 design time)
+
+1. **Cross-mikser ref shape — mirror or proxy?**
+   - (a) Mirror: source-party plugin pulls all peer entities into local catalog. Refs resolve locally. Cost: memory grows with peer corpus size.
+   - (b) Proxy: refs resolve via HTTP at lookup time. Cost: latency on every render that touches a cross-mikser ref.
+   - Working proposal: **mirror with TTL** — pull peer entities lazily on first ref, cache with checkpoint-based invalidation. Worst-case memory is bounded by what's actually referenced.
+
+2. **Discovery mechanism.**
+   - Config-pinned peer URLs (the snippet above) for v10. Anything more (mDNS, registry, DNS-SD) lives in 10.x or 11.0 after we see what real deployments need. Working proposal: **explicit URLs only for v10.**
+
+3. **Authentication between peers.**
+   - The api plugin already supports per-endpoint tokens. Reuse that: each peer in `join()` config carries a token. Working proposal: **bearer token per peer, configured in `join()`.**
+
+4. **What if two peers claim entities with the same id?**
+   - The fully-qualified form is `id@peer-name`. Local entities have no `@`. Working proposal: **namespacing by peer name avoids collision entirely** — cross-mikser refs always carry the peer name.
+
+5. **What about cycles?**
+   - A subscribes to B, B subscribes to A. Working proposal: **explicitly allowed**. Each side mirrors the other; the checkpoint mechanism prevents replay loops.
+
+## Out of scope for 10.0
+
+- **Write-across-peers** — `party.create(entity, { in: 'invoices' })` from peer A creating an entity in peer B's catalog. Probably 10.x or 11.0. Single-direction read is enough to validate the architecture.
+- **Distributed consensus / conflict resolution** — each peer owns its catalog, period. Nobody writes to anyone else's.
+- **Replication / HA** — that's a different problem. A party isn't a cluster; if a peer goes down, its entities go cold, not get failed-over.
+- **Plugin marketplace / discovery service** — same answer as 9.0. Not building this.
+
+## ADR series
+
+`party-mikser-io` gets its own decision log under `party-mikser-io/documentation/decisions/`, starting from 0001. Not extending the `mikser-io` ADR series — keeps the substrate test (#1) clean. `mikser-io` ADRs cover the single-instance engine; `party-mikser-io` ADRs cover the coordination plane.
+
+Cross-references between the two ADR series are fine; the boundary stays at "does this code reference anything outside this mikser's `runtime`?"
+
+## Concretely, what work this is (rough order)
+
+1. **`mikser-io` engine** — add `runtime.name` (read from `mikser.config.js`). One file change.
+2. **`mikser-io-api`** — add `/api/identity`, `/api/checkpoint`, `/api/refs/:id`, and `?since=` on subscribe. ~110 LOC.
+3. **`party-mikser-io`** — new repo. `join()`, peer registry, liveness loop. The boring substrate of the coordination layer. ~400 LOC.
+4. **`mikser-io-source-party`** — new repo (or sub-package). Source plugin that mounts a peer's catalog. The mirror cache + checkpoint invalidation logic. ~300 LOC.
+5. **Cross-mikser ref resolution** — extends `mikser-io-source-party`. Parses `id@peer` syntax in `$`-refs and resolves through the mirror. ~150 LOC.
+6. **Recipe documentation** — README composition examples: PDF mikser + CSV mikser + HTML mikser, piped together. The example that proves the v10 pitch works end-to-end.
+7. **`mikser-io-example-party`** — new example repo. Three miksers in one repo (subfolders), each with its own config, demonstrating the full composition.
+
+Items 1-2 ship in `mikser-io` 10.0 itself. Items 3-7 are the v10 ecosystem release.
+
+## What this earns us, when it lands
+
+The 9.0 pitch was "file-based knowledge substrate with AI superpowers." v10 extends that with one word: **composable**.
+
+> A file-based knowledge substrate with AI superpowers, composable across instances.
+
+The PDF-in-CSV-out-HTML-elsewhere story works as separate miksers connected through a party — not as one process trying to be everything. That's the architecture the substrate was designed for; v10 just makes it real.

package/9.0-PLAN.md

@@ -0,0 +1,438 @@
+# Mikser 9.0 — Planning
+
+**The shift in one line:** `mikser-io` moves from "AI-native SSG" to "file-based knowledge substrate with AI superpowers." Layouts (and the other rendering plugins currently bundled in `src/plugins/`) move out to their own packages. A new AI substrate plugin family (`mikser-io-transformers` and friends) registers inference pipelines that consumer plugins compose against.
+
+This is a planning document, not an ADR. The ADR(s) land after the move so the wording reflects what actually shipped. This file gets dropped at release.
+
+## Why
+
+The OCR + structured-extraction conversation forced the question: when "drop a folder of PDFs and ask an AI agent about them" works as a primary use case, is layouts still in the substrate's critical path? No. SSG is one composition of the substrate, not the substrate itself.
+
+Re-running ADR-0006's five-test against `layouts`:
+
+| test | result |
+|---|---|
+| 1. Substrate? Engine genuinely needs this | **No** — document/knowledge consumers don't render layouts |
+| 2. Strengthens strategy? Load-bearing for positioning | **Was. Isn't post-shift.** |
+| 3. God-plugin check? Engine becoming kitchen sink | **Yes** — ~900 LOC of pagination + sidecar + sitemap on top of dispatch |
+| 4. Composability? Could ship external | **Yes** — only uses public surfaces |
+| 5. Release cadence? Same as engine | **No** — iterates at SSG-feature cadence |
+
+Three fails. Test 2 became a fail when positioning shifted. The architecture doesn't believe in layouts-in-core anymore; only inertia does.
+
+## What stays in the engine
+
+`src/plugins/` slims from 14 directories to 5:
+
+```
+documents · files · front-matter · yaml · json
+```
+
+These are entity-sourcing primitives — any consumer reading from a working folder needs them. Plus the substrate code (`runtime.js`, `engine.js`, `lifecycle.js`, `catalog.js`, `refs.js`, `manifest.js`, `journal.js`, `database/`, `source.js`, `subscriptions.js`, `track.js`, `render.js`/`postprocess.js` as dispatch shims — the actual rendering lives in plugins).
+
+## What moves out
+
+Each becomes its own package with its own version cadence:
+
+| package | what moves | approx. LOC |
+|---|---|---|
+| `mikser-io-layouts` | layout matching + pagination + sidecar protocol | ~900 |
+| `mikser-io-assets` | preset processing + binary pipelines | ~600 |
+| `mikser-io-resources` | external resource fetching | ~400 |
+| `mikser-io-preview` | in-memory render cache + `GET /preview/` | ~250 |
+| `mikser-io-data` | JSON catalog exports | ~150 |
+| `mikser-io-observer` | HTTP source polling | ~200 |
+| `mikser-io-mapper` | config-driven field rewrites | ~100 |
+| `mikser-io-validator` | custom validation hooks | ~100 |
+| `mikser-io-commands` | cron jobs | ~150 |
+| `mikser-io-shares` | multi-output replication | ~200 |
+
+Engine LOC drops ~40%. None of these plugins is wrong to ship — they're just not substrate.
+
+## New: AI substrate
+
+The substrate exposes a verb-level pipelines surface:
+
+```js
+runtime.options.pipelines = {
+    embed:      (text, opts?)                  => Promise<number[]>,
+    imageEmbed: (imageBuffer, opts?)           => Promise<number[]>,
+    ocr:        (input, opts?)                 => Promise<string>,
+    extract:    (input, { schema, prompt }, opts?) => Promise<unknown>,
+    classify:   (text, labels, opts?)          => Promise<{ label, score }[]>,
+    rerank:     (query, docs, opts?)           => Promise<{ idx, score }[]>,
+    summarize:  (text, opts?)                  => Promise<string>,
+    transcribe: (audioBuffer, opts?)           => Promise<string>,
+    translate:  (text, { from, to }, opts?)    => Promise<string>,
+}
+```
+
+All pipeline functions accept `{ signal }` so consumer plugins can pass through the cycle's abort signal. Watch-mode cancellations short-circuit pending inference cleanly.
+
+### Provider plugins
+
+Each provider plugin registers only the pipelines it can support:
+
+| plugin | typical pipelines | runtime requirement |
+|---|---|---|
+| `mikser-io-transformers` | embed, imageEmbed, ocr, extract, classify, rerank, summarize, transcribe | `@xenova/transformers`; first-run model download |
+| `mikser-io-openai` | embed, extract (via JSON mode), classify, summarize, transcribe | `OPENAI_API_KEY` |
+| `mikser-io-ollama` | embed, summarize (depends on pulled models) | ollama daemon |
+| `mikser-io-anthropic` | extract, classify, summarize | `ANTHROPIC_API_KEY` |
+
+Consumer plugins gate on what's registered:
+
+```js
+onLoaded(() => {
+    if (!runtime.options.pipelines?.ocr) {
+        logger.warn(
+            'mikser-io-ocr needs an ocr pipeline — install mikser-io-transformers ' +
+            '(or another provider that registers ocr).'
+        )
+        return
+    }
+    // wire up
+})
+```
+
+Same gate pattern the api / preview / mcp plugins already use against `runtime.options.app` / `runtime.options.mcp`.
+
+## New consumer plugins
+
+| package | needs pipeline | what it does |
+|---|---|---|
+| `mikser-io-vector` | embed | semantic search over text (refactored to depend on substrate) |
+| `mikser-io-ocr` | ocr | populate `entity.content` from PDF/image source files |
+| `mikser-io-extract` | extract | schema-driven structured extraction into `entity.meta` |
+| `mikser-io-image-search` | imageEmbed | CLIP-based image search; text query → matching images |
+| `mikser-io-rerank` | rerank | post-process vector search results with cross-encoder |
+| `mikser-io-summarize` | summarize | auto-populate `meta.summary` when authors don't |
+| `mikser-io-classify` | classify | zero-shot tag suggestions per entity create/update |
+| `mikser-io-transcribe` | transcribe | podcasts/videos → searchable transcripts |
+| `mikser-io-translate` | translate | multilingual content |
+
+Each is small — 100-400 LOC. The substrate handles model lifecycle (download, cache, progress, dimension introspection); the consumer handles the entity-lifecycle integration.
+
+## Format liberation: source ↔ render symmetry
+
+Mikser already has rendering plugins that emit entities AS specific formats (`render-hbs`, `render-eta`, `render-liquid`, `render-markdown`, `render-file`). The 9.0 lens reveals the missing other half: matching source plugins that parse FROM specific formats INTO entities. The substrate becomes a **format translation layer** — anything in, anything out, all composing on the same catalog.
+
+### The symmetry
+
+| source plugin (format → entities) | render plugin (entities → format) | format |
+|---|---|---|
+| `documents` (md/html/yml/json) | `render-hbs`, `render-eta`, `render-liquid`, `render-markdown` | text documents |
+| `files` (any binary) | `post-pdf`, `post-mjml` | binary outputs |
+| `mikser-io-csv` *(new)* | `mikser-io-render-csv` *(new)* | CSV |
+| existing (api plugin's data exports) | `mikser-io-data` *(extracted in 9.0)* | JSON snapshots |
+| `mikser-io-ocr` + `mikser-io-extract` (PDF/image → entities) | `mikser-io-render-pdf` *(future)* | PDF documents |
+| future: `mikser-io-ical`, `mikser-io-vcard`, `mikser-io-rss` | future: matching renderers | calendars, contacts, feeds |
+
+The substrate doesn't care which side a format lives on. The catalog's job is to hold queryable entities; source plugins fill it from whatever the world produces; render plugins emit whatever the world consumes.
+
+### The round-trip that demonstrates it
+
+A folder of scanned PDF invoices walks through the entire stack and lands as a CSV for the accountant — without anyone writing pipeline code:
+
+```
+PDFs in documents/invoices/
+    ↓ documents plugin             (file → entity, raw)
+    ↓ mikser-io-ocr                (OCR pipeline populates entity.content)
+    ↓ mikser-io-extract            (schema-driven, populates entity.meta)
+        → catalog now holds typed invoice entities
+    ↓ mikser-io-render-csv         (entities → rows in invoices.csv)
+    + mikser-io-vector             (semantic search across invoices)
+    + mikser-io-mcp                (agent answers "which Q1 vendors charged us most")
+    + mikser-io-api                (HTTP /api/public/entities/invoices)
+
+Each individual file → live-reload on watch-mode
+A schema change → re-extraction on next cycle
+A new PDF dropped in → automatic ingest + index + export
+```
+
+Edit a PDF, save it, the CSV regenerates with the corrected row a couple seconds later. The accountant pulls `invoices.csv` from a synced folder. The AI agent answers questions. The API exposes the data. **All from the same canonical source — the PDFs on disk.**
+
+### Why this is a positioning shift, not just a feature list
+
+Other tools trap your data in their source format:
+
+- **Notion** — your data lives in their proprietary database tables; export is a deliberate, lossy operation
+- **Airtable** — same shape with a different paint job
+- **Glean** — indexes your data but you can't re-emit it in another format cleanly
+- **OneDrive / Dropbox / Box** — store files but don't make their content queryable
+
+Mikser flips this. Source format is where data enters; **what shape it takes after that is the user's call.** PDF in → CSV out. CSV in → HTML out. Markdown in → semantic embeddings out. Anything in → anything out, with watch-mode keeping every output current as the inputs change.
+
+The 9.0 plugin set turns this from architectural potential into a shipped product. With `csv`, `render-csv`, `ocr`, `extract`, `transformers`, `vector`, and the existing render-* family, mikser is a real "knowledge format translation substrate" rather than "an SSG that has some AI hooks."
+
+### What lands in 9.0 specifically
+
+| package | role | LOC est. |
+|---|---|---|
+| `mikser-io-csv` | source: CSV file → row entities (one entity per row; `idColumn` config for stable ids) | ~250 |
+| `mikser-io-render-csv` | render: entities → CSV file (RFC 4180 escaping; column ordering from sidecar) | ~150 |
+
+Future render/source pairs (post-9.0, demand-driven): `render-pdf`, `render-rss`, `render-ical`, `ical` source, `vcard` source. Each ~150-300 LOC. The substrate carries them.
+
+## Schemas plugin pulls real weight here
+
+`mikser-io-schemas` has been sitting idle. Under the 9.0 shape it becomes load-bearing:
+
+- Declares entity shapes via zod
+- `extract: true` flag per schema turns the declaration into an auto-extraction target
+- `mikser-io-extract` walks new entities, looks up the schema for the entity's `meta.layout`, calls `runtime.options.pipelines.extract(input, { schema })`, validates the result, populates `entity.meta`
+- Validated structured data becomes queryable through standard indexed columns
+
+Drop a folder of PDF invoices → schema declares `{vendor, date, items, total}` → catalog has fully-typed invoice rows → `findEntities({'meta.layout': 'invoice', 'meta.total': {$gt: 5000}})` returns them.
+
+The schemas plugin also needs the lifecycle-ordering bug fixed before this lands (currently `onValidate` fires before `front-matter` populates `meta`). Separate but related work.
+
+## Recipe compositions
+
+**SSG** (today's flagship — same shape, different distribution):
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'layouts',
+    'render-hbs', 'render-markdown', 'render-href',
+    'assets', 'resources', 'preview',
+    'data',
+]
+```
+
+**Document management system**:
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'transformers', 'ocr', 'extract',
+    'schemas',
+    'vector',
+    'api', 'mcp',
+]
+```
+
+**Personal knowledge management**:
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'transformers', 'ocr', 'extract', 'schemas',
+    'vector',
+    'mcp',
+]
+```
+
+No HTTP — Claude Desktop talks MCP directly. An Obsidian competitor in ~8 plugins.
+
+**Photo library with semantic search**:
+
+```js
+plugins: [
+    'files',
+    'transformers', 'image-search',
+    'api', 'mcp',
+]
+```
+
+No `documents` even. Just files + image embeddings + access.
+
+**Tabular data workflow** (the round-trip from the format-liberation section):
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'csv',                                       // CSV files → row entities
+    'transformers', 'ocr', 'extract',            // PDFs → row entities via schema
+    'schemas',
+    'render-csv', 'render-hbs',                  // entities → CSV + HTML
+    'data',
+    'api', 'mcp',
+]
+```
+
+PDF invoices and CSV exports land as the same entity shape. Editor renders a dashboard HTML page; accountant downloads regenerated CSV; agent answers semantic questions. One source of truth, three output formats.
+
+**Newsletter platform**:
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'layouts', 'render-hbs',
+    'post-mjml', 'post-mailchimp',
+    'data',
+]
+```
+
+Each composition is a coherent product. None of them carry weight they don't need.
+
+## Migration shape
+
+ADR-0002 (files-as-truth) plus the schema-version recovery pattern shipped in 8.3.7 carry the migration. The engine knows the world has changed and tells the user how to fix it.
+
+1. **`mikser-io@9.0.0`** — `src/plugins/` slimmed to the five engine plugins. Legacy stub modules at the old paths throw a clear error:
+
+   ```
+   Plugin "layouts" not found in src/plugins/. It has moved to its own
+   package — run: npm install mikser-io-layouts
+   ```
+
+   `schema_version` stamp bumps; auto-recover wipes the cache on first run (no manual `--clear` needed, per 8.3.7).
+
+2. **New plugin repos shipped at the same time**:
+   - `mikser-io-layouts` (the big one)
+   - `mikser-io-assets`, `mikser-io-resources`, `mikser-io-preview`, `mikser-io-data`
+   - `mikser-io-observer`, `mikser-io-mapper`, `mikser-io-validator`, `mikser-io-commands`, `mikser-io-shares`
+
+3. **New AI plugin repos**:
+   - `mikser-io-transformers` (substrate)
+   - `mikser-io-ocr`, `mikser-io-extract` (consumer plugins for the document workflow)
+   - Others land on rolling cadence as demand justifies
+
+4. **`mikser-io-vector` bumps to 2.0.0** — drops its OpenAI HTTP client, depends on the pipelines substrate. Old config keys (`vector.openai.apiKey`) keep working through a one-time deprecation cycle if `mikser-io-openai` is loaded; recommended path is `mikser-io-transformers` (no API key, fully offline).
+
+5. **README rewrite** — "Getting Started" splits into recipes. The current single-default-everyone-gets-SSG path becomes one of several. Each recipe shows its plugin list, the workflow it enables, and links to an example repo.
+
+6. **Example repos**:
+   - `mikser-io-example-blog` (existing — the SSG recipe)
+   - `mikser-io-example-dms` (new — document management)
+   - `mikser-io-example-pkm` (new — personal knowledge management)
+   - `mikser-io-example-photos` (new — image library)
+
+## `mikser --install` — explicit plugin installation
+
+The plugin count balloons from "everyone uses the default" to "everyone composes a recipe." A document management recipe is 10 packages; an SSG recipe is 12. Without a clean install story, every recipe shift is "edit `package.json`, `npm install`, hit a missing plugin, repeat."
+
+But the engine shouldn't shell out to a package manager during normal runs. No magic, no surprise installs as a side effect of `mikser`. Package management is its own step, run explicitly.
+
+### Shape
+
+`mikser --install` reads `mikser.config.js`, identifies which plugins aren't resolvable via the existing four-step lookup chain, batches them into one install call, and exits. No `onInitialize`, no `onLoaded`, no render, no watch — the substrate doesn't even open `mikser.sqlite`. Just dependency resolution and the package install.
+
+```
+$ mikser --install
+
+🟡 Plugins not installed locally: layouts, vector, transformers
+   Resolving to: mikser-io-layouts, mikser-io-vector, mikser-io-transformers
+   Running: npm install --save mikser-io-layouts mikser-io-vector mikser-io-transformers
+   [npm output...]
+✓ Installed 3 plugins (4.2s)
+
+$ mikser
+🟢 Continuing normally...
+```
+
+Plain `mikser` without `--install` does **not** auto-install. If a plugin is missing, the engine fails with a clear error pointing at the install command:
+
+```
+$ mikser
+
+🔴 Plugin "layouts" not found. Install with:
+   mikser --install
+
+   (or directly: npm install mikser-io-layouts)
+```
+
+The fallback path is the same as the schema-version recovery shipped in 8.3.7 — engine explains exactly what's wrong and exactly how to fix it. The fix is one command.
+
+### Implementation notes
+
+- **Package name resolution.** Plugin name `layouts` → package `mikser-io-layouts`. Same convention as today's loader. User can override per-plugin in config when they need a non-standard package name:
+
+  ```js
+  plugins: [
+      'documents',
+      { name: 'custom', package: 'my-org-mikser-custom' },
+  ]
+  ```
+
+- **Package manager detection.** Detect from lock file in the working folder:
+  - `package-lock.json` → npm
+  - `yarn.lock` → yarn
+  - `pnpm-lock.yaml` → pnpm
+  - `bun.lockb` → bun
+  - Default to npm if no lock file detected. CLI override via `--package-manager` for edge cases.
+
+- **Batched install.** Scan all missing plugins, install in one call. Ten missing plugins → one network round-trip, not ten.
+
+- **Persistence via `--save`.** Updates `package.json` so the deps are recorded. Future `mikser` invocations find them via normal node resolution. No state hidden in `runtime/` or anywhere else.
+
+- **Version pinning.** Pin to the engine's matching major: `mikser-io@9.x` installs `mikser-io-layouts@^9`. Prevents accidental cross-major drift. Users can override per-plugin with explicit version in the config object form (`{name, package, version}`).
+
+- **Safe-by-name.** Only resolve plugin names matching the `mikser-io-*` convention (or the explicit `package:` override). Catches typos before installing random packages — `plugins: ['layoutz']` errors at resolution time instead of silently installing `mikser-io-layoutz`.
+
+- **Network failure → clear error.** No network, no install. Standard npm error with the engine's note: "Run when you have network connectivity, or install the listed packages manually."
+
+- **`--install` is loud.** The install step logs every resolved package name, the exact command run, the duration, and the result. The user explicitly asked for this; show them what happened.
+
+### CLI
+
+```
+mikser                            # normal run; fails clearly on missing plugins
+mikser --install                  # resolve + install missing plugins; exit
+mikser --install --package-manager=pnpm   # override lock-file detection
+```
+
+That's the whole surface. No `--no-auto-install` (nothing to disable). No `NODE_ENV` branch (no behavior to gate on environment).
+
+### Why no auto-install
+
+Two compounding reasons:
+
+1. **No magic during normal runs.** `mikser` is for building. `mikser --install` is for installing. One command does one thing. The engine doesn't shell out to npm as a side effect of starting the lifecycle — that would surprise CI, surprise production deploys, surprise air-gapped environments, and surprise anyone debugging "why did `mikser` take 30 seconds to start?"
+
+2. **It matches the rest of the npm ecosystem.** `node app.js` doesn't auto-install missing packages. `next build` doesn't. `vite` doesn't. Adding auto-install to mikser would be the odd one out for no real win — the install command is one line, the error message tells you exactly what to run, the workflow is "edit config → `mikser --install` → `mikser`" which is the same shape every JS project already follows.
+
+The friction the auto-install would have saved is one explicit command after editing the plugin list. The cost would have been: engine that may install packages during `mikser`, harder-to-reason-about CI, NODE_ENV branches, watch-mode interaction edge cases, and a "what just got installed?" question on every cold cycle.
+
+The trade isn't worth it. `--install` as the explicit, opt-in command is the right shape.
+
+### What this isn't
+
+- **Not a package manager.** Mikser shells out to npm/yarn/pnpm/bun for the install. No proprietary install logic, no opinion about lockfiles, no version solver. The engine just identifies which packages need to be installed and asks the user's existing tool to do it.
+- **Not a cross-major migration tool.** Pins to the matching major. Pre-9 plugins don't get upgraded through this path; users explicitly `npm install mikser-io-X@9` if they need to.
+
+## Out of scope / deferred
+
+- **ADR-0010** ("the engine is the substrate; rendering is a plugin"). Written when the move is done so the wording reflects what shipped, not a forward-projection. CLAUDE.md note added at that point too.
+- **Plugin marketplace / discovery**. Not building this. README recipes + ecosystem docs handle discovery. Anyone shipping a third-party plugin uses the existing `mikser-io-<name>` convention.
+- **Automated migration tool**. The legacy stub errors give the `npm install` command. That's the migration tool. Users running `mikser` after upgrade get told what to install per missing plugin; one `npm install` per package gets them running again.
+- **Mikser-to-mikser composition** (v10 territory, separate package). The scaling answer for big corpora isn't "make one mikser hold N million entities"; it's "compose N miksers, each holding a domain-sized corpus, talking through a coordination layer." That coordination layer ships as **`party-mikser-io`** — its own package, its own ADR series, its own release cadence. Naming it separately is deliberate: cross-instance concerns (discovery, peer identity, routing, cross-mikser refs, watch propagation, liveness, partition behavior, peer auth) belong to a distributed-systems substrate that iterates at "we hit a production partition bug" speed, not at engine-substrate speed. Folding any of it into `mikser-io` would re-fail ADR-0006 test #5 (release cadence) the moment it ships. The metaphor: each mikser is a guest in a party — voluntary attendance, each brings what it owns, the host knows who's there but nobody's centrally in charge. Until `party-mikser-io` ships, `mikser-io`'s design center stays "one folder, one mikser, domain-sized catalog." This deferral also kills any 9.0-era pressure to add output streaming, iterator-first API redesigns, or journal chunked commits — those exist to make a single process scale, which isn't where v10 is heading.
+
+## Open questions to settle before shipping
+
+1. **Naming.** "Engine plugins" vs "substrate" vs "core" — what's the term we use in the README to distinguish what ships in `mikser-io` from what doesn't? Working proposal: **"substrate"** for the engine-only pieces (catalog, refs, manifest, journal, the 5 entity-sourcing plugins), **"plugins"** for everything else (internal or external — no asymmetry in language because there's no asymmetry in mechanism).
+
+2. **Pipeline signal contract.** All pipeline functions accept `{ signal }` for abort. Required by the substrate's TypeScript types? Or convention with the substrate enforcing at registration time? Working proposal: **required** — the substrate validates registered functions have the signature.
+
+3. **Multiple provider plugins loaded at once.** Can `mikser-io-transformers` AND `mikser-io-openai` both register `embed`? If yes, which one wins for the shared verb name, and can consumers explicitly request one vendor? Working proposal: **registration order wins for the shared name**. Consumer can reach a specific provider via `runtime.options.pipelines.embed.openai(text)` for vendor-specific cases. The substrate exposes both.
+
+4. **Schemas plugin integration with extract.** `mikser-io-extract` needs to read schema declarations from `mikser-io-schemas` to know what shape to coerce into. Two options:
+   - (a) schemas plugin exposes `runtime.options.schemas.lookup(entity)` → returns the matching zod schema. Extract consumes that surface.
+   - (b) schemas plugin enriches entities with the schema reference attached; extract reads it off the entity.
+   Working proposal: **(a)** — lookup surface. Keeps the schema declaration in one place; extract reads on demand.
+
+5. **Version bump cascade.** `mikser-io` 9.0, `mikser-io-vector` 2.0, layouts/assets/etc. 1.0 (initial extraction releases). Coordinated or independent? Working proposal: **coordinated for the initial 9.0 set** (so users can `npm install` a consistent slate); independent cadence after.
+
+6. **What about `mikser-io-render-*`?** Those already live external. The question: do they stay shipped from their own repos as before, or get consolidated under `mikser-io-layouts` as siblings? Working proposal: **stay external as today** — they're rendering-engine adapters (hbs / eta / liquid / markdown / file), one per template language, and consolidating them serves nobody.
+
+
+## Concretely, what work this is
+
+Roughly in dependency order:
+
+1. **mikser-io engine** — slim `src/plugins/`, add legacy stub error files at the old paths, document the `runtime.options.pipelines` surface in `documentation/api-reference.md`. Pipeline functions are no-op until a provider plugin registers them. **`mikser --install` command** (see section above) lands here too — same release, so users have a one-command way to populate a recipe's plugin set after editing config.
+2. **mikser-io-transformers** — new repo. Wire @xenova/transformers; register embed, ocr, extract, classify, rerank, summarize, transcribe. Handle model download, cache directory, progress bar, dimension introspection.
+3. **mikser-io-vector 2.0** — drop OpenAI HTTP client. Depend on substrate. Validate dimension against the registered embed pipeline.
+4. **mikser-io-layouts** — new repo. Extract `src/plugins/layouts.js` verbatim. CI + tests.
+5. **mikser-io-assets, mikser-io-resources, mikser-io-preview, mikser-io-data** — new repos. Verbatim extraction with their own version cadence.
+6. **mikser-io-ocr** — new repo. Consumer plugin for the ocr pipeline.
+7. **mikser-io-extract** — new repo. Consumer plugin for the extract pipeline. Wire into mikser-io-schemas's `lookup()` surface.
+8. **mikser-io-csv** — new repo. Source plugin: each CSV row becomes a queryable entity. `idColumn` config for stable per-row ids; checksum-gated re-emit on file change.
+9. **mikser-io-render-csv** — new repo. Render plugin. RFC 4180 escaping; aggregations from sidecar `findEntities()` calls; first non-text-document render plugin in the family.
+10. **mikser-io-schemas** — fix lifecycle ordering (front-matter populates meta before validation fires). Add `extract: true` flag handling. Add `lookup(entity)` surface for the extract plugin.
+11. **README rewrite** — recipe-driven Getting Started. Multiple compositions shown.
+12. **mikser-io-example-dms, -pkm, -photos, -invoices** — new example repos. The invoices example shows the PDF → entity → CSV round-trip end-to-end.
+
+Items 1-3 are blocking for the 9.0 release. Items 4-8 ship at 9.0 too (so users have something to install when the engine tells them a plugin moved). Items 9-10 follow but should aim for the same release window — without recipe docs, users don't know which composition to install.

package/CLAUDE.md

@@ -10,6 +10,16 @@ paths. No migration markdown. No `task: pool` legacy aliasing for
 `task: inline`. Update READMEs and ADRs in place as source-of-truth
 changes; rewrite, don't supersede. The freedom is the point.
 
+**v9 has to pay out before v10 starts.** Not "ship v9 and start v10."
+**Use v9 in real daily work until the workflow pays out**, then
+v10. The v10 architecture (party-mikser-io) is more fun to design
+than v9 documentation is to write — the gate exists because that
+asymmetry kills projects. Concrete pass/fail signals in
+`10.0-PLAN.md` under "The gate." If a session brings up v10 design
+work, check the gate first; if v9 isn't paying out yet, the
+question to answer is "what about v9 isn't working in daily use?"
+— not "what should v10 look like?"
+
 **Position mikser by what it is, not by speed.** Hugo wins the speed
 race; mikser doesn't compete there. Position: AI-native, lifecycle-
 observable, files-as-source-of-truth, full-cycle introspection. Speed

package/CLOUD-PLAN.md

@@ -0,0 +1,147 @@
+# Mikser Cloud — Planning (stub)
+
+**The shift in one line:** mikser stops being only a CLI/library you install and becomes a hosted product. Pre-configured appliances (invoices, blog, photos, pkm, ...) run as managed containers; a hosted `party-mikser-io` instance orchestrates them per tenant. The OSS substrate doesn't change — cloud is a product built on it, not an extension of it.
+
+This is a strategic stub, not a technical plan. Cloud isn't versioned alongside `mikser-io` (engine versions and product versions iterate independently). The document exists to stake the architectural boundary and name the hard problems before any cloud code gets written.
+
+## The discipline this stub enforces
+
+**OSS earns cloud, not the other way around.** Cloud doesn't get built until `mikser-io` 10.0 has been used in real workloads by users who didn't pay for it. Building cloud on an untested substrate compounds risk — every cloud assumption built on an unvalidated substrate assumption is a refactor waiting to happen.
+
+The boundary check at any cloud-era PR: **does this code reference cloud concepts? If yes → wrong repo unless that repo is `mikser-cloud` or an appliance repo.** No multi-tenancy, no billing hooks, no platform observability in `mikser-io` or `party-mikser-io`.
+
+## What cloud cleanly is
+
+| layer | what ships | where it lives |
+|---|---|---|
+| **Hosted runtime** | Containers running mikser appliances. Persistent storage for files + `runtime/`. Health probes. Restart semantics. | `mikser-cloud` |
+| **Appliances** | Pre-configured mikser images, one per problem domain. Each is `mikser-io` + opinionated plugin set + sample config + starter folder structure. | `mikser-appliance-<name>` per repo |
+| **Party as a service** | Per-tenant hosted `party-mikser-io` instance. Routes discovery / refs / subscribe between the tenant's appliances. | `mikser-cloud` (uses party-mikser-io as a library) |
+| **Control plane** | Web UI: spin up an appliance, connect it to others, expose to an MCP agent, billing, auth, tenant isolation. | `mikser-cloud` |
+
+## Initial appliance set
+
+Each is its own product (own repo, own README, own sample data, own support burden). Underestimating this is the easy mistake — five appliances = five products to maintain.
+
+| appliance | plugin set | the problem it solves |
+|---|---|---|
+| `mikser-appliance-blog` | documents, files, layouts, front-matter, yaml, render-hbs, render-markdown, vector, api, mcp | personal/team blog with semantic search and AI agent integration |
+| `mikser-appliance-invoices` | files, csv, ocr, extract, transformers, schemas, render-csv, api, mcp | drop PDF invoices → typed catalog → exportable CSV → agent answers questions |
+| `mikser-appliance-photos` | files, assets, transformers (image embeddings), vector, api, mcp | drop photo folder → semantic search by description |
+| `mikser-appliance-pkm` | documents, front-matter, yaml, json, layouts, render-markdown, render-hbs, vector, mcp | obsidian-shaped knowledge base with AI agent |
+| `mikser-appliance-dms` | files, ocr, extract, transformers, schemas, vector, api, mcp | document management: drop arbitrary docs → typed extraction → searchable archive |
+
+Each appliance is opinionated. The point of an appliance is "you don't configure plugins, you don't pick a layout engine, you just use it." Power users still get `mikser-io` standalone with full control; appliances are the easy path.
+
+## Hard problems cloud forces us to confront
+
+Not dealbreakers, but real. Each one needs a designed answer before cloud launches.
+
+### Files-as-source-of-truth in a container
+
+The whole `mikser-io` pitch is "drop files in a folder." In a container, where IS that folder?
+
+Options (each with cost):
+
+- **Mounted persistent volume (EBS / pd / etc.)** — simplest mental model. Files live on a real disk per appliance. User edits via web upload or sync agent. Cost: scales per-appliance, snapshot/backup is the platform's problem.
+- **S3-backed FS (s3fs / goofys / mountpoint-s3)** — files live in object storage. Cheap, infinite, but file-system semantics are partial. Chokidar might not work cleanly; rename + metadata operations can be slow or non-atomic.
+- **Git-synced** — appliance pulls from a user-provided git repo. File source-of-truth lives in git, not the container. Best for power users; weird for non-technical ones.
+- **Web upload only** — appliance has a control-plane UI for managing files. No local "folder" per se. Cost: builds a CMS-shaped UI, which is a product surface unto itself.
+
+Working assumption: **persistent volume + web upload UI for non-power-users, optional git sync for power users.** S3-backed FS isn't worth the file-system-semantics tax. **Decision deferred** until first appliance is built.
+
+### Multi-tenancy in the party host
+
+A central party host serving N tenants must route discovery + refs + subscribe per tenant. One user's appliances can't see another's, ever.
+
+`party-mikser-io` itself doesn't need to know about tenants — it's per-process. The cloud's deployment is **one party-mikser-io process per tenant**, sitting in front of that tenant's appliance set. Cost: N processes for N tenants. At scale, that's a real ops bill.
+
+Alternative: a multi-tenant party host with tenant-aware routing. Cost: complicates `party-mikser-io` with tenant-awareness it shouldn't have. Working bias: **per-tenant process** — keeps party-mikser-io clean. Revisit if economics force consolidation.
+
+### Persistent state lifecycle
+
+`runtime/mikser.sqlite` is the catalog cache. ADR-0002: files are truth, the cache is derived. But re-deriving 50k entities + refs + manifest on container restart takes minutes — that's a cold-start UX problem.
+
+Cloud needs the cache to survive restarts. The container's persistent volume holds `runtime/` alongside the file folder. On restart, `mikser` reuses the cache and rebuilds incrementally. This already works in `mikser-io`; cloud just needs the volume mount.
+
+The non-obvious bit: **`runtime/` is per-appliance, not per-tenant**. If a tenant has 3 appliances, that's 3 sqlite files in 3 volumes. Don't try to consolidate.
+
+### Watch-mode vs push
+
+Chokidar is right for a laptop. In cloud, file changes come from API uploads / git pulls / webhook syncs. The trigger model is different.
+
+Easiest path: the sync layer drops files into the volume; chokidar sees them and triggers rebuild. Works out of the box. The sync layer is the cloud platform's problem, not the engine's.
+
+Worth verifying once cloud starts: chokidar on a mounted volume behaves predictably across pod restarts.
+
+### Billing surface
+
+Per-appliance-hour? Per-entity? Per-render? Per-MCP-call?
+
+Each metric is a different observability investment. Per-render and per-MCP-call require engine-side counters that don't exist today. Per-entity is weird (what's an entity, a row of a CSV?). Per-appliance-hour is the simplest honest unit and matches how the user thinks ("I have 3 appliances running").
+
+Working assumption: **per-appliance-hour, with a generous free tier on the smallest plan size.** Per-call billing for the API tier comes later if usage patterns justify it.
+
+### Open-core boundary
+
+The standard hard question. Two cleanly defensible answers:
+
+- **Convenience-only cloud (working bias).** Everything in OSS works standalone. Cloud sells hosting + the appliance catalog + the web UI + the hosted party-mikser-io. No engine feature is paid-only. The OSS substrate stays the same software cloud runs on; contributors stay aligned.
+- **Open-core with paid features.** Some engine features (advanced auth, audit logging, SLA-level support) become cloud/enterprise-only. Cost: contributor alienation, harder to make external use feel first-class.
+
+The convenience-only model has worked for many shops (Cloudron, PostHog, Plausible). The open-core model has worked for others (GitLab, Sentry). For mikser specifically — file-based, agent-native, small surface area — **convenience-only is probably right**, but the decision deserves a real conversation when cloud actually starts.
+
+### Appliance lifecycle as a product surface
+
+Each appliance is its own product. That means:
+
+- Its own README + docs
+- Sample folder structure shipped with the image
+- Sample data for the demo experience
+- Schema definitions (for OCR + extract appliances)
+- Plugin pinning + version compatibility
+- Security review per release
+- Support tickets specific to that appliance's domain
+
+Five appliances at launch = five product tracks. Realistically, **start with one or two** (probably `blog` and `invoices`), validate the appliance-shape pattern, then expand.
+
+## What cloud needs from the substrate (forward-look only — no commit)
+
+Things that *might* land in `mikser-io` or `party-mikser-io` to enable cloud. Each one passes the substrate test independently — they're not cloud-specific:
+
+| candidate | substrate justification |
+|---|---|
+| `GET /api/health` endpoint in mikser-io-api | Operationally useful for any deployment, not just cloud. Probably trivial. |
+| Env-var-based config overrides in mikser-io | Already useful for OSS users running in CI. Cloud just consumes the same surface. |
+| `--rebuild` mode (one-shot rebuild on signal, no chokidar) | Useful for any CI / scheduled-build scenario. |
+| Structured operation metrics (entities created/updated/rendered per cycle) — already partially via pino logs | Useful for any operator wanting observability. |
+| `mikser.config.js` deriving from env vars cleanly | Already standard JS — no engine change needed. |
+| `party-mikser-io` as a standalone server binary, not just a library | Useful for any self-hosted multi-mikser deployment, not just cloud. |
+
+None of these get added pre-emptively for cloud. Each goes through the five-test framework on its own merits, and lands when an OSS user benefits.
+
+## What cloud explicitly is NOT
+
+- **Not an Anthropic / Notion / Glean clone.** Mikser cloud sells hosted mikser, with mikser's specific positioning (file-based, agent-native, OSS substrate). It's not a general note-taking SaaS.
+- **Not a Backend-as-a-Service.** No exposed-to-end-users database API. Each tenant gets appliances, not raw catalog access (except via the appliance's own MCP + api surfaces).
+- **Not a multi-tenant single mikser.** No shared substrate between tenants. Each tenant gets their own container(s); ADR-0002 (files-as-source-of-truth) means a shared catalog across tenants would conflate user data, which is operationally and legally fraught.
+- **Not a competitor to plugin authors.** Anyone shipping a `mikser-io-<name>` plugin can also be an appliance author. The appliance catalog is open; we're not gatekeeping the ecosystem.
+
+## Out of scope (defer until cloud actually starts)
+
+- Pricing — every dimension of it. Tiers, free-tier limits, overages, enterprise contracts. Real numbers need real cost data, which we won't have until we run actual appliances.
+- Compliance posture (SOC 2, GDPR, HIPAA) — depends on which markets cloud targets. Decided when go-to-market is real.
+- Geographic regions — same.
+- White-label / self-hosted cloud for enterprises — possible product variant, but premature.
+- Marketplace for third-party appliances — possible, but starting with first-party only is the sane shape.
+
+## When this stub becomes a real plan
+
+When two things are true:
+
+1. `mikser-io` 10.0 has shipped and has at least one real-world deployment that isn't a toy example.
+2. `party-mikser-io` has shipped and someone outside the immediate dev circle has composed two miksers without help.
+
+If both of those are true, the substrate is validated enough that cloud can build on it confidently. This stub becomes `CLOUD-PLAN.md` proper, with timelines, appliance prioritization, and pricing dimensions.
+
+Until both are true, this document exists to keep cloud as a coherent direction without distorting current engine work. That's the whole purpose of the stub.

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "mikser-io",
-	"version": "8.3.6",
+	"version": "8.3.9",
 	"description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
 	"main": "index.js",
 	"scripts": {

package/src/database/index.js

@@ -25,7 +25,7 @@
 // version, open() throws with a clear "run mikser --clear" message.
 
 import path from 'node:path'
-import { mkdirSync } from 'node:fs'
+import { mkdirSync, unlinkSync, existsSync } from 'node:fs'
 import Database from 'better-sqlite3'
 import runtime from '../runtime.js'
 import { onLoaded } from '../lifecycle.js'
@@ -49,6 +49,38 @@ const DEFAULT_FILENAME = 'mikser.sqlite'
 // vector_documents in mikser-io-vector).
 const schemas = new Map()
 
+// Provisioning callbacks registered by plugins. Run on every open(),
+// between the raw `new Database(dbPath)` (with PRAGMAs + meta bootstrap)
+// and the schema apply pass. Each callback receives a context object:
+//
+//   {
+//     firstRun:        boolean — db file didn't exist before this open
+//                                (or :memory:, or was wiped by --clear /
+//                                schema mismatch)
+//     upgraded:        boolean — stored schema_version differed from
+//                                current (db was just wiped + recreated)
+//     previousVersion: string | null — the version stamp we found before
+//                                       this open, null if firstRun
+//     currentVersion:  string — what this mikser binary expects
+//     handle:          Database — the raw better-sqlite3 handle, for
+//                                 loadExtension / collations / custom
+//                                 functions / one-time data seeding
+//     logger:          pino|null — for surfacing user-visible setup events
+//   }
+//
+// Use cases:
+//   - Load runtime extensions before they're referenced by schemas
+//     (sqlite-vec's vec0 module is the canonical example)
+//   - Register custom collations, application_id, etc.
+//   - First-run data seeding (use `if (ctx.firstRun)` to gate)
+//   - Upgrade-time data migrations (gate on `ctx.upgraded` and check
+//     `ctx.previousVersion`)
+//
+// Plugins register at module-eval time, same shape as registerSchema.
+// Order doesn't matter unless callbacks depend on each other; if you
+// have such a dep, both should be in the same plugin.
+const provisioners = []
+
 // Active database handle. Set during the first onLoaded; persists across
 // cycles (sqlite stays open for the lifetime of the process). Public API
 // readers see null only before the first onLoaded fires.
@@ -102,6 +134,53 @@ export function registerSchema(name, sqlScript) {
     }
 }
 
+// Register a database-provisioning callback. Fires once per open(),
+// AFTER PRAGMAs and the mikser_meta bootstrap (so version detection
+// and the schema-mismatch wipe have already happened) and BEFORE
+// schema apply (so callbacks can install virtual-table modules,
+// register collations, etc. that the schemas about to apply may
+// reference).
+//
+// Callback receives a context object with everything needed to act
+// situationally:
+//
+//   onProvision(({ firstRun, upgraded, previousVersion, currentVersion, handle, logger }) => {
+//       // Install runtime extensions on every open:
+//       sqliteVec.load(handle)
+//
+//       // Seed default data on first run only:
+//       if (firstRun) {
+//           handle.exec("INSERT INTO mikser_meta (key, value) VALUES ('site_id', '...')")
+//       }
+//
+//       // Migrate after a specific upgrade:
+//       if (upgraded && previousVersion?.startsWith('8.2')) {
+//           handle.exec("UPDATE my_plugin_table SET ...")
+//       }
+//   })
+//
+// Plugins register at module-eval time, same shape as registerSchema.
+// Lazy-applies on the live handle if the database is already open
+// (matches registerSchema's late-registration semantics) — the
+// callback fires with firstRun=false, upgraded=false (since the open
+// already happened cleanly), and the live handle.
+export function onProvision(callback) {
+    if (typeof callback !== 'function') {
+        throw new Error('onProvision: callback must be a function (ctx) => void')
+    }
+    provisioners.push(callback)
+    if (db?.isOpen) {
+        callback({
+            firstRun: false,
+            upgraded: false,
+            previousVersion: db.provisioning?.currentVersion ?? null,
+            currentVersion: db.provisioning?.currentVersion ?? null,
+            handle: db.handle,
+            logger: runtime.engine?.logger ?? null,
+        })
+    }
+}
+
 // Return the active database handle. Hot-path callers (catalog, refs,
 // manifest) should cache the reference once at onLoaded rather than
 // calling per-operation.
@@ -118,8 +197,12 @@ export function useDatabase() {
 // the full onLoaded chain). The runtime path uses this internally from
 // onLoaded below.
 export function createSqliteDatabase({
-    runtimeFolder, version, logger, config = {}, schemas,
+    runtimeFolder, version, logger, config = {}, schemas, provisioners: provisionersArg,
 }) {
+    // Tests inject their own provisioners; the runtime path falls
+    // through to the module-level `provisioners` array that plugins
+    // populate via onProvision() at module-eval.
+    const provisionersToRun = provisionersArg ?? provisioners
     // Resolve the on-disk path. Honor an absolute `config.filename`,
     // otherwise resolve relative to the runtime folder. `:memory:`
     // works for tests — better-sqlite3 treats it as ephemeral.
@@ -132,6 +215,11 @@ export function createSqliteDatabase({
             : path.join(runtimeFolder, DEFAULT_FILENAME)
 
     let handle = null
+    // Provisioning context exposed on the returned wrapper so any code
+    // with a handle (catalog onLoaded, plugin onLoaded, etc.) can ask
+    // "was this open a fresh db / a version upgrade / a normal cycle?"
+    // without subscribing to onProvision.
+    let provisioningCtx = null
 
     function open() {
         if (handle) return  // idempotent — caller may invoke twice across cycles
@@ -140,34 +228,91 @@ export function createSqliteDatabase({
             mkdirSync(path.dirname(dbPath), { recursive: true })
         }
 
-        handle = new Database(dbPath)
+        // Detect firstRun BEFORE we touch the filesystem. :memory:
+        // always counts as firstRun (no persistence across opens).
+        const fileExistedBeforeOpen = dbPath !== ':memory:' && existsSync(dbPath)
 
-        handle.exec('PRAGMA journal_mode = WAL')
-        handle.exec('PRAGMA synchronous = NORMAL')
-        handle.exec('PRAGMA foreign_keys = ON')
+        const setupConnection = () => {
+            handle.exec('PRAGMA journal_mode = WAL')
+            handle.exec('PRAGMA synchronous = NORMAL')
+            handle.exec('PRAGMA foreign_keys = ON')
+            handle.exec(`
+                CREATE TABLE IF NOT EXISTS mikser_meta (
+                    key   TEXT PRIMARY KEY,
+                    value TEXT NOT NULL
+                )
+            `)
+        }
 
-        // meta table always exists — holds the schema_version stamp
-        // and any future engine-wide key/value state.
-        handle.exec(`
-            CREATE TABLE IF NOT EXISTS mikser_meta (
-                key   TEXT PRIMARY KEY,
-                value TEXT NOT NULL
-            )
-        `)
+        handle = new Database(dbPath)
+        setupConnection()
 
         const recorded = handle.prepare('SELECT value FROM mikser_meta WHERE key = ?')
             .get('schema_version')?.value
+        let upgradedFromVersion = null
         if (recorded && recorded !== version) {
+            // Schema mismatch on upgrade or downgrade. Per ADR-0002 the
+            // files on disk are the source of truth and this database
+            // is a derived cache, so the right behavior is to wipe the
+            // cache and let the next cycle rebuild it from source —
+            // not to halt the build with an error.
+            //
+            // Loud warning so operators can see it happened and know to
+            // expect a cold-start rebuild on this run. No data loss
+            // beyond the cache itself; everything in mikser.sqlite is
+            // recoverable from the working folder.
+            logger?.warn(
+                'Database schema mismatch: stored=%s, current=%s. Wiping the cache and rebuilding from sources (files are the source of truth — no source data is affected).',
+                recorded, version,
+            )
             handle.close()
             handle = null
-            throw new Error(
-                `Database schema is ${recorded}; this mikser-io expects ${version}. ` +
-                `Run \`mikser --clear\` to rebuild from sources.`,
-            )
+
+            if (dbPath !== ':memory:') {
+                // sqlite WAL leaves -wal and -shm sidecar files. Remove
+                // them along with the main file so the next open starts
+                // from a guaranteed-clean slate.
+                for (const suffix of ['', '-wal', '-shm']) {
+                    try { unlinkSync(dbPath + suffix) } catch { /* file may not exist */ }
+                }
+            }
+
+            upgradedFromVersion = recorded
+            handle = new Database(dbPath)
+            setupConnection()
         }
         handle.prepare('INSERT OR REPLACE INTO mikser_meta (key, value) VALUES (?, ?)')
             .run('schema_version', version)
 
+        // Build provisioning context. firstRun is true when the file
+        // didn't exist before this open OR when the schema mismatch
+        // wiped it (and the previous open's stamp is gone) — both shapes
+        // present an empty-state database to provisioners.
+        provisioningCtx = {
+            firstRun: !fileExistedBeforeOpen || upgradedFromVersion !== null,
+            upgraded: upgradedFromVersion !== null,
+            previousVersion: upgradedFromVersion,
+            currentVersion: version,
+            handle,
+            logger: logger ?? null,
+        }
+
+        // Fire provisioning callbacks before schema apply, so they can
+        // load runtime extensions (sqlite-vec's vec0, etc.) that the
+        // schemas about to apply might reference, register custom
+        // collations, or do first-run / upgrade work that needs to
+        // happen before the engine tables exist.
+        for (const provision of provisionersToRun) {
+            try {
+                provision(provisioningCtx)
+            } catch (err) {
+                handle.close()
+                handle = null
+                provisioningCtx = null
+                throw new Error(`onProvision callback failed: ${err.message}`)
+            }
+        }
+
         // Apply each subsystem's registered schema script. Idempotent
         // CREATE statements mean replay-safe across opens.
         for (const [name, sqlScript] of schemas) {
@@ -177,6 +322,7 @@ export function createSqliteDatabase({
             } catch (err) {
                 handle.close()
                 handle = null
+                provisioningCtx = null
                 throw new Error(`Schema "${name}" failed to apply: ${err.message}`)
             }
         }
@@ -204,6 +350,12 @@ export function createSqliteDatabase({
         open,
         close,
         get isOpen() { return handle !== null },
+        // Provisioning context from the most recent open. Plugins'
+        // onLoaded handlers can read this without subscribing to
+        // onProvision — useful when the work needs the catalog or
+        // refs schemas to exist first (which onProvision callbacks
+        // can't assume because they fire before schema apply).
+        get provisioning() { return provisioningCtx },
         // Pass-through primitives for subsystems. They prepare their
         // own statements at the underlying better-sqlite3 handle.
         prepare(sql) {