mikser-io 8.3.7 → 8.3.9

package/10.0-PLAN.md

@@ -0,0 +1,182 @@
+# Mikser 10.0 — Planning (stub)
+
+**The shift in one line:** scale moves from "make one mikser hold more" to "compose more miksers." A new coordination layer ships as `party-mikser-io` — separate package, separate ADR series, separate release cadence. `mikser-io` itself stays single-instance-pure.
+
+This is a v10 placeholder — direction-setting, not a finished design. The hard pieces (cross-mikser refs, discovery, partition behavior) get worked out as `party-mikser-io` takes shape. This file gets dropped at release; the surviving decisions become ADRs.
+
+## The gate: v9 has to pay out first
+
+**No v10 work begins until v9 is incorporated into our own daily work and the workflow pays out.** Not "v9 has shipped." Not "v9 has been tested." **Used, daily, for our own real work, until the format-liberation pitch is something we'd defend with our own time.**
+
+This is the strongest gate in the roadmap. It exists because the v10 architecture is more interesting to build than the v9 documentation is to write, and that asymmetry kills projects with strong substrates. The discipline is to refuse v10 design work — even thinking about it as an escape — until v9 has earned the next layer through real use.
+
+What "pays out" means operationally — concrete signals to watch:
+
+| signal | reading |
+|---|---|
+| New work naturally flows into mikser, not into "I should mikser-ize this later" | ✅ working |
+| You stop wanting to switch back to your previous setup | ✅ working |
+| The format-liberation flows (PDF in → CSV out, CSV in → HTML out) are things you actually run, not just demos | ✅ working |
+| The MCP agent integration makes daily work faster, not just possible | ✅ working |
+| Watch-mode + incremental rebuild holds up under your real editing patterns | ✅ working |
+| You stop writing helper scripts that work around mikser instead of through it | ✅ working |
+| Friends/colleagues see what you have running and ask how to use it themselves | ✅ working — the strongest signal |
+
+Failure signals — any of these means v9 hasn't paid out yet, fix before v10:
+
+| signal | reading |
+|---|---|
+| You manually run `mikser --clear` more than weekly | ❌ incremental rebuild has rough edges |
+| You keep editing files outside the mikser folder because watch-mode is slow / breaks | ❌ watch-mode regression somewhere |
+| Specific plugins break in ways that make you avoid them | ❌ plugin needs work, not v10 work |
+| You stop using MCP integration because grep is faster | ❌ MCP surface isn't carrying its weight |
+| You compose pipelines outside mikser (one-off scripts) because composing in mikser is awkward | ❌ recipe documentation or substrate ergonomics gap |
+| You don't find yourself reaching for it for new tasks | ❌ the substrate isn't actually changing how you work |
+
+When the working signals dominate over a sustained period of real use, v10 design starts. Not before.
+
+## Why v10 (not 9.x)
+
+The 9.0 plan deliberately deferred mikser-to-mikser composition to v10. The reasoning is in `9.0-PLAN.md` under "Out of scope":
+
+> Cross-instance concerns (discovery, peer identity, routing, cross-mikser refs, watch propagation, liveness, partition behavior, peer auth) belong to a distributed-systems substrate that iterates at "we hit a production partition bug" speed, not at engine-substrate speed. Folding any of it into `mikser-io` would re-fail ADR-0006 test #5 (release cadence) the moment it ships.
+
+v10 is the version where that substrate ships, alongside the small `mikser-io` additions that make it possible.
+
+Until then: one folder, one mikser, domain-sized catalog. That's the v9 design center, and it earns the v10 work by staying small.
+
+## The split
+
+```
+mikser-io           → single-instance substrate (unchanged from 9.0)
+party-mikser-io     → coordination layer between mikser instances
+```
+
+The repo-split check is one line: **does this code reference anything outside this mikser's `runtime`? If yes → wrong repo.** That makes ADR-0006 test #1 (substrate?) concrete for distributed code, which is otherwise ambiguous.
+
+| concern | lives in | why |
+|---|---|---|
+| Catalog / refs / manifest / journal | `mikser-io` | every mikser needs this |
+| Source plugins, render dispatch, lifecycle | `mikser-io` | same |
+| Discovery (how A finds B) | `party-mikser-io` | mikser-io never needs to know peers exist |
+| Identity (`mikser.name` as a stable handle) | `party-mikser-io` | mikser-io has no notion of "self" beyond `runtime` |
+| Routing (which mikser owns entity X) | `party-mikser-io` | local catalog queries don't ask this |
+| Cross-mikser refs (`$author` in A → entity in B) | `party-mikser-io` | refs.js stays local-graph-pure |
+| Watch propagation across instances | `party-mikser-io` | chokidar covers one folder; cross-mikser is a different transport |
+| Liveness / health / partition behavior | `party-mikser-io` | distributed-systems posture, not file-engine posture |
+| Auth / trust between peers | `party-mikser-io` | engine auth is per-endpoint; peer auth is its own model |
+
+## The metaphor
+
+A "party" — voluntary attendance, each guest brings what they own, the host knows who's there but nobody's centrally in charge.
+
+More accurate than:
+
+- **Cluster** — implies homogeneity (every node same code, same role). A photos mikser and an invoices mikser are different roles.
+- **Mesh** — implies n-to-n by default. Real parties talk to small explicit peer sets.
+- **Federation** — implies central authority or shared schema. Each mikser owns its catalog and exposes whatever shape it wants.
+
+Reads correctly at the call site:
+
+```js
+import { join } from 'party-mikser-io'
+
+await join({
+    as: 'invoices',
+    peers: [
+        { name: 'documents', url: 'http://localhost:8081' },
+        { name: 'photos',    url: 'http://localhost:8082' },
+    ],
+})
+```
+
+## What lands in `mikser-io` to support this
+
+These are the only `mikser-io` 10.0 changes — small, surgical, and they don't introduce peer-awareness into the engine. They expose information party-mikser-io needs in order to coordinate.
+
+### `mikser-io-api` surface additions
+
+Three new endpoints. None of them know about peers; they just expose what a peer would need to read.
+
+| endpoint | purpose | rough LOC |
+|---|---|---|
+| `GET /api/identity` | mikser name (from config) + version + capabilities (plugin list + their versions) | ~20 |
+| `GET /api/checkpoint` | current catalog snapshot id (for partition-recovery sync) | ~30 |
+| `GET /api/refs/:id?direction=in|out` | inbound or outbound refs for an entity | ~40 |
+
+The `subscribe` endpoint also gains a `?since=<checkpoint>` query param for replay-on-reconnect. ~20 LOC.
+
+Total: ~110 LOC of additive endpoints. Nothing existing changes. Each one is independently testable.
+
+### `mikser.config.js` gets a `name` field
+
+```js
+export default {
+    name: 'invoices',   // stable identifier; used by /api/identity and party.join({as})
+    plugins: [...],
+}
+```
+
+Engine reads it, exposes via `runtime.name`. Default falls back to the working folder's basename. That's it — `mikser-io` doesn't act on it; party-mikser-io does.
+
+## What `party-mikser-io` ships in 10.0
+
+Not designing this in detail yet — the design happens as the package gets built. The shape at this stage is:
+
+1. **`join(config)`** — handshake with each declared peer via `/api/identity`. Verifies peer is reachable and reports name/capabilities.
+2. **`runtime.options.party.peers`** — map of peer name → metadata. Available to plugins after join completes.
+3. **`mikser-io-source-party`** — source plugin: mounts a peer's `/api` SSE feed as a local collection. Live-reload via the existing subscribe semantics.
+4. **Cross-mikser ref resolution** — `$author@documents` syntax (entity id + peer name). Resolution proxies through the source-party plugin's mirrored cache. **Design call deferred** to first implementation pass.
+5. **Liveness checks** — periodic `/api/checkpoint` poll per peer. Stale peers logged but don't break the build (their entities go cold; resync on reconnect).
+
+## Open questions (working proposals at v10 design time)
+
+1. **Cross-mikser ref shape — mirror or proxy?**
+   - (a) Mirror: source-party plugin pulls all peer entities into local catalog. Refs resolve locally. Cost: memory grows with peer corpus size.
+   - (b) Proxy: refs resolve via HTTP at lookup time. Cost: latency on every render that touches a cross-mikser ref.
+   - Working proposal: **mirror with TTL** — pull peer entities lazily on first ref, cache with checkpoint-based invalidation. Worst-case memory is bounded by what's actually referenced.
+
+2. **Discovery mechanism.**
+   - Config-pinned peer URLs (the snippet above) for v10. Anything more (mDNS, registry, DNS-SD) lives in 10.x or 11.0 after we see what real deployments need. Working proposal: **explicit URLs only for v10.**
+
+3. **Authentication between peers.**
+   - The api plugin already supports per-endpoint tokens. Reuse that: each peer in `join()` config carries a token. Working proposal: **bearer token per peer, configured in `join()`.**
+
+4. **What if two peers claim entities with the same id?**
+   - The fully-qualified form is `id@peer-name`. Local entities have no `@`. Working proposal: **namespacing by peer name avoids collision entirely** — cross-mikser refs always carry the peer name.
+
+5. **What about cycles?**
+   - A subscribes to B, B subscribes to A. Working proposal: **explicitly allowed**. Each side mirrors the other; the checkpoint mechanism prevents replay loops.
+
+## Out of scope for 10.0
+
+- **Write-across-peers** — `party.create(entity, { in: 'invoices' })` from peer A creating an entity in peer B's catalog. Probably 10.x or 11.0. Single-direction read is enough to validate the architecture.
+- **Distributed consensus / conflict resolution** — each peer owns its catalog, period. Nobody writes to anyone else's.
+- **Replication / HA** — that's a different problem. A party isn't a cluster; if a peer goes down, its entities go cold, not get failed-over.
+- **Plugin marketplace / discovery service** — same answer as 9.0. Not building this.
+
+## ADR series
+
+`party-mikser-io` gets its own decision log under `party-mikser-io/documentation/decisions/`, starting from 0001. Not extending the `mikser-io` ADR series — keeps the substrate test (#1) clean. `mikser-io` ADRs cover the single-instance engine; `party-mikser-io` ADRs cover the coordination plane.
+
+Cross-references between the two ADR series are fine; the boundary stays at "does this code reference anything outside this mikser's `runtime`?"
+
+## Concretely, what work this is (rough order)
+
+1. **`mikser-io` engine** — add `runtime.name` (read from `mikser.config.js`). One file change.
+2. **`mikser-io-api`** — add `/api/identity`, `/api/checkpoint`, `/api/refs/:id`, and `?since=` on subscribe. ~110 LOC.
+3. **`party-mikser-io`** — new repo. `join()`, peer registry, liveness loop. The boring substrate of the coordination layer. ~400 LOC.
+4. **`mikser-io-source-party`** — new repo (or sub-package). Source plugin that mounts a peer's catalog. The mirror cache + checkpoint invalidation logic. ~300 LOC.
+5. **Cross-mikser ref resolution** — extends `mikser-io-source-party`. Parses `id@peer` syntax in `$`-refs and resolves through the mirror. ~150 LOC.
+6. **Recipe documentation** — README composition examples: PDF mikser + CSV mikser + HTML mikser, piped together. The example that proves the v10 pitch works end-to-end.
+7. **`mikser-io-example-party`** — new example repo. Three miksers in one repo (subfolders), each with its own config, demonstrating the full composition.
+
+Items 1-2 ship in `mikser-io` 10.0 itself. Items 3-7 are the v10 ecosystem release.
+
+## What this earns us, when it lands
+
+The 9.0 pitch was "file-based knowledge substrate with AI superpowers." v10 extends that with one word: **composable**.
+
+> A file-based knowledge substrate with AI superpowers, composable across instances.
+
+The PDF-in-CSV-out-HTML-elsewhere story works as separate miksers connected through a party — not as one process trying to be everything. That's the architecture the substrate was designed for; v10 just makes it real.

package/9.0-PLAN.md

@@ -174,59 +174,6 @@ The 9.0 plugin set turns this from architectural potential into a shipped produc
 
 Future render/source pairs (post-9.0, demand-driven): `render-pdf`, `render-rss`, `render-ical`, `ical` source, `vcard` source. Each ~150-300 LOC. The substrate carries them.
 
-## Streaming render output
-
-Row-as-entity (`mikser-io-csv`, 1M-row catalogs) breaks an assumption the current render path quietly relies on: that the output of `render(entity)` is a string or `Buffer` you can hand to `writeFile`. A render that serializes 1M rows to a single 500MB CSV string is the worst kind of failure mode — it works in dev with 100 rows, OOMs in production.
-
-The substrate already streams **input** (`iterateEntities` is a seek-paginated async generator, ADR-0009 phase 5). Output is the other half.
-
-### What changes in the render dispatch
-
-`render.js` currently expects each renderer to return `string | Buffer`. Under 9.0 the contract widens:
-
-```js
-return string                            // current path — buffer to disk
-return Buffer                            // current path — buffer to disk
-return Readable                          // new — pipe to a write stream
-return AsyncIterable<string | Buffer>    // new — drained into a write stream
-```
-
-The dispatcher inspects the return value and routes:
-
-- string/Buffer → `writeFile(destination, result)` (unchanged hot path; everything that exists today keeps working)
-- Readable or async iterable → `await pipeline(result, createWriteStream(destination))`
-
-Postprocess plugins (`render-file` for inlining, `mjml`, `pdf`, etc.) opt out of the streaming path automatically when they need a buffered input — they declare `bufferedInput: true` and the dispatcher collects the stream first. Most postprocess plugins already need the full document (HTML rewrite, MJML compile), so this is the common case; streaming is the optimization for renderers whose output is naturally append-only.
-
-### What this enables
-
-| renderer | streaming wins because |
-|---|---|
-| `render-csv` | row count is corpus-scale; output is line-by-line append; no postprocess for CSV |
-| `render-rss` / `render-atom` | item list is corpus-scale; XML is append-only per item |
-| `render-ical` | event list is corpus-scale |
-| `data` (JSON catalog export) | the corpus IS the output; current code buffers the whole catalog |
-| future `render-ndjson` | naturally one entity per line |
-| `render-hbs` / `render-eta` / `render-markdown` | NOT a win — template engines produce one document, postprocess (front-matter inlining, link rewriting) needs the full string anyway |
-
-### Memory budget the streaming path holds
-
-At 1M rows, `render-csv` running in streaming mode should never hold more than one row's worth of formatted output in memory. The peak isn't the row count, it's the slowest downstream consumer (disk write speed). On a SATA SSD, a 500MB CSV writes at ~400MB/sec — under 2 seconds wall-clock for the full file, with the JS heap never crossing 50MB for the render itself.
-
-The buffered path on the same workload: 500MB CSV string in V8 = ~1GB after the UTF-16 doubling, then a second 500MB allocation for the `Buffer` conversion, then GC pressure. Streaming isn't a 10% optimization here; it's the difference between working and OOM.
-
-### What this doesn't change
-
-- HTML rendering paths (the dominant workload) keep returning strings. No measurable cost.
-- The journal's auto-persist diff still works — it diffs the entity object, not the output.
-- Manifest checksumming streams over the output too (`createHash` accepts piped data) — no extra read.
-
-### Scope for 9.0
-
-Land the dispatcher change in the engine alongside `mikser-io-csv` + `mikser-io-render-csv`. The CSV plugins are the first consumers; they validate the contract. `data` plugin migrates to streaming as a second step (separate release). All existing renderers stay on the string path; the widening is additive.
-
-Add to the work list: dispatcher widening + `render-csv` shipping in streaming mode as the proof.
-
 ## Schemas plugin pulls real weight here
 
 `mikser-io-schemas` has been sitting idle. Under the 9.0 shape it becomes load-bearing:
@@ -451,6 +398,7 @@ The trade isn't worth it. `--install` as the explicit, opt-in command is the rig
 - **ADR-0010** ("the engine is the substrate; rendering is a plugin"). Written when the move is done so the wording reflects what shipped, not a forward-projection. CLAUDE.md note added at that point too.
 - **Plugin marketplace / discovery**. Not building this. README recipes + ecosystem docs handle discovery. Anyone shipping a third-party plugin uses the existing `mikser-io-<name>` convention.
 - **Automated migration tool**. The legacy stub errors give the `npm install` command. That's the migration tool. Users running `mikser` after upgrade get told what to install per missing plugin; one `npm install` per package gets them running again.
+- **Mikser-to-mikser composition** (v10 territory, separate package). The scaling answer for big corpora isn't "make one mikser hold N million entities"; it's "compose N miksers, each holding a domain-sized corpus, talking through a coordination layer." That coordination layer ships as **`party-mikser-io`** — its own package, its own ADR series, its own release cadence. Naming it separately is deliberate: cross-instance concerns (discovery, peer identity, routing, cross-mikser refs, watch propagation, liveness, partition behavior, peer auth) belong to a distributed-systems substrate that iterates at "we hit a production partition bug" speed, not at engine-substrate speed. Folding any of it into `mikser-io` would re-fail ADR-0006 test #5 (release cadence) the moment it ships. The metaphor: each mikser is a guest in a party — voluntary attendance, each brings what it owns, the host knows who's there but nobody's centrally in charge. Until `party-mikser-io` ships, `mikser-io`'s design center stays "one folder, one mikser, domain-sized catalog." This deferral also kills any 9.0-era pressure to add output streaming, iterator-first API redesigns, or journal chunked commits — those exist to make a single process scale, which isn't where v10 is heading.
 
 ## Open questions to settle before shipping
 
@@ -469,19 +417,12 @@ The trade isn't worth it. `--install` as the explicit, opt-in command is the rig
 
 6. **What about `mikser-io-render-*`?** Those already live external. The question: do they stay shipped from their own repos as before, or get consolidated under `mikser-io-layouts` as siblings? Working proposal: **stay external as today** — they're rendering-engine adapters (hbs / eta / liquid / markdown / file), one per template language, and consolidating them serves nobody.
 
-7. **Postprocess opt-in vs opt-out for buffered input.** The streaming render dispatcher needs to know whether a postprocess plugin can consume a stream. Two shapes:
-   - (a) postprocess plugins declare `bufferedInput: true` on registration; default is "can stream"
-   - (b) postprocess plugins declare `streamInput: true`; default is "needs buffer" (current behavior)
-   Working proposal: **(b)** — default to buffered so existing plugins (mjml, pdf, front-matter-rewrite) keep working without code changes. Plugins opt into streaming only when the format genuinely supports it (CSV row-rewrite, NDJSON line-transform). One-line migration cost beats silently breaking the plugin set.
-
-8. **Worker path for streaming renders.** TASKS.WORKER serializes return values through Piscina's structured clone — a Readable stream doesn't survive that boundary. Working proposal: **streaming renders are INLINE/SERIAL only for 9.0**. A renderer that declares streaming output is force-dispatched on the main thread regardless of its `task:` frontmatter. Documented as a limitation, not a bug. Revisit when Piscina ships transferable-stream support or we move to MessagePort-based piping (post-9.0).
-
 
 ## Concretely, what work this is
 
 Roughly in dependency order:
 
-1. **mikser-io engine** — slim `src/plugins/`, add legacy stub error files at the old paths, document the `runtime.options.pipelines` surface in `documentation/api-reference.md`. Pipeline functions are no-op until a provider plugin registers them. **`mikser --install` command** (see section above) lands here too — same release, so users have a one-command way to populate a recipe's plugin set after editing config. **Streaming render output**: widen the render dispatcher in `src/render.js` to accept `Readable` and `AsyncIterable` returns; pipe to `createWriteStream` instead of `writeFile`. Postprocess plugins declare `bufferedInput: true` to opt back into the buffered path. String/Buffer returns unchanged.
+1. **mikser-io engine** — slim `src/plugins/`, add legacy stub error files at the old paths, document the `runtime.options.pipelines` surface in `documentation/api-reference.md`. Pipeline functions are no-op until a provider plugin registers them. **`mikser --install` command** (see section above) lands here too — same release, so users have a one-command way to populate a recipe's plugin set after editing config.
 2. **mikser-io-transformers** — new repo. Wire @xenova/transformers; register embed, ocr, extract, classify, rerank, summarize, transcribe. Handle model download, cache directory, progress bar, dimension introspection.
 3. **mikser-io-vector 2.0** — drop OpenAI HTTP client. Depend on substrate. Validate dimension against the registered embed pipeline.
 4. **mikser-io-layouts** — new repo. Extract `src/plugins/layouts.js` verbatim. CI + tests.
@@ -489,7 +430,7 @@ Roughly in dependency order:
 6. **mikser-io-ocr** — new repo. Consumer plugin for the ocr pipeline.
 7. **mikser-io-extract** — new repo. Consumer plugin for the extract pipeline. Wire into mikser-io-schemas's `lookup()` surface.
 8. **mikser-io-csv** — new repo. Source plugin: each CSV row becomes a queryable entity. `idColumn` config for stable per-row ids; checksum-gated re-emit on file change.
-9. **mikser-io-render-csv** — new repo. Render plugin. RFC 4180 escaping; aggregations from sidecar `findEntities()` calls; first non-text-document render plugin in the family. **Ships in streaming mode** — returns an async iterable from `render()`, validating the widened dispatcher contract end-to-end. The "1M-row CSV builds in <2s without OOM" smoke test lives in this repo.
+9. **mikser-io-render-csv** — new repo. Render plugin. RFC 4180 escaping; aggregations from sidecar `findEntities()` calls; first non-text-document render plugin in the family.
 10. **mikser-io-schemas** — fix lifecycle ordering (front-matter populates meta before validation fires). Add `extract: true` flag handling. Add `lookup(entity)` surface for the extract plugin.
 11. **README rewrite** — recipe-driven Getting Started. Multiple compositions shown.
 12. **mikser-io-example-dms, -pkm, -photos, -invoices** — new example repos. The invoices example shows the PDF → entity → CSV round-trip end-to-end.

package/CLAUDE.md

@@ -10,6 +10,16 @@ paths. No migration markdown. No `task: pool` legacy aliasing for
 `task: inline`. Update READMEs and ADRs in place as source-of-truth
 changes; rewrite, don't supersede. The freedom is the point.
 
+**v9 has to pay out before v10 starts.** Not "ship v9 and start v10."
+**Use v9 in real daily work until the workflow pays out**, then
+v10. The v10 architecture (party-mikser-io) is more fun to design
+than v9 documentation is to write — the gate exists because that
+asymmetry kills projects. Concrete pass/fail signals in
+`10.0-PLAN.md` under "The gate." If a session brings up v10 design
+work, check the gate first; if v9 isn't paying out yet, the
+question to answer is "what about v9 isn't working in daily use?"
+— not "what should v10 look like?"
+
 **Position mikser by what it is, not by speed.** Hugo wins the speed
 race; mikser doesn't compete there. Position: AI-native, lifecycle-
 observable, files-as-source-of-truth, full-cycle introspection. Speed

package/CLOUD-PLAN.md

@@ -0,0 +1,147 @@
+# Mikser Cloud — Planning (stub)
+
+**The shift in one line:** mikser stops being only a CLI/library you install and becomes a hosted product. Pre-configured appliances (invoices, blog, photos, pkm, ...) run as managed containers; a hosted `party-mikser-io` instance orchestrates them per tenant. The OSS substrate doesn't change — cloud is a product built on it, not an extension of it.
+
+This is a strategic stub, not a technical plan. Cloud isn't versioned alongside `mikser-io` (engine versions and product versions iterate independently). The document exists to stake the architectural boundary and name the hard problems before any cloud code gets written.
+
+## The discipline this stub enforces
+
+**OSS earns cloud, not the other way around.** Cloud doesn't get built until `mikser-io` 10.0 has been used in real workloads by users who didn't pay for it. Building cloud on an untested substrate compounds risk — every cloud assumption built on an unvalidated substrate assumption is a refactor waiting to happen.
+
+The boundary check at any cloud-era PR: **does this code reference cloud concepts? If yes → wrong repo unless that repo is `mikser-cloud` or an appliance repo.** No multi-tenancy, no billing hooks, no platform observability in `mikser-io` or `party-mikser-io`.
+
+## What cloud cleanly is
+
+| layer | what ships | where it lives |
+|---|---|---|
+| **Hosted runtime** | Containers running mikser appliances. Persistent storage for files + `runtime/`. Health probes. Restart semantics. | `mikser-cloud` |
+| **Appliances** | Pre-configured mikser images, one per problem domain. Each is `mikser-io` + opinionated plugin set + sample config + starter folder structure. | `mikser-appliance-<name>` per repo |
+| **Party as a service** | Per-tenant hosted `party-mikser-io` instance. Routes discovery / refs / subscribe between the tenant's appliances. | `mikser-cloud` (uses party-mikser-io as a library) |
+| **Control plane** | Web UI: spin up an appliance, connect it to others, expose to an MCP agent, billing, auth, tenant isolation. | `mikser-cloud` |
+
+## Initial appliance set
+
+Each is its own product (own repo, own README, own sample data, own support burden). Underestimating this is the easy mistake — five appliances = five products to maintain.
+
+| appliance | plugin set | the problem it solves |
+|---|---|---|
+| `mikser-appliance-blog` | documents, files, layouts, front-matter, yaml, render-hbs, render-markdown, vector, api, mcp | personal/team blog with semantic search and AI agent integration |
+| `mikser-appliance-invoices` | files, csv, ocr, extract, transformers, schemas, render-csv, api, mcp | drop PDF invoices → typed catalog → exportable CSV → agent answers questions |
+| `mikser-appliance-photos` | files, assets, transformers (image embeddings), vector, api, mcp | drop photo folder → semantic search by description |
+| `mikser-appliance-pkm` | documents, front-matter, yaml, json, layouts, render-markdown, render-hbs, vector, mcp | obsidian-shaped knowledge base with AI agent |
+| `mikser-appliance-dms` | files, ocr, extract, transformers, schemas, vector, api, mcp | document management: drop arbitrary docs → typed extraction → searchable archive |
+
+Each appliance is opinionated. The point of an appliance is "you don't configure plugins, you don't pick a layout engine, you just use it." Power users still get `mikser-io` standalone with full control; appliances are the easy path.
+
+## Hard problems cloud forces us to confront
+
+Not dealbreakers, but real. Each one needs a designed answer before cloud launches.
+
+### Files-as-source-of-truth in a container
+
+The whole `mikser-io` pitch is "drop files in a folder." In a container, where IS that folder?
+
+Options (each with cost):
+
+- **Mounted persistent volume (EBS / pd / etc.)** — simplest mental model. Files live on a real disk per appliance. User edits via web upload or sync agent. Cost: scales per-appliance, snapshot/backup is the platform's problem.
+- **S3-backed FS (s3fs / goofys / mountpoint-s3)** — files live in object storage. Cheap, infinite, but file-system semantics are partial. Chokidar might not work cleanly; rename + metadata operations can be slow or non-atomic.
+- **Git-synced** — appliance pulls from a user-provided git repo. File source-of-truth lives in git, not the container. Best for power users; weird for non-technical ones.
+- **Web upload only** — appliance has a control-plane UI for managing files. No local "folder" per se. Cost: builds a CMS-shaped UI, which is a product surface unto itself.
+
+Working assumption: **persistent volume + web upload UI for non-power-users, optional git sync for power users.** S3-backed FS isn't worth the file-system-semantics tax. **Decision deferred** until first appliance is built.
+
+### Multi-tenancy in the party host
+
+A central party host serving N tenants must route discovery + refs + subscribe per tenant. One user's appliances can't see another's, ever.
+
+`party-mikser-io` itself doesn't need to know about tenants — it's per-process. The cloud's deployment is **one party-mikser-io process per tenant**, sitting in front of that tenant's appliance set. Cost: N processes for N tenants. At scale, that's a real ops bill.
+
+Alternative: a multi-tenant party host with tenant-aware routing. Cost: complicates `party-mikser-io` with tenant-awareness it shouldn't have. Working bias: **per-tenant process** — keeps party-mikser-io clean. Revisit if economics force consolidation.
+
+### Persistent state lifecycle
+
+`runtime/mikser.sqlite` is the catalog cache. ADR-0002: files are truth, the cache is derived. But re-deriving 50k entities + refs + manifest on container restart takes minutes — that's a cold-start UX problem.
+
+Cloud needs the cache to survive restarts. The container's persistent volume holds `runtime/` alongside the file folder. On restart, `mikser` reuses the cache and rebuilds incrementally. This already works in `mikser-io`; cloud just needs the volume mount.
+
+The non-obvious bit: **`runtime/` is per-appliance, not per-tenant**. If a tenant has 3 appliances, that's 3 sqlite files in 3 volumes. Don't try to consolidate.
+
+### Watch-mode vs push
+
+Chokidar is right for a laptop. In cloud, file changes come from API uploads / git pulls / webhook syncs. The trigger model is different.
+
+Easiest path: the sync layer drops files into the volume; chokidar sees them and triggers rebuild. Works out of the box. The sync layer is the cloud platform's problem, not the engine's.
+
+Worth verifying once cloud starts: chokidar on a mounted volume behaves predictably across pod restarts.
+
+### Billing surface
+
+Per-appliance-hour? Per-entity? Per-render? Per-MCP-call?
+
+Each metric is a different observability investment. Per-render and per-MCP-call require engine-side counters that don't exist today. Per-entity is weird (what's an entity, a row of a CSV?). Per-appliance-hour is the simplest honest unit and matches how the user thinks ("I have 3 appliances running").
+
+Working assumption: **per-appliance-hour, with a generous free tier on the smallest plan size.** Per-call billing for the API tier comes later if usage patterns justify it.
+
+### Open-core boundary
+
+The standard hard question. Two cleanly defensible answers:
+
+- **Convenience-only cloud (working bias).** Everything in OSS works standalone. Cloud sells hosting + the appliance catalog + the web UI + the hosted party-mikser-io. No engine feature is paid-only. The OSS substrate stays the same software cloud runs on; contributors stay aligned.
+- **Open-core with paid features.** Some engine features (advanced auth, audit logging, SLA-level support) become cloud/enterprise-only. Cost: contributor alienation, harder to make external use feel first-class.
+
+The convenience-only model has worked for many shops (Cloudron, PostHog, Plausible). The open-core model has worked for others (GitLab, Sentry). For mikser specifically — file-based, agent-native, small surface area — **convenience-only is probably right**, but the decision deserves a real conversation when cloud actually starts.
+
+### Appliance lifecycle as a product surface
+
+Each appliance is its own product. That means:
+
+- Its own README + docs
+- Sample folder structure shipped with the image
+- Sample data for the demo experience
+- Schema definitions (for OCR + extract appliances)
+- Plugin pinning + version compatibility
+- Security review per release
+- Support tickets specific to that appliance's domain
+
+Five appliances at launch = five product tracks. Realistically, **start with one or two** (probably `blog` and `invoices`), validate the appliance-shape pattern, then expand.
+
+## What cloud needs from the substrate (forward-look only — no commit)
+
+Things that *might* land in `mikser-io` or `party-mikser-io` to enable cloud. Each one passes the substrate test independently — they're not cloud-specific:
+
+| candidate | substrate justification |
+|---|---|
+| `GET /api/health` endpoint in mikser-io-api | Operationally useful for any deployment, not just cloud. Probably trivial. |
+| Env-var-based config overrides in mikser-io | Already useful for OSS users running in CI. Cloud just consumes the same surface. |
+| `--rebuild` mode (one-shot rebuild on signal, no chokidar) | Useful for any CI / scheduled-build scenario. |
+| Structured operation metrics (entities created/updated/rendered per cycle) — already partially via pino logs | Useful for any operator wanting observability. |
+| `mikser.config.js` deriving from env vars cleanly | Already standard JS — no engine change needed. |
+| `party-mikser-io` as a standalone server binary, not just a library | Useful for any self-hosted multi-mikser deployment, not just cloud. |
+
+None of these get added pre-emptively for cloud. Each goes through the five-test framework on its own merits, and lands when an OSS user benefits.
+
+## What cloud explicitly is NOT
+
+- **Not an Anthropic / Notion / Glean clone.** Mikser cloud sells hosted mikser, with mikser's specific positioning (file-based, agent-native, OSS substrate). It's not a general note-taking SaaS.
+- **Not a Backend-as-a-Service.** No exposed-to-end-users database API. Each tenant gets appliances, not raw catalog access (except via the appliance's own MCP + api surfaces).
+- **Not a multi-tenant single mikser.** No shared substrate between tenants. Each tenant gets their own container(s); ADR-0002 (files-as-source-of-truth) means a shared catalog across tenants would conflate user data, which is operationally and legally fraught.
+- **Not a competitor to plugin authors.** Anyone shipping a `mikser-io-<name>` plugin can also be an appliance author. The appliance catalog is open; we're not gatekeeping the ecosystem.
+
+## Out of scope (defer until cloud actually starts)
+
+- Pricing — every dimension of it. Tiers, free-tier limits, overages, enterprise contracts. Real numbers need real cost data, which we won't have until we run actual appliances.
+- Compliance posture (SOC 2, GDPR, HIPAA) — depends on which markets cloud targets. Decided when go-to-market is real.
+- Geographic regions — same.
+- White-label / self-hosted cloud for enterprises — possible product variant, but premature.
+- Marketplace for third-party appliances — possible, but starting with first-party only is the sane shape.
+
+## When this stub becomes a real plan
+
+When two things are true:
+
+1. `mikser-io` 10.0 has shipped and has at least one real-world deployment that isn't a toy example.
+2. `party-mikser-io` has shipped and someone outside the immediate dev circle has composed two miksers without help.
+
+If both of those are true, the substrate is validated enough that cloud can build on it confidently. This stub becomes `CLOUD-PLAN.md` proper, with timelines, appliance prioritization, and pricing dimensions.
+
+Until both are true, this document exists to keep cloud as a coherent direction without distorting current engine work. That's the whole purpose of the stub.

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "mikser-io",
-	"version": "8.3.7",
+	"version": "8.3.9",
 	"description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
 	"main": "index.js",
 	"scripts": {

package/src/database/index.js

@@ -25,7 +25,7 @@
 // version, open() throws with a clear "run mikser --clear" message.
 
 import path from 'node:path'
-import { mkdirSync, unlinkSync } from 'node:fs'
+import { mkdirSync, unlinkSync, existsSync } from 'node:fs'
 import Database from 'better-sqlite3'
 import runtime from '../runtime.js'
 import { onLoaded } from '../lifecycle.js'
@@ -49,6 +49,38 @@ const DEFAULT_FILENAME = 'mikser.sqlite'
 // vector_documents in mikser-io-vector).
 const schemas = new Map()
 
+// Provisioning callbacks registered by plugins. Run on every open(),
+// between the raw `new Database(dbPath)` (with PRAGMAs + meta bootstrap)
+// and the schema apply pass. Each callback receives a context object:
+//
+//   {
+//     firstRun:        boolean — db file didn't exist before this open
+//                                (or :memory:, or was wiped by --clear /
+//                                schema mismatch)
+//     upgraded:        boolean — stored schema_version differed from
+//                                current (db was just wiped + recreated)
+//     previousVersion: string | null — the version stamp we found before
+//                                       this open, null if firstRun
+//     currentVersion:  string — what this mikser binary expects
+//     handle:          Database — the raw better-sqlite3 handle, for
+//                                 loadExtension / collations / custom
+//                                 functions / one-time data seeding
+//     logger:          pino|null — for surfacing user-visible setup events
+//   }
+//
+// Use cases:
+//   - Load runtime extensions before they're referenced by schemas
+//     (sqlite-vec's vec0 module is the canonical example)
+//   - Register custom collations, application_id, etc.
+//   - First-run data seeding (use `if (ctx.firstRun)` to gate)
+//   - Upgrade-time data migrations (gate on `ctx.upgraded` and check
+//     `ctx.previousVersion`)
+//
+// Plugins register at module-eval time, same shape as registerSchema.
+// Order doesn't matter unless callbacks depend on each other; if you
+// have such a dep, both should be in the same plugin.
+const provisioners = []
+
 // Active database handle. Set during the first onLoaded; persists across
 // cycles (sqlite stays open for the lifetime of the process). Public API
 // readers see null only before the first onLoaded fires.
@@ -102,6 +134,53 @@ export function registerSchema(name, sqlScript) {
     }
 }
 
+// Register a database-provisioning callback. Fires once per open(),
+// AFTER PRAGMAs and the mikser_meta bootstrap (so version detection
+// and the schema-mismatch wipe have already happened) and BEFORE
+// schema apply (so callbacks can install virtual-table modules,
+// register collations, etc. that the schemas about to apply may
+// reference).
+//
+// Callback receives a context object with everything needed to act
+// situationally:
+//
+//   onProvision(({ firstRun, upgraded, previousVersion, currentVersion, handle, logger }) => {
+//       // Install runtime extensions on every open:
+//       sqliteVec.load(handle)
+//
+//       // Seed default data on first run only:
+//       if (firstRun) {
+//           handle.exec("INSERT INTO mikser_meta (key, value) VALUES ('site_id', '...')")
+//       }
+//
+//       // Migrate after a specific upgrade:
+//       if (upgraded && previousVersion?.startsWith('8.2')) {
+//           handle.exec("UPDATE my_plugin_table SET ...")
+//       }
+//   })
+//
+// Plugins register at module-eval time, same shape as registerSchema.
+// Lazy-applies on the live handle if the database is already open
+// (matches registerSchema's late-registration semantics) — the
+// callback fires with firstRun=false, upgraded=false (since the open
+// already happened cleanly), and the live handle.
+export function onProvision(callback) {
+    if (typeof callback !== 'function') {
+        throw new Error('onProvision: callback must be a function (ctx) => void')
+    }
+    provisioners.push(callback)
+    if (db?.isOpen) {
+        callback({
+            firstRun: false,
+            upgraded: false,
+            previousVersion: db.provisioning?.currentVersion ?? null,
+            currentVersion: db.provisioning?.currentVersion ?? null,
+            handle: db.handle,
+            logger: runtime.engine?.logger ?? null,
+        })
+    }
+}
+
 // Return the active database handle. Hot-path callers (catalog, refs,
 // manifest) should cache the reference once at onLoaded rather than
 // calling per-operation.
@@ -118,8 +197,12 @@ export function useDatabase() {
 // the full onLoaded chain). The runtime path uses this internally from
 // onLoaded below.
 export function createSqliteDatabase({
-    runtimeFolder, version, logger, config = {}, schemas,
+    runtimeFolder, version, logger, config = {}, schemas, provisioners: provisionersArg,
 }) {
+    // Tests inject their own provisioners; the runtime path falls
+    // through to the module-level `provisioners` array that plugins
+    // populate via onProvision() at module-eval.
+    const provisionersToRun = provisionersArg ?? provisioners
     // Resolve the on-disk path. Honor an absolute `config.filename`,
     // otherwise resolve relative to the runtime folder. `:memory:`
     // works for tests — better-sqlite3 treats it as ephemeral.
@@ -132,6 +215,11 @@ export function createSqliteDatabase({
             : path.join(runtimeFolder, DEFAULT_FILENAME)
 
     let handle = null
+    // Provisioning context exposed on the returned wrapper so any code
+    // with a handle (catalog onLoaded, plugin onLoaded, etc.) can ask
+    // "was this open a fresh db / a version upgrade / a normal cycle?"
+    // without subscribing to onProvision.
+    let provisioningCtx = null
 
     function open() {
         if (handle) return  // idempotent — caller may invoke twice across cycles
@@ -140,6 +228,10 @@ export function createSqliteDatabase({
             mkdirSync(path.dirname(dbPath), { recursive: true })
         }
 
+        // Detect firstRun BEFORE we touch the filesystem. :memory:
+        // always counts as firstRun (no persistence across opens).
+        const fileExistedBeforeOpen = dbPath !== ':memory:' && existsSync(dbPath)
+
         const setupConnection = () => {
             handle.exec('PRAGMA journal_mode = WAL')
             handle.exec('PRAGMA synchronous = NORMAL')
@@ -157,6 +249,7 @@ export function createSqliteDatabase({
 
         const recorded = handle.prepare('SELECT value FROM mikser_meta WHERE key = ?')
             .get('schema_version')?.value
+        let upgradedFromVersion = null
         if (recorded && recorded !== version) {
             // Schema mismatch on upgrade or downgrade. Per ADR-0002 the
             // files on disk are the source of truth and this database
@@ -184,12 +277,42 @@ export function createSqliteDatabase({
                 }
             }
 
+            upgradedFromVersion = recorded
             handle = new Database(dbPath)
             setupConnection()
         }
         handle.prepare('INSERT OR REPLACE INTO mikser_meta (key, value) VALUES (?, ?)')
             .run('schema_version', version)
 
+        // Build provisioning context. firstRun is true when the file
+        // didn't exist before this open OR when the schema mismatch
+        // wiped it (and the previous open's stamp is gone) — both shapes
+        // present an empty-state database to provisioners.
+        provisioningCtx = {
+            firstRun: !fileExistedBeforeOpen || upgradedFromVersion !== null,
+            upgraded: upgradedFromVersion !== null,
+            previousVersion: upgradedFromVersion,
+            currentVersion: version,
+            handle,
+            logger: logger ?? null,
+        }
+
+        // Fire provisioning callbacks before schema apply, so they can
+        // load runtime extensions (sqlite-vec's vec0, etc.) that the
+        // schemas about to apply might reference, register custom
+        // collations, or do first-run / upgrade work that needs to
+        // happen before the engine tables exist.
+        for (const provision of provisionersToRun) {
+            try {
+                provision(provisioningCtx)
+            } catch (err) {
+                handle.close()
+                handle = null
+                provisioningCtx = null
+                throw new Error(`onProvision callback failed: ${err.message}`)
+            }
+        }
+
         // Apply each subsystem's registered schema script. Idempotent
         // CREATE statements mean replay-safe across opens.
         for (const [name, sqlScript] of schemas) {
@@ -199,6 +322,7 @@ export function createSqliteDatabase({
             } catch (err) {
                 handle.close()
                 handle = null
+                provisioningCtx = null
                 throw new Error(`Schema "${name}" failed to apply: ${err.message}`)
             }
         }
@@ -226,6 +350,12 @@ export function createSqliteDatabase({
         open,
         close,
         get isOpen() { return handle !== null },
+        // Provisioning context from the most recent open. Plugins'
+        // onLoaded handlers can read this without subscribing to
+        // onProvision — useful when the work needs the catalog or
+        // refs schemas to exist first (which onProvision callbacks
+        // can't assume because they fire before schema apply).
+        get provisioning() { return provisioningCtx },
         // Pass-through primitives for subsystems. They prepare their
         // own statements at the underlying better-sqlite3 handle.
         prepare(sql) {