mikser-io 8.3.2 → 8.3.7

package/9.0-PLAN.md

@@ -0,0 +1,497 @@
+# Mikser 9.0 — Planning
+
+**The shift in one line:** `mikser-io` moves from "AI-native SSG" to "file-based knowledge substrate with AI superpowers." Layouts (and the other rendering plugins currently bundled in `src/plugins/`) move out to their own packages. A new AI substrate plugin family (`mikser-io-transformers` and friends) registers inference pipelines that consumer plugins compose against.
+
+This is a planning document, not an ADR. The ADR(s) land after the move so the wording reflects what actually shipped. This file gets dropped at release.
+
+## Why
+
+The OCR + structured-extraction conversation forced the question: when "drop a folder of PDFs and ask an AI agent about them" works as a primary use case, is layouts still in the substrate's critical path? No. SSG is one composition of the substrate, not the substrate itself.
+
+Re-running ADR-0006's five-test against `layouts`:
+
+| test | result |
+|---|---|
+| 1. Substrate? Engine genuinely needs this | **No** — document/knowledge consumers don't render layouts |
+| 2. Strengthens strategy? Load-bearing for positioning | **Was. Isn't post-shift.** |
+| 3. God-plugin check? Engine becoming kitchen sink | **Yes** — ~900 LOC of pagination + sidecar + sitemap on top of dispatch |
+| 4. Composability? Could ship external | **Yes** — only uses public surfaces |
+| 5. Release cadence? Same as engine | **No** — iterates at SSG-feature cadence |
+
+Three fails. Test 2 became a fail when positioning shifted. The architecture doesn't believe in layouts-in-core anymore; only inertia does.
+
+## What stays in the engine
+
+`src/plugins/` slims from 14 directories to 5:
+
+```
+documents · files · front-matter · yaml · json
+```
+
+These are entity-sourcing primitives — any consumer reading from a working folder needs them. Plus the substrate code (`runtime.js`, `engine.js`, `lifecycle.js`, `catalog.js`, `refs.js`, `manifest.js`, `journal.js`, `database/`, `source.js`, `subscriptions.js`, `track.js`, `render.js`/`postprocess.js` as dispatch shims — the actual rendering lives in plugins).
+
+## What moves out
+
+Each becomes its own package with its own version cadence:
+
+| package | what moves | approx. LOC |
+|---|---|---|
+| `mikser-io-layouts` | layout matching + pagination + sidecar protocol | ~900 |
+| `mikser-io-assets` | preset processing + binary pipelines | ~600 |
+| `mikser-io-resources` | external resource fetching | ~400 |
+| `mikser-io-preview` | in-memory render cache + `GET /preview/` | ~250 |
+| `mikser-io-data` | JSON catalog exports | ~150 |
+| `mikser-io-observer` | HTTP source polling | ~200 |
+| `mikser-io-mapper` | config-driven field rewrites | ~100 |
+| `mikser-io-validator` | custom validation hooks | ~100 |
+| `mikser-io-commands` | cron jobs | ~150 |
+| `mikser-io-shares` | multi-output replication | ~200 |
+
+Engine LOC drops ~40%. None of these plugins is wrong to ship — they're just not substrate.
+
+## New: AI substrate
+
+The substrate exposes a verb-level pipelines surface:
+
+```js
+runtime.options.pipelines = {
+    embed:      (text, opts?)                  => Promise<number[]>,
+    imageEmbed: (imageBuffer, opts?)           => Promise<number[]>,
+    ocr:        (input, opts?)                 => Promise<string>,
+    extract:    (input, { schema, prompt }, opts?) => Promise<unknown>,
+    classify:   (text, labels, opts?)          => Promise<{ label, score }[]>,
+    rerank:     (query, docs, opts?)           => Promise<{ idx, score }[]>,
+    summarize:  (text, opts?)                  => Promise<string>,
+    transcribe: (audioBuffer, opts?)           => Promise<string>,
+    translate:  (text, { from, to }, opts?)    => Promise<string>,
+}
+```
+
+All pipeline functions accept `{ signal }` so consumer plugins can pass through the cycle's abort signal. Watch-mode cancellations short-circuit pending inference cleanly.
+
+### Provider plugins
+
+Each provider plugin registers only the pipelines it can support:
+
+| plugin | typical pipelines | runtime requirement |
+|---|---|---|
+| `mikser-io-transformers` | embed, imageEmbed, ocr, extract, classify, rerank, summarize, transcribe | `@xenova/transformers`; first-run model download |
+| `mikser-io-openai` | embed, extract (via JSON mode), classify, summarize, transcribe | `OPENAI_API_KEY` |
+| `mikser-io-ollama` | embed, summarize (depends on pulled models) | ollama daemon |
+| `mikser-io-anthropic` | extract, classify, summarize | `ANTHROPIC_API_KEY` |
+
+Consumer plugins gate on what's registered:
+
+```js
+onLoaded(() => {
+    if (!runtime.options.pipelines?.ocr) {
+        logger.warn(
+            'mikser-io-ocr needs an ocr pipeline — install mikser-io-transformers ' +
+            '(or another provider that registers ocr).'
+        )
+        return
+    }
+    // wire up
+})
+```
+
+Same gate pattern the api / preview / mcp plugins already use against `runtime.options.app` / `runtime.options.mcp`.
+
+## New consumer plugins
+
+| package | needs pipeline | what it does |
+|---|---|---|
+| `mikser-io-vector` | embed | semantic search over text (refactored to depend on substrate) |
+| `mikser-io-ocr` | ocr | populate `entity.content` from PDF/image source files |
+| `mikser-io-extract` | extract | schema-driven structured extraction into `entity.meta` |
+| `mikser-io-image-search` | imageEmbed | CLIP-based image search; text query → matching images |
+| `mikser-io-rerank` | rerank | post-process vector search results with cross-encoder |
+| `mikser-io-summarize` | summarize | auto-populate `meta.summary` when authors don't |
+| `mikser-io-classify` | classify | zero-shot tag suggestions per entity create/update |
+| `mikser-io-transcribe` | transcribe | podcasts/videos → searchable transcripts |
+| `mikser-io-translate` | translate | multilingual content |
+
+Each is small — 100-400 LOC. The substrate handles model lifecycle (download, cache, progress, dimension introspection); the consumer handles the entity-lifecycle integration.
+
+## Format liberation: source ↔ render symmetry
+
+Mikser already has rendering plugins that emit entities AS specific formats (`render-hbs`, `render-eta`, `render-liquid`, `render-markdown`, `render-file`). The 9.0 lens reveals the missing other half: matching source plugins that parse FROM specific formats INTO entities. The substrate becomes a **format translation layer** — anything in, anything out, all composing on the same catalog.
+
+### The symmetry
+
+| source plugin (format → entities) | render plugin (entities → format) | format |
+|---|---|---|
+| `documents` (md/html/yml/json) | `render-hbs`, `render-eta`, `render-liquid`, `render-markdown` | text documents |
+| `files` (any binary) | `post-pdf`, `post-mjml` | binary outputs |
+| `mikser-io-csv` *(new)* | `mikser-io-render-csv` *(new)* | CSV |
+| existing (api plugin's data exports) | `mikser-io-data` *(extracted in 9.0)* | JSON snapshots |
+| `mikser-io-ocr` + `mikser-io-extract` (PDF/image → entities) | `mikser-io-render-pdf` *(future)* | PDF documents |
+| future: `mikser-io-ical`, `mikser-io-vcard`, `mikser-io-rss` | future: matching renderers | calendars, contacts, feeds |
+
+The substrate doesn't care which side a format lives on. The catalog's job is to hold queryable entities; source plugins fill it from whatever the world produces; render plugins emit whatever the world consumes.
+
+### The round-trip that demonstrates it
+
+A folder of scanned PDF invoices walks through the entire stack and lands as a CSV for the accountant — without anyone writing pipeline code:
+
+```
+PDFs in documents/invoices/
+    ↓ documents plugin             (file → entity, raw)
+    ↓ mikser-io-ocr                (OCR pipeline populates entity.content)
+    ↓ mikser-io-extract            (schema-driven, populates entity.meta)
+        → catalog now holds typed invoice entities
+    ↓ mikser-io-render-csv         (entities → rows in invoices.csv)
+    + mikser-io-vector             (semantic search across invoices)
+    + mikser-io-mcp                (agent answers "which Q1 vendors charged us most")
+    + mikser-io-api                (HTTP /api/public/entities/invoices)
+
+Each individual file → live-reload on watch-mode
+A schema change → re-extraction on next cycle
+A new PDF dropped in → automatic ingest + index + export
+```
+
+Edit a PDF, save it, the CSV regenerates with the corrected row a couple seconds later. The accountant pulls `invoices.csv` from a synced folder. The AI agent answers questions. The API exposes the data. **All from the same canonical source — the PDFs on disk.**
+
+### Why this is a positioning shift, not just a feature list
+
+Other tools trap your data in their source format:
+
+- **Notion** — your data lives in their proprietary database tables; export is a deliberate, lossy operation
+- **Airtable** — same shape with a different paint job
+- **Glean** — indexes your data but you can't re-emit it in another format cleanly
+- **OneDrive / Dropbox / Box** — store files but don't make their content queryable
+
+Mikser flips this. Source format is where data enters; **what shape it takes after that is the user's call.** PDF in → CSV out. CSV in → HTML out. Markdown in → semantic embeddings out. Anything in → anything out, with watch-mode keeping every output current as the inputs change.
+
+The 9.0 plugin set turns this from architectural potential into a shipped product. With `csv`, `render-csv`, `ocr`, `extract`, `transformers`, `vector`, and the existing render-* family, mikser is a real "knowledge format translation substrate" rather than "an SSG that has some AI hooks."
+
+### What lands in 9.0 specifically
+
+| package | role | LOC est. |
+|---|---|---|
+| `mikser-io-csv` | source: CSV file → row entities (one entity per row; `idColumn` config for stable ids) | ~250 |
+| `mikser-io-render-csv` | render: entities → CSV file (RFC 4180 escaping; column ordering from sidecar) | ~150 |
+
+Future render/source pairs (post-9.0, demand-driven): `render-pdf`, `render-rss`, `render-ical`, `ical` source, `vcard` source. Each ~150-300 LOC. The substrate carries them.
+
+## Streaming render output
+
+Row-as-entity (`mikser-io-csv`, 1M-row catalogs) breaks an assumption the current render path quietly relies on: that the output of `render(entity)` is a string or `Buffer` you can hand to `writeFile`. A render that serializes 1M rows to a single 500MB CSV string is the worst kind of failure mode — it works in dev with 100 rows, OOMs in production.
+
+The substrate already streams **input** (`iterateEntities` is a seek-paginated async generator, ADR-0009 phase 5). Output is the other half.
+
+### What changes in the render dispatch
+
+`render.js` currently expects each renderer to return `string | Buffer`. Under 9.0 the contract widens:
+
+```js
+return string                            // current path — buffer to disk
+return Buffer                            // current path — buffer to disk
+return Readable                          // new — pipe to a write stream
+return AsyncIterable<string | Buffer>    // new — drained into a write stream
+```
+
+The dispatcher inspects the return value and routes:
+
+- string/Buffer → `writeFile(destination, result)` (unchanged hot path; everything that exists today keeps working)
+- Readable or async iterable → `await pipeline(result, createWriteStream(destination))`
+
+Postprocess plugins (`render-file` for inlining, `mjml`, `pdf`, etc.) opt out of the streaming path automatically when they need a buffered input — they declare `bufferedInput: true` and the dispatcher collects the stream first. Most postprocess plugins already need the full document (HTML rewrite, MJML compile), so this is the common case; streaming is the optimization for renderers whose output is naturally append-only.
+
+### What this enables
+
+| renderer | streaming wins because |
+|---|---|
+| `render-csv` | row count is corpus-scale; output is line-by-line append; no postprocess for CSV |
+| `render-rss` / `render-atom` | item list is corpus-scale; XML is append-only per item |
+| `render-ical` | event list is corpus-scale |
+| `data` (JSON catalog export) | the corpus IS the output; current code buffers the whole catalog |
+| future `render-ndjson` | naturally one entity per line |
+| `render-hbs` / `render-eta` / `render-markdown` | NOT a win — template engines produce one document, postprocess (front-matter inlining, link rewriting) needs the full string anyway |
+
+### Memory budget the streaming path holds
+
+At 1M rows, `render-csv` running in streaming mode should never hold more than one row's worth of formatted output in memory. The peak isn't the row count, it's the slowest downstream consumer (disk write speed). On a SATA SSD, a 500MB CSV writes at ~400MB/sec — under 2 seconds wall-clock for the full file, with the JS heap never crossing 50MB for the render itself.
+
+The buffered path on the same workload: 500MB CSV string in V8 = ~1GB after the UTF-16 doubling, then a second 500MB allocation for the `Buffer` conversion, then GC pressure. Streaming isn't a 10% optimization here; it's the difference between working and OOM.
+
+### What this doesn't change
+
+- HTML rendering paths (the dominant workload) keep returning strings. No measurable cost.
+- The journal's auto-persist diff still works — it diffs the entity object, not the output.
+- Manifest checksumming streams over the output too (`createHash` accepts piped data) — no extra read.
+
+### Scope for 9.0
+
+Land the dispatcher change in the engine alongside `mikser-io-csv` + `mikser-io-render-csv`. The CSV plugins are the first consumers; they validate the contract. `data` plugin migrates to streaming as a second step (separate release). All existing renderers stay on the string path; the widening is additive.
+
+Add to the work list: dispatcher widening + `render-csv` shipping in streaming mode as the proof.
+
+## Schemas plugin pulls real weight here
+
+`mikser-io-schemas` has been sitting idle. Under the 9.0 shape it becomes load-bearing:
+
+- Declares entity shapes via zod
+- `extract: true` flag per schema turns the declaration into an auto-extraction target
+- `mikser-io-extract` walks new entities, looks up the schema for the entity's `meta.layout`, calls `runtime.options.pipelines.extract(input, { schema })`, validates the result, populates `entity.meta`
+- Validated structured data becomes queryable through standard indexed columns
+
+Drop a folder of PDF invoices → schema declares `{vendor, date, items, total}` → catalog has fully-typed invoice rows → `findEntities({'meta.layout': 'invoice', 'meta.total': {$gt: 5000}})` returns them.
+
+The schemas plugin also needs the lifecycle-ordering bug fixed before this lands (currently `onValidate` fires before `front-matter` populates `meta`). Separate but related work.
+
+## Recipe compositions
+
+**SSG** (today's flagship — same shape, different distribution):
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'layouts',
+    'render-hbs', 'render-markdown', 'render-href',
+    'assets', 'resources', 'preview',
+    'data',
+]
+```
+
+**Document management system**:
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'transformers', 'ocr', 'extract',
+    'schemas',
+    'vector',
+    'api', 'mcp',
+]
+```
+
+**Personal knowledge management**:
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'transformers', 'ocr', 'extract', 'schemas',
+    'vector',
+    'mcp',
+]
+```
+
+No HTTP — Claude Desktop talks MCP directly. An Obsidian competitor in ~8 plugins.
+
+**Photo library with semantic search**:
+
+```js
+plugins: [
+    'files',
+    'transformers', 'image-search',
+    'api', 'mcp',
+]
+```
+
+No `documents` even. Just files + image embeddings + access.
+
+**Tabular data workflow** (the round-trip from the format-liberation section):
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'csv',                                       // CSV files → row entities
+    'transformers', 'ocr', 'extract',            // PDFs → row entities via schema
+    'schemas',
+    'render-csv', 'render-hbs',                  // entities → CSV + HTML
+    'data',
+    'api', 'mcp',
+]
+```
+
+PDF invoices and CSV exports land as the same entity shape. Editor renders a dashboard HTML page; accountant downloads regenerated CSV; agent answers semantic questions. One source of truth, three output formats.
+
+**Newsletter platform**:
+
+```js
+plugins: [
+    'documents', 'files', 'front-matter', 'yaml',
+    'layouts', 'render-hbs',
+    'post-mjml', 'post-mailchimp',
+    'data',
+]
+```
+
+Each composition is a coherent product. None of them carry weight they don't need.
+
+## Migration shape
+
+ADR-0002 (files-as-truth) plus the schema-version recovery pattern shipped in 8.3.7 carry the migration. The engine knows the world has changed and tells the user how to fix it.
+
+1. **`mikser-io@9.0.0`** — `src/plugins/` slimmed to the five engine plugins. Legacy stub modules at the old paths throw a clear error:
+
+   ```
+   Plugin "layouts" not found in src/plugins/. It has moved to its own
+   package — run: npm install mikser-io-layouts
+   ```
+
+   `schema_version` stamp bumps; auto-recover wipes the cache on first run (no manual `--clear` needed, per 8.3.7).
+
+2. **New plugin repos shipped at the same time**:
+   - `mikser-io-layouts` (the big one)
+   - `mikser-io-assets`, `mikser-io-resources`, `mikser-io-preview`, `mikser-io-data`
+   - `mikser-io-observer`, `mikser-io-mapper`, `mikser-io-validator`, `mikser-io-commands`, `mikser-io-shares`
+
+3. **New AI plugin repos**:
+   - `mikser-io-transformers` (substrate)
+   - `mikser-io-ocr`, `mikser-io-extract` (consumer plugins for the document workflow)
+   - Others land on rolling cadence as demand justifies
+
+4. **`mikser-io-vector` bumps to 2.0.0** — drops its OpenAI HTTP client, depends on the pipelines substrate. Old config keys (`vector.openai.apiKey`) keep working through a one-time deprecation cycle if `mikser-io-openai` is loaded; recommended path is `mikser-io-transformers` (no API key, fully offline).
+
+5. **README rewrite** — "Getting Started" splits into recipes. The current single-default-everyone-gets-SSG path becomes one of several. Each recipe shows its plugin list, the workflow it enables, and links to an example repo.
+
+6. **Example repos**:
+   - `mikser-io-example-blog` (existing — the SSG recipe)
+   - `mikser-io-example-dms` (new — document management)
+   - `mikser-io-example-pkm` (new — personal knowledge management)
+   - `mikser-io-example-photos` (new — image library)
+
+## `mikser --install` — explicit plugin installation
+
+The plugin count balloons from "everyone uses the default" to "everyone composes a recipe." A document management recipe is 10 packages; an SSG recipe is 12. Without a clean install story, every recipe shift is "edit `package.json`, `npm install`, hit a missing plugin, repeat."
+
+But the engine shouldn't shell out to a package manager during normal runs. No magic, no surprise installs as a side effect of `mikser`. Package management is its own step, run explicitly.
+
+### Shape
+
+`mikser --install` reads `mikser.config.js`, identifies which plugins aren't resolvable via the existing four-step lookup chain, batches them into one install call, and exits. No `onInitialize`, no `onLoaded`, no render, no watch — the substrate doesn't even open `mikser.sqlite`. Just dependency resolution and the package install.
+
+```
+$ mikser --install
+
+🟡 Plugins not installed locally: layouts, vector, transformers
+   Resolving to: mikser-io-layouts, mikser-io-vector, mikser-io-transformers
+   Running: npm install --save mikser-io-layouts mikser-io-vector mikser-io-transformers
+   [npm output...]
+✓ Installed 3 plugins (4.2s)
+
+$ mikser
+🟢 Continuing normally...
+```
+
+Plain `mikser` without `--install` does **not** auto-install. If a plugin is missing, the engine fails with a clear error pointing at the install command:
+
+```
+$ mikser
+
+🔴 Plugin "layouts" not found. Install with:
+   mikser --install
+
+   (or directly: npm install mikser-io-layouts)
+```
+
+The fallback path is the same as the schema-version recovery shipped in 8.3.7 — engine explains exactly what's wrong and exactly how to fix it. The fix is one command.
+
+### Implementation notes
+
+- **Package name resolution.** Plugin name `layouts` → package `mikser-io-layouts`. Same convention as today's loader. User can override per-plugin in config when they need a non-standard package name:
+
+  ```js
+  plugins: [
+      'documents',
+      { name: 'custom', package: 'my-org-mikser-custom' },
+  ]
+  ```
+
+- **Package manager detection.** Detect from lock file in the working folder:
+  - `package-lock.json` → npm
+  - `yarn.lock` → yarn
+  - `pnpm-lock.yaml` → pnpm
+  - `bun.lockb` → bun
+  - Default to npm if no lock file detected. CLI override via `--package-manager` for edge cases.
+
+- **Batched install.** Scan all missing plugins, install in one call. Ten missing plugins → one network round-trip, not ten.
+
+- **Persistence via `--save`.** Updates `package.json` so the deps are recorded. Future `mikser` invocations find them via normal node resolution. No state hidden in `runtime/` or anywhere else.
+
+- **Version pinning.** Pin to the engine's matching major: `mikser-io@9.x` installs `mikser-io-layouts@^9`. Prevents accidental cross-major drift. Users can override per-plugin with explicit version in the config object form (`{name, package, version}`).
+
+- **Safe-by-name.** Only resolve plugin names matching the `mikser-io-*` convention (or the explicit `package:` override). Catches typos before installing random packages — `plugins: ['layoutz']` errors at resolution time instead of silently installing `mikser-io-layoutz`.
+
+- **Network failure → clear error.** No network, no install. Standard npm error with the engine's note: "Run when you have network connectivity, or install the listed packages manually."
+
+- **`--install` is loud.** The install step logs every resolved package name, the exact command run, the duration, and the result. The user explicitly asked for this; show them what happened.
+
+### CLI
+
+```
+mikser                            # normal run; fails clearly on missing plugins
+mikser --install                  # resolve + install missing plugins; exit
+mikser --install --package-manager=pnpm   # override lock-file detection
+```
+
+That's the whole surface. No `--no-auto-install` (nothing to disable). No `NODE_ENV` branch (no behavior to gate on environment).
+
+### Why no auto-install
+
+Two compounding reasons:
+
+1. **No magic during normal runs.** `mikser` is for building. `mikser --install` is for installing. One command does one thing. The engine doesn't shell out to npm as a side effect of starting the lifecycle — that would surprise CI, surprise production deploys, surprise air-gapped environments, and surprise anyone debugging "why did `mikser` take 30 seconds to start?"
+
+2. **It matches the rest of the npm ecosystem.** `node app.js` doesn't auto-install missing packages. `next build` doesn't. `vite` doesn't. Adding auto-install to mikser would be the odd one out for no real win — the install command is one line, the error message tells you exactly what to run, the workflow is "edit config → `mikser --install` → `mikser`" which is the same shape every JS project already follows.
+
+The friction the auto-install would have saved is one explicit command after editing the plugin list. The cost would have been: engine that may install packages during `mikser`, harder-to-reason-about CI, NODE_ENV branches, watch-mode interaction edge cases, and a "what just got installed?" question on every cold cycle.
+
+The trade isn't worth it. `--install` as the explicit, opt-in command is the right shape.
+
+### What this isn't
+
+- **Not a package manager.** Mikser shells out to npm/yarn/pnpm/bun for the install. No proprietary install logic, no opinion about lockfiles, no version solver. The engine just identifies which packages need to be installed and asks the user's existing tool to do it.
+- **Not a cross-major migration tool.** Pins to the matching major. Pre-9 plugins don't get upgraded through this path; users explicitly `npm install mikser-io-X@9` if they need to.
+
+## Out of scope / deferred
+
+- **ADR-0010** ("the engine is the substrate; rendering is a plugin"). Written when the move is done so the wording reflects what shipped, not a forward-projection. CLAUDE.md note added at that point too.
+- **Plugin marketplace / discovery**. Not building this. README recipes + ecosystem docs handle discovery. Anyone shipping a third-party plugin uses the existing `mikser-io-<name>` convention.
+- **Automated migration tool**. The legacy stub errors give the `npm install` command. That's the migration tool. Users running `mikser` after upgrade get told what to install per missing plugin; one `npm install` per package gets them running again.
+
+## Open questions to settle before shipping
+
+1. **Naming.** "Engine plugins" vs "substrate" vs "core" — what's the term we use in the README to distinguish what ships in `mikser-io` from what doesn't? Working proposal: **"substrate"** for the engine-only pieces (catalog, refs, manifest, journal, the 5 entity-sourcing plugins), **"plugins"** for everything else (internal or external — no asymmetry in language because there's no asymmetry in mechanism).
+
+2. **Pipeline signal contract.** All pipeline functions accept `{ signal }` for abort. Required by the substrate's TypeScript types? Or convention with the substrate enforcing at registration time? Working proposal: **required** — the substrate validates registered functions have the signature.
+
+3. **Multiple provider plugins loaded at once.** Can `mikser-io-transformers` AND `mikser-io-openai` both register `embed`? If yes, which one wins for the shared verb name, and can consumers explicitly request one vendor? Working proposal: **registration order wins for the shared name**. Consumer can reach a specific provider via `runtime.options.pipelines.embed.openai(text)` for vendor-specific cases. The substrate exposes both.
+
+4. **Schemas plugin integration with extract.** `mikser-io-extract` needs to read schema declarations from `mikser-io-schemas` to know what shape to coerce into. Two options:
+   - (a) schemas plugin exposes `runtime.options.schemas.lookup(entity)` → returns the matching zod schema. Extract consumes that surface.
+   - (b) schemas plugin enriches entities with the schema reference attached; extract reads it off the entity.
+   Working proposal: **(a)** — lookup surface. Keeps the schema declaration in one place; extract reads on demand.
+
+5. **Version bump cascade.** `mikser-io` 9.0, `mikser-io-vector` 2.0, layouts/assets/etc. 1.0 (initial extraction releases). Coordinated or independent? Working proposal: **coordinated for the initial 9.0 set** (so users can `npm install` a consistent slate); independent cadence after.
+
+6. **What about `mikser-io-render-*`?** Those already live external. The question: do they stay shipped from their own repos as before, or get consolidated under `mikser-io-layouts` as siblings? Working proposal: **stay external as today** — they're rendering-engine adapters (hbs / eta / liquid / markdown / file), one per template language, and consolidating them serves nobody.
+
+7. **Postprocess opt-in vs opt-out for buffered input.** The streaming render dispatcher needs to know whether a postprocess plugin can consume a stream. Two shapes:
+   - (a) postprocess plugins declare `bufferedInput: true` on registration; default is "can stream"
+   - (b) postprocess plugins declare `streamInput: true`; default is "needs buffer" (current behavior)
+   Working proposal: **(b)** — default to buffered so existing plugins (mjml, pdf, front-matter-rewrite) keep working without code changes. Plugins opt into streaming only when the format genuinely supports it (CSV row-rewrite, NDJSON line-transform). One-line migration cost beats silently breaking the plugin set.
+
+8. **Worker path for streaming renders.** TASKS.WORKER serializes return values through Piscina's structured clone — a Readable stream doesn't survive that boundary. Working proposal: **streaming renders are INLINE/SERIAL only for 9.0**. A renderer that declares streaming output is force-dispatched on the main thread regardless of its `task:` frontmatter. Documented as a limitation, not a bug. Revisit when Piscina ships transferable-stream support or we move to MessagePort-based piping (post-9.0).
+
+
+## Concretely, what work this is
+
+Roughly in dependency order:
+
+1. **mikser-io engine** — slim `src/plugins/`, add legacy stub error files at the old paths, document the `runtime.options.pipelines` surface in `documentation/api-reference.md`. Pipeline functions are no-op until a provider plugin registers them. **`mikser --install` command** (see section above) lands here too — same release, so users have a one-command way to populate a recipe's plugin set after editing config. **Streaming render output**: widen the render dispatcher in `src/render.js` to accept `Readable` and `AsyncIterable` returns; pipe to `createWriteStream` instead of `writeFile`. Postprocess plugins declare `bufferedInput: true` to opt back into the buffered path. String/Buffer returns unchanged.
+2. **mikser-io-transformers** — new repo. Wire @xenova/transformers; register embed, ocr, extract, classify, rerank, summarize, transcribe. Handle model download, cache directory, progress bar, dimension introspection.
+3. **mikser-io-vector 2.0** — drop OpenAI HTTP client. Depend on substrate. Validate dimension against the registered embed pipeline.
+4. **mikser-io-layouts** — new repo. Extract `src/plugins/layouts.js` verbatim. CI + tests.
+5. **mikser-io-assets, mikser-io-resources, mikser-io-preview, mikser-io-data** — new repos. Verbatim extraction with their own version cadence.
+6. **mikser-io-ocr** — new repo. Consumer plugin for the ocr pipeline.
+7. **mikser-io-extract** — new repo. Consumer plugin for the extract pipeline. Wire into mikser-io-schemas's `lookup()` surface.
+8. **mikser-io-csv** — new repo. Source plugin: each CSV row becomes a queryable entity. `idColumn` config for stable per-row ids; checksum-gated re-emit on file change.
+9. **mikser-io-render-csv** — new repo. Render plugin. RFC 4180 escaping; aggregations from sidecar `findEntities()` calls; first non-text-document render plugin in the family. **Ships in streaming mode** — returns an async iterable from `render()`, validating the widened dispatcher contract end-to-end. The "1M-row CSV builds in <2s without OOM" smoke test lives in this repo.
+10. **mikser-io-schemas** — fix lifecycle ordering (front-matter populates meta before validation fires). Add `extract: true` flag handling. Add `lookup(entity)` surface for the extract plugin.
+11. **README rewrite** — recipe-driven Getting Started. Multiple compositions shown.
+12. **mikser-io-example-dms, -pkm, -photos, -invoices** — new example repos. The invoices example shows the PDF → entity → CSV round-trip end-to-end.
+
+Items 1-3 are blocking for the 9.0 release. Items 4-8 ship at 9.0 too (so users have something to install when the engine tells them a plugin moved). Items 9-10 follow but should aim for the same release window — without recipe docs, users don't know which composition to install.

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "mikser-io",
-	"version": "8.3.2",
+	"version": "8.3.7",
 	"description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
 	"main": "index.js",
 	"scripts": {

package/src/catalog.js

@@ -71,9 +71,38 @@ registerSchema('mikser_entities', `
     CREATE INDEX IF NOT EXISTS idx_mikser_entities_uri         ON mikser_entities(uri);
 `)
 
+// Per-process dedupe of the no-filter findEntities/iterateEntities
+// warning. Keyed by the rendering entity id (which is what gets
+// blamed in the recorded refClosure). Once per offending site is
+// enough — the warning is educational, not load-bearing.
+const _warnedNullFilter = new Set()
+
 function recordQuery(filter) {
     const ctx = queryContext.getStore()
     if (!ctx?.track) return
+    // Null/undefined filter records as `null` in the snapshot's
+    // refClosure, which manifest.shouldSkip and manifest.queryAffected
+    // treat as "any mutation could have affected this render."
+    // Architecturally correct, but it means an aggregate layout whose
+    // sidecar calls findEntities() with no args invalidates on every
+    // single CREATE/UPDATE/DELETE — including spurious ones (plugins
+    // re-emitting unchanged entities, etc.).
+    //
+    // Warn once per rendering entity so authors can narrow the filter.
+    // The fix is almost always to add the collection / type / format
+    // dimension the sidecar actually cares about; the JS-side .filter()
+    // chain that usually follows findEntities() is the signal that the
+    // filter belongs in the SQL.
+    if ((filter === undefined || filter === null) && ctx.entityId) {
+        if (!_warnedNullFilter.has(ctx.entityId)) {
+            _warnedNullFilter.add(ctx.entityId)
+            const logger = useLogger()
+            logger?.warn(
+                'findEntities()/iterateEntities() called with no filter from %s — recorded query dep invalidates on every mutation. For "all renderable entities" use findEntities({"meta.href": {$exists: true}}). For narrower scopes use any indexed column ({collection, type, format, "meta.layout", "meta.lang"}); pushing the filter into SQL keeps invalidation precise.',
+                ctx.entityId,
+            )
+        }
+    }
     ctx.track.query(normalizeFilter(filter))
 }
 

package/src/database/index.js

@@ -25,7 +25,7 @@
 // version, open() throws with a clear "run mikser --clear" message.
 
 import path from 'node:path'
-import { mkdirSync } from 'node:fs'
+import { mkdirSync, unlinkSync } from 'node:fs'
 import Database from 'better-sqlite3'
 import runtime from '../runtime.js'
 import { onLoaded } from '../lifecycle.js'
@@ -140,30 +140,52 @@ export function createSqliteDatabase({
             mkdirSync(path.dirname(dbPath), { recursive: true })
         }
 
-        handle = new Database(dbPath)
-
-        handle.exec('PRAGMA journal_mode = WAL')
-        handle.exec('PRAGMA synchronous = NORMAL')
-        handle.exec('PRAGMA foreign_keys = ON')
+        const setupConnection = () => {
+            handle.exec('PRAGMA journal_mode = WAL')
+            handle.exec('PRAGMA synchronous = NORMAL')
+            handle.exec('PRAGMA foreign_keys = ON')
+            handle.exec(`
+                CREATE TABLE IF NOT EXISTS mikser_meta (
+                    key   TEXT PRIMARY KEY,
+                    value TEXT NOT NULL
+                )
+            `)
+        }
 
-        // meta table always exists — holds the schema_version stamp
-        // and any future engine-wide key/value state.
-        handle.exec(`
-            CREATE TABLE IF NOT EXISTS mikser_meta (
-                key   TEXT PRIMARY KEY,
-                value TEXT NOT NULL
-            )
-        `)
+        handle = new Database(dbPath)
+        setupConnection()
 
         const recorded = handle.prepare('SELECT value FROM mikser_meta WHERE key = ?')
             .get('schema_version')?.value
         if (recorded && recorded !== version) {
+            // Schema mismatch on upgrade or downgrade. Per ADR-0002 the
+            // files on disk are the source of truth and this database
+            // is a derived cache, so the right behavior is to wipe the
+            // cache and let the next cycle rebuild it from source —
+            // not to halt the build with an error.
+            //
+            // Loud warning so operators can see it happened and know to
+            // expect a cold-start rebuild on this run. No data loss
+            // beyond the cache itself; everything in mikser.sqlite is
+            // recoverable from the working folder.
+            logger?.warn(
+                'Database schema mismatch: stored=%s, current=%s. Wiping the cache and rebuilding from sources (files are the source of truth — no source data is affected).',
+                recorded, version,
+            )
             handle.close()
             handle = null
-            throw new Error(
-                `Database schema is ${recorded}; this mikser-io expects ${version}. ` +
-                `Run \`mikser --clear\` to rebuild from sources.`,
-            )
+
+            if (dbPath !== ':memory:') {
+                // sqlite WAL leaves -wal and -shm sidecar files. Remove
+                // them along with the main file so the next open starts
+                // from a guaranteed-clean slate.
+                for (const suffix of ['', '-wal', '-shm']) {
+                    try { unlinkSync(dbPath + suffix) } catch { /* file may not exist */ }
+                }
+            }
+
+            handle = new Database(dbPath)
+            setupConnection()
         }
         handle.prepare('INSERT OR REPLACE INTO mikser_meta (key, value) VALUES (?, ?)')
             .run('schema_version', version)

package/src/engine.js

@@ -410,7 +410,20 @@ export async function setup(options) {
                             sidecarQueries: context?.sidecarQueries,
                         })
                         entry.deps = edges
-                        runtime.refs?.replaceDynamic(entity.id, edges)
+                        // Pagination produces synthetic pageEntities
+                        // (index.2.html, index.3.html, ...) whose ids
+                        // are NOT in mikser_entities — they exist only
+                        // at render time. Roll their dynamic refs up to
+                        // entity.parent (set by layouts.onBeforeRender
+                        // for pages 2+) so the mikser_refs FK to
+                        // mikser_entities holds. The parent's own
+                        // render also writes to the same source_id;
+                        // INSERT OR IGNORE in stmtInsertEdge handles the
+                        // dedup across pages. Invalidation re-dispatches
+                        // the parent and the pagination expansion
+                        // produces the children from there, so granular
+                        // per-page refs aren't needed.
+                        runtime.refs?.replaceDynamic(entity.parent ?? entity.id, edges)
                         await runtime.complete(entry)
                         await updateEntry({ id, output: entry.output, deps: edges })
                     }

package/src/plugins/files.js

@@ -1,6 +1,8 @@
 import path from 'node:path'
 import { mkdir, symlink, unlink, lstat, realpath } from 'fs/promises'
 import { globby } from 'globby'
+import pMap from 'p-map'
+import { checksumsByCollection } from '../catalog.js'
 
 export default ({
     runtime,
@@ -125,25 +127,44 @@ export default ({
 
         const paths = await globby('**/*', { cwd: runtime.options.filesFolder })
         trackProgress('Files import', paths.length)
-        return Promise.all(paths.map(async relativePath => {
+        // Bulk-prefetch the catalog's existing (id → checksum) map for
+        // this collection once at scan start, so the gate below reads
+        // from it per-file instead of doing per-file SQL lookups. Same
+        // pattern source.js uses for documents/layouts via useSource.
+        // Without this gate the plugin re-emitted createEntity on every
+        // cycle for every file regardless of changes, inflating the
+        // journal with phantom mutations and triggering downstream
+        // re-dispatch of aggregate layouts whose recorded query deps
+        // matched the collection.
+        const priorChecksums = checksumsByCollection(collection)
+        await pMap(paths, async relativePath => {
             const { uri, source } = await ensureLink(relativePath)
             let name = relativePath
             if (runtime.config.files?.outputFolder) {
                 name = path.join(runtime.config.files.outputFolder, relativePath)
             }
+            const id = path.join(`/${collection}`, relativePath)
+            const newChecksum = await checksum(source)
+            updateProgress()
+            // Gate: if the catalog already has this entity with the same
+            // checksum, the file hasn't changed since the last cycle.
+            // Skip emitting a CREATE — the catalog row stays correct,
+            // the journal stays accurate (mutations = actual changes),
+            // and downstream aggregate-layout invalidation isn't fired
+            // spuriously.
+            if (priorChecksums.get(id) === newChecksum) return
             await createEntity({
-                id: path.join(`/${collection}`, relativePath),
+                id,
                 uri,
                 collection,
                 type,
                 format: path.extname(relativePath).substring(1).toLowerCase(),
                 name,
                 source,
-                checksum: await checksum(source),
-                link: await link(source)
+                checksum: newChecksum,
+                link: await link(source),
             })
-            updateProgress()
-        }))
+        }, { concurrency: 16 })
     })
 
     return {