mikser-io 8.2.0 → 8.3.0

package/CLAUDE.md

@@ -0,0 +1,303 @@
+# CLAUDE.md — mikser-io
+
+Read this before proposing changes. Posture, architectural conventions,
+and the landmarks that survive across sessions.
+
+## Posture
+
+**Until v10, mikser has no users.** No back-compat. No deprecation
+paths. No migration markdown. No `task: pool` legacy aliasing for
+`task: inline`. Update READMEs and ADRs in place as source-of-truth
+changes; rewrite, don't supersede. The freedom is the point.
+
+**Position mikser by what it is, not by speed.** Hugo wins the speed
+race; mikser doesn't compete there. Position: AI-native, lifecycle-
+observable, files-as-source-of-truth, full-cycle introspection. Speed
+claim cap: "fast enough that watch-mode rebuilds feel instant."
+Honest range: 200–800 docs/sec depending on corpus size
+(see `test/perf/`).
+
+**Direct critique preferred.** Skip "great question," skip "solid but,"
+skip softening. When something didn't pay off, say so. Match user
+brevity.
+
+## Module map (`src/`)
+
+- `runtime.js` — singleton, holds engine state (`runtime.catalog`,
+  `runtime.refs`) and `runtime.options`. Lifecycle hook arrays live
+  here.
+- `lifecycle.js` — hook registration: `onLoad`/`onLoaded`/`onProcess`/
+  `onProcessed`/`onPersist`/`onBeforeRender`/`onRender`/`onAfterRender`/
+  `onBeforePostprocess`/`onPostprocess`/`onComplete`/`onFinalize`/
+  `onFinalized`/`onCancel`/`onCancelled`. Plus `runtime.create`/
+  `.update`/`.delete` (journal helpers).
+- `journal.js` — per-cycle queue, persisted to `mikser_journal` rows in
+  `runtime/mikser.sqlite`. Drained at onFinalized; survives crashes so
+  `--resume` (`-R`) can pick up unfinalized entries on the next start.
+  Same public surface: `addEntry`/`addEntries`/`updateEntry`/
+  `useJournal`/`clearJournal`. Inserts `JSON.stringify` the row body for
+  snapshot isolation (replaces the prior `structuredClone`). Walks are
+  chunked (`CHUNK_SIZE=500`) so peak journal memory stays bounded
+  regardless of corpus.
+  **Auto-persist:** `useJournal` diffs the yielded entity after each
+  iteration and UPDATEs the row if it changed. Plugin authors mutate
+  the yielded entity and move on — no explicit `updateEntry({id,entity})`
+  required. `updateEntry` is still exported for engine-internal writes
+  (output, deps) and as a no-op safety valve for plugins that prefer to
+  call it; if the entity hasn't drifted, the auto-persist skips.
+- `catalog.js` — entity persistence in the `mikser_entities` table of
+  `runtime/mikser.sqlite`. Indexed columns: id (PK), collection,
+  type, format, name, meta_href, meta_layout, meta_lang, meta_cache,
+  time, uri. Full entity body in `data` (JSON TEXT). 10k-entry LRU
+  cache in front of `findById`. Public ops: `findEntity`,
+  `findEntities`, `iterateEntities` (streaming async generator over the
+  same query shape, seek-paginated — use it when results may be
+  corpus-scale and the caller doesn't need an array), `queryEntities`,
+  `readEntity`, `subscribe`, `assertExpand`. Expand internals
+  (`expandLimits`, `expandAndProject`, `findRef`) are PRIVATE.
+- `subscriptions.js` — `subscribe()` primitive. Two modes: journal-
+  walk dispatch (default) and graph dispatch via
+  `runtime.refs.subscribeGraph` (when `expand` is set).
+- `refs.js` — inverse-reference graph (`$`-keyed refs per ADR-0007).
+  Persisted as `mikser_refs` rows with FK to `mikser_entities`
+  (`ON DELETE CASCADE`). Indexed on `target` and `source`. Exposed
+  at `runtime.refs.*`: `inboundFor`, `outboundFor`, `allRefs`,
+  `size`, `rename`, `subscribeGraph`, `inverseClosureOf`. Plus
+  `refExists` module-level. Prepared statements through the shared
+  sqlite handle.
+- `engine.js` — `setup()`, lifecycle wiring, render + postprocess
+  dispatchers, manifest tracking. Owns the Piscina worker pools
+  (`renderWorkers`, `postprocessWorkers`); both are lazy
+  (`minThreads: 0` + `idleTimeout: 30_000`) so INLINE-only workloads
+  pay no worker overhead. `workerSafeOptions(runtime.options)`
+  strips plugin-surface functions before TASKS.WORKER dispatch so
+  Piscina's structured clone doesn't choke.
+- `database/` — `createSqliteDatabase()`, `registerSchema()`,
+  `useDatabase()` (the `mikser_meta` table stamps schema_version).
+  `sift-to-sql.js` translates sift filters to SQL WHERE clauses
+  against `INDEXED_COLUMNS`; un-pushed clauses fall through to
+  JS-side sift. `query-context.js` is the AsyncLocalStorage that
+  lets catalog queries auto-report into the render-time `track`.
+- `manifest.js` — render snapshots in `mikser_snapshots` table
+  (PK `(id, destination)`, `refClosure` as JSON, partial index on
+  `parent`). `recordedHashes` aggregates dep hashes via
+  `json_each` in C rather than parsing every row in JS.
+- `server.js` — Express bring-up: CLI flags (`--server`, `--cors`,
+  `--no-cors`), trust-proxy, CORS (with extensible header arrays for
+  plugins to push onto), late-binding static mount + listen.
+- `logger.js` — pino + pino-pretty (inline) + gauge progress + custom
+  Writable for progress coordination + `pino.multistream` for
+  third-party shipping (`runtime.config.logging.transports`).
+- `utils.js` — shared pure helpers: `mimeForEntity`, `isLoopback`,
+  `expandEntity`, `projectMeta`, `useCollection`, `useRenderer` (via
+  render.js), `isTextEntity`, `readEntityContent`, `extractRefs`,
+  `isRefKey`, `writeEntity`, `matchEntity`, `getFormatInfo`,
+  `changeExtension`, `checksum`, `normalize`, `formatErrorContext`,
+  `formatLogArgs`, `ExpandError`, `AbortError`.
+- `render.js` / `postprocess.js` — Piscina worker entry points AND the
+  default-export functions the INLINE/SERIAL dispatcher calls directly.
+  Each receives entity + options + config + state; the WORKER path also
+  receives a MessageChannel `port` that forwards pino records back to
+  the engine's logger. Each worker opens its own read-only sqlite
+  handle on first task (`ensureWorkerDb` in render.js) so template
+  helpers like `runtime.lookupHref` stay sync. Never touch the journal
+  directly.
+- `config.js` — loads `mikser.config.js` at `onLoad`.
+- `plugins.js` — loads user plugins at `onLoad`. Plugin factories
+  receive the full `core` exports as their first argument.
+- `manager.js` — file watching (chokidar) and cron scheduling.
+- `source.js` — `useSource` codifies the folder-of-files pattern.
+- `constants.js` — `OPERATION` (CREATE/UPDATE/DELETE/RENDER/
+  POSTPROCESS), `ACTION` (sync action types), `TASKS` (`INLINE`/
+  `SERIAL`/`WORKER` — dispatch modes).
+
+## Plugin map (`src/plugins/`)
+
+- `documents` — file→entity sync for the documents collection
+- `layouts` — layout matching + sitemap + `inspect()` primitive
+  (exposed at `runtime.options.layouts.inspect`)
+- `files` — file→entity sync for the files collection
+- `assets` — asset references and copy
+- `resources` — resource references
+- `front-matter` — YAML frontmatter extraction (HTML/MD files)
+- `yaml` — YAML format support (.yml/.yaml entities)
+- `json` — JSON format support
+- `api` — HTTP catalog access. Pure transport — exposes nothing on
+  `runtime.options.*`. Per-query disk cache.
+- `preview` — in-memory render cache + GET /preview/:filename route.
+  Exposed at `runtime.options.preview = { store, get, stats, config }`.
+- `data` — JSON snapshots of entities/context/catalog to disk
+- `observer` / `mapper` / `validator` / `commands` / `shares` —
+  utility plugins
+
+## Naming conventions
+
+- **Engine state** at `runtime.<name>` — `runtime.refs`, `runtime.catalog`.
+- **Plugin surfaces** at `runtime.options.<plugin>` —
+  `runtime.options.preview`, `runtime.options.layouts.inspect`.
+- **Engine functions** as module-level exports from `mikser-io`:
+  `import { queryEntities, subscribe, useRenderer, useCollection,
+  readEntityContent, isTextEntity } from 'mikser-io'`.
+- **Plugin packages**: `mikser-io-<name>` (mikser-io-mcp, mikser-io-vector,
+  mikser-io-schemas, etc.).
+- **MCP tools**: `mikser_<verb>` or `mikser_<subsystem>_<verb>`:
+  `mikser_query_entities`, `mikser_read_entity`, `mikser_update_entity`,
+  `mikser_delete_entity`, `mikser_render`, `mikser_refs_inbound`,
+  `mikser_refs_outbound`, `mikser_refs_broken`, `mikser_refs_rename`,
+  `mikser_layouts_inspect`, `mikser_preview_render`,
+  `mikser_preview_ui`, `mikser_ui_action`, `mikser_ping`.
+  **Never `mikser_api_*`** — that prefix is dead.
+- **TASKS constants**: `INLINE` (main-thread async), `SERIAL`
+  (p-queue concurrency 1), `WORKER` (Piscina pool). **Not** the old
+  `POOL`/`QUEUE`/`WORKER` (which misled readers — `POOL` was actually
+  main-thread, not Piscina).
+
+## Code style
+
+- **No historical-narrative comments.** Describe what's there now.
+  "Used to live in X, moved in 8.2.0" → git log territory.
+- **No separator-line comments** (`// ---- Section -----`).
+- **No `_` prefix on engine-managed paths** (`runtimeFolder/foo`,
+  not `runtimeFolder/_foo`).
+- **Engine-set entity fields under `entity.options`**, not
+  `_`-prefixed top-level.
+- **No cross-plugin imports.** Plugins compose through lifecycle +
+  `runtime.options.*`. Shared pure helpers go in `src/utils.js`.
+  Audit: `grep -rEn "from '\./|await import\('\./" src/plugins/*.js`
+  should return nothing.
+- **Single source of truth fixes.** For cross-plugin bugs, ask
+  "what's the canonical copy?" before symptom-site patching.
+- **README claims are promises.** Every line in mikser-io's README
+  is a commitment — true today or actively being made true. Stale
+  claims are bugs.
+
+## ADRs (canonical decisions)
+
+- **0001-0005** — foundational: content-layer-not-the-app,
+  files-as-source-of-truth, plugins-independent-engine-stable,
+  compose-via-protocols, engine-infrastructure-runs-before-plugin-hooks
+- **0006** — five-test framework for adding to core: (1) substrate,
+  (2) strengthens strategy, (3) god-plugin check, (4) composability,
+  (5) release cadence. Conjunctive. Express passes; MCP failed test
+  #5 and ships as mikser-io-mcp.
+- **0007** — `$`-prefixed reference declaration + `expand`
+  resolution. Implemented in `catalog.js` + `refs.js`. Caps
+  configurable under `catalog.expand.{maxDepth,maxPaths,maxResolved}`
+  (defaults 5/20/100).
+- **0008** — MCP-UI rendering + action delivery. Lives in
+  `mikser-io-mcp/documentation/decisions/` (not core — moved with
+  MCP).
+- **0009** — Sqlite is the engine's persistence substrate. Single
+  `runtime/mikser.sqlite` holds `mikser_entities` / `mikser_refs` /
+  `mikser_snapshots` / `mikser_journal` (+ `mikser_meta` for the
+  schema-version stamp). Sift→SQL pushdown + LRU for findById;
+  worker-side read-only sqlite for sync template helpers.
+  `registerSchema(name, sql)` + `useDatabase()` is the plugin-side
+  persistence pattern. Journal-on-sqlite (Phase 7) enables `--resume`;
+  auto-persist (Phase 9) means plugins mutate the yielded entity and
+  the journal writes back without an explicit `updateEntry` call.
+
+## MCP
+
+Lives in `mikser-io-mcp` plugin (separate repo). Activate by listing
+`'mcp'` **first** in your `mikser.config.js` plugins array:
+
+```js
+export default {
+    plugins: ['mcp', /* ...other plugins */],
+    mcp: {
+        path: '/mcp',          // optional; default '/mcp'
+        // endpoints: { ... }  // optional; per-endpoint token + scope
+    },
+}
+```
+
+Must be first because its factory creates `runtime.options.mcp`
+synchronously, and other plugins gate their MCP tool registration on
+`if (runtime.options.mcp)` in their own `onLoaded`.
+
+There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
+
+## Perf
+
+- Rig: `npm run test:perf` (generates 10k corpus, runs render-only
+  pipeline). Configurable: `node test/perf/generate.js 50000`.
+  `SIZE=realistic node test/perf/generate.js 10000` switches to fat
+  entities (full SEO meta, hero/gallery image objects, $-refs to
+  author/category/related, longer body — ~7KB per catalog entry
+  instead of ~3KB). Add `task: worker` to a layout's frontmatter to
+  dispatch its renders + postprocess through Piscina.
+- Current honest numbers (Apple Silicon, 4-thread default, INLINE
+  dispatch; see ADR-0009 for the substrate the numbers below run on):
+  - 14k realistic cold (--clear):   33s,    RSS 1.4GB peak
+  - 14k realistic warm clean:       2.6s,   RSS 156MB peak
+  - 14k realistic warm + 1 change:  3.0s,   RSS 156MB peak
+  - 110k realistic cold (--clear):  5.5 min
+  - 110k realistic warm clean:      25s,    RSS 3.2GB
+- vs the Map+NDJSON baseline (origin/main 6922b33) at 14k realistic:
+  cold ~4× faster, warm ~2× faster, warm RSS ~9× smaller. 110k is a
+  workload Map+NDJSON couldn't reach — process OOMed before ADR-0009.
+- Catalog scan cost was the 2024-era objection to sqlite. The
+  resolution lives in `src/database/sift-to-sql.js`: indexed sift
+  clauses ($eq/$in/$lt/$exists/etc. on collection / type / format /
+  name / meta_href / meta_layout / meta_lang / meta_cache / time /
+  uri) push down to SQL, so layouts.onLoaded and source.sweep don't
+  materialize the table per cycle. `findById` is a 10k-entry LRU in
+  front of PK lookup. Without those two, sqlite is strictly slower
+  than the old Map (we measured it — see ADR-0009).
+- Piscina is lazy (`minThreads: 0` + `idleTimeout: 30_000`). INLINE
+  dispatch is the default for both render and postprocess; layouts
+  that opt into TASKS.WORKER (`task: worker` in frontmatter) get a
+  thread per first task. At 14k the lazy init dropped peak RSS
+  ~130MB on workloads that never use WORKER (which is most).
+- **Profile before optimizing.** `node --cpu-prof app.js
+  --working-folder test/perf --clear` produces `.cpuprofile` for
+  Chrome DevTools. Intuition has a real miss rate (multiple perf
+  hypotheses across this rewrite turned out wrong; the profile
+  caught them every time).
+- What actually helps when RSS is too high: trim entity weight
+  (don't store source `content` in the catalog if the renderer
+  re-reads it; don't keep computed fields you can recompute),
+  filter your `data.entities` exports so the catalog isn't
+  carrying the rendered shape, or drop `--threads` to 1-2 if the
+  build is memory-bound and cold time is acceptable.
+
+## When extending
+
+- **New engine capability?** Run through ADR-0006's five tests. Bar
+  is high. Express is the only earned addition.
+- **New plugin?** Own repo, named `mikser-io-<name>`. Composes
+  against `runtime.options.app` / `runtime.options.mcp` / lifecycle
+  hooks. Never imports another plugin's source.
+- **New MCP tool?** Add to `mikser-io-mcp/index.js` via
+  `mcp.simpleTool(name, description, zodSchema, handler)`. Tool name
+  follows `mikser_*` convention.
+- **New lifecycle hook?** Almost certainly no. Existing hooks cover
+  all known patterns. If you think you need one, post the use case
+  to ADR-0006 review.
+
+## Test suites
+
+- `npm run test:unit` — 363 unit tests across plugins + utilities
+- `npm run test:scenarios` — 18 subprocess-spawned end-to-end runs
+  (manifest skip, refs replay, watch-mode change/delete). Spawns
+  mikser fresh per scenario so module-level catalog/refs/manifest
+  state can't leak between tests.
+- `npm run test:smoke` — full lifecycle build of `test/fixture/`
+  (with vector + decap + post-mjml + post-pdf if env supports).
+  Exercises both INLINE postprocess (PDF) and WORKER postprocess
+  (MJML via `task: worker` on `welcome.yml`).
+- `npm run test:perf` — render-pipeline perf rig (corpus generation
+  + clean-build timing). `SIZE=realistic` + entity count knobs
+  documented in **Perf**.
+
+## Reference
+
+- `documentation/architecture.md` — module map (audit before relying
+  on specifics; may have drift)
+- `documentation/decisions/` — ADRs
+- `documentation/configuration.md` — config reference
+- `documentation/api-reference.md` — public API
+- `test/perf/` — render-pipeline perf rig
+- Sibling repos: `mikser-io-mcp`, `mikser-io-vector`, `mikser-io-schemas`,
+  `mikser-io-sdk-{api,react,svelte,vue,vector}`, `mikser-io-example-blog`

package/README.md

@@ -36,7 +36,7 @@ Build mikser into the parts of your application that are content-shaped. Keep th
 
 **Incremental builds that scale.** Mikser tracks every entity in a journal. When a file changes, only the affected entities re-process — not the whole site graph. On 10k+ documents this dramatically outpaces tools that rebuild more on every change.
 
-**Concurrent rendering.** Renders fan out across a worker pool that keeps every CPU core hot. Multi-format outputs (HTML, PDF, MJML email, etc.) generate in parallel from the same source.
+**Concurrent rendering.** Renders run async by default and CPU-heavy layouts (MJML compile, image processing, custom transforms) opt into a Piscina worker pool per layout via `task: worker` in frontmatter. Multi-format outputs (HTML, PDF, MJML email, etc.) generate from the same source; the pool is lazy — no workers spawn until they're asked for.
 
 **Asset pipelines are whatever Node can do.** Most static frameworks (Astro, Next.js, Hugo) ship image optimization and stop there — video transcoding, AI upscaling, watermarking all need a separate service. Mikser runs user-written modules over binary inputs: ~10 lines around `sharp` resize an image, ~10 around `fluent-ffmpeg` transcode a video, ~30 around the Replicate API upscale with AI. Anything an npm package can do, your pipeline can do — including pulling uploads from a DAM or CDN through the same flow.
 
@@ -225,11 +225,25 @@ What you get from how this project is built:
 - **Builds are deterministic; no async middleware layer.** The journal is the only synchronization primitive — no event bus, no IoC container, no orchestrator running plugins in surprising order. The lifecycle is a list of named phases; "what ran when?" has an answer you can read off the source.
 - **The whole engine is one read.** The [Architecture Overview](./documentation/overview.md) walks the full pipeline top to bottom. Onboarding a new engineer is an afternoon, not a tour through fifteen reference docs.
 
+## Mikser among static site generators
+
+**The only SSG that's both fast enough for daily use and deep enough for AI agents to drive.**
+
+| SSG | Speed | Feature surface | The tradeoff |
+|---|---|---|---|
+| Hugo | Fastest full rebuild | Minimal | Fast but limited; rebuilds everything every run |
+| Eleventy | OK | Broad, no introspection | Flexible but slow at corpus scale |
+| Astro | OK | Modern, framework-coupled | Tied to a frontend framework |
+| Next.js SSG | meh | Full framework | Framework first, content second |
+| **Mikser** | Incremental beats Hugo's full rebuild | Broad, deeply observable, AI-native | None of the speed-vs-features kind |
+
+Every other SSG asks you to trade something — raw speed for features (Hugo), features for framework lock-in (Astro, Next), introspection for any of the above (Eleventy). Mikser doesn't make that trade. Hugo still wins a full cold rebuild — and that's worth knowing — but most cycles aren't full cold rebuilds. CI deploys, watch-mode edits, "ran mikser, nothing changed" — these are the daily case, where Mikser's persistent manifest skips what's still current while Hugo rebuilds everything from scratch. The rest of the feature surface (MCP introspection on every lifecycle phase, files-as-source-of-truth, 20-phase composability, multi-format outputs from one corpus) is what no other "fast" SSG carries.
+
 ## Acknowledgments
 
 The earliest version of mikser was inspired by [DocPad](https://github.com/docpad/docpad) (Benjamin Lupton, with Michael Duane Mooring and Rob Loach). DocPad's "freeway, not a box" philosophy — files on disk, any pre-processor or template engine, plugin-by-convention extension — shaped how mikser started.
 
-Mikser itself has a previous chapter: the [legacy 7.x line](https://github.com/almero-digital-marketing/mikser) (last release 2022) introduced the real-time SSG model the current engine still carries forward. The redesign dropped MongoDB (the catalog lives in-process now, not in a database), modernized to Node ESM with a structured 20-phase lifecycle, added the live SSE channel that powers the framework SDKs, and replaced cluster-based rendering with an async worker pool. Same intent — content as files, real-time previews, multi-format output at scale — clearer foundations.
+Mikser itself has a previous chapter: the [legacy 7.x line](https://github.com/almero-digital-marketing/mikser) (last release 2022) introduced the real-time SSG model the current engine still carries forward. The redesign dropped MongoDB for a single in-process sqlite database, modernized to Node ESM with a structured 20-phase lifecycle, added the live SSE channel that powers the framework SDKs, and replaced cluster-based rendering with a lazy worker pool. Same intent — content as files, real-time previews, multi-format output at scale — clearer foundations.
 
 ## Documentation Index
 

package/index.js

@@ -1,10 +1,13 @@
 export { default as runtime } from './src/runtime.js'
 export * as constants from './src/constants.js'
 export * from './src/utils.js'
-export * from './src/journal.js'
 export * from './src/lifecycle.js'
+export * from './src/database/index.js'
+export * from './src/journal.js'
 export * from './src/catalog.js'
 export * from './src/refs.js'
+export * from './src/manifest.js'
+export * from './src/track.js'
 export * from './src/subscriptions.js'
 export * from './src/config.js'
 export * from './src/plugins.js'

package/package.json

@@ -1,14 +1,15 @@
 {
 	"name": "mikser-io",
-	"version": "8.2.0",
+	"version": "8.3.0",
 	"description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
 	"main": "index.js",
 	"scripts": {
 		"debug": "node --no-warnings app.js --server --watch --working-folder test/fixture",
 		"test:unit": "node --test --test-reporter=spec 'test/unit/**/*.test.js'",
 		"test:smoke": "node --no-warnings app.js --working-folder test/fixture",
+		"test:scenarios": "node --test --test-reporter=spec --test-timeout=60000 'test/scenarios/**/*.test.js'",
 		"test:perf": "node test/perf/generate.js && node --no-warnings app.js --working-folder test/perf",
-		"test": "npm run test:unit && npm run test:smoke"
+		"test": "npm run test:unit && npm run test:scenarios && npm run test:smoke"
 	},
 	"bin": {
 		"mikser": "app.js"
@@ -24,6 +25,7 @@
 		"@budibase/handlebars-helpers": "^0.14.3",
 		"await-semaphore": "^0.1.3",
 		"axios": "^1.17.0",
+		"better-sqlite3": "^12.10.0",
 		"chokidar": "^5.0.0",
 		"commander": "^15.0.0",
 		"cors": "^2.8.5",
@@ -39,7 +41,6 @@
 		"is-url": "^1.2.4",
 		"line-reader": "^0.4.0",
 		"lodash": "^4.18.1",
-		"lowdb": "^7.0.1",
 		"minimatch": "^10.2.5",
 		"node-cron": "^4.2.1",
 		"p-map": "^7.0.4",