mikser-io 8.0.0 → 8.2.0

package/README.md

@@ -79,7 +79,7 @@ The honest caveat: this advantage is real on **content-shaped work** — adding
 
 ## Control mikser from your AI agent
 
-Add `--mcp` to your mikser command and any MCP-speaking client — Claude Desktop, Claude Code, ChatGPT, custom agents — connects to the running engine. From inside a chat, your AI can:
+Install the [`mikser-io-mcp`](https://github.com/almero-digital-marketing/mikser-io-mcp) plugin and any MCP-speaking client — Claude Desktop, Claude Code, ChatGPT, custom agents — connects to the running engine. From inside a chat, your AI can:
 
 - read every entity in the catalog
 - write new content files (markdown, layouts, configuration) — writes land on disk and the next cycle picks them up
@@ -90,8 +90,16 @@ Add `--mcp` to your mikser command and any MCP-speaking client — Claude Deskto
 
 Plugins extend the tool surface the same way they mount HTTP routes; install the plugin, the agent gets new verbs. No glue code, no per-project agent wiring.
 
+```js
+// mikser.config.js
+export default {
+    plugins: ['mcp', /* … */],
+    // optional: mcp: { path: '/mcp', endpoints: { … } }
+}
+```
+
 ```bash
-mikser --server --mcp           # mounts MCP at /mcp on the same port as --server
+mikser --server                 # MCP mounts at /mcp on the same port
 ```
 
 What that feels like in practice: *"draft three hero-section variants and show me previews"* — three layouts written, three previews returned inline, one chat turn. *"Why did the build break?"* — the agent reads the rolling log buffer and answers from the same view your terminal sees. *"Update this article's tone and show me the preview"* — the agent edits the file and surfaces the rendered article inline; you click Approve or Reject, the agent acts on your choice. Operator, AI, and any observer dashboard share the same engine because mikser is single-tenant by design.
@@ -109,7 +117,7 @@ When an AI agent edits ten files, the next question is: *did it do what I asked?
 
 The shift this enables: AI review stops being *"read every change"* and becomes *"spot-check the agent's confidence."* The agent verifies its own work; the human samples and approves. That's the workflow that lets a content team actually use AI at scale — change the tone across the entire site in a morning, ship it after a coffee.
 
-Full tool reference and twelve worked scenarios in [MCP — talking to mikser from AI](./documentation/mcp.md).
+Full tool reference and twelve worked scenarios in the [`mikser-io-mcp` plugin docs](https://github.com/almero-digital-marketing/mikser-io-mcp#readme).
 
 ## Plugins on top of the engine
 
@@ -132,14 +140,14 @@ The engine is what stays stable — the lifecycle, the catalog, the file-based c
 |---|---|
 | `data` | JSON snapshots of entities / context / catalog, written to disk for static serving |
 | `api` | REST endpoints with sift-backed queries, per-endpoint tokens, optional render, opt-in [per-query disk cache](./documentation/caching.md) for reverse-proxy failover |
-| `preview` | In-memory render cache + `GET /preview/:filename` route. Companion to the [`mikser_preview`](./documentation/mcp.md) MCP tool — transient render bytes served at a clickable URL, no filesystem footprint |
+| `preview` | In-memory render cache + `GET /preview/:filename` route. Companion to the `mikser_preview_render` MCP tool (in [`mikser-io-mcp`](https://github.com/almero-digital-marketing/mikser-io-mcp)) — transient render bytes served at a clickable URL, no filesystem footprint |
 
 **Integrations:**
 
 | Plugin | What it does |
 |---|---|
 | [`mikser-io-vector`](https://github.com/almero-digital-marketing/mikser-io-vector) | OpenAI embeddings + semantic search (sqlite-vec or pgvector) |
-| [`mikser-io-plugin-schemas`](https://github.com/almero-digital-marketing/mikser-io-plugin-schemas) | Zod-backed entity validation + auto-generated TypeScript declarations for the SDK. Auto-detects `$`-keyed references and warns on broken ones — see [ADR-0007](./documentation/decisions/0007-references-declaration-and-expansion.md) |
+| [`mikser-io-schemas`](https://github.com/almero-digital-marketing/mikser-io-schemas) | Zod-backed entity validation + auto-generated TypeScript declarations for the SDK. Auto-detects `$`-keyed references and warns on broken ones — see [ADR-0007](./documentation/decisions/0007-references-declaration-and-expansion.md) |
 | [`mikser-io-archive`](https://github.com/almero-digital-marketing/mikser-io-archive) | Persist matching entities to YAML — audit trail, versioned content history, downstream export |
 | `mapper` | Run config-supplied transforms over matched entities each cycle (in-core, generic transformation layer) |
 | [`mikser-io-live`](https://github.com/almero-digital-marketing/mikser-io-live) | Lightweight dev server with browser auto-refresh — pair with `--watch` for the classic save→reload loop |
@@ -162,7 +170,7 @@ The `api`, `vector`, and `schemas` plugins are paired with client-side SDKs so a
 | [`mikser-io-sdk-api`](https://github.com/almero-digital-marketing/mikser-io-sdk-api) | `api` | `entities(name).list / query / urlFor / pages / update / delete / render / live` — Mongo-style filter operators backed by sift, sort, projection, pagination, SSE-driven live subscriptions, and `expand: [...]` to inline-resolve `$`-keyed references in one round-trip (multi-hop chains, `*` array iteration) |
 | [`mikser-io-sdk-vector`](https://github.com/almero-digital-marketing/mikser-io-sdk-vector) | `vector` | `vector(storeName).findSimilar(text, { limit })` — semantic search hits with the original mapped object attached |
 
-**Framework integrations** — all three wrap `mikser-io-sdk-api` in framework-idiomatic shapes. Same surface: `useDocument` / `useDocuments` live data, multilingual `useHref` / `useAlternates`, asset resolution via `useAsset`, generic on entity type so `mikser-io-plugin-schemas`-emitted types compose:
+**Framework integrations** — all three wrap `mikser-io-sdk-api` in framework-idiomatic shapes. Same surface: `useDocument` / `useDocuments` live data, multilingual `useHref` / `useAlternates`, asset resolution via `useAsset`, generic on entity type so `mikser-io-schemas`-emitted types compose:
 
 | Package | Framework | Notes |
 |---|---|---|
@@ -170,7 +178,7 @@ The `api`, `vector`, and `schemas` plugins are paired with client-side SDKs so a
 | [`mikser-io-sdk-react`](https://github.com/almero-digital-marketing/mikser-io-sdk-react) | React 18+ / 19+ | Hooks, `<MikserProvider>` Context, React Router v6+ integration via `useMikserRoutes` → `useRoutes()`. |
 | [`mikser-io-sdk-svelte`](https://github.com/almero-digital-marketing/mikser-io-sdk-svelte) | Svelte 5 (runes) | `$state` / `$effect` reactives, SvelteKit-friendly `generateMikserRoutes` for `entries()` prerender, `useMikserPages` for live nav. |
 
-Each SDK ships TypeScript declarations so client projects get autocomplete on filters, envelopes, and the `MikserError` thrown on non-2xx responses. Pair any of the framework SDKs with the `entities.d.ts` emitted by `mikser-io-plugin-schemas` for typed entity meta per layout. Install only the one(s) a project needs.
+Each SDK ships TypeScript declarations so client projects get autocomplete on filters, envelopes, and the `MikserError` thrown on non-2xx responses. Pair any of the framework SDKs with the `entities.d.ts` emitted by `mikser-io-schemas` for typed entity meta per layout. Install only the one(s) a project needs.
 
 ## Quick Start
 
@@ -201,7 +209,7 @@ The shape mikser fits cleanly:
 
 - **Marketing sites with editorial teams** — content authors work in files (via their editor, a Git client, or `mikser-io-decap`), engineers ship features without negotiating with a CMS schema, the site stays portable.
 - **Multilingual publishing platforms** — the `useHref()` / `useAlternates()` pattern in `sdk-vue` decouples logical references from per-locale URLs. One source tree, many language deployments.
-- **Content-heavy product catalogues** — `documents` + `mikser-io-plugin-schemas` + `data` plugin + a Vue frontend = typed product listings with live updates, semantic search via `vector`, and static-CDN-friendly JSON snapshots all at once.
+- **Content-heavy product catalogues** — `documents` + `mikser-io-schemas` + `data` plugin + a Vue frontend = typed product listings with live updates, semantic search via `vector`, and static-CDN-friendly JSON snapshots all at once.
 - **AI-augmented media pipelines** — `assets` plugin presets call out to Replicate / OpenAI / local models to upscale images, transcribe audio, transcode video. The pipeline is JS code, so anything Node can do is in scope.
 - **Mixed-output publishing** — the same source document renders to HTML, PDF (via `post-pdf`), MJML email (via `post-mjml`), and JSON snapshots. One catalog, many output formats, all concurrent.
 - **Headless backends for static frontends** — pair the `api` plugin with `sdk-api` for SSE-driven live frontends; pair the `data` plugin output with any static host for pre-rendered consumption.
@@ -235,7 +243,7 @@ Mikser itself has a previous chapter: the [legacy 7.x line](https://github.com/a
 | [Entities](./documentation/entities.md)               | Users & Developers | Entity model, operations, journal, catalog         |
 | [Rendering](./documentation/rendering.md)             | Users & Developers | Render pipeline, render plugins, render modes      |
 | [Watch Mode](./documentation/watch-mode.md)           | Users              | File watching, scheduled tasks, incremental builds |
-| [MCP](./documentation/mcp.md)                         | Users              | The `--mcp` server — tool surface, `mikser://` resources, twelve worked AI-driven scenarios |
+| [MCP](https://github.com/almero-digital-marketing/mikser-io-mcp#readme) | Users              | The `mikser-io-mcp` plugin — tool surface, `mikser://` resources, twelve worked AI-driven scenarios |
 | [Caching](./documentation/caching.md)                 | Users (production) | The `cache: true` disk cache + working nginx config for reverse-proxy failover |
 | [Architecture](./documentation/architecture.md)       | Developers         | Module-level reference — what's in each file       |
 | [API Reference](./documentation/api-reference.md)     | Developers         | Complete public API reference                      |

package/index.js

@@ -5,11 +5,11 @@ export * from './src/journal.js'
 export * from './src/lifecycle.js'
 export * from './src/catalog.js'
 export * from './src/refs.js'
+export * from './src/subscriptions.js'
 export * from './src/config.js'
 export * from './src/plugins.js'
 export * from './src/manager.js'
-export * from './src/tracking.js'
+export * from './src/logger.js'
 export * from './src/engine.js'
-export * from './src/api.js'
-export * from './src/source.js'
-export * from './src/mcp.js'
+export * from './src/render.js'
+export * from './src/source.js'

package/package.json

@@ -1,12 +1,13 @@
 {
 	"name": "mikser-io",
-	"version": "8.0.0",
+	"version": "8.2.0",
 	"description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
 	"main": "index.js",
 	"scripts": {
-		"debug": "node --no-warnings app.js --debug --server --watch --working-folder test/fixture",
+		"debug": "node --no-warnings app.js --server --watch --working-folder test/fixture",
 		"test:unit": "node --test --test-reporter=spec 'test/unit/**/*.test.js'",
 		"test:smoke": "node --no-warnings app.js --working-folder test/fixture",
+		"test:perf": "node test/perf/generate.js && node --no-warnings app.js --working-folder test/perf",
 		"test": "npm run test:unit && npm run test:smoke"
 	},
 	"bin": {
@@ -21,22 +22,21 @@
 	"license": "ISC",
 	"dependencies": {
 		"@budibase/handlebars-helpers": "^0.14.3",
-		"@modelcontextprotocol/sdk": "^1.29.0",
 		"await-semaphore": "^0.1.3",
 		"axios": "^1.17.0",
 		"chokidar": "^5.0.0",
-		"cli-progress": "^3.12.0",
 		"commander": "^15.0.0",
+		"cors": "^2.8.5",
 		"dayjs": "^1.11.21",
 		"deepdash": "^5.3.9",
 		"escape-string-regexp": "^5.0.0",
 		"execa": "^9.6.1",
 		"front-matter": "^4.0.2",
+		"gauge": "^5.0.2",
 		"globby": "^16.2.0",
 		"handlebars": "^4.7.9",
 		"hasha": "^7.0.0",
 		"is-url": "^1.2.4",
-		"knex": "^3.2.10",
 		"line-reader": "^0.4.0",
 		"lodash": "^4.18.1",
 		"lowdb": "^7.0.1",
@@ -48,7 +48,6 @@
 		"pino-pretty": "^13.1.3",
 		"piscina": "^5.1.4",
 		"sift": "^17.1.3",
-		"sqlite3": "^6.0.1",
 		"truncate-stream": "^1.0.2",
 		"yaml": "^2.9.0"
 	},

package/src/catalog.js

@@ -7,8 +7,8 @@ import { Low } from 'lowdb'
 import path from 'node:path'
 import { JSONFile } from 'lowdb/node'
 import _ from 'lodash'
-
-let catalog
+import sift from 'sift'
+import { expandEntity, projectMeta } from './utils.js'
 
 // Same reasoning as journal.js — initialize the catalog in
 // onInitialized so every plugin hook from onLoad onwards can safely
@@ -16,9 +16,14 @@ let catalog
 // catalog.js imports useJournal from journal.js, so journal.js's
 // onInitialized registers first within the phase — its hook runs
 // before this one.
+//
+// The catalog lives at `runtime.catalog` — single source of truth.
+// All operations below read through that property so tests can stub
+// `runtime.catalog` with an in-memory equivalent and the same call
+// shapes work without code changes.
 onInitialized(async () => {
 	const adapter = new JSONFile(path.join(runtime.options.runtimeFolder, `catalog.json`))
-	catalog = new Low(adapter, {
+	const catalog = new Low(adapter, {
 		entities: [],
 	})
 	catalog.chain = _.chain(catalog).get('data')
@@ -27,6 +32,7 @@ onInitialized(async () => {
 
 onPersist(async () => {
 	const logger = useLogger()
+	const catalog = runtime.catalog
 	for await (let { operation, entity } of useJournal('Catalog')) {
 		switch (operation) {
 			case OPERATION.CREATE:
@@ -57,17 +63,169 @@ onPersist(async () => {
 })
 
 onFinalized(async () => {
-	await catalog.write()
+	await runtime.catalog.write()
 })
 
 export async function findEntity(query) {
 	if (!query) return
-	return catalog.chain.get('entities').find(query).value()
+	return runtime.catalog.chain.get('entities').find(query).value()
 }
 
 export async function findEntities(query) {
 	if (!query) {
-		return catalog.chain.get('entities').value()
+		return runtime.catalog.chain.get('entities').value()
+	}
+	return runtime.catalog.chain.get('entities').filter(query).value()
+}
+
+// High-level CRUD-with-query surface — sift filters, sort, pagination,
+// dotted-path projection, plus optional inline-expand of $-keyed
+// references (ADR-0007). Used by the api plugin's HTTP handlers, the
+// mikser-io-mcp plugin's tools, and any library-mode caller.
+//
+// `findEntity`/`findEntities` above are raw catalog access (no filter,
+// no pagination, callback-style query). The queryEntities / readEntity
+// pair below are the higher-level operations that compose findEntities
+// with sift + expand + projection.
+//
+// Lives in catalog.js (not in the api plugin) because catalog operations
+// are engine-level: render workers, library embedders, and other plugins
+// all need to query the catalog with the same semantics. The api plugin
+// is one consumer that adds HTTP routing + an on-disk cache on top.
+
+// Match the same heuristic the schemas plugin uses for ref-existence
+// checks — id, meta.href, or stripped-extension id. Centralised so the
+// expand walker resolves refs the same way everywhere.
+async function findRef(ref) {
+	if (!ref || typeof ref !== 'string') return null
+	const matches = await findEntities(e =>
+		!!e && (
+			e.id === ref ||
+			e.meta?.href === ref ||
+			(typeof e.id === 'string' && e.id.replace(/\.[^./]+$/, '') === ref)
+		),
+	)
+	return matches[0] ?? null
+}
+
+// Per-call expansion caps. Plumbed from mikser.config.js:
+//   catalog: { expand: { maxDepth, maxPaths, maxResolved } }
+// Defaults match ADR-0007 B7. Centralised here so every expand-aware
+// operation reads the same numbers. Private — callers reach it via
+// `assertExpand` (pre-flight validation) or via queryEntities (which
+// applies the same caps inside the walker).
+function expandLimits() {
+	const cfg = runtime?.config?.catalog?.expand ?? {}
+	return {
+		maxDepth:    typeof cfg.maxDepth    === 'number' ? cfg.maxDepth    : 5,
+		maxPaths:    typeof cfg.maxPaths    === 'number' ? cfg.maxPaths    : 20,
+		maxResolved: typeof cfg.maxResolved === 'number' ? cfg.maxResolved : 100,
+	}
+}
+
+// Validate an `expand` spec against the configured caps. Throws on
+// violation; returns nothing on success. Used by `subscribe()` at
+// registration time to reject bad expand before opening any transport
+// channels. The same caps are enforced inside the walker, so this is
+// purely about early/clean error surfacing.
+export function assertExpand(expand) {
+	if (!expand?.length) return
+	const { maxPaths, maxDepth } = expandLimits()
+	if (expand.length > maxPaths) {
+		throw new Error(
+			`expand has ${expand.length} paths, exceeds maxPaths (${maxPaths})`,
+		)
+	}
+	for (const p of expand) {
+		const parts = p.split('.').filter(Boolean)
+		if (parts.length > maxDepth) {
+			throw new Error(
+				`Path '${p}' has length ${parts.length}, exceeds maxDepth (${maxDepth})`,
+			)
+		}
+	}
+}
+
+// Apply expand-then-project to one entity. When `expand` has entries
+// the walker inlines resolved refs first; in all cases the final meta
+// is normalized (`$`-keys stripped) so the wire shape matches what the
+// SDK consumers and templates expect per ADR-0007 A3. Private — the
+// only callers are queryEntities (per-item) and readEntity (via
+// queryEntities). External code expands by going through one of those.
+async function expandAndProject(entity, expand) {
+	let result = entity
+	if (expand?.length) {
+		result = await expandEntity(entity, expand, {
+			findRef,
+			...expandLimits(),
+		})
 	}
-	return catalog.chain.get('entities').filter(query).value()
+	if (!result?.meta) return result
+	return { ...result, meta: projectMeta(result.meta) }
+}
+
+// Paginated, sorted, filtered, projected query over the catalog. The
+// `scope` arg is an optional pre-filter predicate (used by the api
+// plugin's per-endpoint allowedEntities lambda); library callers
+// usually pass nothing. Returns `{ items, total, skip, limit, hasNext }`.
+export async function queryEntities({
+	filter, sort, fields, skip, limit, expand, scope,
+} = {}) {
+	const effectiveLimit = Math.min(100, Math.max(1, limit ?? 25))
+	const effectiveSkip = Math.max(0, skip ?? 0)
+
+	let all = await findEntities()
+	if (scope) all = all.filter(scope)
+
+	if (filter && Object.keys(filter).length) {
+		const match = sift(filter)
+		all = all.filter(match)
+	}
+
+	const total = all.length
+
+	if (sort && Object.keys(sort).length) {
+		const entries = Object.entries(sort)
+		all.sort((a, b) => {
+			for (const [key, dir] of entries) {
+				const av = _.get(a, key)
+				const bv = _.get(b, key)
+				if (av == null && bv == null) continue
+				if (av == null) return 1
+				if (bv == null) return -1
+				if (av < bv) return -dir
+				if (av > bv) return dir
+			}
+			return 0
+		})
+	}
+
+	let items = all.slice(effectiveSkip, effectiveSkip + effectiveLimit)
+	items = await Promise.all(items.map(item => expandAndProject(item, expand)))
+	if (fields?.length) items = items.map(e => _.pick(e, fields))
+
+	return {
+		items,
+		total,
+		skip: effectiveSkip,
+		limit: effectiveLimit,
+		hasNext: effectiveSkip + effectiveLimit < total,
+	}
+}
+
+// Read a single entity by catalog id. Returns the entity (post-expand,
+// post-project) or null when not found. Throws on missing id.
+// Source-file content loading is a separate concern — compose with
+// `readEntityContent` from utils.js if you want `entity.content`
+// populated (with the text/binary gate) or call `readFile(entity.uri)`
+// directly if you want raw bytes.
+export async function readEntity({ id, expand } = {}) {
+	if (!id) throw new Error('id is required')
+	const result = await queryEntities({
+		filter: { id },
+		skip: 0,
+		limit: 1,
+		expand,
+	})
+	return result.items[0] ?? null
 }

package/src/constants.js

@@ -13,8 +13,30 @@ export const ACTION = {
     TRIGGER: 'trigger',
 }
 
+// Render / postprocess dispatch modes. Set per-entity via
+// `meta.task: 'inline' | 'serial' | 'worker'`, or per-task by a plugin.
+//
+//   INLINE  — runs the render/postprocess in the main event loop via
+//             `await`. The outer dispatcher iterates with concurrency =
+//             runtime.options.threads (default 4), so multiple inline
+//             tasks can be in-flight at once; but they share the main
+//             thread, so CPU-bound work doesn't actually run in
+//             parallel. Right default for cheap renders (HTML / md /
+//             yaml templating) where IPC overhead would dominate.
+//
+//   SERIAL  — runs in the main event loop too, but via p-queue with
+//             concurrency 1. One at a time, no interleaving. Right
+//             choice for tasks that touch a shared, non-reentrant
+//             resource (a single browser instance, a sequential
+//             encoder, etc.).
+//
+//   WORKER  — runs on a real OS thread via the Piscina pool sized to
+//             runtime.options.threads. True parallelism. Right choice
+//             for expensive CPU-bound renders (PDF via headless
+//             Chromium, MJML compilation, image compose) where the
+//             render cost amortizes the IPC overhead.
 export const TASKS = {
-    QUEUE: 'queue',
+    INLINE: 'inline',
+    SERIAL: 'serial',
     WORKER: 'worker',
-    POOL: 'pool',
 }