mikser-io 6.23.1 → 6.27.0

package/README.md

@@ -40,6 +40,8 @@ Build mikser into the parts of your application that are content-shaped. Keep th
 
 **Run anywhere.** The same CLI handles one-shot builds, watch-mode dev loops, and a long-running HTTP server with a shared Express app. `npx mikser` ships a static site; `mikser --watch` is the dev loop; `mikser --server` exposes a live admin/API.
 
+**Survives backend outages.** The `api` plugin's [per-query disk cache](./documentation/caching.md) writes every cacheable list response to `out/` keyed by the request URL. A stock-nginx reverse proxy can fail over to the cached file when mikser is unreachable — same URL, transparent to the client, no Lua or extra modules required. Production frontends keep rendering routes during deploys, brief outages, and the upstream-blip-of-the-week. Opt in per endpoint with `cache: true`; see [Caching and reverse-proxy failover](./documentation/caching.md) for the working config.
+
 **Library mode.** Mikser is also a library. `useRenderer`, `useCollection`, `findSimilar`, and direct lifecycle hooks let you embed the engine inside an existing Node app instead of running it as a CLI.
 
 **Open source.** MIT-licensed, on GitHub, no telemetry, no auth wall, no SaaS dependency. What you see is what runs.
@@ -58,6 +60,8 @@ Files-as-source isn't just a portability story — it makes the project unusuall
 
 **Plugin-by-example.** Authoring a new plugin? There are 15+ existing ones in the same shape to pattern-match against. Convention is dense enough that new plugins look like the old ones without coaching.
 
+**One-shot bootstrap via Claude Code.** The [`mikser-io-claude-plugin`](https://github.com/almero-digital-marketing/mikser-io-claude-plugin) wraps the setup story above into a single skill. Register the repo as a Claude Code marketplace and install the plugin (`/plugin marketplace add almero-digital-marketing/mikser-io-claude-plugin` then `/plugin install mikser-io-claude-plugin@mikser-io`), then in any Vue 3, React, or SvelteKit project — or in a blank directory — say "add mikser to this app." It detects the framework (or scaffolds a fresh starter via `create-vite` / `sv create`), wires the matching framework SDK, composes with your existing router rather than replacing it, and optionally lays down a `mikser-content/` sibling folder with Zod schemas and starter documents so the backend works on first run.
+
 The honest caveat: this advantage is real on **content-shaped work** — adding pages, restructuring collections, generating new layouts, building frontends. It doesn't make mikser better for non-content tasks (concurrency bugs in the worker pool, database tuning elsewhere in your stack); those are plain Node debugging like anywhere else. The visibility advantage also degrades past ~10k documents — at that scale the agent queries via the SDK instead of grepping the tree, which is still good but less "see everything at once."
 
 ## Plugins on top of the engine
@@ -80,7 +84,7 @@ The engine is what stays stable — the lifecycle, the catalog, the file-based c
 | Plugin | What it does |
 |---|---|
 | `data` | JSON snapshots of entities / context / catalog, written to disk for static serving |
-| `api` | REST endpoints with sift-backed queries, per-endpoint tokens, optional render |
+| `api` | REST endpoints with sift-backed queries, per-endpoint tokens, optional render, opt-in [per-query disk cache](./documentation/caching.md) for reverse-proxy failover |
 
 **Integrations:**
 
@@ -111,7 +115,7 @@ The `api`, `vector`, and `schemas` plugins are paired with client-side SDKs so a
 
 | Package | Framework | Notes |
 |---|---|---|
-| [`mikser-io-sdk-vue`](https://github.com/almero-digital-marketing/mikser-io-sdk-vue) | Vue 3 | Composables, vue-router integration (`createMikserRouter`, `generateMikserRoutes`), provide/inject for the client. |
+| [`mikser-io-sdk-vue`](https://github.com/almero-digital-marketing/mikser-io-sdk-vue) | Vue 3 | Composables, vue-router integration (`useMikserRoutes` to augment an existing router, `generateMikserRoutes` for SSG prerender), provide/inject for the client. |
 | [`mikser-io-sdk-react`](https://github.com/almero-digital-marketing/mikser-io-sdk-react) | React 18+ / 19+ | Hooks, `<MikserProvider>` Context, React Router v6+ integration via `useMikserRoutes` → `useRoutes()`. |
 | [`mikser-io-sdk-svelte`](https://github.com/almero-digital-marketing/mikser-io-sdk-svelte) | Svelte 5 (runes) | `$state` / `$effect` reactives, SvelteKit-friendly `generateMikserRoutes` for `entries()` prerender, `useMikserPages` for live nav. |
 
@@ -181,6 +185,7 @@ A few things this project takes seriously:
 | [Entities](./documentation/entities.md)               | Users & Developers | Entity model, operations, journal, catalog         |
 | [Rendering](./documentation/rendering.md)             | Users & Developers | Render pipeline, render plugins, render modes      |
 | [Watch Mode](./documentation/watch-mode.md)           | Users              | File watching, scheduled tasks, incremental builds |
+| [Caching](./documentation/caching.md)                 | Users (production) | The `cache: true` disk cache + working nginx config for reverse-proxy failover |
 | [Architecture](./documentation/architecture.md)       | Developers         | Module-level reference — what's in each file       |
 | [API Reference](./documentation/api-reference.md)     | Developers         | Complete public API reference                      |
 | [Decisions (ADRs)](./documentation/decisions/)        | Developers         | Load-bearing architectural choices and what protects them |

package/app.js

@@ -1,3 +1,4 @@
+#!/usr/bin/env node
 import { setup } from "./index.js"
 
 async function main() {

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "mikser-io",
-	"version": "6.23.1",
+	"version": "6.27.0",
 	"description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
 	"main": "index.js",
 	"scripts": {

package/src/plugins/api.js

@@ -1,5 +1,6 @@
 import path from 'node:path'
-import { access } from 'node:fs/promises'
+import { access, writeFile, mkdir, rm } from 'node:fs/promises'
+import { createHash } from 'node:crypto'
 import _ from 'lodash'
 import sift from 'sift'
 import { useRenderer, useCollection } from '../api.js'
@@ -118,6 +119,88 @@ async function runQuery({ filter, sort, fields, skip, limit, scope, findEntities
     return { items, total }
 }
 
+// Derive the cache filename from the raw query string the client sent.
+//   no query string (`/entities`)       → 'index'
+//   query string ('a=1&b=2')            → 'a=1&b=2'
+//
+// We deliberately keep the raw, undecoded form because:
+//   - that's what nginx's `$args` variable contains
+//   - stock nginx can pass it straight into `try_files` without
+//     hashing modules or Lua
+//   - the URL is its own cache key — no extra contract for clients
+//     or proxies to agree on beyond "use the request URL"
+//
+// Trade-off: structurally-equal queries with different parameter
+// orders produce different cache files (`?a=1&b=2` vs `?b=2&a=1`).
+// Soft inefficiency, not a correctness bug — the SDK uses a stable
+// order and most consumers are SDK-driven. Filesystem rules apply:
+// most chars (`=`, `&`, `[`, `]`, `%`) are safe; 255-byte filename
+// limits cap query length on typical fs. Document these in the
+// caching README.
+function cacheNameForQueryString(rawQueryString) {
+    if (!rawQueryString) return 'index'
+    return rawQueryString
+}
+
+// Wide-list defaults — tuned for "an SPA accidentally pulled the whole
+// catalog because it forgot a `fields` projection." Operators can grep
+// the warning out of logs; SDK users see the same shape client-side via
+// the SDK's dev-mode warning.
+const WIDE_RESPONSE_ITEMS = 100
+const WIDE_RESPONSE_BYTES = 256 * 1024
+
+function maybeWarnWide({ logger, endpoint, envelope, req }) {
+    const items = envelope?.items?.length ?? 0
+    if (items <= WIDE_RESPONSE_ITEMS) return
+    const bytes = Buffer.byteLength(JSON.stringify(envelope))
+    if (bytes <= WIDE_RESPONSE_BYTES && items <= WIDE_RESPONSE_ITEMS) return
+    const queryStr = (req.originalUrl || req.url || '').split('?')[1] || '(none)'
+    const sizeLabel = bytes >= 1024 * 1024
+        ? `${(bytes / 1024 / 1024).toFixed(1)} MB`
+        : `${Math.round(bytes / 1024)} KB`
+    logger.warn(
+        'Api[%s] wide list response: %d items, %s — query=%s',
+        endpoint, items, sizeLabel, queryStr,
+    )
+    logger.warn(
+        '  ↳ Consider a `fields:` projection, or move this query to a `data.catalog.<name>` snapshot loaded via the SDK\'s `initialUrl`.',
+    )
+}
+
+// Write the response envelope to <out>/<base>/<endpoint>/entities/<name>.json.
+// The path matches what nginx's `try_files /<base>/<endpoint>/entities/$args.json
+// /<base>/<endpoint>/entities/index.json` looks up on upstream failure.
+async function writeQueryCache({ outputFolder, base, name, cacheName, envelope, logger }) {
+    const relBase = base.replace(/^\//, '')
+    const dir = path.join(outputFolder, relBase, name, 'entities')
+    const file = path.join(dir, `${cacheName}.json`)
+    try {
+        await mkdir(dir, { recursive: true })
+        await writeFile(file, JSON.stringify(envelope), 'utf8')
+        logger.trace('Api[%s] cache write: %s (%d items)', name, file, envelope.items.length)
+    } catch (err) {
+        // Filename-length / illegal-char failures land here. Logged but
+        // not propagated — the live response is already on the wire.
+        logger.error('Api[%s] cache write failed (%s): %s', name, file, err.message)
+    }
+}
+
+// Clear the entire per-endpoint cache directory. Called when any
+// catalog entity changes — coarse but correct, and on-demand writes
+// rebuild the cache as queries come back in. The cost of warm-up
+// after invalidation is bounded by traffic, which is what you'd want
+// from a write-through cache anyway.
+async function clearEndpointCache({ outputFolder, base, name, logger }) {
+    const relBase = base.replace(/^\//, '')
+    const dir = path.join(outputFolder, relBase, name, 'entities')
+    try {
+        await rm(dir, { recursive: true, force: true })
+        logger.trace('Api[%s] cache cleared: %s', name, dir)
+    } catch (err) {
+        logger.error('Api[%s] cache clear failed (%s): %s', name, dir, err.message)
+    }
+}
+
 // MIME type lookup used when streaming a postprocessor's output back over
 // HTTP. The renderer's output extension lives on entity.destination
 // (assigned by the layouts plugin), so we use it as the source of truth.
@@ -211,6 +294,13 @@ export default ({
     findEntities,
     constants: { OPERATION },
 }) => {
+    // Shared between onLoaded (populates) and onFinalize (consumes).
+    // Hoisted so the lifecycle hooks see the same registry; the body
+    // inside onLoaded is what actually fills it in based on
+    // runtime.config.api.endpoints.
+    let apiBase = '/api'
+    const cachedEndpoints = []
+
     onLoaded(async () => {
         const logger = useLogger()
 
@@ -237,10 +327,14 @@ export default ({
             throw new Error('Express is required for the api plugin — run: npm install express')
         })
 
-        const base = runtime.config.api?.base ?? '/api'
+        apiBase = runtime.config.api?.base ?? '/api'
+        const base = apiBase   // alias so existing local references still work
         const globalPageSize = runtime.config.api?.pageSize ?? 10
         const globalRenderTimeout = runtime.config.api?.renderTimeout ?? 30_000
 
+        // cachedEndpoints is hoisted above (shared with onFinalize).
+        // Per-endpoint setup loop pushes into it when cache: true is set.
+
         // The plugin mirrors the vector / data plugin shape: a named map
         // of endpoints, each with its own optional `token`, `query`
         // scope, allowed `operations`, and overrides for pageSize /
@@ -267,6 +361,69 @@ export default ({
             const pageSize = ep.pageSize ?? globalPageSize
             const renderTimeout = ep.renderTimeout ?? globalRenderTimeout
 
+            // Server-enforced field projection. When set, every list /
+            // query / subscribe response is narrowed to exactly these
+            // dotted paths — regardless of what the client asks for.
+            //
+            // The right tool when an endpoint backs a broad list query
+            // and only a handful of fields per entity are actually
+            // useful to the client — sitemap, navigation menus,
+            // faceted search dimensions. A SPA's first paint shouldn't
+            // download every markdown body just to find out which
+            // routes exist.
+            //
+            // Skip it for endpoints that serve individual full
+            // documents (the typical `public` shape used by
+            // useDocument(id)) — those queries return one entity at a
+            // time so narrowing has no real benefit.
+            const allowedFields = Array.isArray(ep.fields) && ep.fields.length
+                ? ep.fields
+                : null
+
+            // Pick the effective projection for a request. If the
+            // endpoint declares allowedFields, that's the ceiling: a
+            // client requesting more gets only the intersection. A
+            // client requesting nothing gets exactly allowedFields.
+            // No allowedFields → whatever the client asked for (or
+            // all fields when omitted).
+            function resolveFields(requested) {
+                if (!allowedFields) return requested ?? null
+                if (!requested || !requested.length) return allowedFields
+                const allowSet = new Set(allowedFields)
+                return requested.filter(f => allowSet.has(f))
+            }
+
+            // Endpoints with cache: true cache GET /entities responses
+            // to disk on a per-query-string basis. Path scheme:
+            //   <out>/<base>/<name>/entities/<raw-query-string>.json
+            //   <out>/<base>/<name>/entities/index.json    (no params)
+            //
+            // On any catalog change, the whole cache directory is
+            // dropped — the next requests through repopulate whatever's
+            // needed. Coarse but correct, no per-query tracking.
+            //
+            // Reverse-proxy failover (stock nginx, no Lua):
+            //   location /api/sitemap/entities {
+            //       proxy_pass http://localhost:3001;
+            //       proxy_intercept_errors on;
+            //       error_page 502 503 504 = @cache;
+            //   }
+            //   location @cache {
+            //       root /var/www/out;
+            //       try_files /api/sitemap/entities/$args.json
+            //                 /api/sitemap/entities/index.json
+            //                 =502;
+            //   }
+            //
+            // POST /entities/query responses aren't cached — there's no
+            // URL the proxy could derive a cache file path from. Use
+            // GET for cacheable queries (the SDK's client.list({...}) →
+            // GET URL is the canonical pattern).
+            if (ep.cache === true) {
+                cachedEndpoints.push({ name })
+            }
+            const cacheEnabled = ep.cache === true
+
             const auth = (req, res, next) => {
                 if (!ep.token) return next()
                 if (req.headers.authorization === `Bearer ${ep.token}`) return next()
@@ -298,7 +455,7 @@ export default ({
                     const { items, total } = await runQuery({
                         filter: parsed.filter,
                         sort: parsed.sort,
-                        fields: parsed.fields,
+                        fields: resolveFields(parsed.fields),
                         skip,
                         limit,
                         scope: query,
@@ -307,12 +464,35 @@ export default ({
 
                     const page = Math.floor(skip / limit) + 1
                     const totalPages = Math.ceil(total / limit) || 1
-                    res.json({
+                    const envelope = {
                         items, page, limit, total, totalPages,
                         hasNext: skip + limit < total,
                         hasPrev: skip > 0,
-                    })
+                    }
+                    res.json(envelope)
                     logger.trace('Api[%s] list %dms (%d/%d items)', name, Date.now() - t0, items.length, total)
+                    maybeWarnWide({ logger, endpoint: name, envelope, req })
+
+                    // Write-through cache: write the response to a file
+                    // path that mirrors the request URL — same `$args`
+                    // nginx's `try_files` sees, no hashing on either
+                    // side. See the caching docs for the nginx config
+                    // that wires the failover.
+                    if (cacheEnabled && runtime.options.outputFolder) {
+                        const url = req.originalUrl || req.url || ''
+                        const qIdx = url.indexOf('?')
+                        const rawQueryString = qIdx >= 0 ? url.slice(qIdx + 1) : ''
+                        const cacheName = cacheNameForQueryString(rawQueryString)
+                        // Fire-and-forget — the response is already sent.
+                        writeQueryCache({
+                            outputFolder: runtime.options.outputFolder,
+                            base: apiBase,
+                            name,
+                            cacheName,
+                            envelope,
+                            logger,
+                        }).catch(() => {})
+                    }
                 } catch (err) {
                     logger.error('Api[%s] list error (%dms): %s', name, Date.now() - t0, err.message)
                     res.status(500).json({ error: err.message })
@@ -330,19 +510,31 @@ export default ({
                     const skip = rawSkip ?? (Math.max(1, parseInt(rawPage) || 1) - 1) * limit
 
                     const { items, total } = await runQuery({
-                        filter, sort, fields, skip, limit,
+                        filter, sort,
+                        fields: resolveFields(fields),
+                        skip, limit,
                         scope: query,
                         findEntities,
                     })
 
                     const page = Math.floor(skip / limit) + 1
                     const totalPages = Math.ceil(total / limit) || 1
-                    res.json({
+                    const envelope = {
                         items, page, limit, total, totalPages,
                         hasNext: skip + limit < total,
                         hasPrev: skip > 0,
-                    })
+                    }
+                    res.json(envelope)
                     logger.trace('Api[%s] query %dms (%d/%d items)', name, Date.now() - t0, items.length, total)
+                    maybeWarnWide({ logger, endpoint: name, envelope, req })
+
+                    // POST queries aren't disk-cached. The reverse proxy
+                    // failover scheme is URL-based (cache file path =
+                    // request URL), but POST has no URL-equivalent for
+                    // its body. Cacheable queries should use GET — which
+                    // the SDK's list() does by default. Body-only queries
+                    // are by definition complex/non-canonicalizable so
+                    // they're treated as live-only.
                 } catch (err) {
                     logger.error('Api[%s] query error (%dms): %s', name, Date.now() - t0, err.message)
                     res.status(500).json({ error: err.message })
@@ -377,6 +569,7 @@ export default ({
                     endpointName: name,
                     scope: query,
                     filter: filterFn,
+                    allowedFields,
                     res,
                 })
 
@@ -455,27 +648,63 @@ export default ({
     // before plugin onFinalized hooks. By Finalize we still have the
     // cycle's journal entries; by Finalized they're already gone.
     onFinalize(async (signal) => {
-        if (!subscriptions.size) return
         const logger = useLogger()
         const evMap = {
             [OPERATION.CREATE]: 'create',
             [OPERATION.UPDATE]: 'update',
             [OPERATION.DELETE]: 'delete',
         }
+
+        // Cache invalidation is intentionally coarse: if ANY entity
+        // changed in this cycle, rebuild every cached endpoint. Per-
+        // endpoint scope matching would shave a few file writes off a
+        // churning catalog but adds bug surface for no real win —
+        // buildDefaultEnvelope is microseconds (in-memory sift),
+        // writeFile is milliseconds. Simpler and safer to just rebuild.
+        let anyChange = false
+
+        // Single journal iteration drives both SSE push and the
+        // any-change flag — one pass per cycle.
         for await (const { operation, entity } of useJournal(
             'Api subscriptions',
             [OPERATION.CREATE, OPERATION.UPDATE, OPERATION.DELETE],
             signal,
         )) {
+            anyChange = true
+
+            // SSE push to live subscribers
             for (const [subId, sub] of subscriptions) {
                 if (sub.scope && !sub.scope(entity)) continue
                 if (sub.filter && !sub.filter(entity)) continue
+                // Apply the endpoint's field projection to SSE payloads
+                // too — so live updates leak no more than list/query
+                // responses do.
+                const projected = sub.allowedFields
+                    ? _.pick(entity, sub.allowedFields)
+                    : entity
                 const payload = operation === OPERATION.DELETE
                     ? { id: entity.id }
-                    : { id: entity.id, entity }
+                    : { id: entity.id, entity: projected }
                 sseSend(sub.res, evMap[operation], payload)
                 logger.trace('Api subscription %s %s: %s', subId, evMap[operation], entity.id)
             }
         }
+
+        // Clear each cached endpoint's directory on any entity change.
+        // Subsequent list requests re-warm the cache via the write-
+        // through path in the GET/POST handlers above. Coarse but
+        // correct — any stale entry, anywhere in the per-endpoint
+        // cache, gets dropped without us having to track which queries
+        // were affected by which entity changes.
+        if (anyChange && cachedEndpoints.length > 0 && runtime.options.outputFolder) {
+            for (const ep of cachedEndpoints) {
+                await clearEndpointCache({
+                    outputFolder: runtime.options.outputFolder,
+                    base: apiBase,
+                    name: ep.name,
+                    logger,
+                })
+            }
+        }
     })
 }