mikser-io 6.23.0 → 6.27.0

package/README.md

@@ -4,9 +4,13 @@
 
 # Mikser
 
-**Mikser is the content layer of your application.** Business logic, user accounts, transactions live in their own services; mikser handles the parts that *are* content — pages, docs, the published catalog, multi-format outputs. The SDKs are the seam between them.
+**Mikser is the content layer of your application.** Business logic, user accounts, transactions live in their own services; mikser handles the parts that *are* content — pages, docs, the published catalog, multi-format outputs. [Vue](https://github.com/almero-digital-marketing/mikser-io-sdk-vue), [React](https://github.com/almero-digital-marketing/mikser-io-sdk-react), and [Svelte](https://github.com/almero-digital-marketing/mikser-io-sdk-svelte) SDKs are the seam between them — same surface across all three (`useDocument`, `useDocuments`, multilingual `useHref`, live SSE updates), each in its framework's idiomatic shape.
 
-Built for Node.js around a strict lifecycle, a composable plugin system, and direct control over every output. Every document, asset, and template flows through the same deterministic pipeline. Plugins hook in at any phase; nothing runs outside the cycle. It scales from a single markdown blog to a multi-language, multi-format publishing platform — and stays predictable in both directions.
+Built for Node.js around a strict lifecycle, a composable plugin system, and direct control over every output. Every document, asset, and template flows through the same deterministic pipeline. Plugins hook in at any phase; nothing runs outside the cycle. It scales from a single markdown blog to a multi-language, multi-format publishing platform with image / video / AI pipelines, live SSE-driven editors, semantic search, and typed frontend contracts — and stays predictable in both directions.
+
+It's MIT-licensed, runs on Node 18+, has zero hosted dependencies, and the entire content tree it manages is a folder of `.md` and `.yml` files you can copy, diff, and version-control. **The portability promise is the architecture, not a feature.**
+
+> **New to mikser?** Read the [Architecture Overview](./documentation/overview.md) — one document, end-to-end walkthrough of how a file becomes a deployed page across all twenty lifecycle phases. It's the doc most projects need first.
 
 ## Where it fits
 
@@ -30,10 +34,14 @@ Build mikser into the parts of your application that are content-shaped. Keep th
 
 **Concurrent rendering.** Renders fan out across a worker pool that keeps every CPU core hot. Multi-format outputs (HTML, PDF, MJML email, etc.) generate in parallel from the same source.
 
+**Image, video, and AI pipelines authored as plugins — not configured.** The `assets` plugin runs user-written preset modules over binary inputs. A preset is a plain Node module: ~10 lines around `sharp` resize an image, ~10 lines around `fluent-ffmpeg` transcode a video, ~30 lines around the Replicate API run an AI upscaler. The pipeline isn't a fixed menu of operations — it's whatever Node can call. Most SSGs cap your asset processing at "resize and convert format." Mikser caps it at "what can Node do." Compose with the `resources` plugin and your DAM, CDN, or company content server becomes an upstream input — uploaded files end up transcoded, watermarked, AI-enhanced, and deployed without manual handoff. See [the assets / resources docs](./documentation/plugins.md#assets) for end-to-end examples.
+
 **One lifecycle, everything composes.** Plugins hook into 20+ named lifecycle phases. A search-indexing plugin shares the same journal iteration as an email-rendering plugin and a PDF-postprocessing plugin — no glue code, no orchestration layer. The engine doesn't know which plugins are loaded; plugins don't have to know about each other.
 
 **Run anywhere.** The same CLI handles one-shot builds, watch-mode dev loops, and a long-running HTTP server with a shared Express app. `npx mikser` ships a static site; `mikser --watch` is the dev loop; `mikser --server` exposes a live admin/API.
 
+**Survives backend outages.** The `api` plugin's [per-query disk cache](./documentation/caching.md) writes every cacheable list response to `out/` keyed by the request URL. A stock-nginx reverse proxy can fail over to the cached file when mikser is unreachable — same URL, transparent to the client, no Lua or extra modules required. Production frontends keep rendering routes during deploys, brief outages, and the upstream-blip-of-the-week. Opt in per endpoint with `cache: true`; see [Caching and reverse-proxy failover](./documentation/caching.md) for the working config.
+
 **Library mode.** Mikser is also a library. `useRenderer`, `useCollection`, `findSimilar`, and direct lifecycle hooks let you embed the engine inside an existing Node app instead of running it as a CLI.
 
 **Open source.** MIT-licensed, on GitHub, no telemetry, no auth wall, no SaaS dependency. What you see is what runs.
@@ -52,6 +60,8 @@ Files-as-source isn't just a portability story — it makes the project unusuall
 
 **Plugin-by-example.** Authoring a new plugin? There are 15+ existing ones in the same shape to pattern-match against. Convention is dense enough that new plugins look like the old ones without coaching.
 
+**One-shot bootstrap via Claude Code.** The [`mikser-io-claude-plugin`](https://github.com/almero-digital-marketing/mikser-io-claude-plugin) wraps the setup story above into a single skill. Register the repo as a Claude Code marketplace and install the plugin (`/plugin marketplace add almero-digital-marketing/mikser-io-claude-plugin` then `/plugin install mikser-io-claude-plugin@mikser-io`), then in any Vue 3, React, or SvelteKit project — or in a blank directory — say "add mikser to this app." It detects the framework (or scaffolds a fresh starter via `create-vite` / `sv create`), wires the matching framework SDK, composes with your existing router rather than replacing it, and optionally lays down a `mikser-content/` sibling folder with Zod schemas and starter documents so the backend works on first run.
+
 The honest caveat: this advantage is real on **content-shaped work** — adding pages, restructuring collections, generating new layouts, building frontends. It doesn't make mikser better for non-content tasks (concurrency bugs in the worker pool, database tuning elsewhere in your stack); those are plain Node debugging like anywhere else. The visibility advantage also degrades past ~10k documents — at that scale the agent queries via the SDK instead of grepping the tree, which is still good but less "see everything at once."
 
 ## Plugins on top of the engine
@@ -74,13 +84,14 @@ The engine is what stays stable — the lifecycle, the catalog, the file-based c
 | Plugin | What it does |
 |---|---|
 | `data` | JSON snapshots of entities / context / catalog, written to disk for static serving |
-| `api` | REST endpoints with sift-backed queries, per-endpoint tokens, optional render |
+| `api` | REST endpoints with sift-backed queries, per-endpoint tokens, optional render, opt-in [per-query disk cache](./documentation/caching.md) for reverse-proxy failover |
 
 **Integrations:**
 
 | Plugin | What it does |
 |---|---|
-| `vector` | OpenAI embeddings + semantic search (sqlite-vec or pgvector) |
+| [`mikser-io-vector`](https://github.com/almero-digital-marketing/mikser-io-vector) | OpenAI embeddings + semantic search (sqlite-vec or pgvector) |
+| [`mikser-io-plugin-schemas`](https://github.com/almero-digital-marketing/mikser-io-plugin-schemas) | Zod-backed entity validation + auto-generated TypeScript declarations for the SDK |
 | `archive`, `mapper`, `live`, `aml` | Specialty integrations |
 
 **Integration probes** — wrap a substantial external project as a plugin to confirm the lifecycle is open enough to host it without core changes. Treat these as feasibility evidence, not as a statement about where mikser is heading:
@@ -91,14 +102,24 @@ The engine is what stays stable — the lifecycle, the catalog, the file-based c
 
 ## Client SDKs
 
-The `api` and `vector` plugins are paired with small client-side SDKs so a frontend (or another Node app) can talk to a running mikser server without rolling its own `fetch` glue. Zero dependencies, runs in browsers / Node 18+ / Deno / Bun / Workers.
+The `api`, `vector`, and `schemas` plugins are paired with client-side SDKs so a frontend (or another Node app) can talk to a running mikser server without rolling its own `fetch` glue or type contracts. Zero dependencies, runs in browsers / Node 18+ / Deno / Bun / Workers.
+
+**Transport-level:**
 
 | Package | For the plugin | What you get |
 |---|---|---|
-| [`mikser-io-sdk-api`](https://github.com/almero-digital-marketing/mikser-io-sdk-api) | `api` | `entities(name).list / query / urlFor / pages / update / delete / render` — Mongo-style filter operators backed by sift, sort, projection, pagination |
+| [`mikser-io-sdk-api`](https://github.com/almero-digital-marketing/mikser-io-sdk-api) | `api` | `entities(name).list / query / urlFor / pages / update / delete / render / live` — Mongo-style filter operators backed by sift, sort, projection, pagination, SSE-driven live subscriptions |
 | [`mikser-io-sdk-vector`](https://github.com/almero-digital-marketing/mikser-io-sdk-vector) | `vector` | `vector(storeName).findSimilar(text, { limit })` — semantic search hits with the original mapped object attached |
 
-Each SDK ships TypeScript declarations so client projects get autocomplete on filters, envelopes, and the `MikserError` thrown on non-2xx responses. Install only the one(s) a project needs.
+**Framework integrations** — all three wrap `mikser-io-sdk-api` in framework-idiomatic shapes. Same surface: `useDocument` / `useDocuments` live data, multilingual `useHref` / `useAlternates`, asset resolution via `useAsset`, generic on entity type so `mikser-io-plugin-schemas`-emitted types compose:
+
+| Package | Framework | Notes |
+|---|---|---|
+| [`mikser-io-sdk-vue`](https://github.com/almero-digital-marketing/mikser-io-sdk-vue) | Vue 3 | Composables, vue-router integration (`useMikserRoutes` to augment an existing router, `generateMikserRoutes` for SSG prerender), provide/inject for the client. |
+| [`mikser-io-sdk-react`](https://github.com/almero-digital-marketing/mikser-io-sdk-react) | React 18+ / 19+ | Hooks, `<MikserProvider>` Context, React Router v6+ integration via `useMikserRoutes` → `useRoutes()`. |
+| [`mikser-io-sdk-svelte`](https://github.com/almero-digital-marketing/mikser-io-sdk-svelte) | Svelte 5 (runes) | `$state` / `$effect` reactives, SvelteKit-friendly `generateMikserRoutes` for `entries()` prerender, `useMikserPages` for live nav. |
+
+Each SDK ships TypeScript declarations so client projects get autocomplete on filters, envelopes, and the `MikserError` thrown on non-2xx responses. Pair any of the framework SDKs with the `entities.d.ts` emitted by `mikser-io-plugin-schemas` for typed entity meta per layout. Install only the one(s) a project needs.
 
 ## Quick Start
 
@@ -130,19 +151,44 @@ npx mikser --server     # build + serve at :3001
 - **Runtime Singleton** — A plain module-level object holds all global state and coordinates the lifecycle. The ES module cache guarantees every importer gets the same instance.
 - **Watch Mode** — In watch mode, file changes trigger incremental re-processing without restarting.
 
+## What you can build with it
+
+The shape mikser fits cleanly:
+
+- **Marketing sites with editorial teams** — content authors work in files (via their editor, a Git client, or `mikser-io-decap`), engineers ship features without negotiating with a CMS schema, the site stays portable.
+- **Multilingual publishing platforms** — the `useHref()` / `useAlternates()` pattern in `sdk-vue` decouples logical references from per-locale URLs. One source tree, many language deployments.
+- **Content-heavy product catalogues** — `documents` + `mikser-io-plugin-schemas` + `data` plugin + a Vue frontend = typed product listings with live updates, semantic search via `vector`, and static-CDN-friendly JSON snapshots all at once.
+- **AI-augmented media pipelines** — `assets` plugin presets call out to Replicate / OpenAI / local models to upscale images, transcribe audio, transcode video. The pipeline is JS code, so anything Node can do is in scope.
+- **Mixed-output publishing** — the same source document renders to HTML, PDF (via `post-pdf`), MJML email (via `post-mjml`), and JSON snapshots. One catalog, many output formats, all concurrent.
+- **Headless backends for static frontends** — pair the `api` plugin with `sdk-api` for SSE-driven live frontends; pair the `data` plugin output with any static host for pre-rendered consumption.
+
+The shape mikser **doesn't** fit cleanly: anything with non-technical content authors who can't or won't work with files, anything with non-content business logic at the core, anything needing multi-tenant / per-user auth. Those aren't bugs — they're outside the design envelope. See [`decisions/0001-content-layer-not-the-app.md`](./documentation/decisions/0001-content-layer-not-the-app.md) for the explicit scope decision.
+
+## Engineering discipline
+
+A few things this project takes seriously:
+
+- **ADRs for load-bearing decisions.** The [`decisions/`](./documentation/decisions/) folder names which choices are structural — files-as-source, journal+catalog split, plugin-as-factory, compose-via-protocols — and explains what protects them. Read those before proposing a feature that pushes against one.
+- **Engine stability, plugin churn.** The 15+ plugins in the ecosystem add capability without core changes. The integration probes (e.g. `decap`, mounting a third-party CMS in ~150 lines) are deliberate evidence that the extension model holds.
+- **Deterministic builds.** The journal is the synchronization primitive. There's no event-passing layer, no IoC container, no plugin orchestrator. The engine knows how to fire phases in order; everything else falls out of that.
+- **The mental model is one document.** The [Architecture Overview](./documentation/overview.md) is one read for the full top-to-bottom picture. The reference docs exist for lookup; the overview exists for comprehension.
+
 ## Documentation Index
 
 | Document                                              | Audience           | Description                                        |
 | ----------------------------------------------------- | ------------------ | -------------------------------------------------- |
+| [Architecture Overview](./documentation/overview.md)  | Everyone           | **Start here.** End-to-end walkthrough of how a file becomes a deployed page across all lifecycle phases. |
 | [Getting Started](./documentation/getting-started.md) | Users              | Installation, first project, basic usage           |
 | [Configuration](./documentation/configuration.md)     | Users              | All CLI options and config file reference          |
 | [Lifecycle](./documentation/lifecycle.md)             | Users & Developers | Complete lifecycle phases and hook system          |
-| [Plugins](./documentation/plugins.md)                 | Users & Developers | Built-in plugins, writing custom plugins           |
+| [Plugins](./documentation/plugins.md)                 | Users & Developers | Built-in plugins, writing custom plugins, the assets / resources / AI pipeline |
 | [Entities](./documentation/entities.md)               | Users & Developers | Entity model, operations, journal, catalog         |
 | [Rendering](./documentation/rendering.md)             | Users & Developers | Render pipeline, render plugins, render modes      |
 | [Watch Mode](./documentation/watch-mode.md)           | Users              | File watching, scheduled tasks, incremental builds |
-| [Architecture](./documentation/architecture.md)       | Developers         | System design, module structure, extension points  |
+| [Caching](./documentation/caching.md)                 | Users (production) | The `cache: true` disk cache + working nginx config for reverse-proxy failover |
+| [Architecture](./documentation/architecture.md)       | Developers         | Module-level reference — what's in each file       |
 | [API Reference](./documentation/api-reference.md)     | Developers         | Complete public API reference                      |
+| [Decisions (ADRs)](./documentation/decisions/)        | Developers         | Load-bearing architectural choices and what protects them |
 
 ## License
 

package/app.js

@@ -1,3 +1,4 @@
+#!/usr/bin/env node
 import { setup } from "./index.js"
 
 async function main() {

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "mikser-io",
-	"version": "6.23.0",
+	"version": "6.27.0",
 	"description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
 	"main": "index.js",
 	"scripts": {

package/src/plugins/api.js

@@ -1,5 +1,6 @@
 import path from 'node:path'
-import { access } from 'node:fs/promises'
+import { access, writeFile, mkdir, rm } from 'node:fs/promises'
+import { createHash } from 'node:crypto'
 import _ from 'lodash'
 import sift from 'sift'
 import { useRenderer, useCollection } from '../api.js'
@@ -118,6 +119,88 @@ async function runQuery({ filter, sort, fields, skip, limit, scope, findEntities
     return { items, total }
 }
 
+// Derive the cache filename from the raw query string the client sent.
+//   no query string (`/entities`)       → 'index'
+//   query string ('a=1&b=2')            → 'a=1&b=2'
+//
+// We deliberately keep the raw, undecoded form because:
+//   - that's what nginx's `$args` variable contains
+//   - stock nginx can pass it straight into `try_files` without
+//     hashing modules or Lua
+//   - the URL is its own cache key — no extra contract for clients
+//     or proxies to agree on beyond "use the request URL"
+//
+// Trade-off: structurally-equal queries with different parameter
+// orders produce different cache files (`?a=1&b=2` vs `?b=2&a=1`).
+// Soft inefficiency, not a correctness bug — the SDK uses a stable
+// order and most consumers are SDK-driven. Filesystem rules apply:
+// most chars (`=`, `&`, `[`, `]`, `%`) are safe; 255-byte filename
+// limits cap query length on typical fs. Document these in the
+// caching README.
+function cacheNameForQueryString(rawQueryString) {
+    if (!rawQueryString) return 'index'
+    return rawQueryString
+}
+
+// Wide-list defaults — tuned for "an SPA accidentally pulled the whole
+// catalog because it forgot a `fields` projection." Operators can grep
+// the warning out of logs; SDK users see the same shape client-side via
+// the SDK's dev-mode warning.
+const WIDE_RESPONSE_ITEMS = 100
+const WIDE_RESPONSE_BYTES = 256 * 1024
+
+function maybeWarnWide({ logger, endpoint, envelope, req }) {
+    const items = envelope?.items?.length ?? 0
+    if (items <= WIDE_RESPONSE_ITEMS) return
+    const bytes = Buffer.byteLength(JSON.stringify(envelope))
+    if (bytes <= WIDE_RESPONSE_BYTES && items <= WIDE_RESPONSE_ITEMS) return
+    const queryStr = (req.originalUrl || req.url || '').split('?')[1] || '(none)'
+    const sizeLabel = bytes >= 1024 * 1024
+        ? `${(bytes / 1024 / 1024).toFixed(1)} MB`
+        : `${Math.round(bytes / 1024)} KB`
+    logger.warn(
+        'Api[%s] wide list response: %d items, %s — query=%s',
+        endpoint, items, sizeLabel, queryStr,
+    )
+    logger.warn(
+        '  ↳ Consider a `fields:` projection, or move this query to a `data.catalog.<name>` snapshot loaded via the SDK\'s `initialUrl`.',
+    )
+}
+
+// Write the response envelope to <out>/<base>/<endpoint>/entities/<name>.json.
+// The path matches what nginx's `try_files /<base>/<endpoint>/entities/$args.json
+// /<base>/<endpoint>/entities/index.json` looks up on upstream failure.
+async function writeQueryCache({ outputFolder, base, name, cacheName, envelope, logger }) {
+    const relBase = base.replace(/^\//, '')
+    const dir = path.join(outputFolder, relBase, name, 'entities')
+    const file = path.join(dir, `${cacheName}.json`)
+    try {
+        await mkdir(dir, { recursive: true })
+        await writeFile(file, JSON.stringify(envelope), 'utf8')
+        logger.trace('Api[%s] cache write: %s (%d items)', name, file, envelope.items.length)
+    } catch (err) {
+        // Filename-length / illegal-char failures land here. Logged but
+        // not propagated — the live response is already on the wire.
+        logger.error('Api[%s] cache write failed (%s): %s', name, file, err.message)
+    }
+}
+
+// Clear the entire per-endpoint cache directory. Called when any
+// catalog entity changes — coarse but correct, and on-demand writes
+// rebuild the cache as queries come back in. The cost of warm-up
+// after invalidation is bounded by traffic, which is what you'd want
+// from a write-through cache anyway.
+async function clearEndpointCache({ outputFolder, base, name, logger }) {
+    const relBase = base.replace(/^\//, '')
+    const dir = path.join(outputFolder, relBase, name, 'entities')
+    try {
+        await rm(dir, { recursive: true, force: true })
+        logger.trace('Api[%s] cache cleared: %s', name, dir)
+    } catch (err) {
+        logger.error('Api[%s] cache clear failed (%s): %s', name, dir, err.message)
+    }
+}
+
 // MIME type lookup used when streaming a postprocessor's output back over
 // HTTP. The renderer's output extension lives on entity.destination
 // (assigned by the layouts plugin), so we use it as the source of truth.
@@ -211,6 +294,13 @@ export default ({
     findEntities,
     constants: { OPERATION },
 }) => {
+    // Shared between onLoaded (populates) and onFinalize (consumes).
+    // Hoisted so the lifecycle hooks see the same registry; the body
+    // inside onLoaded is what actually fills it in based on
+    // runtime.config.api.endpoints.
+    let apiBase = '/api'
+    const cachedEndpoints = []
+
     onLoaded(async () => {
         const logger = useLogger()
 
@@ -237,10 +327,14 @@ export default ({
             throw new Error('Express is required for the api plugin — run: npm install express')
         })
 
-        const base = runtime.config.api?.base ?? '/api'
+        apiBase = runtime.config.api?.base ?? '/api'
+        const base = apiBase   // alias so existing local references still work
         const globalPageSize = runtime.config.api?.pageSize ?? 10
         const globalRenderTimeout = runtime.config.api?.renderTimeout ?? 30_000
 
+        // cachedEndpoints is hoisted above (shared with onFinalize).
+        // Per-endpoint setup loop pushes into it when cache: true is set.
+
         // The plugin mirrors the vector / data plugin shape: a named map
         // of endpoints, each with its own optional `token`, `query`
         // scope, allowed `operations`, and overrides for pageSize /
@@ -267,6 +361,69 @@ export default ({
             const pageSize = ep.pageSize ?? globalPageSize
             const renderTimeout = ep.renderTimeout ?? globalRenderTimeout
 
+            // Server-enforced field projection. When set, every list /
+            // query / subscribe response is narrowed to exactly these
+            // dotted paths — regardless of what the client asks for.
+            //
+            // The right tool when an endpoint backs a broad list query
+            // and only a handful of fields per entity are actually
+            // useful to the client — sitemap, navigation menus,
+            // faceted search dimensions. A SPA's first paint shouldn't
+            // download every markdown body just to find out which
+            // routes exist.
+            //
+            // Skip it for endpoints that serve individual full
+            // documents (the typical `public` shape used by
+            // useDocument(id)) — those queries return one entity at a
+            // time so narrowing has no real benefit.
+            const allowedFields = Array.isArray(ep.fields) && ep.fields.length
+                ? ep.fields
+                : null
+
+            // Pick the effective projection for a request. If the
+            // endpoint declares allowedFields, that's the ceiling: a
+            // client requesting more gets only the intersection. A
+            // client requesting nothing gets exactly allowedFields.
+            // No allowedFields → whatever the client asked for (or
+            // all fields when omitted).
+            function resolveFields(requested) {
+                if (!allowedFields) return requested ?? null
+                if (!requested || !requested.length) return allowedFields
+                const allowSet = new Set(allowedFields)
+                return requested.filter(f => allowSet.has(f))
+            }
+
+            // Endpoints with cache: true cache GET /entities responses
+            // to disk on a per-query-string basis. Path scheme:
+            //   <out>/<base>/<name>/entities/<raw-query-string>.json
+            //   <out>/<base>/<name>/entities/index.json    (no params)
+            //
+            // On any catalog change, the whole cache directory is
+            // dropped — the next requests through repopulate whatever's
+            // needed. Coarse but correct, no per-query tracking.
+            //
+            // Reverse-proxy failover (stock nginx, no Lua):
+            //   location /api/sitemap/entities {
+            //       proxy_pass http://localhost:3001;
+            //       proxy_intercept_errors on;
+            //       error_page 502 503 504 = @cache;
+            //   }
+            //   location @cache {
+            //       root /var/www/out;
+            //       try_files /api/sitemap/entities/$args.json
+            //                 /api/sitemap/entities/index.json
+            //                 =502;
+            //   }
+            //
+            // POST /entities/query responses aren't cached — there's no
+            // URL the proxy could derive a cache file path from. Use
+            // GET for cacheable queries (the SDK's client.list({...}) →
+            // GET URL is the canonical pattern).
+            if (ep.cache === true) {
+                cachedEndpoints.push({ name })
+            }
+            const cacheEnabled = ep.cache === true
+
             const auth = (req, res, next) => {
                 if (!ep.token) return next()
                 if (req.headers.authorization === `Bearer ${ep.token}`) return next()
@@ -289,6 +446,7 @@ export default ({
             const { render } = useRenderer(runtime, { defaultTimeout: renderTimeout })
 
             router.get('/entities', allow('list'), auth, async (req, res) => {
+                const t0 = Date.now()
                 try {
                     const parsed = parseQueryString(req.query)
                     const limit = Math.min(100, Math.max(1, parsed.limit ?? pageSize))
@@ -297,7 +455,7 @@ export default ({
                     const { items, total } = await runQuery({
                         filter: parsed.filter,
                         sort: parsed.sort,
-                        fields: parsed.fields,
+                        fields: resolveFields(parsed.fields),
                         skip,
                         limit,
                         scope: query,
@@ -306,13 +464,37 @@ export default ({
 
                     const page = Math.floor(skip / limit) + 1
                     const totalPages = Math.ceil(total / limit) || 1
-                    res.json({
+                    const envelope = {
                         items, page, limit, total, totalPages,
                         hasNext: skip + limit < total,
                         hasPrev: skip > 0,
-                    })
+                    }
+                    res.json(envelope)
+                    logger.trace('Api[%s] list %dms (%d/%d items)', name, Date.now() - t0, items.length, total)
+                    maybeWarnWide({ logger, endpoint: name, envelope, req })
+
+                    // Write-through cache: write the response to a file
+                    // path that mirrors the request URL — same `$args`
+                    // nginx's `try_files` sees, no hashing on either
+                    // side. See the caching docs for the nginx config
+                    // that wires the failover.
+                    if (cacheEnabled && runtime.options.outputFolder) {
+                        const url = req.originalUrl || req.url || ''
+                        const qIdx = url.indexOf('?')
+                        const rawQueryString = qIdx >= 0 ? url.slice(qIdx + 1) : ''
+                        const cacheName = cacheNameForQueryString(rawQueryString)
+                        // Fire-and-forget — the response is already sent.
+                        writeQueryCache({
+                            outputFolder: runtime.options.outputFolder,
+                            base: apiBase,
+                            name,
+                            cacheName,
+                            envelope,
+                            logger,
+                        }).catch(() => {})
+                    }
                 } catch (err) {
-                    logger.error('Api[%s] list error: %s', name, err.message)
+                    logger.error('Api[%s] list error (%dms): %s', name, Date.now() - t0, err.message)
                     res.status(500).json({ error: err.message })
                 }
             })
@@ -321,26 +503,40 @@ export default ({
             // doesn't fit cleanly in a URL: $and/$or, nested operators,
             // regex, projections, etc. Same shape as a Mongo find.
             router.post('/entities/query', allow('list'), auth, async (req, res) => {
+                const t0 = Date.now()
                 try {
                     const { filter = {}, sort, fields, page: rawPage = 1, limit: rawLimit, skip: rawSkip } = req.body ?? {}
                     const limit = Math.min(100, Math.max(1, rawLimit ?? pageSize))
                     const skip = rawSkip ?? (Math.max(1, parseInt(rawPage) || 1) - 1) * limit
 
                     const { items, total } = await runQuery({
-                        filter, sort, fields, skip, limit,
+                        filter, sort,
+                        fields: resolveFields(fields),
+                        skip, limit,
                         scope: query,
                         findEntities,
                     })
 
                     const page = Math.floor(skip / limit) + 1
                     const totalPages = Math.ceil(total / limit) || 1
-                    res.json({
+                    const envelope = {
                         items, page, limit, total, totalPages,
                         hasNext: skip + limit < total,
                         hasPrev: skip > 0,
-                    })
+                    }
+                    res.json(envelope)
+                    logger.trace('Api[%s] query %dms (%d/%d items)', name, Date.now() - t0, items.length, total)
+                    maybeWarnWide({ logger, endpoint: name, envelope, req })
+
+                    // POST queries aren't disk-cached. The reverse proxy
+                    // failover scheme is URL-based (cache file path =
+                    // request URL), but POST has no URL-equivalent for
+                    // its body. Cacheable queries should use GET — which
+                    // the SDK's list() does by default. Body-only queries
+                    // are by definition complex/non-canonicalizable so
+                    // they're treated as live-only.
                 } catch (err) {
-                    logger.error('Api[%s] query error: %s', name, err.message)
+                    logger.error('Api[%s] query error (%dms): %s', name, Date.now() - t0, err.message)
                     res.status(500).json({ error: err.message })
                 }
             })
@@ -373,6 +569,7 @@ export default ({
                     endpointName: name,
                     scope: query,
                     filter: filterFn,
+                    allowedFields,
                     res,
                 })
 
@@ -451,27 +648,63 @@ export default ({
     // before plugin onFinalized hooks. By Finalize we still have the
     // cycle's journal entries; by Finalized they're already gone.
     onFinalize(async (signal) => {
-        if (!subscriptions.size) return
         const logger = useLogger()
         const evMap = {
             [OPERATION.CREATE]: 'create',
             [OPERATION.UPDATE]: 'update',
             [OPERATION.DELETE]: 'delete',
         }
+
+        // Cache invalidation is intentionally coarse: if ANY entity
+        // changed in this cycle, rebuild every cached endpoint. Per-
+        // endpoint scope matching would shave a few file writes off a
+        // churning catalog but adds bug surface for no real win —
+        // buildDefaultEnvelope is microseconds (in-memory sift),
+        // writeFile is milliseconds. Simpler and safer to just rebuild.
+        let anyChange = false
+
+        // Single journal iteration drives both SSE push and the
+        // any-change flag — one pass per cycle.
         for await (const { operation, entity } of useJournal(
             'Api subscriptions',
             [OPERATION.CREATE, OPERATION.UPDATE, OPERATION.DELETE],
             signal,
         )) {
+            anyChange = true
+
+            // SSE push to live subscribers
             for (const [subId, sub] of subscriptions) {
                 if (sub.scope && !sub.scope(entity)) continue
                 if (sub.filter && !sub.filter(entity)) continue
+                // Apply the endpoint's field projection to SSE payloads
+                // too — so live updates leak no more than list/query
+                // responses do.
+                const projected = sub.allowedFields
+                    ? _.pick(entity, sub.allowedFields)
+                    : entity
                 const payload = operation === OPERATION.DELETE
                     ? { id: entity.id }
-                    : { id: entity.id, entity }
+                    : { id: entity.id, entity: projected }
                 sseSend(sub.res, evMap[operation], payload)
                 logger.trace('Api subscription %s %s: %s', subId, evMap[operation], entity.id)
             }
         }
+
+        // Clear each cached endpoint's directory on any entity change.
+        // Subsequent list requests re-warm the cache via the write-
+        // through path in the GET/POST handlers above. Coarse but
+        // correct — any stale entry, anywhere in the per-endpoint
+        // cache, gets dropped without us having to track which queries
+        // were affected by which entity changes.
+        if (anyChange && cachedEndpoints.length > 0 && runtime.options.outputFolder) {
+            for (const ep of cachedEndpoints) {
+                await clearEndpointCache({
+                    outputFolder: runtime.options.outputFolder,
+                    base: apiBase,
+                    name: ep.name,
+                    logger,
+                })
+            }
+        }
     })
 }