mikser-io-sdk-api 2.4.0 → 2.5.1

package/README.md

@@ -103,7 +103,94 @@ Operations outside the endpoint's allowlist return `403`; missing or wrong token
 
 ## Entities
 
-`mikser.entities(endpointName, { token })` returns a per-endpoint client. The endpoint name matches a key in your `api.endpoints` config on the server.
+`mikser.entities(endpointName, options)` returns a per-endpoint client. The endpoint name matches a key in your `api.endpoints` config on the server. Supported options:
+
+| Option            | Default | What it does                                                                                  |
+|---|---|---|
+| `token`           | `null`  | Bearer token for endpoints declared with a token. Sent on every request.                       |
+| `initialUrl`      | `null`  | Static-snapshot URL — see below.                                                              |
+| `fallbackToList`  | `true`  | If `initialUrl` is set and the fetch fails (404 in dev, wrong shape, network error), fall back to a live `list()`. Set `false` for production deploys where missing the snapshot should be a hard error. |
+
+### `initialUrl` — pair with the `data` plugin for fast first paint
+
+If you have a known, predictable query that runs on every page load — a route table for an SPA, a navigation menu, a category list — paying an API round-trip for it on first paint is wasted work. The right shape is to publish that data as a static file at build/finalize time and have the SDK pick it up.
+
+Mikser's `data` plugin does the publishing side. A `catalog.<name>` entry writes one JSON file per name under `out/data/`; `<name>` is whatever you want to call it — name it after the role the snapshot plays in your app (`sitemap`, `menu`, `tags`, `categories`, etc.). Example for a router-driving snapshot:
+
+```js
+// mikser.config.js
+plugins: ['documents', 'front-matter', 'data', 'api', /* ... */],
+
+data: {
+    catalog: {
+        // out/data/sitemap.json — one entry per published, component-having
+        // document, projected to just the routing fields. `sitemap` is the
+        // catalog-entry name; the SDK below uses the matching URL path.
+        sitemap: {
+            query: e => e.type === 'document' && e.meta?.published && e.meta?.component,
+            pick: ['id', 'destination', 'meta.component', 'meta.route', 'meta.title'],
+        },
+    },
+},
+
+api: {
+    endpoints: {
+        public: {
+            query: e => e.type === 'document' && e.meta?.published,
+            operations: ['list', 'subscribe'],
+            cache: true,
+        },
+    },
+},
+```
+
+On the client, point `entities()` at the matching file. The URL path mirrors the catalog name: `data.catalog.sitemap` → `/data/sitemap.json`, `data.catalog.menu` → `/data/menu.json`, and so on.
+
+```js
+const documents = createClient({ baseUrl: 'https://cms.example.com' })
+    .entities('public', { initialUrl: '/data/sitemap.json' })
+```
+
+What `initialUrl` changes:
+
+- **`live(filter, onChange, options)`** fires `onChange` with the snapshot immediately (no API round-trip), then opens the SSE subscribe stream as usual. So `useMikserRoutes` / `useMikserPages` get a populated route table before the network even settles.
+- **`listAll()`** consults the snapshot first and only falls back to paginated `list()` calls if the snapshot is missing or `fallbackToList: true` and the fetch failed.
+- **`list()`**, `urlFor()`, `query()`, `update()`, `delete()`, `render()`, `subscribe()` are unchanged — they always go to the API.
+
+The data plugin emits each entry as `{ refId, name, date, data: {...picked} }`. The SDK strips that wrapper automatically and hands `onChange` / `listAll` a plain array of the `data` payloads, so a static snapshot looks identical to a live response.
+
+This is **not** a cache. The snapshot is only consulted for the initial fill; ongoing changes come over SSE on the actual API endpoint. For runtime fail-safety on per-id reads, that's what the api plugin's `cache: true` is for — see [mikser-io's caching docs](https://github.com/almero-digital-marketing/mikser-io/blob/main/documentation/caching.md).
+
+**Edge case: first-paint flash for fast-changing snapshots.** First paint renders the snapshot — which may be N seconds old, depending on when mikser last wrote it — and the SSE stream then arrives and reconciles any changes since. For a route table that's invisible; routes don't move every second. For snapshots of fast-changing data (recent activity, in-stock badges, live counters) the user may briefly see stale content before SSE catches up. If that flash is visible UX, either design for it (e.g. show a subtle "syncing" indicator until the first SSE event arrives, or render a loading state when the snapshot age exceeds your tolerance) or skip `initialUrl` for that particular query — paying the API roundtrip is the right tradeoff when the response can't be allowed to be stale.
+
+**Snapshot-bypass warning.** Snapshots only apply when the `live()` / `listAll()` call is trivial — no filter, no sort, no skip. Add any of those and the SDK silently falls back to the live API. Since that's the kind of thing a developer can change without noticing, the SDK emits a one-time `console.warn` per `(endpoint, call kind, what-was-set)` shape:
+
+```
+[mikser-sdk] initialUrl is set on "public" but this live() call uses filter+sort
+  — snapshot bypassed, falling back to live list().
+  Snapshots only apply when the call is trivial (no filter/sort/skip).
+  Either remove the filter+sort from this call, or accept the API roundtrip if
+  filtering is intentional.
+  Suppress: pass { quiet: true } on the call, or set MIKSER_QUIET=1.
+```
+
+Same suppression channels as the wide-list warning above.
+
+### Dev-mode warning: accidentally wide queries
+
+The common failure mode for a CMS-backed app is "I just wanted a nav menu but pulled every full document over the wire." To catch it at write time, `list()` and `live()` emit a one-time dev-mode `console.warn` when a response has more than 50 items and the query has no `fields:` projection:
+
+```
+[mikser-sdk] list() returned 247 items (~4.2 MB) from "public" with no `fields` projection.
+  Add { fields: [...] } to narrow it, or — if this query runs on every page load —
+  move it to a `data.catalog.<name>` snapshot on the mikser side and load via:
+    entities('public', { initialUrl: '/data/<name>.json' })
+  Suppress: pass { quiet: true } on the call, or set MIKSER_QUIET=1.
+```
+
+The warning is deduped per `(endpoint, filter, sort)` shape, so an SSE-driven list that updates 30 times only fires once. It's silenced when `process.env.NODE_ENV === 'production'`, when `MIKSER_QUIET` is set, or per-call via `{ quiet: true }` on `list(query, opts)` / `live(filter, onChange, opts)`.
+
+The mikser-io server emits a matching warning in its logs for the same shape — useful when the wide query came from curl, another SDK, or an environment where `console` isn't visible.
 
 ### `list(query)` — body-based
 

package/index.d.ts

@@ -16,6 +16,32 @@ export interface ClientOptions {
 export interface EntityOptions {
     /** Bearer token sent on every request to this endpoint. */
     token?: string
+    /**
+     * URL (relative to baseUrl or absolute) of a pre-built static JSON
+     * snapshot of the entity set. When set, `live()` and `listAll()`
+     * read from this URL on first call instead of hitting the live API.
+     *
+     * Largely obsolete with the api plugin's per-query disk cache
+     * (mikser-io ^6.25.0) — the cache writes to the same URL the live
+     * API serves, and a reverse proxy can fail over to it transparently.
+     * Use `initialUrl` only when:
+     *   - the snapshot URL is a different host (CDN, edge cache,
+     *     pre-rendered static asset on disk)
+     *   - you want to skip the live API roundtrip on boot even when
+     *     mikser is up (microoptimization)
+     *
+     * Otherwise, just configure your reverse proxy to fail over to the
+     * cache on backend errors and the live URL Just Works.
+     */
+    initialUrl?: string
+    /**
+     * When the snapshot fetch fails (404 in dev, network error,
+     * unrecognised shape), default behaviour is to silently fall
+     * back to a fresh `list()` call. Set false for production
+     * environments where you require the snapshot to be present.
+     * @default true
+     */
+    fallbackToList?: boolean
 }
 
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mikser-io-sdk-api",
-  "version": "2.4.0",
+  "version": "2.5.1",
   "description": "Client SDK for mikser-io's api plugin — query the document catalog from the browser or Node",
   "main": "index.js",
   "types": "index.d.ts",

package/src/entities.js

@@ -7,20 +7,197 @@ import { bearer, jsonOrThrow } from './http.js'
 import { joinUrl, sortToParam, filterToParams } from './url.js'
 import { parseSseEvent } from './sse.js'
 
+// Conservative URL-length ceiling for the GET form of list(). Real
+// browsers and proxies vary (Chrome ~32k, IIS ~16k, nginx default
+// 8k, some CDNs 4k), but the safest interop floor for list-with-
+// many-filter-params is well under 2k. Queries past this fall back
+// to POST automatically.
+const GET_MAX_URL = 1800
+
+// Wide-list warning — surfaces in dev mode when a list/live call
+// returns more than this without a `fields` projection. Catches the
+// common "I just wanted a nav menu but pulled every full document"
+// failure mode at the first place it manifests (developer's DevTools
+// console). Server-side has a matching warning that fires for all
+// clients regardless of SDK use.
+const WIDE_RESPONSE_ITEMS = 50
+const _warnedShapes = new Set()
+
+function isProductionEnv() {
+    try {
+        return typeof process !== 'undefined'
+            && process.env?.NODE_ENV === 'production'
+    } catch { return false }
+}
+function isQuiet() {
+    try {
+        return typeof process !== 'undefined' && process.env?.MIKSER_QUIET
+    } catch { return false }
+}
+
+function maybeWarnWide({ endpoint, query, envelopeOrItems, quiet }) {
+    if (quiet || isProductionEnv() || isQuiet()) return
+    const items = Array.isArray(envelopeOrItems)
+        ? envelopeOrItems
+        : envelopeOrItems?.items
+    if (!Array.isArray(items) || items.length <= WIDE_RESPONSE_ITEMS) return
+    const hasFields = Array.isArray(query?.fields) && query.fields.length > 0
+    if (hasFields) return
+    const shape = `${endpoint}|${JSON.stringify(query?.filter ?? null)}|${JSON.stringify(query?.sort ?? null)}`
+    if (_warnedShapes.has(shape)) return
+    _warnedShapes.add(shape)
+    let sizeNote = ''
+    try {
+        const bytes = JSON.stringify(items).length
+        sizeNote = bytes >= 1024 * 1024
+            ? ` (~${(bytes / 1024 / 1024).toFixed(1)} MB)`
+            : ` (~${Math.round(bytes / 1024)} KB)`
+    } catch { /* size note is best-effort */ }
+    console.warn(
+        `[mikser-sdk] list() returned ${items.length} items${sizeNote} from "${endpoint}" with no \`fields\` projection.\n` +
+        `  Add { fields: [...] } to narrow it, or — if this query runs on every page load —\n` +
+        `  move it to a \`data.catalog.<name>\` snapshot on the mikser side and load via:\n` +
+        `    entities('${endpoint}', { initialUrl: '/data/<name>.json' })\n` +
+        `  Suppress: pass { quiet: true } on the call, or set MIKSER_QUIET=1.`,
+    )
+}
+
+// Snapshot-bypass warning — fires when `initialUrl` is configured but
+// the call is non-trivial (has filter / sort / skip), so the snapshot
+// can't be used and the SDK falls back to the live API. The bypass is
+// correct behavior, but it's silent: developers often set initialUrl
+// once and then add a sort to one of their useDocuments calls without
+// noticing the snapshot is no longer involved. Deduped per
+// (endpoint, kind, what-was-set) so a page with 3 filtered calls
+// produces 3 warnings, not 30.
+const _bypassedShapes = new Set()
+function maybeWarnSnapshotBypass({ endpoint, kind, filter, sort, skip, quiet }) {
+    if (quiet || isProductionEnv() || isQuiet()) return
+    const reasons = []
+    if (filter) reasons.push('filter')
+    if (sort)   reasons.push('sort')
+    if (skip != null) reasons.push('skip')
+    if (reasons.length === 0) return
+    const reasonLabel = reasons.join('+')
+    const shape = `${endpoint}|${kind}|${reasonLabel}`
+    if (_bypassedShapes.has(shape)) return
+    _bypassedShapes.add(shape)
+    const fallback = kind === 'live' ? 'live list()' : 'paginated fetch'
+    console.warn(
+        `[mikser-sdk] initialUrl is set on "${endpoint}" but this ${kind}() call uses ${reasonLabel} — snapshot bypassed, falling back to ${fallback}.\n` +
+        `  Snapshots only apply when the call is trivial (no filter/sort/skip).\n` +
+        `  Either remove the ${reasonLabel} from this call, or accept the API roundtrip if filtering is intentional.\n` +
+        `  Suppress: pass { quiet: true } on the call, or set MIKSER_QUIET=1.`,
+    )
+}
+
 export function createEntitiesClient({ baseUrl, basePath, fetch: doFetch, headers: defaultHeaders }) {
-    return function entities(name, { token } = {}) {
+    return function entities(name, opts = {}) {
+        const {
+            token,
+            // initialUrl: optional URL (relative to baseUrl or absolute) for a
+            // pre-built static JSON snapshot — typically produced by the
+            // `data` plugin's catalog.<name> output. When set, live() fires
+            // onChange with the snapshot immediately, then opens the SSE
+            // subscribe stream as usual. listAll() also consults the
+            // snapshot before falling back to paginated list calls.
+            //
+            // The fetched JSON is unwrapped automatically — the data
+            // plugin emits `[{ refId, name, date, data }, ...]`; we
+            // return the `.data` payloads. Plain arrays and { items }
+            // envelopes are passed through unchanged.
+            //
+            // Pre-built snapshots are the fast first-paint path:
+            // CDN-cached, no API roundtrip, no SSE latency tax on boot.
+            // Pair the snapshot's data-plugin filter with your live()
+            // filter so the initial state matches what SSE will send.
+            initialUrl,
+            // When initialUrl fetch fails (404 in dev, network error,
+            // wrong shape), default behavior is to log and fall back to
+            // a fresh list() call — keeps dev mode trivial. Set false
+            // for environments where you require the snapshot.
+            fallbackToList = true,
+        } = opts
         const endpointBase = `${basePath}/${name}`
         const queryUrl     = joinUrl(baseUrl, `${endpointBase}/entities/query`)
         const listUrl      = joinUrl(baseUrl, `${endpointBase}/entities`)
         const subscribeUrl = joinUrl(baseUrl, `${endpointBase}/entities/subscribe`)
         const renderUrl    = joinUrl(baseUrl, `${endpointBase}/render`)
 
+        const resolvedInitialUrl = initialUrl
+            ? (/^https?:\/\//.test(initialUrl) ? initialUrl : joinUrl(baseUrl, initialUrl))
+            : null
+
+        // Cached snapshot promise — concurrent live() / listAll() calls
+        // share one fetch. Cleared on error so a flaky connection can
+        // recover on the next call.
+        let snapshotPromise = null
+        async function loadSnapshot() {
+            if (!resolvedInitialUrl) return null
+            if (snapshotPromise) return snapshotPromise
+            snapshotPromise = (async () => {
+                try {
+                    const res = await doFetch(resolvedInitialUrl, {
+                        method: 'GET',
+                        headers: { accept: 'application/json', ...defaultHeaders },
+                    })
+                    if (!res.ok) {
+                        if (!fallbackToList) {
+                            throw new MikserError(res.status, res.statusText, null, resolvedInitialUrl)
+                        }
+                        return null
+                    }
+                    const payload = await res.json()
+                    return unwrapSnapshot(payload)
+                } catch (err) {
+                    // Clear the cached promise so the next call can retry.
+                    snapshotPromise = null
+                    if (!fallbackToList) throw err
+                    return null
+                }
+            })()
+            return snapshotPromise
+        }
+
         /**
-         * Body-based query. Send everything sift accepts —
-         * $and / $or / $regex, projections, sorts. Returns the standard
-         * envelope: { items, page, limit, total, totalPages, hasNext, hasPrev }.
+         * Run a list query. Returns the standard envelope:
+         * { items, page, limit, total, totalPages, hasNext, hasPrev }.
+         *
+         * Transport selection: tries GET first because GET responses
+         * are what the api plugin's per-query disk cache writes (and
+         * what a reverse proxy can serve as failover when mikser is
+         * down). Falls back to POST when:
+         *   - the encoded URL exceeds GET_MAX_URL (browsers and most
+         *     proxies start refusing past ~2KB; we use a conservative
+         *     limit of 1800)
+         *   - GET is explicitly disabled via opts.method = 'POST'
+         *
+         * GET can express anything sift accepts via the `.$op` URL-param
+         * suffix scheme (see urlFor). Use POST when you want to be
+         * explicit about not engaging the cache path, e.g. for queries
+         * with secrets in the body that shouldn't sit in proxy logs.
          */
-        async function list(query = {}) {
+        async function list(query = {}, opts = {}) {
+            const forcePost = opts.method === 'POST'
+
+            if (!forcePost) {
+                const url = urlFor(query)
+                if (url.length <= GET_MAX_URL) {
+                    const res = await doFetch(url, {
+                        method: 'GET',
+                        headers: {
+                            accept: 'application/json',
+                            ...defaultHeaders,
+                            ...bearer(token),
+                        },
+                    })
+                    const envelope = await jsonOrThrow(res, url)
+                    maybeWarnWide({ endpoint: name, query, envelopeOrItems: envelope, quiet: opts.quiet })
+                    return envelope
+                }
+            }
+
+            // POST fallback — long URLs and forced POST land here.
             const res = await doFetch(queryUrl, {
                 method: 'POST',
                 headers: {
@@ -30,7 +207,9 @@ export function createEntitiesClient({ baseUrl, basePath, fetch: doFetch, header
                 },
                 body: JSON.stringify(query),
             })
-            return jsonOrThrow(res, queryUrl)
+            const envelope = await jsonOrThrow(res, queryUrl)
+            maybeWarnWide({ endpoint: name, query, envelopeOrItems: envelope, quiet: opts.quiet })
+            return envelope
         }
 
         /**
@@ -78,6 +257,27 @@ export function createEntitiesClient({ baseUrl, basePath, fetch: doFetch, header
          * is wasteful. Use pages() directly and stream-process there.
          */
         async function listAll(query = {}) {
+            // Snapshot fast path: if no filter/sort/skip is requested
+            // beyond what the snapshot was built with, return the
+            // pre-built array directly. The caller's filter/sort would
+            // require re-querying the live endpoint anyway, so any
+            // non-trivial query falls through to the paginated fetch.
+            const trivial = !query.filter && !query.sort && !query.skip
+            if (trivial && resolvedInitialUrl) {
+                const snapshot = await loadSnapshot()
+                if (snapshot) {
+                    return query.fields
+                        ? snapshot.map(item => pickFields(item, query.fields))
+                        : snapshot
+                }
+                // Snapshot unavailable — fall through to paginated fetch.
+            }
+            if (!trivial && resolvedInitialUrl) {
+                maybeWarnSnapshotBypass({
+                    endpoint: name, kind: 'listAll',
+                    filter: query.filter, sort: query.sort, skip: query.skip,
+                })
+            }
             const items = []
             for await (const env of pages({ limit: 1000, ...query })) {
                 items.push(...env.items)
@@ -170,6 +370,7 @@ export function createEntitiesClient({ baseUrl, basePath, fetch: doFetch, header
         function live(filter, onChange, options = {}) {
             const {
                 sort, fields, limit, skip,
+                quiet,
                 signal: externalSignal,
                 onError = (err) => console.error('mikser-io-sdk-api live error:', err),
             } = options
@@ -185,10 +386,40 @@ export function createEntitiesClient({ baseUrl, basePath, fetch: doFetch, header
 
             const loop = (async () => {
                 try {
-                    const env = await list({ filter, sort, fields, limit, skip })
-                    if (disposed || ac.signal.aborted) return
-                    items = env.items
-                    onChange(items)
+                    // Snapshot fast path. Use it only when the caller's
+                    // query is trivial enough that the pre-built array
+                    // reflects what list() would return — anything more
+                    // specific (filter, sort, skip) goes through list()
+                    // so the caller's intent is honored.
+                    let usedSnapshot = false
+                    const trivial = !filter && !sort && !skip
+                    if (trivial && resolvedInitialUrl) {
+                        const snapshot = await loadSnapshot()
+                        if (snapshot) {
+                            if (disposed || ac.signal.aborted) return
+                            items = fields ? snapshot.map(i => pickFields(i, fields)) : snapshot
+                            onChange(items)
+                            usedSnapshot = true
+                        }
+                    }
+                    if (!trivial && resolvedInitialUrl) {
+                        maybeWarnSnapshotBypass({
+                            endpoint: name, kind: 'live',
+                            filter, sort, skip, quiet,
+                        })
+                    }
+                    if (!usedSnapshot) {
+                        // Pass { quiet } so list()'s wide-warning honors
+                        // the live() caller's quiet opt. live() also
+                        // does its own snapshot-path warning below for
+                        // the case where the initial fill came from
+                        // /data/<name>.json — those callers already
+                        // opted into the narrow shape, so no warning.
+                        const env = await list({ filter, sort, fields, limit, skip }, { quiet })
+                        if (disposed || ac.signal.aborted) return
+                        items = env.items
+                        onChange(items)
+                    }
 
                     for await (const event of watch({ filter }, { signal: ac.signal })) {
                         if (disposed) return
@@ -288,3 +519,56 @@ export function createEntitiesClient({ baseUrl, basePath, fetch: doFetch, header
         return { list, listAll, urlFor, pages, watch, live, update, delete: remove, render }
     }
 }
+
+// Unwrap a snapshot payload into a plain array of entity-like objects.
+// Recognises three shapes:
+//   - data-plugin catalog output:  [{ refId, name, date, data }, ...]
+//   - plain array of entities:     [{ id, meta, ... }, ...]
+//   - list() envelope:             { items: [...], page, ... }
+// Returns null on unrecognised shapes so the caller can decide to fall
+// back to a fresh list() call (when fallbackToList is true).
+function unwrapSnapshot(payload) {
+    if (Array.isArray(payload)) {
+        if (payload.length === 0) return []
+        // Heuristic: data-plugin entries have `refId` + `data`; treat any
+        // object with `data` as a wrapped entry and unwrap. Anything else
+        // is a plain array.
+        if (payload[0] && typeof payload[0] === 'object' && 'data' in payload[0]) {
+            return payload.map(entry => entry.data).filter(Boolean)
+        }
+        return payload
+    }
+    if (payload && typeof payload === 'object' && Array.isArray(payload.items)) {
+        return payload.items
+    }
+    return null
+}
+
+// Project an entity object to only the fields requested. Dotted paths
+// supported ("meta.title" → object with `meta.title` set, other meta
+// keys dropped). Used by the snapshot fast paths in list/live so a
+// caller asking for narrow fields still gets narrow data from the
+// snapshot — saves a re-fetch and keeps memory/bundles smaller.
+function pickFields(entity, fields) {
+    const out = {}
+    for (const field of fields) {
+        const parts = field.split('.')
+        let src = entity, dst = out
+        for (let i = 0; i < parts.length - 1; i++) {
+            const part = parts[i]
+            if (src == null || typeof src !== 'object') { src = undefined; break }
+            src = src[part]
+            if (dst[part] == null || typeof dst[part] !== 'object') dst[part] = {}
+            dst = dst[part]
+        }
+        if (src !== undefined) {
+            const last = parts[parts.length - 1]
+            if (src != null && typeof src === 'object' && last in src) {
+                dst[last] = src[last]
+            } else if (parts.length === 1 && entity[last] !== undefined) {
+                out[last] = entity[last]
+            }
+        }
+    }
+    return out
+}