mikser-io 9.70.0 → 9.72.0

package/CLAUDE.md

@@ -214,6 +214,35 @@ brevity.
   when it looks like one (`'hash' in format`); without that, `asset 'web'
   '/x.jpg'` built `/assets/web/x.[object Object]` — the bug the finalize
   check found on its first run, present since the helper took a format.
+- `matchesLibrary` (utils.js) — a resources library key is a REGEX source
+  (`escapeStringRegexp(url)`), and it has two consumers: the plugin's
+  discovery walk, which decides what to DOWNLOAD, and the `resource` render
+  helper, which builds the url. They read the same string with two matchers
+  — discovery used `matchEntity`, a GLOB demanding a full match, so a key
+  derived from `url` (a bare prefix, no trailing wildcard) matched nothing
+  and NO url-declared library was ever fetched, while the helper kept
+  building links to the missing files. Green build, missing images; found
+  when the reference check read the output back. Both now call one function
+  so they cannot drift.
+- `references.js` — the OTHER half of the broken-link answer: reads the
+  EMITTED output (html + css) and resolves every `src` / `href` / `poster`
+  / `srcset` / `url()` the way a browser does. Complements
+  `reportMissingAssets`, which reads the render track — that one sees urls
+  that never reach an html file (a feed, a sitemap) and knows the entity;
+  this one sees paths written by hand and can tell BROKEN (resolves to no
+  file) from OVER-DEEP (resolves only because a `..` run was floored at the
+  site root — loads today, breaks one nesting level deeper). Where both can
+  see a file the scan wins and the track check is skipped, so one problem
+  is one warning. Codes: `reference-broken` / `reference-over-deep` plus
+  summaries. `runtime.config.siteRoots` declares which subtrees deploy as
+  their own domain root; it CANNOT be derived — it is a fact about
+  deployment, not about the bytes — and resolving a per-language build
+  against the output root reports every working url as broken. Skips other
+  origins, `data:`, fragments, percent-encoded externals
+  (`https%3A%2F%2F...` in a query param) and unrendered template syntax.
+  Decodes `"` first: a CSS custom property in an inline style is
+  `url("../x.svg")`, and left encoded the entity text becomes the
+  path. 434 references over a real site in ~20ms.
 - Postprocess failures are RENDER ERRORS. The dispatcher's catch used to
   log one uncoded `Postprocess error:` line and stop there — exit code 0,
   nothing in `--json` `errors`, `🟢 Mikser completed`. A build missing

package/docs/configuration.md

@@ -64,6 +64,31 @@ These options are part of `runtime.options` and apply to the engine itself.
 | `server.requestTimeout` | — | number | node default (`300000`) | Milliseconds a single request may take, set on the underlying `http.Server`. Node's 5-minute default is effectively an **upload size limit expressed in seconds** — a large file over a slow link is indistinguishable from a stalled request, so it is cut off and the caller sees a truncated write rather than a readable error. Raised to two hours automatically when any route registers as `streaming` (an upload surface such as `mikser-io-drive`), because Node's default mismeasures exactly those; set this explicitly to override. `0` disables the cap: reasonable on a trusted-network build server, bad facing the internet, where it removes the only bound on how long a client can hold a connection open doing nothing. `headersTimeout` is clamped to stay at or below it. The server is also exposed as `runtime.options.httpServer`. |
 | `url` | `-u, --url <url>` | string | — | Public URL where this mikser is reachable (e.g. `https://blog.me.com`). Validated, trailing slash stripped, stamped on `runtime.options.url`. Read by webhook-capable plugins for push-vs-poll gating (`url.startsWith('https://')`); used by anything that surfaces absolute URLs externally — MCP preview URLs returned to agents, forms share links, email tracking pixels. Plugins that just need internal URLs keep using `runtime.options.port`. |
 
+### `siteRoots`
+
+Which subtrees of the output folder are deployed as their own domain root.
+
+```js
+export default {
+    // out/bg becomes lmed.bg, out/en becomes lmed.info, out/mk becomes lmed.mk
+    siteRoots: ['bg', 'en', 'mk'],
+}
+```
+
+Read only by the broken-reference check. It resolves urls the way a browser
+does, and a browser cannot climb above the origin root — it discards the
+extra `..` and loads the file. Where the site root actually is therefore
+decides whether `../../x.svg` on a given page is correct, merely over-deep,
+or broken.
+
+Default is the output folder itself, which is right for the ordinary case of
+one site per build. Declare this only when a build emits several sites, as a
+per-language deploy does — resolving those against the output root instead
+misses the over-escape entirely and reports the working urls as broken.
+
+Nothing can infer it: it is a fact about where the bytes get deployed, not
+about the bytes.
+
 ## Engine Substrate
 
 The catalog, inverse-ref graph, render snapshot manifest, and per-cycle

package/docs/diagnostics.md

@@ -873,10 +873,26 @@ surfaces that turn silence into a statement:
   `resource()` *build* a URL from a naming convention rather than looking
   an entity up, so they cannot fail: a preset that never ran, or a
   template naming an extension the preset no longer emits, yields a
-  well-formed URL to nothing. Every such call is recorded on the render
-  track and checked against the output folder at finalize; what is missing
-  is warned under `asset-missing`, naming the path and the pages that
-  linked it.
+  well-formed URL to nothing. Two checks run at finalize, from different
+  evidence:
+  - **The emitted output**, read back and resolved the way a browser
+    would — `src`, `href`, `poster`, `srcset` and CSS `url()` across
+    html and css. Anything resolving to no file warns under
+    `reference-broken`, naming the url and the pages carrying it. This
+    one sees paths written by hand, not just helper output.
+  - **The render track**, which records every `asset()` / `resource()`
+    call and tests the destination it built. This catches a url that
+    never reaches an html file at all — one emitted into a feed or a
+    sitemap — and warns under `asset-missing`. Where both can see the
+    same file, the output scan reports it and this one stays quiet.
+- **A link that works only by accident** — a url with one `..` too many
+  still loads, because a browser discards a climb above the origin root
+  rather than failing. It is one level of nesting away from a 404, and
+  it means the emitted depth does not match the page. Reported under
+  `reference-over-deep`, separately from the outright failures. Which
+  root to floor at is deployment intent and cannot be derived, so
+  declare it — see `siteRoots` in
+  [configuration](./configuration.md#siteroots).
 
 ## See also
 

package/docs/rendering.md

@@ -392,7 +392,15 @@ naming the path and the pages that linked it, with a summary under
 deployed into the output as a link resolves normally.
 
 It is a warning, not an error: a build can legitimately link a file that
-some later step supplies. What it removes is the silence.
+some later step supplies, and a missing asset must not stop a dev server.
+What it removes is the silence.
+
+A second check reads the **emitted output** rather than the render track —
+every `src` / `href` / `poster` / `srcset` / CSS `url()` in the html and
+css that shipped, resolved the way a browser resolves it. It covers paths
+written by hand, which no helper ever saw, and it is the only one that can
+see a url which loads solely because the browser floored a `..` run at the
+site root. See [diagnostics](./diagnostics.md#when-mikser-is-silent).
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mikser-io",
-  "version": "9.70.0",
+  "version": "9.72.0",
   "description": "A mixer for content: entities in, configurable render pipelines, outputs of any kind. Static sites are the canonical recipe, not the definition — the same engine renders PDFs, emails and whatever a renderer plugin produces. Files are the source of truth, every lifecycle phase is observable, and the build graph is queryable by an agent.",
   "main": "index.js",
   "exports": {

package/src/engine.js

@@ -12,6 +12,7 @@ import { globby } from 'globby'
 import { OPERATION, TASKS } from './constants.js'
 import { changeExtension, formatErrorContext, projectMeta, lookupKeys } from './utils.js'
 import { reportRendered, reportSkipped, reportError, renderErrorCount, emitReport, finishCycle, reportAssetUse, assetUse } from './report.js'
+import { checkReferences } from './references.js'
 import { toolSchemas, invokeTool, toolResultText, toolResultFailed } from './tools.js'
 import { registerBuiltinTools } from './builtin-tools.js'
 import { useDatabase } from './database/index.js'
@@ -89,6 +90,61 @@ function workerSafeOptions(opts) {
     return result
 }
 
+// Warn for anything the EMITTED output points at that is not there.
+//
+// Complements the helper-call check below rather than repeating it: this reads
+// what shipped, so it also sees paths written by hand, and it resolves them the
+// way a browser does, which is the only way to see a url that works solely
+// because a `..` run was floored at the site root.
+//
+// A floored url is not broken today. It is the same markup one level deeper
+// away from being broken, and it means the emitted depth does not match the
+// page — so it is reported separately rather than folded in with the failures.
+//
+// Warn, never fail: a missing asset must not stop a dev server. Both lists
+// carry stable codes into `--json` so a deploy script can decide for itself.
+// Returns the set of broken targets so the helper-call check can skip them.
+async function reportBrokenReferences(logger) {
+    const outputFolder = runtime.options.outputFolder
+    if (!outputFolder || !existsSync(outputFolder)) return new Set()
+
+    const siteRoots = runtime.config?.siteRoots ?? []
+    const { broken, overDeep, checked } = await checkReferences(outputFolder, { siteRoots })
+    if (!checked) return new Set()
+
+    const SHOWN = 10
+    const named = (files) =>
+        files.slice(0, 3).join(', ') + (files.length > 3 ? ` and ${files.length - 3} more` : '')
+
+    for (const { url, target, files } of broken.slice(0, SHOWN)) {
+        logger.warn({ code: 'reference-broken', url, target, files },
+            'Resolves to nothing: %s (from %s) — %s', url, named(files), target)
+    }
+    if (broken.length) {
+        logger.warn({ code: 'reference-broken-summary', broken: broken.length, checked },
+            '%d of %d reference(s) in the output resolve to nothing%s. A URL helper builds the '
+            + 'path rather than looking it up, so these are links to files nothing produced.',
+            broken.length, checked, broken.length > SHOWN ? `, ${SHOWN} shown` : '')
+    }
+
+    for (const { url, target, files } of overDeep.slice(0, SHOWN)) {
+        logger.warn({ code: 'reference-over-deep', url, target, files },
+            'Over-deep, loads only because the browser floors it: %s (from %s) — %s',
+            url, named(files), target)
+    }
+    if (overDeep.length) {
+        logger.warn({ code: 'reference-over-deep-summary', overDeep: overDeep.length, checked },
+            '%d of %d reference(s) climb above the site root and load only because a browser '
+            + 'discards the extra `..`%s. They break as soon as the same markup renders one '
+            + 'level deeper.%s',
+            overDeep.length, checked, overDeep.length > SHOWN ? `, ${SHOWN} shown` : '',
+            siteRoots.length ? '' : ' No siteRoots are declared, so this resolved against the '
+                + 'output root — declare runtime.config.siteRoots if a subtree is deployed as its own domain.')
+    }
+
+    return new Set(broken.map(b => b.target))
+}
+
 // Warn for anything a render linked to that is not in the output.
 //
 // Deliberately phrased as what was OBSERVED. Only entities that rendered this
@@ -96,7 +152,7 @@ function workerSafeOptions(opts) {
 // and says nothing about the rest — the same reasoning the assets plugin
 // already applies to its preset warning, and for the same reason: a warning
 // that overclaims gets filtered, and the filtered-out line is the real one.
-async function reportMissingAssets(logger) {
+async function reportMissingAssets(logger, alreadyReported = new Set()) {
     const used = assetUse()
     if (!used.length) return
     const outputFolder = runtime.options.outputFolder
@@ -105,6 +161,10 @@ async function reportMissingAssets(logger) {
     const missing = []
     for (const [destination, ids] of used) {
         const file = path.join(outputFolder, destination.replace(/^\//, ''))
+        // The output scan resolves the same file the way a browser does and
+        // names the pages that link it, which is strictly more useful. Where
+        // both would fire, one warning is enough.
+        if (alreadyReported.has(destination.replace(/^\//, ''))) continue
         if (!existsSync(file)) missing.push([destination, ids])
     }
     if (!missing.length) return
@@ -1138,7 +1198,8 @@ export async function setup(options) {
         // Checked at the end of the cycle because that is the first moment the
         // answer is stable: derivatives are produced during the cycle, so
         // asking any earlier would report files that were about to appear.
-        await reportMissingAssets(useLogger())
+        const brokenTargets = await reportBrokenReferences(useLogger())
+        await reportMissingAssets(useLogger(), brokenTargets)
 
         // After the cycle, and only under --json. stdout has been kept clear
         // for exactly this (the logger writes to stderr under --json), so the

package/src/plugins/render/resource.js

@@ -1,10 +1,14 @@
 import path from 'node:path'
+import { matchesLibrary } from '../../utils.js'
 
 export function load({ runtime, entity, state, options, track }) {
     runtime.resource = (url) => {
         const { resourceLib } = state.resources
         for (let library in resourceLib) {
-            if (url.match(library)) {
+            // Same matcher the resources plugin uses to decide what to
+            // DOWNLOAD. When these disagreed, this built urls for files the
+            // plugin never fetched.
+            if (matchesLibrary(url, library)) {
                 const { origin } = new URL(url)
                 const name = url.replace(origin, `${resourceLib[library]}`)
                 const relative = url.replace(origin, `${state.resources.resourcesFolder}/${resourceLib[library]}`)

package/src/plugins/resources.js

@@ -10,6 +10,7 @@ import * as stream from 'stream'
 import { promisify } from 'util'
 import isUrl from 'is-url'
 import map from 'p-map'
+import { matchesLibrary } from '../utils.js'
 
 export function resources(options = {}) {
     return ({
@@ -24,7 +25,6 @@ export function resources(options = {}) {
         checksum,
         trackProgress,
         updateProgress,
-        matchEntity,
         constants: { OPERATION },
     }) => {
     const collection = 'resources'
@@ -52,6 +52,11 @@ export function resources(options = {}) {
 
         for (let library in (options.libraries || [])) {
             let resource = options.libraries[library]
+                // The key is a REGULAR EXPRESSION source, which is what the
+            // escapeStringRegexp call says: you only escape a string you are
+            // about to compile. The render helper has always read it that way
+            // (`url.match(library)`), so a library declared by `url` is a
+            // prefix pattern matching anything under it.
             runtime.state.resources.resourceLib[resource.match || escapeStringRegexp(resource.url)] = library
         }
     })
@@ -66,7 +71,15 @@ export function resources(options = {}) {
                 _.eachDeep(entity.meta, resource => {
                     if (typeof resource == 'string') {
                         for (let library in resourceLib) {
-                            if (matchEntity(resource, library)) {
+                            // Regex, matching the render helper. This used
+                            // matchEntity, which is a GLOB demanding a full
+                            // match — so a key derived from `url` (a bare
+                            // prefix, no trailing wildcard) matched nothing and
+                            // NO url-declared library was ever downloaded. The
+                            // helper still built urls for them, so pages linked
+                            // files nothing fetched and the build stayed green:
+                            // one string read with two incompatible matchers.
+                            if (matchesLibrary(resource, library)) {
                                 resourceMap[entity.id].push({ library, resource, entity })
                             }
                         }

package/src/references.js

@@ -0,0 +1,187 @@
+// What the build actually shipped, checked against what it actually wrote.
+//
+// The URL helpers BUILD paths from a naming convention rather than resolving
+// an entity, so they cannot fail — `asset` composes
+// `<assetsFolder>/<preset>/<path>` with whatever extension it was handed and
+// never asks whether that file exists. A wrong preset name, a wrong extension,
+// or a source whose derivative silently failed to render all produce a
+// well-formed url pointing at nothing, and every existing surface stays green:
+// nothing threw, --verify compares snapshots against what was rendered rather
+// than against what those renders point at, and mikser_refs_broken tracks
+// document-to-document refs, not urls.
+//
+// This reads the emitted bytes instead. Everything it needs is on disk at the
+// end of a cycle and nothing has to be inferred.
+//
+// It is deliberately NOT the same check as `asset-missing` in engine.js. That
+// one records helper CALLS on the render track and tests the output-root
+// absolute destination each one built; it knows the referencing entity, and it
+// sees urls that never reach an html file at all (a sitemap, a feed). This one
+// sees everything that shipped, including paths written by hand, and resolves
+// them the way a browser would — which is the only way to catch a url that
+// resolves solely because the browser floored a `..` run at the site root.
+
+import path from 'node:path'
+import { existsSync } from 'node:fs'
+import { readFile } from 'node:fs/promises'
+import { globby } from 'globby'
+
+// Documents that can carry a reference. Anything else in the output is either
+// an asset itself or something whose internal structure this has no business
+// guessing at.
+const SCANNED = ['**/*.html', '**/*.htm', '**/*.css']
+
+// Attributes whose value is a single url.
+const ATTR = /(?:src|href|poster|data-bg)\s*=\s*["']([^"']*)["']/gi
+// srcset / imagesrcset: a comma-separated list of `url [descriptor]`.
+const SRCSET = /(?:img|image)?srcset\s*=\s*["']([^"']*)["']/gi
+// css url(), in a stylesheet and in an inline style attribute alike.
+const CSS_URL = /url\(\s*(['"]?)([^'")]*)\1\s*\)/gi
+
+// A url this check has nothing to say about: another origin, an inline
+// payload, a fragment or an in-page action. `//host/path` is protocol-relative
+// and therefore external too.
+function isExternal(url) {
+    if (!url) return true
+    const u = url.trim()
+    if (!u) return true
+    if (u.startsWith('#') || u.startsWith('//')) return true
+    // Template syntax that reached the output unrendered — `{{link}}`,
+    // `${x}`, `<%= y %>`. It is not a path, so "resolves to nothing" says
+    // nothing useful about it; the real problem is that it did not render,
+    // which is a different question than this one is asking. Documentation
+    // pages showing escaped template syntax are the common source, and they
+    // are not broken at all.
+    if (/\{\{|\}\}|\$\{|<%/.test(u)) return true
+    // A scheme — http:, data:, mailto:, tel:, javascript:.
+    if (/^[a-z][a-z0-9+.-]*:/i.test(u)) return true
+    // The same thing percent-encoded, which is how an external url arrives
+    // when it was built as a query parameter — `https%3A%2F%2F...` in a maps
+    // link. It has no scheme until it is decoded, so the test above misses it
+    // and the whole encoded string gets resolved as a path segment.
+    try {
+        if (/^[a-z][a-z0-9+.-]*:/i.test(decodeURIComponent(u))) return true
+    } catch { /* malformed escape — treat as a path and let it resolve */ }
+    return false
+}
+
+// Quotes inside an attribute value arrive encoded, and a CSS custom property
+// in an inline style is the common way that happens:
+//
+//     style="--icon-src:url(&quot;../media/raw/icons/x.svg&quot;)"
+//
+// Without decoding, the captured url is the entity text itself, which resolves
+// nowhere and reports as broken — a false positive that would have buried the
+// real ones. Only the quote and ampersand forms are decoded; turning &lt; back
+// into a bracket could invent markup that was deliberately escaped.
+function decodeEntities(source) {
+    return source
+        .replace(/&quot;|&#34;/g, '"')
+        .replace(/&apos;|&#39;/g, "'")
+        .replace(/&amp;/g, '&')
+}
+
+// Everything a page points at, as raw url strings.
+export function extractReferences(rawSource) {
+    const source = decodeEntities(rawSource)
+    const found = new Set()
+    for (const [, url] of source.matchAll(ATTR)) found.add(url)
+    for (const [, , url] of source.matchAll(CSS_URL)) found.add(url)
+    for (const [, list] of source.matchAll(SRCSET)) {
+        for (const candidate of list.split(',')) {
+            const url = candidate.trim().split(/\s+/)[0]
+            if (url) found.add(url)
+        }
+    }
+    return [...found].filter(u => !isExternal(u))
+}
+
+// Resolve the way a browser does, which is the whole point.
+//
+// A browser walks the page's directory segments, pops one per `..`, and
+// DISCARDS a `..` that would climb above the origin root — it does not error
+// and it does not escape. So a url with one `..` too many still loads, and the
+// page looks correct while carrying a path that breaks the moment the same
+// markup is used one level deeper. That flooring is what `overDeep` records.
+//
+// `pageDir` and the result are both relative to `root`.
+export function resolveUrl(pageDir, url, { root = '' } = {}) {
+    const clean = url.split('#')[0].split('?')[0]
+    const absolute = clean.startsWith('/')
+    const segments = clean.split('/').filter(s => s !== '' && s !== '.')
+
+    const parts = absolute ? [] : pageDir.split('/').filter(Boolean)
+    let overDeep = false
+    for (const segment of segments) {
+        if (segment !== '..') { parts.push(segment); continue }
+        if (parts.length) parts.pop()
+        else overDeep = true      // a climb above the root, floored
+    }
+    return { target: path.join(root, ...parts), overDeep }
+}
+
+// Which declared site root a file belongs to.
+//
+// lmed emits one subtree per language and deploys each as its own domain root
+// (out/bg becomes lmed.bg), so the site root is out/<lang>/ and every url
+// carries one extra `..` for the language segment that the browser then floors.
+// Resolving against the output root instead would miss the over-escape entirely
+// and report working urls as broken. Nothing can derive this — it is deployment
+// intent — so it is declared, and the default is the output root itself.
+export function siteRootFor(relativeFile, roots) {
+    let best = ''
+    for (const root of roots) {
+        if (!root) continue
+        if (relativeFile.startsWith(`${root}/`) && root.length > best.length) best = root
+    }
+    return best
+}
+
+// Everything the output points at that is not there.
+//
+// Returns { broken, overDeep, checked }, where each entry is
+// { url, target, files } — the target with the pages that named it, because
+// "this is missing" is only actionable next to "and these link it".
+export async function checkReferences(outputFolder, { siteRoots = [] } = {}) {
+    const files = await globby(SCANNED, {
+        cwd: outputFolder,
+        followSymbolicLinks: false,
+        suppressErrors: true,
+    })
+
+    const broken = new Map()
+    const overDeep = new Map()
+    let checked = 0
+    // Existence is the expensive part and the same target repeats across a
+    // site — one lookup each.
+    const exists = new Map()
+
+    for (const file of files) {
+        let source
+        try { source = await readFile(path.join(outputFolder, file), 'utf8') }
+        catch { continue }
+
+        const root = siteRootFor(file, siteRoots)
+        // The page's directory, relative to its own site root.
+        const pageDir = path.dirname(file).slice(root.length).replace(/^\/+/, '')
+
+        for (const url of extractReferences(source)) {
+            const { target, overDeep: floored } = resolveUrl(pageDir, url, { root })
+            checked++
+
+            if (!exists.has(target)) {
+                exists.set(target, existsSync(path.join(outputFolder, target)))
+            }
+            // Broken outranks over-deep: a url that resolves nowhere is the
+            // failure, and adding that it is also one level too deep is noise.
+            const bucket = !exists.get(target) ? broken : (floored ? overDeep : null)
+            if (!bucket) continue
+
+            const key = `${target} ${url}`
+            if (!bucket.has(key)) bucket.set(key, { url, target, files: [] })
+            bucket.get(key).files.push(file)
+        }
+    }
+
+    return { broken: [...broken.values()], overDeep: [...overDeep.values()], checked }
+}

package/src/utils.js

@@ -1316,3 +1316,30 @@ export function junkFilter() {
         return registered.match.some(test => test(name))
     }
 }
+
+// Does a value fall under a resources library?
+//
+// The library key is a REGULAR EXPRESSION source — `resources()` derives it
+// with escapeStringRegexp(url), and you only escape a string you are about to
+// compile. It has two consumers: the plugin's discovery walk, which decides
+// what to download, and the `resource` render helper, which builds the url.
+// They read the same string with two different matchers — discovery used a
+// GLOB, which demands a full match, so a key derived from `url` (a bare prefix
+// with no trailing wildcard) matched nothing. Nothing was ever downloaded for
+// a url-declared library, while the helper happily built links to the files
+// that were not fetched. Green build, missing images.
+//
+// One function, so the two cannot drift again.
+const libraryPatterns = new Map()
+export function matchesLibrary(value, pattern) {
+    if (typeof value !== 'string' || !pattern) return false
+    if (!libraryPatterns.has(pattern)) {
+        let re
+        try { re = new RegExp(pattern) }
+        // A hand-written `match` that is not valid regex would otherwise throw
+        // mid-walk and take the build down.
+        catch { re = { test: () => false } }
+        libraryPatterns.set(pattern, re)
+    }
+    return libraryPatterns.get(pattern).test(value)
+}