mikser-io 9.84.0 → 9.88.0

package/docs/diagnostics.md

@@ -767,7 +767,7 @@ answer is about your request. It is also faster, because a forwarded command
 never imports the config or the plugin graph, which is most of what a one-shot
 spends its time on.
 
-Three things it refuses or reports rather than guessing:
+Four things it refuses or reports rather than guessing:
 
 - **A different config.** If you resolve `mikser.config.prod.js` and the
   instance is running `mikser.config.js`, it refuses. Building with the wrong
@@ -779,6 +779,14 @@ Three things it refuses or reports rather than guessing:
   to *become* the instance, which is not something a running one can do for
   you — it would have to open a port in your process. They exit 1 and say so,
   rather than building and leaving nothing on the port.
+- **`--clear`.** Clearing removes the output folder and closes, unlinks and
+  reopens the cache database — a boot operation, which is why the wipe lives
+  where the database is opened and nowhere else. A running instance holds that
+  handle, has its manifest and plugin state loaded from what would be deleted,
+  and may be serving out of the folder being removed. It exits 1 and tells you
+  to stop the instance. It used to be accepted and ignored, so the build
+  reported success with nothing cleared; a half-clear would have been worse
+  than either, since afterwards nobody could say what state the folder was in.
 There is no opt-out. A flag for running a second engine on a held folder
 only ever enabled the accident this surface prevents, and stopping the
 instance serves every case it was reached for.

package/docs/plugins.md

@@ -198,15 +198,40 @@ files({
 ```
 
 **Entity properties set:**
-- `id`: `/{folder}/{relativePath}`
+- `id`: `/files/{relativePath}` — never carries the `outputFolder` prefix
 - `collection`: `'files'`
 - `type`: `'file'`
 - `format`: File extension
-- `destination`: Target path in output folder
+- `uri`: the **source** file, the same meaning it has in every other
+  collection. It named the published symlink until 9.86.0, which made `uri`
+  mean the source for a document and the destination for a file — see below
+- `source`: the same path; kept because presets read it
+- `name` / `meta.url`: where the file is **published**, carrying the
+  `outputFolder` prefix when one is configured
 - `checksum`: MD5 checksum
+- `link`: the resolved target, when the source is itself a symlink
 
 **Watch support:** Yes.
 
+**Deleted files are reconciled by the scan**, not only by the watcher. A file
+removed while nothing was watching used to leave a dangling symlink in the
+deployed output, its catalog row, and anything derived from it — on every
+later build. Neither a plain rebuild nor `--force` removed them; only
+`--clear` did.
+
+The scan now compares what it found against what the catalog holds and removes
+the difference, scoped by `uri` to the files folder so entities another plugin
+emitted into this collection (a CSV, a drive, an API) are left alone. It says
+`Files removed: N no longer on disk` when it acts and nothing when it does not.
+
+`--force` reconciles deletions too, as of 9.88.0. It used to skip this and the
+equivalent sweep for documents and layouts, which made it the one flag that
+could not fix what it is reached for — nothing about `--force` wipes the
+catalog (that is `--clear`), so a row for a file that no longer exists simply
+survived. Only `--clear` cleaned up.
+
+The matching cleanup for derivatives lives in `assets` — see below.
+
 ---
 
 ### `layouts` *(sibling: `mikser-io-layouts`)*
@@ -419,6 +444,26 @@ The assets plugin processes whatever's on disk. Source files don't have to start
 
 Yes — the plugin watches both the source files and the preset modules. Editing a preset re-processes every input that matches it; editing a source re-processes just that input.
 
+
+**Derivatives with no source are removed.** A derivative outlived its source:
+deleting a file removed its catalog row and its published copy and left the
+derived file, because the delete handler dropped the in-memory mapping and
+nothing on disk. Narrowing a preset's `match` left one the same way. A stale
+derivative passes every check — the url resolves and the bytes are there —
+so the only cleanup was `--clear`, or deleting it by hand.
+
+The post-cycle pass that already walks the revision markers now also asks, per
+derivative, whether it still has a source that this preset still covers. It is
+answered from the **catalog**, not from the delete event: a delete entry is
+sparse (`{ id, type, collection }`) in every source plugin, so it cannot say
+where the derivative went, while the catalog answers whatever route the orphan
+arrived by — including ones that predate the fix. Reported as
+`Assets removed: N derivative(s) with no source`.
+
+It does nothing when the catalog is empty. Every check concludes "no source,
+therefore orphan", and an empty catalog answers that for every derivative on
+the site, so a failed import would otherwise delete the whole assets folder.
+
 ---
 
 ### `resources`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mikser-io",
-  "version": "9.84.0",
+  "version": "9.88.0",
   "files": [
     "app.js",
     "index.js",

package/src/instance.js

@@ -330,6 +330,35 @@ function refuseUnknownFlags(socket, request) {
     return true
 }
 
+// `--clear` against a running instance.
+//
+// Clearing is a BOOT operation. It removes the output folder, and it closes,
+// unlinks and reopens the cache database — which is why the wipe lives at the
+// point the database is opened and nowhere else. An instance has that handle
+// open with prepared statements held against it all over the engine, its
+// manifest and plugin state loaded from what is about to be deleted, and a
+// server answering requests out of the folder being removed. There is no
+// moment mid-run where this can be done and still mean what the flag means.
+//
+// It travelled on the request and was never read, so `mikser --clear` against
+// a watcher rebuilt normally and exited 0 having cleared nothing: the flag
+// looked honoured. Refusing says the only thing that helps, which is that the
+// instance has to stop first.
+//
+// A half-clear would be worse than either: it would report success for an
+// operation the caller can no longer describe.
+function refuseClear(socket, request) {
+    if (!request.clear) return false
+    frame(socket, {
+        type: 'refused',
+        reason: 'a mikser instance is running in this folder, and --clear cannot run while it does.',
+        detail: 'Clearing removes the output folder and reopens the cache database, both of which happen '
+            + 'once at startup — the running instance holds that database open and serves out of that '
+            + 'folder. Stop it and run this again.',
+    })
+    return true
+}
+
 function refuseStale(socket, movedFile) {
     frame(socket, {
         type: 'refused',
@@ -451,6 +480,7 @@ export function serveInstance() {
                     // original incident, and it is wrong whether or not it
                     // writes anything.
                     if (refuseUnknownFlags(socket, request)) return
+                    if (refuseClear(socket, request)) return
                     const wrongConfig = configMismatch(request.config)
                     if (wrongConfig) return refuseConfig(socket, request, wrongConfig)
                     const movedFile = await configStale()

package/src/plugins/assets.js

@@ -141,8 +141,32 @@ export function assets(options = {}) {
     // entity, and it is the hottest call in a render. Nothing is looked up
     // when the url is built. This runs at most a handful of times, after
     // everything has settled, and only when something is already wrong.
+    // A derivative wears the PRESET's extension, so a source is found by stem:
+    // `media/hero.webp` came from `media/hero.jpg`.
+    const stemOf = (value) => value.slice(0, value.length - path.extname(value).length)
+
+    // stem(entity.name) -> entities, built once per cycle.
+    //
+    // Two callers ask the same question from opposite directions — "why is
+    // this derivative missing" and "does this derivative still have a source"
+    // — so they share one index rather than each walking the catalog.
     let sourceIndex = null
     let sourceIndexCycle
+    async function sourceByStem() {
+        const cycle = runtime.state?.cycle?.id ?? null
+        if (sourceIndexCycle !== cycle) { sourceIndex = null; sourceIndexCycle = cycle }
+        if (!sourceIndex) {
+            sourceIndex = new Map()
+            for await (const candidate of iterateEntities({ collection: { $ne: collection } })) {
+                if (typeof candidate.name !== 'string') continue
+                const key = stemOf(candidate.name)
+                if (!sourceIndex.has(key)) sourceIndex.set(key, [])
+                sourceIndex.get(key).push(candidate)
+            }
+        }
+        return sourceIndex
+    }
+
     async function explainMissing(destination) {
         const assetsName = runtime.options.assets
         const parts = String(destination).replace(/^\/+/, '').split('/')
@@ -155,27 +179,12 @@ export function assets(options = {}) {
             return `no preset named '${preset}' is configured (configured: ${configured.join(', ') || 'none'})`
         }
 
-        // The derivative wears the PRESET's extension, so the source is found
-        // by stem — `media/hero.webp` came from `media/hero.jpg`.
-        const stem = (value) => value.slice(0, value.length - path.extname(value).length)
-
-        // Built once per cycle and only on this path: a build with nothing
-        // broken never walks the catalog for it.
-        const cycle = runtime.state?.cycle?.id ?? null
-        if (sourceIndexCycle !== cycle) { sourceIndex = null; sourceIndexCycle = cycle }
-        if (!sourceIndex) {
-            sourceIndex = new Map()
-            for await (const candidate of iterateEntities({ collection: { $ne: collection } })) {
-                if (typeof candidate.name !== 'string') continue
-                const key = stem(candidate.name)
-                if (!sourceIndex.has(key)) sourceIndex.set(key, [])
-                sourceIndex.get(key).push(candidate)
-            }
-        }
-
-        const source = (sourceIndex.get(stem(name)) ?? [])[0]
+        // Built only on this path: a build with nothing broken never walks
+        // the catalog for it.
+        const index = await sourceByStem()
+        const source = (index.get(stemOf(name)) ?? [])[0]
         if (!source) {
-            return `no source file is named '${stem(name)}' — nothing would produce this derivative `
+            return `no source file is named '${stemOf(name)}' — nothing would produce this derivative `
                 + 'under any preset'
         }
 
@@ -726,25 +735,79 @@ export function assets(options = {}) {
         } catch { /* a count is not worth failing a build over */ }
 
         let revisions = await globby('**/*.md5', { cwd: runtime.options.assetsFolder })
+
+        // Is the catalog answerable at all?
+        //
+        // Every check below concludes "no source, therefore orphan", and an
+        // EMPTY catalog answers that for every derivative on the site. A
+        // failed import or a scan that has not run yet would then delete the
+        // whole assets folder — a rebuild of every derivative at best, and at
+        // worst it happens on the machine that serves them. So the sweep only
+        // runs when there is something to be absent from.
+        const catalogSize = countEntities({ collection: { $ne: collection } })
+        const orphaned = []
+
         for (let revision of revisions) {
             const [preset] = revision.split(path.sep)
             const [assetsRevision] = revision.split('.').slice(-2, -1)
 
             if (!presets[preset]) {
                 const assetsPresetFolder = path.join(runtime.options.assetsFolder, preset)
-                const assetsPresetRemoved = false
+                // `let`. This was `const` with an assignment inside the try,
+                // so removing a preset folder threw TypeError into the empty
+                // catch on every pass and the log line was unreachable — the
+                // rm had already happened, so nothing looked wrong.
+                let assetsPresetRemoved = false
                 try {
                     await rm(assetsPresetFolder, { recursive: true, force: true })
                     assetsPresetRemoved = true
-                } catch { }
+                } catch { /* already gone, or not ours to remove */ }
                 if (assetsPresetRemoved) {
                     logger.debug('Assets preset removed: %s', assetsPresetFolder)
                 }
-            } else {
-                if (Number.parseInt(assetsRevision) < presets[preset].checksum) {
-                    await unlink(path.join(runtime.options.assetsFolder, revision))
-                }
+                continue
+            }
+
+            if (Number.parseInt(assetsRevision) < presets[preset].checksum) {
+                await unlink(path.join(runtime.options.assetsFolder, revision))
             }
+
+            // A derivative whose source is gone, or which this preset no
+            // longer covers.
+            //
+            // Deleting the source removed its catalog row and its published
+            // file, and left the derivative — the delete handler dropped the
+            // in-memory mapping and nothing on disk, so the only cleanup was
+            // --clear or doing it by hand. Narrowing a preset's `match` left
+            // one the same way, with nothing said.
+            //
+            // Answered from the catalog rather than from the delete event: a
+            // delete entry is sparse ({ id, type, collection }) by convention
+            // in every source plugin, so it cannot say where the derivative
+            // went. What still HAS a source is a question the catalog answers
+            // whatever route the orphan arrived by.
+            if (!catalogSize) continue
+            const derivative = revision.replace(/\.[^.]+\.md5$/, '')
+            const name = derivative.split(path.sep).slice(1).join('/')
+            const index = await sourceByStem()
+            const source = (index.get(stemOf(name)) ?? [])[0]
+            if (source && presetsSelecting(source).includes(preset)) continue
+            orphaned.push({
+                derivative,
+                marker: revision,
+                reason: source ? `preset '${preset}' no longer covers ${source.id}` : 'its source is gone',
+            })
+        }
+
+        for (const { derivative, marker } of orphaned) {
+            await unlink(path.join(runtime.options.assetsFolder, derivative)).catch(() => { /* already gone */ })
+            await unlink(path.join(runtime.options.assetsFolder, marker)).catch(() => { /* already gone */ })
+        }
+        if (orphaned.length) {
+            logger.info('Assets removed: %d derivative(s) with no source — %s',
+                orphaned.length,
+                orphaned.slice(0, 3).map(o => `${o.derivative} (${o.reason})`).join(', ')
+                + (orphaned.length > 3 ? ` and ${orphaned.length - 3} more` : ''))
         }
     })
 

package/src/plugins/files.js

@@ -3,6 +3,7 @@ import { mkdir, symlink, unlink, lstat, realpath } from 'fs/promises'
 import { globby } from 'globby'
 import pMap from 'p-map'
 import { checksumsByCollection } from '../catalog.js'
+import { sweepDeleted } from '../source.js'
 
 export function files(options = {}) {
     return ({
@@ -38,10 +39,18 @@ export function files(options = {}) {
             return { uri, source }
         }
 
+        // The goal state is "the link is not there", so a link that is
+        // already gone is success, not an error. It genuinely happens: someone
+        // deletes a stale one by hand, and before this the next reconciliation
+        // threw on it.
         async function removeLink(relativePath) {
             let uri = path.join(runtime.options.outputFolder, relativePath)
             if (options.outputFolder) uri = path.join(runtime.options.outputFolder, options.outputFolder, relativePath)
-            await unlink(path.resolve(uri))
+            try {
+                await unlink(path.resolve(uri))
+            } catch (err) {
+                if (err.code !== 'ENOENT') throw err
+            }
         }
 
         async function link(source) {
@@ -58,12 +67,24 @@ export function files(options = {}) {
             const source = path.join(runtime.options.filesFolder, relativePath)
             const format = path.extname(relativePath).substring(1).toLowerCase()
             const id = path.join(`/${collection}`, relativePath)
-            let uri = path.join(runtime.options.outputFolder, relativePath)
             let name = relativePath
             if (options.outputFolder) {
-                uri = path.join(runtime.options.outputFolder, options.outputFolder, relativePath)
                 name = path.join(options.outputFolder, relativePath)
             }
+            // The SOURCE file, as in every other collection.
+            //
+            // It used to be the symlink in the output, which made `uri` mean
+            // one thing for documents and layouts and the opposite here — and
+            // three separate pieces of code carry scars from it: the render
+            // file helper keys its edges on `id` with a comment explaining
+            // that a uri edge "matches nothing" for files, locateEntityFile
+            // rejected every file entity as living outside its own collection
+            // folder, and sweepDeleted could not be called at all because it
+            // scopes by uri rooted at the folder a source owns.
+            //
+            // Where the bytes are PUBLISHED is meta.url, which every consumer
+            // already reads (ADR-0011).
+            const uri = source
 
             let synced = true
             switch (action) {
@@ -145,6 +166,7 @@ export function files(options = {}) {
         })
 
         onImport(async () => {
+            const logger = useLogger()
             await mkdir(runtime.options.outputFolder, { recursive: true })
             if (options.outputFolder) await mkdir(path.join(runtime.options.outputFolder, options.outputFolder), { recursive: true })
 
@@ -167,13 +189,19 @@ export function files(options = {}) {
             // repair path short of deleting them.
             const forced = runtime.options.force || runtime.catalog?.cacheInvalidated
             const priorChecksums = checksumsByCollection(collection)
+            const scanned = new Set()
             await pMap(paths, async relativePath => {
-                const { uri, source } = await ensureLink(relativePath)
+                const { source } = await ensureLink(relativePath)
+                const uri = source
                 let name = relativePath
                 if (options.outputFolder) {
                     name = path.join(options.outputFolder, relativePath)
                 }
                 const id = path.join(`/${collection}`, relativePath)
+                // Recorded before the checksum gate. The gate decides whether
+                // to re-emit; the sweep asks what still EXISTS, and a file
+                // skipped for being unchanged very much exists.
+                scanned.add(id)
                 const newChecksum = await checksum(source)
                 updateProgress()
                 // Gate: if the catalog already has this entity with the same
@@ -196,6 +224,31 @@ export function files(options = {}) {
                     link: await link(source),
                 })
             }, { concurrency: 16 })
+
+            // What the catalog holds that the disk no longer does.
+            //
+            // The scan enumerates what exists and never asked the opposite
+            // question, so deleting a file with nothing watching left three
+            // things behind on every subsequent build: a DANGLING SYMLINK in
+            // the deployed output, the catalog row, and anything derived from
+            // it. Neither a plain rebuild nor --force removed them; only
+            // --clear did, or a watcher that happened to be running at the
+            // moment the file disappeared.
+            //
+            // ownerPrefix is the files folder, which is only a meaningful
+            // scope now that `uri` names the source. It keeps the sweep off
+            // entities another plugin emitted into this collection — a CSV, a
+            // drive, an API — which is the accident the parameter exists to
+            // prevent.
+            //
+            // Skipped under --force, like every other sweep: that guard lives
+            // in sweepDeleted and is shared with documents and layouts.
+            const deleted = await sweepDeleted(collection, scanned, async (entity) => {
+                await removeLink(path.relative(runtime.options.filesFolder, entity.uri))
+                await deleteEntity(entity)
+                logger.debug('files removed (file gone): %s', entity.name)
+            }, runtime.options.filesFolder)
+            if (deleted) logger.info('Files removed: %d no longer on disk', deleted)
         })
 
         return {

package/src/source.js

@@ -130,7 +130,20 @@ export async function gateChecksum(file, id, { reload = false, priorChecksums, b
 // LRU population the scan already does, and we get rid of the 7GB JS
 // heap peak in return.
 export async function sweepDeleted(collection, scanned, onDelete, ownerPrefix) {
-    if (runtime.options.force) return 0
+    // --force does NOT skip this any more.
+    //
+    // It used to, on a stated premise — "operator wants a full rebuild;
+    // deletes still flow naturally on the rebuild" — that is not true and can
+    // be disproved in one command: delete a document, run `mikser --force`,
+    // and both the entity and its output are still there, while an ordinary
+    // build removes them. Nothing about --force wipes the catalog (that is
+    // --clear, via forceWipe), so a row for a file that no longer exists
+    // simply survives.
+    //
+    // Which made --force the one flag that could not fix what it is reached
+    // for. It defeats the checksum GATE; the scanned set is built from the
+    // glob before any gate runs, so it is exactly as complete under --force as
+    // without it, and the sweep is no less accurate for running.
     if (!ownerPrefix) {
         throw new Error(
             'sweepDeleted: ownerPrefix is required — pass the absolute folder ' +