mikser-io 11.3.1 → 11.4.0

package/README.md

@@ -245,7 +245,7 @@ npm install mikser-io
 ```bash
 npx mikser              # one-shot build
 npx mikser --watch      # incremental dev loop
-npx mikser --server     # build + serve at :3001
+npx mikser --server     # build + serve on a free port (or --server 3001 to name one)
 ```
 
 For a working starter — config with a real plugin set, sample `documents/`, expected output — see [Getting Started](./docs/getting-started.md). Or skip straight to "add mikser to this app" via the [Claude Code plugin](#built-for-ai-assisted-development) above.

package/docs/configuration.md

@@ -58,7 +58,7 @@ These options are part of `runtime.options` and apply to the engine itself.
 | `logInstall` | `--log-install` | string | — | Set the level on a **running** instance, so its own rebuilds are verbose too — the case a per-request flag cannot serve. Expires after 30 minutes, dies with the process, and is disclosed in the build report under `logLevel`. |
 | `logReset` | `--log-reset` | boolean | `false` | Return a running instance to its configured level. |
 | `threads` | — | number | `4` | Worker thread count for the Piscina pools (`renderWorkers`, `postprocessWorkers`). Both pools are lazy (`minThreads: 0` + `idleTimeout: 30_000`) so INLINE-only workloads spin up zero workers. |
-| `server` | `-s, --server [port]` | number\|boolean | — | When set, the engine creates a shared Express app on `runtime.options.app` and listens on the given port (default `3001`) after all plugins have mounted their routes. Plugins like `api` attach to it instead of starting their own server. The `outputFolder` is also served as a static catch-all route at `/` (plugin routes match first; anything that doesn't match falls through to the rendered output). Requires `express` to be installed. |
+| `server` | `-s, --server [port]` | number\|boolean | — | When set, the engine creates a shared Express app on `runtime.options.app` and listens on the given port after all plugins have mounted their routes. **Command line only** — the port is read at `initialized`, before the config file is loaded, and `config.server` is the settings object for `cors` / `trustProxy` / `requestTimeout`, not a port. Omit the port (`--server`), or pass `0`, and the OS is asked for one nothing is using — the log line says which. That is the default because a fixed one is a number every account on a shared dev box gets, so the second mikser dies on a collision belonging to neither project. Name a port when something else has to find it (a reverse proxy, a bookmark): `--server 3002` binds 3002 or fails saying who holds it. Plugins like `api` attach to it instead of starting their own server. The `outputFolder` is also served as a static catch-all route at `/` (plugin routes match first; anything that doesn't match falls through to the rendered output). Requires `express` to be installed. |
 | `junk` | — | array\|false | built-in list | OS and file-manager litter, filtered out of both the scan and the watcher. The dot-prefixed files (`.DS_Store`, `._*`) were already invisible — globby defaults to `dot: false` and the watcher ignores leading dots — but the Windows ones are **not** dotfiles: `Thumbs.db` and `desktop.ini` were measurably scanned *and* watched, and became entities. The list is deliberately conservative (OS/file-manager artifacts and application lock files only, no `*.tmp`, `*.bak` or editor backups), because a filter that silently drops content is worse than the litter it prevents. `false` disables it; an array replaces it. See `isJunkPath` / `JUNK_IGNORE` in `src/utils.js`. Plugins that write metadata next to content add their own patterns with `registerJunk({ ignore, match })` — the engine provides the mechanism and the plugin the knowledge of what its files are called (`mikser-io-drive` registers `*.nephelemeta`). Plugin registrations survive an array override, since narrowing the OS list is not a request to start importing a library's sidecars. |
 | — (plugin) | — | object | — | `sources({ styles: { folder: 'styles', extensions: ['css'] } })` registers build inputs as catalog entities, one collection per key. A sidecar can then read them with `findEntities()`, whose queries land in the render's `refClosure` — so editing, adding or removing a part re-renders the bundle and nothing else. Reading the same files with `fs` instead works for one build and silently breaks watch, because the engine has no dependency on a file it never saw. Nothing is linked into `outputFolder`: these are inputs, not output. Named `sources` rather than `inputs` because `entity.inputs` already means something adjacent — bytes an output depends on without being entities at all. |
 | `cors` / `no-cors` | `--cors` / `--no-cors` | boolean | — | Toggle CORS on the engine's shared Express app. See `src/server.js` for the extensible header arrays plugins push onto. |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mikser-io",
-  "version": "11.3.1",
+  "version": "11.4.0",
   "files": [
     "app.js",
     "index.js",

package/src/manifest/cycle.js

@@ -122,6 +122,16 @@ onFinalize(async () => {
     // transaction because better-sqlite3's transaction() callback is
     // sync-only.
     const m = sharedManifest
+    // Nothing to reconcile against, so nothing below means anything.
+    //
+    // Every other statement that touches the manifest sits inside a loop over
+    // journal entries, so this hook used to survive a null one by never
+    // reaching it. The no-output pass reads `_noOutputIds` unconditionally,
+    // which turned that latent case into a TypeError — and it is reachable:
+    // a plugin can schedule a cycle (createdHook does) in a process where the
+    // manifest's own onLoaded never ran, which is exactly how mikser-io-forms'
+    // tests found it.
+    if (!m) return
 
     // 2a. Stage file unlinks for deleted entities + their children.
     const deleted = new Set(deletedIds)

package/src/server.js

@@ -23,6 +23,7 @@
 import path from 'node:path'
 import { fileURLToPath } from 'node:url'
 import { networkInterfaces } from 'node:os'
+import { createServer } from 'node:net'
 
 import runtime from './runtime.js'
 import { useLogger } from './engine/index.js'
@@ -52,12 +53,55 @@ const STREAMING_REQUEST_TIMEOUT = 2 * 60 * 60 * 1000
 
 export function attachServerCliOptions(commander) {
     commander
-        ?.option('-s --server [port]', 'start an Express server on the given port (defaults to 3001)')
+        ?.option('-s --server [port]',
+            'start an Express server on the given port; omit it, or pass 0, for a free one')
         .option('--cors [origin]', 'restrict server CORS to a specific origin (default *)')
         .option('--no-cors', 'disable server CORS headers')
         .option('-u --url <url>', 'public URL where this mikser is reachable (e.g. https://blog.me.com). Webhook-capable plugins use https URLs for push notifications; other plugins use this when generating absolute URLs (email tracking links, forms share links, MCP previews, etc.).')
 }
 
+// What a bare `--server` asks for: a port nothing is using.
+//
+// It used to be 3001, and 3001 is a number two people on one machine both
+// get. The second one dies on a collision that has nothing to do with either
+// project, and neither of them can pick a different number without asking
+// the other — while the one thing they both actually want is "a port".
+//
+// A named port stays exactly as it was: `--server 3002` binds 3002 or fails
+// saying why. This only changes what happens when nobody named one.
+const DEFAULT_PORT = 0
+
+// The port a given `--server` value asks for, with 0 meaning "a free one".
+//
+// Exported for its own test: the trap here is that 0 is falsy, and the
+// previous `Number(x) || 3001` turned an explicit `--server 0` into 3001 —
+// exactly the collision it was asking to avoid. Testing it through the
+// lifecycle hook would mean importing express, which is not a dependency of
+// this package.
+export function requestedPort(server) {
+    const asked = server === true ? DEFAULT_PORT : Number(server)
+    return Number.isInteger(asked) && asked >= 0 ? asked : DEFAULT_PORT
+}
+
+// A port nothing is listening on, according to the OS.
+//
+// Binds to 0, reads what it was given, and lets it go. There is a gap between
+// releasing it and the real listen below, so this is a strong preference
+// rather than a reservation — if something takes the port in between, the
+// EADDRINUSE handler further down says so plainly instead of pretending.
+// Narrow enough not to matter on the machine this exists for; not narrow
+// enough to claim it cannot happen.
+export async function freePort() {
+    return new Promise((resolve, reject) => {
+        const probe = createServer()
+        probe.once('error', reject)
+        probe.listen(0, () => {
+            const { port } = probe.address()
+            probe.close(() => resolve(port))
+        })
+    })
+}
+
 // Wire the server lifecycle hooks. Called by engine.js's setup() AFTER
 // engine's own onInitialized/onLoad registrations so the log-line order
 // stays "engine folder logs → server bring-up" rather than the reverse.
@@ -80,9 +124,29 @@ export function setupServer() {
         })
         runtime.options.app = express()
         ownsApp = true
-        runtime.options.port = runtime.options.server === true
-            ? 3001
-            : Number(runtime.options.server) || 3001
+        runtime.options.port = requestedPort(runtime.options.server)
+
+        if (runtime.options.port === 0) {
+            // Resolved HERE rather than by handing 0 to listen(), even though
+            // the OS would assign one either way, because the port is read
+            // long before the bind: routes.js builds every operator-facing
+            // route URL from it, mikser-io-ngrok reads it to know what to
+            // tunnel. Worse, `options.server` itself is tested for
+            // truthiness in four places across three packages — the engine's
+            // own instance registration, the server bring-up below,
+            // mikser-io-ngrok, and mikser-io-post-email, whose delivery drain
+            // only runs when `watch || server`. A literal 0 left in place is
+            // falsy in all of them, so it would have turned the server off,
+            // and stopped form emails, on its way to choosing a port.
+            //
+            // So it becomes a real number as early as it can, and everything
+            // downstream sees exactly what it would have seen from
+            // `--server <that number>`.
+            const found = await freePort()
+            runtime.options.server = found
+            runtime.options.port = found
+            logger.info('Server port: %d (nothing named one, so this is a free port)', found)
+        }
         logger.debug('Server starting on port %d', runtime.options.port)
 
         // Trust-proxy: when mikser is behind a reverse proxy (nginx,

package/testing/harness.js

@@ -9,6 +9,7 @@ import _ from 'lodash'
 import realRuntime from '../src/runtime.js'
 import { resetServices } from '../src/services.js'
 import { matchEntity, normalize, changeExtension, getFormatInfo, checksum, AbortError } from '../src/utils/index.js'
+import sift from 'sift'
 
 const OPERATION = {
     CREATE: 'create',
@@ -130,6 +131,23 @@ export function createHarness({
         }
     }
 
+    // One matcher for findEntity / findEntities / iterateEntities, and it is
+    // sift — the same library catalog.js runs queries through, so a filter
+    // that works in a unit test works in a build.
+    //
+    // These used to compare `e[key] === value` per key, which silently matched
+    // NOTHING for any mongo-style operator: `{ collection: { $ne: 'assets' } }`
+    // tested a string against an object and every entity failed. A plugin
+    // walking the catalog that way got an empty result and no error, so its
+    // test either passed vacuously or failed somewhere far from the cause —
+    // which is what happened to the assets orphan sweep, whose whole job is
+    // deciding what has no source.
+    const matching = (query) => {
+        if (!query) return [...entities]
+        if (typeof query === 'function') return entities.filter(query)
+        return entities.filter(sift(query))
+    }
+
     const core = {
         runtime,
         useLogger: () => logger,
@@ -169,28 +187,19 @@ export function createHarness({
         findEntity: async (query) => {
             if (!query) return entities[0]
             if (typeof query === 'function') return entities.find(query)
-            return entities.find(e => Object.entries(query).every(([k, v]) => e[k] === v))
+            return matching(query)[0]
         },
         // Synchronous PK lookup mirroring catalog.js's findById. Layouts'
         // onBeforeRender hydrates dispatch ids through this — the harness
         // serves from the same in-memory entities array.
         findById: (id) => entities.find(e => e?.id === id) ?? catalogStub.byId.get(id),
-        findEntities: async (query) => {
-            if (!query) return [...entities]
-            if (typeof query === 'function') return entities.filter(query)
-            return entities.filter(e => Object.entries(query).every(([k, v]) => e[k] === v))
-        },
+        findEntities: async (query) => matching(query),
         iterateEntities: async function* (query) {
             // Stub — yields the same set findEntities would return,
             // one entity at a time. Real impl in catalog.js chunks via
             // sqlite; the harness doesn't need that fidelity because
             // unit-test corpora are tiny.
-            const filtered = !query
-                ? [...entities]
-                : (typeof query === 'function'
-                    ? entities.filter(query)
-                    : entities.filter(e => Object.entries(query).every(([k, v]) => e[k] === v)))
-            for (const e of filtered) yield e
+            for (const e of matching(query)) yield e
         },
 
         // Rendering & postprocessing — capture for assertions