mikser-io 6.17.0 → 6.21.3

package/README.md

@@ -2,23 +2,43 @@
   <img src="mikser-lockup-stacked.svg" alt="mikser" width="198" />
 </p>
 
-# Mikser Documentation
+# Mikser
 
-Mikser is a precision content engine for Node.js — built around a strict lifecycle, a composable plugin system, and zero compromise on output control. Every document, asset, and template flows through the same deterministic pipeline. Plugins hook in at any phase; nothing runs outside the cycle. The result is a system that scales from a single markdown blog to a multi-language, multi-format publishing platform — without ever losing sight of what it produced and why.
+Mikser is a content engine for Node.js built around a strict lifecycle, a composable plugin system, and direct control over every output. Every document, asset, and template flows through the same deterministic pipeline. Plugins hook in at any phase; nothing runs outside the cycle. It scales from a single markdown blog to a multi-language, multi-format publishing platform — and stays predictable in both directions.
 
-## Documentation Index
+## Why mikser
 
-| Document                                              | Audience           | Description                                        |
-| ----------------------------------------------------- | ------------------ | -------------------------------------------------- |
-| [Getting Started](./documentation/getting-started.md) | Users              | Installation, first project, basic usage           |
-| [Configuration](./documentation/configuration.md)     | Users              | All CLI options and config file reference          |
-| [Lifecycle](./documentation/lifecycle.md)             | Users & Developers | Complete lifecycle phases and hook system          |
-| [Plugins](./documentation/plugins.md)                 | Users & Developers | Built-in plugins, writing custom plugins           |
-| [Entities](./documentation/entities.md)               | Users & Developers | Entity model, operations, journal, catalog         |
-| [Rendering](./documentation/rendering.md)             | Users & Developers | Render pipeline, render plugins, render modes      |
-| [Watch Mode](./documentation/watch-mode.md)           | Users              | File watching, scheduled tasks, incremental builds |
-| [Architecture](./documentation/architecture.md)       | Developers         | System design, module structure, extension points  |
-| [API Reference](./documentation/api-reference.md)     | Developers         | Complete public API reference                      |
+**Your content stays yours.** Source files live on disk as `.md`, `.yml`, `.html` with YAML front-matter. The build output is plain static files. No database lock-in, no proprietary export format. The whole content tree is copyable, diffable, and version-controllable with git — your site is portable on day one and on year ten.
+
+**Incremental builds that scale.** Mikser tracks every entity in a journal. When a file changes, only the affected entities re-process — not the whole site graph. On 10k+ documents this dramatically outpaces tools that rebuild more on every change.
+
+**Concurrent rendering.** Renders fan out across a worker pool that keeps every CPU core hot. Multi-format outputs (HTML, PDF, MJML email, etc.) generate in parallel from the same source.
+
+**One lifecycle, everything composes.** Plugins hook into 20+ named lifecycle phases. A search-indexing plugin shares the same journal iteration as a CMS plugin and an email-rendering plugin — no glue code, no orchestration layer.
+
+**Run anywhere.** The same CLI handles one-shot builds, watch-mode dev loops, and a long-running HTTP server with a shared Express app. `npx mikser` ships a static site; `mikser --watch` is the dev loop; `mikser --server` exposes a live admin/API.
+
+**Library mode.** Mikser is also a library. `useRenderer`, `useCollection`, `findSimilar`, and direct lifecycle hooks let you embed the engine inside an existing Node app instead of running it as a CLI.
+
+**Open source.** MIT-licensed, on GitHub, no telemetry, no auth wall, no SaaS dependency. What you see is what runs.
+
+## Plugin ecosystem
+
+Plugins are independent npm packages — install only what a project actually uses.
+
+| Plugin | What it does |
+|---|---|
+| `documents`, `files`, `resources`, `assets` | Content sources |
+| `layouts` | Layout resolution with auto-matching |
+| `render-hbs`, `render-eta`, `render-liquid`, `render-markdown` | Template engines |
+| `render-resource`, `render-asset`, `render-href` | Resource / asset / link rewriting at render time |
+| `post-pdf` | HTML → PDF via headless Chromium |
+| `post-mjml` | MJML email markup → inbox-safe HTML |
+| `data` | JSON snapshots of entities / context / catalog over HTTP |
+| `api` | REST endpoints — list / get / create / update / delete / render |
+| `vector` | OpenAI embeddings + semantic search (sqlite-vec or pgvector) |
+| `decap` | [Decap CMS](https://decapcms.org/) mounted in the same process |
+| `archive`, `mapper`, `live`, `aml` | Specialty integrations |
 
 ## Quick Start
 
@@ -37,7 +57,9 @@ export default {
 ```
 
 ```bash
-npx mikser
+npx mikser              # one-shot build
+npx mikser --watch      # incremental dev loop
+npx mikser --server     # build + serve at :3001
 ```
 
 ## Core Concepts
@@ -47,3 +69,21 @@ npx mikser
 - **Plugins** — Functionality is delivered via plugins. Built-in plugins handle common sources (documents, files, layouts, assets). Custom plugins can be added to any project.
 - **Runtime Singleton** — A plain module-level object holds all global state and coordinates the lifecycle. The ES module cache guarantees every importer gets the same instance.
 - **Watch Mode** — In watch mode, file changes trigger incremental re-processing without restarting.
+
+## Documentation Index
+
+| Document                                              | Audience           | Description                                        |
+| ----------------------------------------------------- | ------------------ | -------------------------------------------------- |
+| [Getting Started](./documentation/getting-started.md) | Users              | Installation, first project, basic usage           |
+| [Configuration](./documentation/configuration.md)     | Users              | All CLI options and config file reference          |
+| [Lifecycle](./documentation/lifecycle.md)             | Users & Developers | Complete lifecycle phases and hook system          |
+| [Plugins](./documentation/plugins.md)                 | Users & Developers | Built-in plugins, writing custom plugins           |
+| [Entities](./documentation/entities.md)               | Users & Developers | Entity model, operations, journal, catalog         |
+| [Rendering](./documentation/rendering.md)             | Users & Developers | Render pipeline, render plugins, render modes      |
+| [Watch Mode](./documentation/watch-mode.md)           | Users              | File watching, scheduled tasks, incremental builds |
+| [Architecture](./documentation/architecture.md)       | Developers         | System design, module structure, extension points  |
+| [API Reference](./documentation/api-reference.md)     | Developers         | Complete public API reference                      |
+
+## License
+
+MIT — see [LICENSE](./LICENSE).

package/package.json

@@ -1,10 +1,10 @@
 {
 	"name": "mikser-io",
-	"version": "6.17.0",
+	"version": "6.21.3",
 	"description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
 	"main": "index.js",
 	"scripts": {
-		"debug": "node --no-warnings app.js --debug --working-folder test/fixture",
+		"debug": "node --no-warnings app.js --debug --server --watch --working-folder test/fixture",
 		"test:unit": "node --test --test-reporter=spec 'test/unit/**/*.test.js'",
 		"test:smoke": "node --no-warnings app.js --working-folder test/fixture",
 		"test": "npm run test:unit && npm run test:smoke"
@@ -56,10 +56,12 @@
 	},
 	"devDependencies": {
 		"fluent-ffmpeg": "^2.1.3",
+		"mikser-io-decap": "file:../mikser-io-decap",
 		"mikser-io-post-mjml": "file:../mikser-io-post-mjml",
 		"mikser-io-render-eta": "file:../mikser-io-render-eta",
 		"mikser-io-render-liquid": "file:../mikser-io-render-liquid",
 		"mikser-io-render-markdown": "file:../mikser-io-render-markdown",
+		"mikser-io-vector": "file:../mikser-io-vector",
 		"sharp": "^0.34.5"
 	},
 	"directories": {

package/src/engine.js

@@ -6,7 +6,7 @@ import { existsSync } from 'fs'
 import _ from 'lodash'
 import Piscina from 'piscina'
 import runtime from './runtime.js'
-import { onInitialize, onInitialized, onRender, onCancel, onCancelled, onFinalized, onLoaded, onAfterRender, onBeforePostprocess, onPostprocess, postprocessEntities } from './lifecycle.js'
+import { onInitialize, onInitialized, onLoad, onRender, onCancel, onCancelled, onFinalized, onLoaded, onAfterRender, onBeforePostprocess, onPostprocess, postprocessEntities } from './lifecycle.js'
 import { useJournal, updateEntry } from './journal.js'
 import { globby } from 'globby'
 import { OPERATION, TASKS } from './constants.js'
@@ -46,6 +46,7 @@ export async function setup(options) {
             .option('-d --debug', 'display debug statements')
             .option('-t --trace', 'display trace statements')
             .option('-e --runtime-folder <folder>', 'set mikser runtime folder relative to working folder', 'runtime')
+            .option('-s --server [port]', 'start an Express server on the given port (defaults to 3001)')
 
         Object.assign(runtime.options, options || runtime.engine.commander.parse(process.argv).opts())
         runtime.options.info = true
@@ -84,6 +85,73 @@ export async function setup(options) {
             }
         }
         await mkdir(runtime.options.runtimeFolder, { recursive: true })
+
+        // Server bring-up: two paths, controlled by which of these are set.
+        //
+        //   runtime.options.app   — pre-supplied Express app (e.g. mikser
+        //                           embedded inside an existing service).
+        //                           The caller owns the listen lifecycle
+        //                           and the static-route policy. Engine
+        //                           stays out of routing/listening so it
+        //                           doesn't clobber their setup; plugins
+        //                           still mount their own routers on it.
+        //
+        //   --server [port]       — engine creates the Express app, mounts
+        //   (runtime.options.server)  static for the output folder, and
+        //                           listens on the port. The actual
+        //                           listen() is deferred via the onLoad
+        //                           hook below so it runs LAST in the
+        //                           onLoaded phase — after every plugin
+        //                           has had a chance to register routes.
+        //
+        // If both are present, the externally-supplied app wins; --server
+        // becomes a no-op (the caller is in charge).
+        if (runtime.options.app) {
+            logger.info('Using externally-supplied Express app on runtime.options.app')
+        } else if (runtime.options.server) {
+            const { default: express } = await import('express').catch(() => {
+                throw new Error('Express is required for --server. Run: npm install express')
+            })
+            runtime.options.app = express()
+            runtime.options.port = runtime.options.server === true
+                ? 3001
+                : Number(runtime.options.server) || 3001
+            logger.info('Server starting on port %d', runtime.options.port)
+        }
+    })
+
+    // Registered here (inside setup) so it runs AFTER plugins.js's onLoad
+    // (which is registered at module-import time). That ordering matters
+    // because plugins.js loads user plugins during its onLoad — each plugin
+    // factory may register onLoaded handlers that mount routes on
+    // runtime.options.app. We want our listen() to run LAST in the
+    // onLoaded phase, which is why we register it from inside another
+    // onLoad — by then plugins have already appended their handlers.
+    onLoad(() => {
+        // Only auto-mount static + auto-listen when the engine owns the
+        // app (created via --server). When an external app was supplied,
+        // `port` is unset and we stay out of the way — the caller manages
+        // both routing decisions and the listen lifecycle themselves.
+        if (!runtime.options.app || runtime.options.port == null) return
+        onLoaded(async () => {
+            const logger = useLogger()
+            const { default: express } = await import('express')
+
+            // Serve the output folder as the catch-all static route.
+            // Mounted LAST in the middleware chain so plugin routes
+            // (e.g. /api/*) match first; anything that didn't match a
+            // plugin's router falls through to the static handler and
+            // gets served from <outputFolder>.
+            runtime.options.app.use(express.static(runtime.options.outputFolder))
+            logger.info('Serving %s as /', runtime.options.outputFolder.replace(runtime.options.workingFolder + '/', ''))
+
+            await new Promise(resolve => {
+                runtime.options.app.listen(runtime.options.port, () => {
+                    logger.info('Server listening on port %d', runtime.options.port)
+                    resolve()
+                })
+            })
+        })
     })
 
     onLoaded(async () => {

package/src/journal.js

@@ -70,12 +70,12 @@ export async function clearJournal(aborted) {
     await journal('operations').del()
     if (!aborted) {
         // Tear down the sqlite connection only when this is genuinely a
-        // one-shot run that's about to exit. Watch mode and any plugin
-        // that keeps the process alive (e.g. the API plugin's HTTP server)
-        // sets `persistent` so subsequent cycles can still write to the
-        // journal — without this, the second /render request would crash
-        // with "Unable to acquire a connection".
-        if (runtime.options.watch !== true && runtime.options.persistent !== true) {
+        // one-shot run that's about to exit. The two ways mikser stays
+        // alive across cycles are watch mode (chokidar handles keep the
+        // event loop ref'd) and an HTTP server (`runtime.options.app`,
+        // set either by --server or by a caller passing setup({ app }));
+        // both need the journal to survive subsequent cycles.
+        if (runtime.options.watch !== true && !runtime.options.app) {
             journal.destroy()
         }
     }

package/src/plugins/api.js

@@ -73,13 +73,23 @@ export default ({
     onLoaded(async () => {
         const logger = useLogger()
 
+        // The api plugin no longer creates its own Express app. It mounts
+        // onto an existing app provided either by --server (engine
+        // creates one) or by a caller that programmatically passed
+        // setup({ app: ... }). Without an app there's nowhere to mount
+        // routes, so fail fast with a message that points at the fix.
+        const app = runtime.options.app
+        if (!app) {
+            throw new Error(
+                'API plugin requires runtime.options.app — run mikser with --server, ' +
+                'or pass { app: yourExpressInstance } to setup() before loading the api plugin'
+            )
+        }
+
         const { default: express } = await import('express').catch(() => {
-            throw new Error('express is required for the api plugin — run: npm install express')
+            throw new Error('Express is required for the api plugin — run: npm install express')
         })
 
-        const ownApp = !runtime.options.app
-        const app = runtime.options.app ?? express()
-
         const router = express.Router()
         router.use(express.json())
 
@@ -167,22 +177,8 @@ export default ({
             }
         })
 
-        // The HTTP server keeps the process alive across many process()
-        // cycles. Tell the journal layer not to tear down the sqlite
-        // connection at the end of each cycle.
-        runtime.options.persistent = true
-
         const base = runtime.config.api?.base ?? '/api'
         app.use(base, router)
-
-        if (ownApp) {
-            const port = runtime.config.api?.port ?? 3001
-            const url = runtime.config.api?.url ?? 'http://localhost'
-            app.listen(port, () => {
-                logger.info('Api listening: %s:%d%s', url, port, base)
-            })
-        } else {
-            logger.info('Api mounted: %s', base)
-        }
+        logger.info('Api mounted: %s', base)
     })
 }

package/src/plugins/post/pdf.js

@@ -14,11 +14,18 @@ export async function setup({ config, logger }) {
     }
 
     const { default: puppeteer } = await import('puppeteer').catch(() => {
-        throw new Error('puppeteer is required for the pdf postprocessor — run: npm install puppeteer')
+        throw new Error('Puppeteer is required for the pdf postprocessor — run: npm install puppeteer')
     })
+    // --no-sandbox / --disable-setuid-sandbox are required on most
+    // headless Linux servers and Docker images where Chrome's sandbox
+    // can't be set up. Applied by default and merged (deduped) with any
+    // user-supplied launch.args so callers can add flags without losing
+    // these. To run WITH the sandbox, override launch.args explicitly.
+    const defaultArgs = ['--no-sandbox', '--disable-setuid-sandbox']
     browser = await puppeteer.launch({
         headless: true,
-        ...config?.launch
+        ...config?.launch,
+        args: [...new Set([...defaultArgs, ...(config?.launch?.args ?? [])])],
     })
     logger.debug('Puppeteer browser launched')
 }

package/src/runtime.js

@@ -77,7 +77,7 @@ const runtime = {
         else if (this.abortController) {
             await this.cancel()
         }
-        this.mutex.use(async () => {
+        await this.mutex.use(async () => {
             try {
                 this.abortController = new AbortController()
                 const { signal } = this.abortController

package/src/utils.js

@@ -55,7 +55,7 @@ export function matchEntity(entity, match) {
         }
     }
     else if (typeof match == 'object') return _.isMatch(entity, match)
-    throw new Error('Ivalid match type')
+    throw new Error('Invalid match type')
 }
 
 export function changeExtension(file, format) {