mikser-io 9.0.0 → 9.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/10.0-PLAN.md CHANGED
@@ -37,7 +37,7 @@ When the working signals dominate over a sustained period of real use, v10 desig
37
37
 
38
38
  ## Why v10 (not 9.x)
39
39
 
40
- The 9.0 plan deliberately deferred mikser-to-mikser composition to v10. The reasoning is in `9.0-PLAN.md` under "Out of scope":
40
+ The 9.0 design deliberately deferred mikser-to-mikser composition to v10. The reasoning:
41
41
 
42
42
  > Cross-instance concerns (discovery, peer identity, routing, cross-mikser refs, watch propagation, liveness, partition behavior, peer auth) belong to a distributed-systems substrate that iterates at "we hit a production partition bug" speed, not at engine-substrate speed. Folding any of it into `mikser-io` would re-fail ADR-0006 test #5 (release cadence) the moment it ships.
43
43
 
package/CLAUDE.md CHANGED
@@ -10,22 +10,51 @@ paths. No migration markdown. No `task: pool` legacy aliasing for
10
10
  `task: inline`. Update READMEs and ADRs in place as source-of-truth
11
11
  changes; rewrite, don't supersede. The freedom is the point.
12
12
 
13
- **v9 has to pay out before v10 starts.** Not "ship v9 and start v10."
14
- **Use v9 in real daily work until the workflow pays out**, then
15
- v10. The v10 architecture (party-mikser-io) is more fun to design
16
- than v9 documentation is to write the gate exists because that
17
- asymmetry kills projects. Concrete pass/fail signals in
18
- `10.0-PLAN.md` under "The gate." If a session brings up v10 design
19
- work, check the gate first; if v9 isn't paying out yet, the
20
- question to answer is "what about v9 isn't working in daily use?"
21
- not "what should v10 look like?"
22
-
23
- **Position mikser by what it is, not by speed.** Hugo wins the speed
24
- race; mikser doesn't compete there. Position: AI-native, lifecycle-
25
- observable, files-as-source-of-truth, full-cycle introspection. Speed
26
- claim cap: "fast enough that watch-mode rebuilds feel instant."
27
- Honest range: 200–800 docs/sec depending on corpus size
28
- (see `test/perf/`).
13
+ **v9 shipped. The gate now is daily-use payoff, not completion.**
14
+ The engine, every sibling plugin, the agency template all live.
15
+ The next step isn't "finish v9," it's **use v9 in real daily work
16
+ until the workflow earns the v10 layer**. The v10 architecture
17
+ (party-mikser-io) is more fun to design than v9 dogfooding is to
18
+ sustain the gate exists because that asymmetry kills projects.
19
+ Concrete pass/fail signals in `10.0-PLAN.md` under "The gate." If
20
+ a session brings up v10 design work, check the gate first; if v9
21
+ isn't paying out yet, the question to answer is "what about v9
22
+ isn't working in daily use?" — not "what should v10 look like?"
23
+
24
+ **mikser is a mixer.** The name is the architecture. It combines
25
+ inputs (entities — from files, forms, queries, anywhere) through
26
+ configurable rendering pipelines (renderer plugins + task-production
27
+ policies) into outputs of any kind. Static-site generation is the
28
+ canonical recipe — what mikser ships with and what most projects
29
+ use today — but it's a recipe, not the definition. The same
30
+ substrate runs:
31
+
32
+ - HTML SSG via `layouts` + `renderHbs` (the recipe everything else
33
+ is currently shaped around)
34
+ - PDFs via `renderHbs` + `post-pdf` (puppeteer-driven)
35
+ - Emails via `renderHbs` → `post-mjml` → `post-email` (chain)
36
+ - 3D renders by composing a hypothetical `renderBlender` against
37
+ `.blend` entities — same engine, different renderer + recipe
38
+ - Video / audio / archives / data-export — same shape
39
+
40
+ The engine has no opinion on what a renderer produces. `render.js`
41
+ and `postprocess.js` are abstract worker entries; `manifest.js`
42
+ tracks snapshots of *any* output; refs invalidation re-runs
43
+ *any* render task. SSG is the dominant case in v9 because it's
44
+ where the dogfooding lives, not because the architecture only
45
+ fits there. When proposing changes, hold the broader view: a
46
+ change that helps SSG but blocks the Blender/video/audio/archive
47
+ cases is constraining the engine's actual reach.
48
+
49
+ **Position mikser by what it is, not by speed.** Hugo wins the
50
+ speed race for SSG specifically; mikser doesn't compete there
51
+ (and "fastest mixer" isn't a useful claim because the bottleneck
52
+ is the renderer, not the engine). Secondary attributes that
53
+ follow from the mixer framing: AI-native, lifecycle-observable,
54
+ files-as-source-of-truth, full-cycle introspection. Speed claim
55
+ cap: "fast enough that watch-mode rebuilds feel instant." Honest
56
+ range: 200–800 docs/sec on the SSG-flavor pipeline depending on
57
+ corpus size (see `test/perf/`).
29
58
 
30
59
  **Direct critique preferred.** Skip "great question," skip "solid but,"
31
60
  skip softening. When something didn't pay off, say so. Match user
@@ -100,7 +129,16 @@ brevity.
100
129
  plugins to push onto), late-binding static mount + listen.
101
130
  - `logger.js` — pino + pino-pretty (inline) + gauge progress + custom
102
131
  Writable for progress coordination + `pino.multistream` for
103
- third-party shipping (`runtime.config.logging.transports`).
132
+ third-party shipping. Two registration surfaces for transports:
133
+ - `runtime.config.logging.transports` — declarative; user config.
134
+ - `addLogTransport({ target, options?, level? })` — public export;
135
+ plugins call this from their factory (queues until logger build)
136
+ or any later hook (live-rebuilds the multistream + swaps
137
+ `runtime.engine.logger`). `useLogger()` reads the swap-target
138
+ fresh on each call so new transports start receiving records
139
+ immediately. Both surfaces compose; same `{level, target, options}`
140
+ shape. Enables Better Stack / Datadog / Loki / Axiom / Sentry as
141
+ standard sibling plugins without engine changes per vendor.
104
142
  - `utils.js` — shared pure helpers: `mimeForEntity`, `isLoopback`,
105
143
  `expandEntity`, `projectMeta`, `useCollection`, `useRenderer` (via
106
144
  render.js), `isTextEntity`, `readEntityContent`, `extractRefs`,
@@ -141,11 +179,35 @@ brevity.
141
179
  POSTPROCESS), `ACTION` (sync action types), `TASKS` (`INLINE`/
142
180
  `SERIAL`/`WORKER` — dispatch modes).
143
181
 
182
+ ## Canonical sibling plugins
183
+
184
+ These ship as separate repos but most projects pull them in. They're
185
+ not bundled because the engine is renderer-agnostic — each plugin
186
+ encodes a particular recipe (SSG, MJML, PDF, email) rather than
187
+ substrate.
188
+
189
+ - **`mikser-io-layouts`** — the canonical SSG-flavor task-production
190
+ policy. Owns multi-match layout assignment (`meta.layout` /
191
+ `meta.layouts` dual key + `autoLayouts` peel ladder), per-layout
192
+ destination Handlebars templates, collision detection, postprocess
193
+ chain parsing (filename `name.html-mjml-email.hbs` → `postprocessors:
194
+ ['mjml','email']` + frontmatter `postprocessor`/`postprocessors`),
195
+ `inspect()` primitive at `runtime.options.layouts.inspect`. The
196
+ engine has no opinion on layouts; this plugin is one way of producing
197
+ render tasks. Other domains (3D rendering, video) would have their
198
+ own task-production plugins.
199
+ - **`mikser-io-render-{hbs,liquid,eta,markdown,metatext}`** — renderer
200
+ plugins. `renderHbs` is bundled in core; the rest are siblings.
201
+ - **`mikser-io-post-{mjml,pdf,email}`** — postprocessor plugins,
202
+ composable in chains (see `documentation/rendering.md#postprocess`).
203
+ - **`mikser-io-mcp`** — AI tooling surface (ADR-0006 test #5).
204
+ - **`mikser-io-vector`** — vector index over the catalog.
205
+ - **`mikser-io-forms`** — HTTP form receivers → entities.
206
+ - **`mikser-io-schemas`** — TypeScript type generation from entity meta.
207
+
144
208
  ## Plugin map (`src/plugins/`)
145
209
 
146
210
  - `documents` — file→entity sync for the documents collection
147
- - `layouts` — layout matching + sitemap + `inspect()` primitive
148
- (exposed at `runtime.options.layouts.inspect`)
149
211
  - `files` — file→entity sync for the files collection
150
212
  - `assets` — asset references and copy
151
213
  - `resources` — resource references
@@ -160,11 +222,75 @@ brevity.
160
222
  - `observer` / `mapper` / `validator` / `commands` / `shares` —
161
223
  utility plugins
162
224
 
225
+ ## Built-in content providers (`src/plugins/providers/`)
226
+
227
+ The substrate-side `readEntityContent(entity)` dispatches by URI
228
+ scheme. Two schemes are handled inline by built-in providers
229
+ (no separate package required, ships with mikser-io):
230
+
231
+ - **No scheme / plain path / `file://`** — filesystem read. Lives
232
+ inline in `src/utils.js readEntityContent` because the engine
233
+ already touches the filesystem everywhere.
234
+ - **`http://` / `https://`** — `src/plugins/providers/http.js`.
235
+ Conditional GET with ETag + Last-Modified, in-memory response
236
+ cache, inflight coalescing on the same URL, binary mirror to
237
+ `runtime/http-cache/<sha-of-url>.<ext>`, operator-supplied
238
+ headers via `entity.meta.httpHeaders`, configurable timeout via
239
+ `entity.meta.httpTimeoutMs`. Test affordance:
240
+ `__resetHttpCacheForTests()` clears the module-level caches
241
+ between unit tests.
242
+
243
+ Any other scheme (`gdrive://`, `notion://`, `s3://`, …) routes
244
+ through the dynamic-import-by-name path — `mikser-io-provider-<scheme>`
245
+ must be installed as a sibling package and export a top-level
246
+ `read(entity)` function. See ADR-0010 / the gdrive provider for
247
+ the convention.
248
+
249
+ ## Multi-emitter collections (sweep ownership)
250
+
251
+ Collections like `documents` / `files` / `assets` aren't owned by a
252
+ single plugin. The file source (via `useSource`) emits into them
253
+ from its scanned folder, but other plugins legitimately emit into
254
+ the same collection: `mikser-io-csv` fans CSV rows into `documents`,
255
+ a gdrive sync would emit there too, an API endpoint can push into
256
+ `files`. The catalog has no opinion on emitter identity — only on
257
+ collection.
258
+
259
+ The delete sweep MUST respect that. `sweepDeleted(collection,
260
+ scanned, onDelete, ownerPrefix)` requires `ownerPrefix` (typically
261
+ `absFolder` for file sources, `layoutsFolder` for layouts) and only
262
+ considers entities whose `entity.uri` is rooted under that prefix.
263
+ Foreign-emitter entities have a different `uri` shape:
264
+ - CSV row entities: empty `uri` (synthetic — meta is the content)
265
+ - HTTP-fetched CSV parent: `uri = 'https://…'`
266
+ - gdrive-sourced docs: `uri = 'gdrive://…'`
267
+
268
+ …and the LIKE clause excludes them. Without the scope, every cycle's
269
+ file-source sweep silently wipes every co-collection emitter's
270
+ entities. Throwing instead of silently sweeping the whole collection
271
+ is intentional — this used to be a class-of-bugs landmine. If you're
272
+ writing a new source-shaped plugin, the scoping is mandatory.
273
+
274
+ Test coverage: `test/unit/source-sweep.test.js`.
275
+
163
276
  ## Naming conventions
164
277
 
165
278
  - **Engine state** at `runtime.<name>` — `runtime.refs`, `runtime.catalog`.
166
279
  - **Plugin surfaces** at `runtime.options.<plugin>` —
167
280
  `runtime.options.preview`, `runtime.options.layouts.inspect`.
281
+ - **Reachability:** `runtime.options.url` is the engine's resolved
282
+ public URL (from `--url` or `config.url`, trailing slash stripped,
283
+ validated). Plugins that need external reachability — webhook
284
+ receivers, absolute links in emails, MCP preview URLs returned to
285
+ agents, forms share links — read this. Standard gating pattern:
286
+ ```js
287
+ const canPush = runtime.options.url?.startsWith('https://')
288
+ if (canPush) registerWebhookAt(`${runtime.options.url}/api/X/webhook`)
289
+ else setupPollingFallback()
290
+ ```
291
+ `runtime.options.port` is the internal listener port (kept for
292
+ loopback/dev URL building); `runtime.options.url` is the external
293
+ origin. When in doubt: external = `url`, internal = `port`.
168
294
  - **Engine functions** as module-level exports from `mikser-io`:
169
295
  `import { queryEntities, subscribe, useRenderer, useCollection,
170
296
  readEntityContent, isTextEntity } from 'mikser-io'`.
@@ -326,6 +452,22 @@ There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
326
452
  - **New lifecycle hook?** Almost certainly no. Existing hooks cover
327
453
  all known patterns. If you think you need one, post the use case
328
454
  to ADR-0006 review.
455
+ - **New content source (gdrive, notion, s3, github, …)?** Two
456
+ things ship together:
457
+ - A regular lifecycle plugin (the sync — emits entities into the
458
+ catalog, sets `entity.uri = '<scheme>://...'` so the dispatch
459
+ knows where to route reads).
460
+ - A named export `read(entity)` from the same package, which
461
+ must be named `mikser-io-provider-<scheme>` so the engine can
462
+ find it. `readEntityContent(entity)` parses the scheme out of
463
+ `entity.uri`, dynamic-imports the package, calls its `read`.
464
+ No registry, no descriptor — same package-name convention as
465
+ `mikser-io-render-*` and `mikser-io-post-*`.
466
+ Built-in: plain local paths (no scheme) and `file://` URIs are
467
+ read via `fs.readFile` directly — no provider package needed for
468
+ the canonical local-filesystem case. `entity.content` already
469
+ populated is a fast-path that skips the dispatch entirely (for
470
+ small remote docs eager-fetched at sync time).
329
471
 
330
472
  ## Test suites
331
473
 
@@ -342,6 +484,48 @@ There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
342
484
  + clean-build timing). `SIZE=realistic` + entity count knobs
343
485
  documented in **Perf**.
344
486
 
487
+ ## Dev workspace
488
+
489
+ The siblings live side-by-side under `/Users/dick/Projects/mikser/`
490
+ and share an npm workspace declared at the parent's `package.json`:
491
+
492
+ ```json
493
+ {
494
+ "private": true,
495
+ "workspaces": ["mikser-io", "mikser-io-*"]
496
+ }
497
+ ```
498
+
499
+ `npm install` at that root hoists everything: each sibling's
500
+ `node_modules/mikser-io` (and any other cross-workspace dep)
501
+ becomes a symlink to the working copy. The whole tree runs
502
+ against ONE module instance of mikser-io.
503
+
504
+ **This isn't optional ergonomics — it's correctness.** Plugins
505
+ use `AsyncLocalStorage` (currently `queryContext` for sidecar
506
+ findEntities tracking; the schema-version meta could grow more).
507
+ Without workspace deduplication, npm 7+ auto-installs the peer
508
+ dep into each sibling's own `node_modules`. Layouts'
509
+ `import { queryContext } from 'mikser-io'` then resolves to its
510
+ bundled copy — a different AsyncLocalStorage instance than the
511
+ engine uses. Sidecar queries don't get tracked → no `query`
512
+ edges in `mikser_snapshots.refClosure` →
513
+ `manifest.queryAffected` returns empty → aggregate layouts
514
+ (index pages, sitemaps, RSS feeds) never invalidate when new
515
+ matching entities land. Silent broken-incremental.
516
+
517
+ `test/scenarios/_harness.js` also auto-coalesces (replaces any
518
+ real `mikser-io-layouts/node_modules/mikser-io` directory with a
519
+ symlink to MIKSER_ROOT on every `setupFixture`) as a backstop —
520
+ the scenarios pass with or without the workspace, but the
521
+ workspace is the canonical setup. Same shape applies to any
522
+ future sibling that uses `queryContext` or other module-level
523
+ state from mikser-io.
524
+
525
+ For production consumers (someone `npm install mikser-io
526
+ mikser-io-layouts` in their own project), npm resolves both from
527
+ the consumer's own project tree — no duplication, no bug.
528
+
345
529
  ## Reference
346
530
 
347
531
  - `documentation/architecture.md` — module map (audit before relying
@@ -350,5 +534,8 @@ There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
350
534
  - `documentation/configuration.md` — config reference
351
535
  - `documentation/api-reference.md` — public API
352
536
  - `test/perf/` — render-pipeline perf rig
353
- - Sibling repos: `mikser-io-mcp`, `mikser-io-vector`, `mikser-io-schemas`,
537
+ - Sibling repos: `mikser-io-layouts` (canonical SSG-flavor render-task
538
+ production policy), `mikser-io-mcp`, `mikser-io-vector`,
539
+ `mikser-io-schemas`, `mikser-io-forms`, `mikser-io-post-{mjml,pdf,email}`,
540
+ `mikser-io-render-{eta,liquid,markdown,metatext}`,
354
541
  `mikser-io-sdk-{api,react,svelte,vue,vector}`, `mikser-io-example-blog`
package/README.md CHANGED
@@ -50,6 +50,28 @@ Build mikser into the parts of your application that are content-shaped. Keep th
50
50
 
51
51
  **Open source.** MIT-licensed, on GitHub, no telemetry, no auth wall, no SaaS dependency. What you see is what runs.
52
52
 
53
+ ## Mix content from anywhere, ship to anywhere
54
+
55
+ A real content stack pulls from more than one system. Marketing copy lives in a CMS or a spreadsheet. Prices and stock live in an ERP. Hero images live in a DAM. Editorial pages live in markdown files in the repo. Most teams either pick one tool and contort the rest to fit it, or build sync services that copy everything into one database — and then more sync services when things drift out of sync.
56
+
57
+ Mikser is built around a different bet: **one queryable substrate that any source can pour into, and any frontend can read from.**
58
+
59
+ **Any source.** Provider plugins let you treat external systems as content sources. A Google Sheet, an ERP feed, a Drive folder, a Notion database, an HTTP webhook, the local repo's `.md` files — each becomes a stream of entities flowing into mikser's catalog with the same shape. The authoring tool doesn't change. The editorial workflow doesn't change. The team writing product copy in Google Sheets keeps writing product copy in Google Sheets — mikser just notices when they save.
60
+
61
+ **Cross-source composition through references.** Mikser tracks references between entities the way a graph database does. A product page can declare `$marketing` pointing at a row pulled from a spreadsheet, `$pricing` at a record pulled from your ERP, `$hero` at an asset pulled from your DAM. A single API call from your frontend asks for the product page *with* its referenced data — and gets the page, the marketing copy, the price, and the hero image merged into one response. One round trip, not three or four, and no consumer-side join logic to maintain.
62
+
63
+ **Live updates are uniform across every source.** Edit a cell in the spreadsheet, change a price in the ERP, replace an asset in the DAM — the pages depending on those entities update within seconds, in every frontend connected to mikser's live channel. You don't write per-source invalidation; mikser already knows which pages reference what.
64
+
65
+ **Your frontend is whatever you want.** Mikser exposes the catalog over HTTP (and over MCP, for AI agents). You query it from React, Vue, Svelte, SvelteKit, Next.js, an iOS app, a kiosk — anything that speaks HTTP. The framework SDKs (`mikser-io-sdk-react`, `mikser-io-sdk-vue`, `mikser-io-sdk-svelte`) make the calls feel native, but they're optional. Headless CMSes free you from the database lock. Frameworks like Astro free you from one kind of authoring lock. Mikser frees you from both at once: **any source on the input side, any framework on the output side.**
66
+
67
+ What this looks like in practice:
68
+
69
+ - Marketing edits product copy in Google Sheets, prices come from your ERP, both flow into the same product page in your storefront — changes appear within seconds, without your e-commerce team writing webhook handlers or cache-invalidation logic.
70
+ - A SaaS pulls customer-facing release notes from `release-notes/*.md` in the repo and feature flags from LaunchDarkly; both render in a dashboard built in the team's own React stack, not a framework somebody else picked.
71
+ - A magazine pulls editorial articles from local markdown, contributor bios from Notion, sponsorship info from Airtable, and renders the same article surface across web, email (MJML), and PDF — one catalog, one set of references, three outputs.
72
+
73
+ Adding a source is mechanical. See [`mikser-io-csv`](https://github.com/almero-digital-marketing/mikser-io-csv) (any HTTP-served CSV, with live polling), [`mikser-io-provider-gdrive`](https://github.com/almero-digital-marketing/mikser-io-provider-gdrive) (Google Drive), and the built-in `http` provider for the shape. If a system has an API, it can be a mikser source, usually in well under 100 lines.
74
+
53
75
  ## Built for AI-assisted development
54
76
 
55
77
  Files-as-source isn't just a portability story — it makes the project unusually friendly to AI coding agents. There's a static-time half (the agent reads your tree the way it reads any repo — no DB connection, no schema upload, no sandboxed query layer to learn) and a runtime half (when the agent needs to write or render, mikser ships its own MCP server in core, so it talks to the live engine instead of a parallel REST shim you have to maintain).
@@ -150,6 +172,7 @@ The engine is what stays stable — the lifecycle, the catalog, the file-based c
150
172
  |---|---|
151
173
  | [`mikser-io-vector`](https://github.com/almero-digital-marketing/mikser-io-vector) | OpenAI embeddings + semantic search (sqlite-vec or pgvector) |
152
174
  | [`mikser-io-schemas`](https://github.com/almero-digital-marketing/mikser-io-schemas) | Zod-backed entity validation + auto-generated TypeScript declarations for the SDK. Auto-detects `$`-keyed references and warns on broken ones — see [ADR-0007](./documentation/decisions/0007-references-declaration-and-expansion.md) |
175
+ | [`mikser-io-forms`](https://github.com/almero-digital-marketing/mikser-io-forms) | Public form-submission endpoints. POST → captcha + schema validation → write a document file plus uploaded files to disk; the `documents` / `files` plugins pick them up via their normal watch loop. Composes with `mikser-io-schemas` for schema-by-name; built-in captcha providers (Google v2/v3, hCaptcha, Turnstile) plus a custom-verify escape hatch |
153
176
  | [`mikser-io-archive`](https://github.com/almero-digital-marketing/mikser-io-archive) | Persist matching entities to YAML — audit trail, versioned content history, downstream export |
154
177
  | `mapper` | Run config-supplied transforms over matched entities each cycle (in-core, generic transformation layer) |
155
178
  | [`mikser-io-live`](https://github.com/almero-digital-marketing/mikser-io-live) | Lightweight dev server with browser auto-refresh — pair with `--watch` for the classic save→reload loop |
package/index.js CHANGED
@@ -28,7 +28,7 @@ export { documents } from './src/plugins/documents.js'
28
28
  export { files } from './src/plugins/files.js'
29
29
  export { frontMatter } from './src/plugins/front-matter.js'
30
30
  export { json } from './src/plugins/json.js'
31
- export { layouts } from './src/plugins/layouts.js'
31
+ // layouts moved to sibling: import from 'mikser-io-layouts'
32
32
  export { mapper } from './src/plugins/mapper.js'
33
33
  export { observer } from './src/plugins/observer.js'
34
34
  export { preview } from './src/plugins/preview.js'
package/package.json CHANGED
@@ -1,12 +1,12 @@
1
1
  {
2
2
  "name": "mikser-io",
3
- "version": "9.0.0",
3
+ "version": "9.0.1",
4
4
  "description": "<p align=\"center\"> <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
5
5
  "main": "index.js",
6
6
  "exports": {
7
- ".": "./index.js",
7
+ ".": "./index.js",
8
8
  "./package.json": "./package.json",
9
- "./*": "./*"
9
+ "./*": "./*"
10
10
  },
11
11
  "scripts": {
12
12
  "debug": "node --no-warnings app.js --server --watch --working-folder test/fixture",
@@ -61,6 +61,7 @@
61
61
  "express": "^5.2.1"
62
62
  },
63
63
  "devDependencies": {
64
+ "@ai-sdk/openai": "^3.0.71",
64
65
  "fluent-ffmpeg": "^2.1.3",
65
66
  "mikser-io-decap": "file:../mikser-io-decap",
66
67
  "mikser-io-post-mjml": "file:../mikser-io-post-mjml",
package/src/engine.js CHANGED
@@ -5,7 +5,7 @@ import { existsSync } from 'fs'
5
5
  import _ from 'lodash'
6
6
  import Piscina from 'piscina'
7
7
  import runtime from './runtime.js'
8
- import { onInitialize, onInitialized, onRender, onCancel, onCancelled, onFinalized, onLoaded, onBeforePostprocess, onPostprocess, postprocessEntities } from './lifecycle.js'
8
+ import { onInitialize, onInitialized, onLoad, onRender, onCancel, onCancelled, onFinalized, onLoaded, onBeforePostprocess, onPostprocess, postprocessEntities } from './lifecycle.js'
9
9
  import { useJournal, updateEntry } from './journal.js'
10
10
  import { globby } from 'globby'
11
11
  import { OPERATION, TASKS } from './constants.js'
@@ -178,6 +178,39 @@ export async function setup(options) {
178
178
  // bring-up logs come after the folder-info lines.
179
179
  setupServer()
180
180
 
181
+ // Public URL resolution. CLI --url wins; otherwise read runtime.config.url
182
+ // populated by the user's mikser.config.js. Stamped on runtime.options.url
183
+ // (no trailing slash) for plugins to consume in their onLoaded.
184
+ //
185
+ // Plugins that need external reachability (webhook receivers, absolute
186
+ // links in emails, MCP preview URLs, forms share links) read this. The
187
+ // standard gating pattern is:
188
+ //
189
+ // const canPush = runtime.options.url?.startsWith('https://')
190
+ // if (canPush) registerWebhookAt(`${runtime.options.url}/api/X/webhook`)
191
+ // else setupPollingFallback()
192
+ //
193
+ // Runs at onLoad — after config.js fires its onLoad (which populates
194
+ // runtime.config) and before any plugin's onLoaded.
195
+ onLoad(async () => {
196
+ const logger = useLogger()
197
+ const cli = runtime.options.url
198
+ const cfg = runtime.config?.url
199
+ const raw = cli ?? cfg
200
+ if (raw == null || raw === '') {
201
+ runtime.options.url = undefined
202
+ return
203
+ }
204
+ try {
205
+ new URL(raw)
206
+ } catch (err) {
207
+ throw new Error(`Invalid --url / config.url value: ${raw} (${err.message})`)
208
+ }
209
+ runtime.options.url = String(raw).replace(/\/+$/, '')
210
+ logger.info('Public URL: %s%s', runtime.options.url,
211
+ runtime.options.url.startsWith('https://') ? ' (webhook-capable)' : ' (http; plugins will not enable push webhooks)')
212
+ })
213
+
181
214
  onLoaded(async () => {
182
215
  const logger = useLogger()
183
216
  logger.debug(runtime.options, 'Mikser options')
@@ -479,15 +512,17 @@ export async function setup(options) {
479
512
 
480
513
  const tasks = []
481
514
  for await (const { entity, options, context, output } of useJournal('Queuing postprocess', [OPERATION.RENDER], signal)) {
482
- if (output?.success && options.postprocessor) {
483
- const ext = await resolveOutputExt(options.postprocessor)
484
- // Engine swaps the extension on whatever destination the
485
- // entity carries. Routing semantics (cleanUrls folder
486
- // structure, alternate placements, etc.) are whoever-set-
487
- // entity.destination's responsibility typically the
488
- // layouts plugin during layout-match. The postprocess
489
- // subsystem has no opinion about routing.
490
- const destination = changeExtension(entity.destination, ext)
515
+ // Read the postprocessor chain. layouts plugin carries
516
+ // it as `options.postprocessors` (array, normalized); the
517
+ // singular `options.postprocessor` is a back-compat alias
518
+ // for the head of the chain. Empty chain no postprocess.
519
+ const chain = options.postprocessors ?? (options.postprocessor ? [options.postprocessor] : [])
520
+ if (output?.success && chain.length) {
521
+ // The FINAL destination extension is the last stage's
522
+ // `output:`. Intermediate stages' extensions are used
523
+ // only for scratch path naming inside src/postprocess.js.
524
+ const finalExt = await resolveOutputExt(chain[chain.length - 1])
525
+ const destination = changeExtension(entity.destination, finalExt)
491
526
 
492
527
  tasks.push({
493
528
  entity: {
@@ -496,7 +531,11 @@ export async function setup(options) {
496
531
  destination
497
532
  },
498
533
  options: {
499
- postprocessor: options.postprocessor,
534
+ // Singular kept for back-compat with the worker
535
+ // serialization shape; chain is the source of
536
+ // truth.
537
+ postprocessor: chain[0],
538
+ postprocessors: chain,
500
539
  tasks: options.tasks,
501
540
  // When the originating render call passed
502
541
  // `save: false`, the layouts plugin wrote the
package/src/lifecycle.js CHANGED
@@ -300,17 +300,21 @@ export function onSync(name, callback) {
300
300
  export function onValidate(operations, callback) {
301
301
  const logger = useLogger()
302
302
  runtime.validators.push(async (entry) => {
303
- if (operations.indexOf(entry.operation) != -1) {
304
- try {
305
- const message = await callback(entry)
306
- if (message) {
307
- logger.warn('Validation problem: [%s] %s %s', entry.operation, entry.entity.name, message)
308
- }
309
- return true
310
- } catch (err) {
311
- logger.error('Validation error: [%s] %s %s', entry.operation, entry.entity.name, err.message)
312
- return false
303
+ // Abstain explicitly when this validator isn't scoped to the
304
+ // entry's operation — returning undefined here means "pass"
305
+ // under runtime.validate's abstain-is-pass contract. The early
306
+ // return makes that intent legible instead of relying on a
307
+ // fall-through.
308
+ if (operations.indexOf(entry.operation) === -1) return
309
+ try {
310
+ const message = await callback(entry)
311
+ if (message) {
312
+ logger.warn('Validation problem: [%s] %s %s', entry.operation, entry.entity.name, message)
313
313
  }
314
+ return true
315
+ } catch (err) {
316
+ logger.error('Validation error: [%s] %s %s', entry.operation, entry.entity.name, err.message)
317
+ return false
314
318
  }
315
319
  })
316
320
  }
package/src/logger.js CHANGED
@@ -37,6 +37,31 @@ import { onLoad } from './lifecycle.js'
37
37
  // want visibility above info noise but aren't warnings.
38
38
  const CUSTOM_LEVELS = { notice: 35 }
39
39
 
40
+ // Plugin-side log transport registry — paired with `addLogTransport`
41
+ // below. Two-state lifecycle:
42
+ //
43
+ // - Before createMikserLogger() runs (the plugin factory phase, when
44
+ // mikser.config.js is being import()'d and the factories embedded
45
+ // in `plugins: []` are calling addLogTransport synchronously), the
46
+ // entries land in pendingTransports. createMikserLogger drains it
47
+ // alongside the declarative runtime.config.logging.transports when
48
+ // it builds the multistream.
49
+ //
50
+ // - After createMikserLogger() has run, `currentStreams` and
51
+ // `currentLevel` are populated. Any later addLogTransport call
52
+ // (e.g. from a plugin's onLoaded hook, or a deferred integration)
53
+ // builds the new stream, pushes it onto currentStreams, and swaps
54
+ // runtime.engine.logger to a fresh pino instance backed by the
55
+ // updated multistream. useLogger() reads runtime.engine.logger
56
+ // fresh on each call, so the new transport starts receiving
57
+ // records immediately for every subsequent log call.
58
+ //
59
+ // The shape `{ level?, target, options? }` mirrors what
60
+ // runtime.config.logging.transports already accepts.
61
+ const pendingTransports = []
62
+ let currentStreams = null
63
+ let currentLevel = 'info'
64
+
40
65
  // Per-level icons prepended via pino-pretty's messageFormat. Empty
41
66
  // strings for levels we don't want to decorate — debug/trace are noisy
42
67
  // enough already. messageFormat receives `log.level` as a number, so we
@@ -135,22 +160,26 @@ export function createMikserLogger(level = 'info') {
135
160
 
136
161
  const streams = [{ level, stream: prettyStream }]
137
162
 
138
- // Mikser-config-driven third-party transports. Each entry is
139
- // `{ level?, target, options? }` — same shape pino.transport()
140
- // accepts. The transport spawns a worker; if its module fails to
141
- // load (missing dep, bad config), we surface it to stderr and keep
142
- // the terminal stream alive rather than crashing the engine.
143
- const transports = runtime.config?.logging?.transports ?? []
144
- for (const t of transports) {
145
- try {
146
- const stream = pino.transport({ target: t.target, options: t.options })
147
- streams.push({ level: t.level ?? level, stream })
148
- } catch (err) {
149
- process.stderr.write(
150
- `Logger: failed to load transport "${t.target}": ${err.message}\n`
151
- )
152
- }
163
+ // Two sources of transports merged into one list:
164
+ //
165
+ // - runtime.config.logging.transports (declarative user-config-
166
+ // driven). The historical surface; still works unchanged.
167
+ // - pendingTransports (plugin-side, drained here). Plugin factories
168
+ // that called addLogTransport at config-load time land here.
169
+ //
170
+ // Each entry is `{ level?, target, options? }` — same shape
171
+ // pino.transport() accepts. Transport workers that fail to load
172
+ // (missing dep, bad config) surface to stderr but don't crash the
173
+ // engine the terminal stream stays alive.
174
+ const declared = runtime.config?.logging?.transports ?? []
175
+ for (const t of [...declared, ...pendingTransports]) {
176
+ const s = buildTransportStream(t, level)
177
+ if (s) streams.push(s)
153
178
  }
179
+ pendingTransports.length = 0
180
+
181
+ currentStreams = streams
182
+ currentLevel = level
154
183
 
155
184
  return pino(
156
185
  {
@@ -161,6 +190,62 @@ export function createMikserLogger(level = 'info') {
161
190
  )
162
191
  }
163
192
 
193
+ // Construct a multistream entry for a transport descriptor. Returns
194
+ // null when the underlying pino.transport() call throws (missing
195
+ // dependency, bad options) — caller treats null as "skip".
196
+ function buildTransportStream(entry, defaultLevel) {
197
+ try {
198
+ return {
199
+ level: entry.level ?? defaultLevel,
200
+ stream: pino.transport({ target: entry.target, options: entry.options }),
201
+ }
202
+ } catch (err) {
203
+ process.stderr.write(
204
+ `Logger: failed to load transport "${entry.target}": ${err.message}\n`
205
+ )
206
+ return null
207
+ }
208
+ }
209
+
210
+ // Add a log transport from anywhere in mikser-io's lifecycle —
211
+ // typically from a plugin's factory (the canonical Better Stack /
212
+ // Datadog / Loki / Axiom / Sentry shape) or from a plugin's onLoaded
213
+ // hook for deferred / runtime-resolved integrations.
214
+ //
215
+ // Behavior depends on when this is called:
216
+ //
217
+ // - Before createMikserLogger() has run (factory phase during
218
+ // config.js's onLoad — plugin factories called inside the user's
219
+ // mikser.config.js evaluate here): the entry queues. The next
220
+ // createMikserLogger() call drains the queue.
221
+ //
222
+ // - After createMikserLogger() has run (plugin onLoaded, any later
223
+ // hook, runtime injection): the new pino.transport stream is
224
+ // built and pushed onto the live multistream, then runtime.engine
225
+ // .logger is swapped to a fresh pino backed by the updated list.
226
+ // useLogger() reads runtime.engine.logger on each call, so all
227
+ // subsequent log emissions reach the new transport.
228
+ //
229
+ // Returns true when the transport was queued or successfully added,
230
+ // false when transport stream construction failed (the load error
231
+ // already went to stderr via buildTransportStream).
232
+ export function addLogTransport(entry) {
233
+ if (currentStreams === null) {
234
+ pendingTransports.push(entry)
235
+ return true
236
+ }
237
+ const s = buildTransportStream(entry, currentLevel)
238
+ if (!s) return false
239
+ currentStreams.push(s)
240
+ if (runtime.engine) {
241
+ runtime.engine.logger = pino(
242
+ { level: 'trace', customLevels: CUSTOM_LEVELS },
243
+ pino.multistream(currentStreams),
244
+ )
245
+ }
246
+ return true
247
+ }
248
+
164
249
  // Replace the bootstrap logger (built by engine.setup() with the
165
250
  // terminal-only stream) with one that includes any third-party
166
251
  // transports from runtime.config.logging.transports. Runs at onLoad
@@ -176,7 +261,9 @@ export function createMikserLogger(level = 'info') {
176
261
  // to closures that have already executed.
177
262
  onLoad(() => {
178
263
  if (!runtime.engine?.logger) return
179
- if (!runtime.config?.logging?.transports?.length) return
264
+ const haveDeclared = (runtime.config?.logging?.transports?.length ?? 0) > 0
265
+ const havePending = pendingTransports.length > 0
266
+ if (!haveDeclared && !havePending) return
180
267
  const level = runtime.engine.logger.level
181
268
  runtime.engine.logger = createMikserLogger(level)
182
269
  })