mikser-io 9.0.0 → 9.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/10.0-PLAN.md +1 -1
- package/CLAUDE.md +207 -20
- package/README.md +23 -0
- package/index.js +1 -1
- package/package.json +4 -3
- package/src/engine.js +50 -11
- package/src/lifecycle.js +14 -10
- package/src/logger.js +103 -16
- package/src/plugins/api.js +8 -5
- package/src/plugins/preview.js +6 -5
- package/src/plugins/providers/http.js +179 -0
- package/src/plugins/render/hbs.js +111 -0
- package/src/postprocess.js +161 -20
- package/src/runtime.js +9 -1
- package/src/server.js +7 -1
- package/src/source.js +44 -12
- package/src/utils.js +116 -31
- package/9.0-PLAN.md +0 -438
- package/src/plugins/layouts.js +0 -818
package/10.0-PLAN.md
CHANGED
|
@@ -37,7 +37,7 @@ When the working signals dominate over a sustained period of real use, v10 desig
|
|
|
37
37
|
|
|
38
38
|
## Why v10 (not 9.x)
|
|
39
39
|
|
|
40
|
-
The 9.0
|
|
40
|
+
The 9.0 design deliberately deferred mikser-to-mikser composition to v10. The reasoning:
|
|
41
41
|
|
|
42
42
|
> Cross-instance concerns (discovery, peer identity, routing, cross-mikser refs, watch propagation, liveness, partition behavior, peer auth) belong to a distributed-systems substrate that iterates at "we hit a production partition bug" speed, not at engine-substrate speed. Folding any of it into `mikser-io` would re-fail ADR-0006 test #5 (release cadence) the moment it ships.
|
|
43
43
|
|
package/CLAUDE.md
CHANGED
|
@@ -10,22 +10,51 @@ paths. No migration markdown. No `task: pool` legacy aliasing for
|
|
|
10
10
|
`task: inline`. Update READMEs and ADRs in place as source-of-truth
|
|
11
11
|
changes; rewrite, don't supersede. The freedom is the point.
|
|
12
12
|
|
|
13
|
-
**v9
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
13
|
+
**v9 shipped. The gate now is daily-use payoff, not completion.**
|
|
14
|
+
The engine, every sibling plugin, the agency template — all live.
|
|
15
|
+
The next step isn't "finish v9," it's **use v9 in real daily work
|
|
16
|
+
until the workflow earns the v10 layer**. The v10 architecture
|
|
17
|
+
(party-mikser-io) is more fun to design than v9 dogfooding is to
|
|
18
|
+
sustain — the gate exists because that asymmetry kills projects.
|
|
19
|
+
Concrete pass/fail signals in `10.0-PLAN.md` under "The gate." If
|
|
20
|
+
a session brings up v10 design work, check the gate first; if v9
|
|
21
|
+
isn't paying out yet, the question to answer is "what about v9
|
|
22
|
+
isn't working in daily use?" — not "what should v10 look like?"
|
|
23
|
+
|
|
24
|
+
**mikser is a mixer.** The name is the architecture. It combines
|
|
25
|
+
inputs (entities — from files, forms, queries, anywhere) through
|
|
26
|
+
configurable rendering pipelines (renderer plugins + task-production
|
|
27
|
+
policies) into outputs of any kind. Static-site generation is the
|
|
28
|
+
canonical recipe — what mikser ships with and what most projects
|
|
29
|
+
use today — but it's a recipe, not the definition. The same
|
|
30
|
+
substrate runs:
|
|
31
|
+
|
|
32
|
+
- HTML SSG via `layouts` + `renderHbs` (the recipe everything else
|
|
33
|
+
is currently shaped around)
|
|
34
|
+
- PDFs via `renderHbs` + `post-pdf` (puppeteer-driven)
|
|
35
|
+
- Emails via `renderHbs` → `post-mjml` → `post-email` (chain)
|
|
36
|
+
- 3D renders by composing a hypothetical `renderBlender` against
|
|
37
|
+
`.blend` entities — same engine, different renderer + recipe
|
|
38
|
+
- Video / audio / archives / data-export — same shape
|
|
39
|
+
|
|
40
|
+
The engine has no opinion on what a renderer produces. `render.js`
|
|
41
|
+
and `postprocess.js` are abstract worker entries; `manifest.js`
|
|
42
|
+
tracks snapshots of *any* output; refs invalidation re-runs
|
|
43
|
+
*any* render task. SSG is the dominant case in v9 because it's
|
|
44
|
+
where the dogfooding lives, not because the architecture only
|
|
45
|
+
fits there. When proposing changes, hold the broader view: a
|
|
46
|
+
change that helps SSG but blocks the Blender/video/audio/archive
|
|
47
|
+
cases is constraining the engine's actual reach.
|
|
48
|
+
|
|
49
|
+
**Position mikser by what it is, not by speed.** Hugo wins the
|
|
50
|
+
speed race for SSG specifically; mikser doesn't compete there
|
|
51
|
+
(and "fastest mixer" isn't a useful claim because the bottleneck
|
|
52
|
+
is the renderer, not the engine). Secondary attributes that
|
|
53
|
+
follow from the mixer framing: AI-native, lifecycle-observable,
|
|
54
|
+
files-as-source-of-truth, full-cycle introspection. Speed claim
|
|
55
|
+
cap: "fast enough that watch-mode rebuilds feel instant." Honest
|
|
56
|
+
range: 200–800 docs/sec on the SSG-flavor pipeline depending on
|
|
57
|
+
corpus size (see `test/perf/`).
|
|
29
58
|
|
|
30
59
|
**Direct critique preferred.** Skip "great question," skip "solid but,"
|
|
31
60
|
skip softening. When something didn't pay off, say so. Match user
|
|
@@ -100,7 +129,16 @@ brevity.
|
|
|
100
129
|
plugins to push onto), late-binding static mount + listen.
|
|
101
130
|
- `logger.js` — pino + pino-pretty (inline) + gauge progress + custom
|
|
102
131
|
Writable for progress coordination + `pino.multistream` for
|
|
103
|
-
third-party shipping
|
|
132
|
+
third-party shipping. Two registration surfaces for transports:
|
|
133
|
+
- `runtime.config.logging.transports` — declarative; user config.
|
|
134
|
+
- `addLogTransport({ target, options?, level? })` — public export;
|
|
135
|
+
plugins call this from their factory (queues until logger build)
|
|
136
|
+
or any later hook (live-rebuilds the multistream + swaps
|
|
137
|
+
`runtime.engine.logger`). `useLogger()` reads the swap-target
|
|
138
|
+
fresh on each call so new transports start receiving records
|
|
139
|
+
immediately. Both surfaces compose; same `{level, target, options}`
|
|
140
|
+
shape. Enables Better Stack / Datadog / Loki / Axiom / Sentry as
|
|
141
|
+
standard sibling plugins without engine changes per vendor.
|
|
104
142
|
- `utils.js` — shared pure helpers: `mimeForEntity`, `isLoopback`,
|
|
105
143
|
`expandEntity`, `projectMeta`, `useCollection`, `useRenderer` (via
|
|
106
144
|
render.js), `isTextEntity`, `readEntityContent`, `extractRefs`,
|
|
@@ -141,11 +179,35 @@ brevity.
|
|
|
141
179
|
POSTPROCESS), `ACTION` (sync action types), `TASKS` (`INLINE`/
|
|
142
180
|
`SERIAL`/`WORKER` — dispatch modes).
|
|
143
181
|
|
|
182
|
+
## Canonical sibling plugins
|
|
183
|
+
|
|
184
|
+
These ship as separate repos but most projects pull them in. They're
|
|
185
|
+
not bundled because the engine is renderer-agnostic — each plugin
|
|
186
|
+
encodes a particular recipe (SSG, MJML, PDF, email) rather than
|
|
187
|
+
substrate.
|
|
188
|
+
|
|
189
|
+
- **`mikser-io-layouts`** — the canonical SSG-flavor task-production
|
|
190
|
+
policy. Owns multi-match layout assignment (`meta.layout` /
|
|
191
|
+
`meta.layouts` dual key + `autoLayouts` peel ladder), per-layout
|
|
192
|
+
destination Handlebars templates, collision detection, postprocess
|
|
193
|
+
chain parsing (filename `name.html-mjml-email.hbs` → `postprocessors:
|
|
194
|
+
['mjml','email']` + frontmatter `postprocessor`/`postprocessors`),
|
|
195
|
+
`inspect()` primitive at `runtime.options.layouts.inspect`. The
|
|
196
|
+
engine has no opinion on layouts; this plugin is one way of producing
|
|
197
|
+
render tasks. Other domains (3D rendering, video) would have their
|
|
198
|
+
own task-production plugins.
|
|
199
|
+
- **`mikser-io-render-{hbs,liquid,eta,markdown,metatext}`** — renderer
|
|
200
|
+
plugins. `renderHbs` is bundled in core; the rest are siblings.
|
|
201
|
+
- **`mikser-io-post-{mjml,pdf,email}`** — postprocessor plugins,
|
|
202
|
+
composable in chains (see `documentation/rendering.md#postprocess`).
|
|
203
|
+
- **`mikser-io-mcp`** — AI tooling surface (ADR-0006 test #5).
|
|
204
|
+
- **`mikser-io-vector`** — vector index over the catalog.
|
|
205
|
+
- **`mikser-io-forms`** — HTTP form receivers → entities.
|
|
206
|
+
- **`mikser-io-schemas`** — TypeScript type generation from entity meta.
|
|
207
|
+
|
|
144
208
|
## Plugin map (`src/plugins/`)
|
|
145
209
|
|
|
146
210
|
- `documents` — file→entity sync for the documents collection
|
|
147
|
-
- `layouts` — layout matching + sitemap + `inspect()` primitive
|
|
148
|
-
(exposed at `runtime.options.layouts.inspect`)
|
|
149
211
|
- `files` — file→entity sync for the files collection
|
|
150
212
|
- `assets` — asset references and copy
|
|
151
213
|
- `resources` — resource references
|
|
@@ -160,11 +222,75 @@ brevity.
|
|
|
160
222
|
- `observer` / `mapper` / `validator` / `commands` / `shares` —
|
|
161
223
|
utility plugins
|
|
162
224
|
|
|
225
|
+
## Built-in content providers (`src/plugins/providers/`)
|
|
226
|
+
|
|
227
|
+
The substrate-side `readEntityContent(entity)` dispatches by URI
|
|
228
|
+
scheme. Two schemes are handled inline by built-in providers
|
|
229
|
+
(no separate package required, ships with mikser-io):
|
|
230
|
+
|
|
231
|
+
- **No scheme / plain path / `file://`** — filesystem read. Lives
|
|
232
|
+
inline in `src/utils.js readEntityContent` because the engine
|
|
233
|
+
already touches the filesystem everywhere.
|
|
234
|
+
- **`http://` / `https://`** — `src/plugins/providers/http.js`.
|
|
235
|
+
Conditional GET with ETag + Last-Modified, in-memory response
|
|
236
|
+
cache, inflight coalescing on the same URL, binary mirror to
|
|
237
|
+
`runtime/http-cache/<sha-of-url>.<ext>`, operator-supplied
|
|
238
|
+
headers via `entity.meta.httpHeaders`, configurable timeout via
|
|
239
|
+
`entity.meta.httpTimeoutMs`. Test affordance:
|
|
240
|
+
`__resetHttpCacheForTests()` clears the module-level caches
|
|
241
|
+
between unit tests.
|
|
242
|
+
|
|
243
|
+
Any other scheme (`gdrive://`, `notion://`, `s3://`, …) routes
|
|
244
|
+
through the dynamic-import-by-name path — `mikser-io-provider-<scheme>`
|
|
245
|
+
must be installed as a sibling package and export a top-level
|
|
246
|
+
`read(entity)` function. See ADR-0010 / the gdrive provider for
|
|
247
|
+
the convention.
|
|
248
|
+
|
|
249
|
+
## Multi-emitter collections (sweep ownership)
|
|
250
|
+
|
|
251
|
+
Collections like `documents` / `files` / `assets` aren't owned by a
|
|
252
|
+
single plugin. The file source (via `useSource`) emits into them
|
|
253
|
+
from its scanned folder, but other plugins legitimately emit into
|
|
254
|
+
the same collection: `mikser-io-csv` fans CSV rows into `documents`,
|
|
255
|
+
a gdrive sync would emit there too, an API endpoint can push into
|
|
256
|
+
`files`. The catalog has no opinion on emitter identity — only on
|
|
257
|
+
collection.
|
|
258
|
+
|
|
259
|
+
The delete sweep MUST respect that. `sweepDeleted(collection,
|
|
260
|
+
scanned, onDelete, ownerPrefix)` requires `ownerPrefix` (typically
|
|
261
|
+
`absFolder` for file sources, `layoutsFolder` for layouts) and only
|
|
262
|
+
considers entities whose `entity.uri` is rooted under that prefix.
|
|
263
|
+
Foreign-emitter entities have a different `uri` shape:
|
|
264
|
+
- CSV row entities: empty `uri` (synthetic — meta is the content)
|
|
265
|
+
- HTTP-fetched CSV parent: `uri = 'https://…'`
|
|
266
|
+
- gdrive-sourced docs: `uri = 'gdrive://…'`
|
|
267
|
+
|
|
268
|
+
…and the LIKE clause excludes them. Without the scope, every cycle's
|
|
269
|
+
file-source sweep silently wipes every co-collection emitter's
|
|
270
|
+
entities. Throwing instead of silently sweeping the whole collection
|
|
271
|
+
is intentional — this used to be a class-of-bugs landmine. If you're
|
|
272
|
+
writing a new source-shaped plugin, the scoping is mandatory.
|
|
273
|
+
|
|
274
|
+
Test coverage: `test/unit/source-sweep.test.js`.
|
|
275
|
+
|
|
163
276
|
## Naming conventions
|
|
164
277
|
|
|
165
278
|
- **Engine state** at `runtime.<name>` — `runtime.refs`, `runtime.catalog`.
|
|
166
279
|
- **Plugin surfaces** at `runtime.options.<plugin>` —
|
|
167
280
|
`runtime.options.preview`, `runtime.options.layouts.inspect`.
|
|
281
|
+
- **Reachability:** `runtime.options.url` is the engine's resolved
|
|
282
|
+
public URL (from `--url` or `config.url`, trailing slash stripped,
|
|
283
|
+
validated). Plugins that need external reachability — webhook
|
|
284
|
+
receivers, absolute links in emails, MCP preview URLs returned to
|
|
285
|
+
agents, forms share links — read this. Standard gating pattern:
|
|
286
|
+
```js
|
|
287
|
+
const canPush = runtime.options.url?.startsWith('https://')
|
|
288
|
+
if (canPush) registerWebhookAt(`${runtime.options.url}/api/X/webhook`)
|
|
289
|
+
else setupPollingFallback()
|
|
290
|
+
```
|
|
291
|
+
`runtime.options.port` is the internal listener port (kept for
|
|
292
|
+
loopback/dev URL building); `runtime.options.url` is the external
|
|
293
|
+
origin. When in doubt: external = `url`, internal = `port`.
|
|
168
294
|
- **Engine functions** as module-level exports from `mikser-io`:
|
|
169
295
|
`import { queryEntities, subscribe, useRenderer, useCollection,
|
|
170
296
|
readEntityContent, isTextEntity } from 'mikser-io'`.
|
|
@@ -326,6 +452,22 @@ There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
|
|
|
326
452
|
- **New lifecycle hook?** Almost certainly no. Existing hooks cover
|
|
327
453
|
all known patterns. If you think you need one, post the use case
|
|
328
454
|
to ADR-0006 review.
|
|
455
|
+
- **New content source (gdrive, notion, s3, github, …)?** Two
|
|
456
|
+
things ship together:
|
|
457
|
+
- A regular lifecycle plugin (the sync — emits entities into the
|
|
458
|
+
catalog, sets `entity.uri = '<scheme>://...'` so the dispatch
|
|
459
|
+
knows where to route reads).
|
|
460
|
+
- A named export `read(entity)` from the same package, which
|
|
461
|
+
must be named `mikser-io-provider-<scheme>` so the engine can
|
|
462
|
+
find it. `readEntityContent(entity)` parses the scheme out of
|
|
463
|
+
`entity.uri`, dynamic-imports the package, calls its `read`.
|
|
464
|
+
No registry, no descriptor — same package-name convention as
|
|
465
|
+
`mikser-io-render-*` and `mikser-io-post-*`.
|
|
466
|
+
Built-in: plain local paths (no scheme) and `file://` URIs are
|
|
467
|
+
read via `fs.readFile` directly — no provider package needed for
|
|
468
|
+
the canonical local-filesystem case. `entity.content` already
|
|
469
|
+
populated is a fast-path that skips the dispatch entirely (for
|
|
470
|
+
small remote docs eager-fetched at sync time).
|
|
329
471
|
|
|
330
472
|
## Test suites
|
|
331
473
|
|
|
@@ -342,6 +484,48 @@ There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
|
|
|
342
484
|
+ clean-build timing). `SIZE=realistic` + entity count knobs
|
|
343
485
|
documented in **Perf**.
|
|
344
486
|
|
|
487
|
+
## Dev workspace
|
|
488
|
+
|
|
489
|
+
The siblings live side-by-side under `/Users/dick/Projects/mikser/`
|
|
490
|
+
and share an npm workspace declared at the parent's `package.json`:
|
|
491
|
+
|
|
492
|
+
```json
|
|
493
|
+
{
|
|
494
|
+
"private": true,
|
|
495
|
+
"workspaces": ["mikser-io", "mikser-io-*"]
|
|
496
|
+
}
|
|
497
|
+
```
|
|
498
|
+
|
|
499
|
+
`npm install` at that root hoists everything: each sibling's
|
|
500
|
+
`node_modules/mikser-io` (and any other cross-workspace dep)
|
|
501
|
+
becomes a symlink to the working copy. The whole tree runs
|
|
502
|
+
against ONE module instance of mikser-io.
|
|
503
|
+
|
|
504
|
+
**This isn't optional ergonomics — it's correctness.** Plugins
|
|
505
|
+
use `AsyncLocalStorage` (currently `queryContext` for sidecar
|
|
506
|
+
findEntities tracking; the schema-version meta could grow more).
|
|
507
|
+
Without workspace deduplication, npm 7+ auto-installs the peer
|
|
508
|
+
dep into each sibling's own `node_modules`. Layouts'
|
|
509
|
+
`import { queryContext } from 'mikser-io'` then resolves to its
|
|
510
|
+
bundled copy — a different AsyncLocalStorage instance than the
|
|
511
|
+
engine uses. Sidecar queries don't get tracked → no `query`
|
|
512
|
+
edges in `mikser_snapshots.refClosure` →
|
|
513
|
+
`manifest.queryAffected` returns empty → aggregate layouts
|
|
514
|
+
(index pages, sitemaps, RSS feeds) never invalidate when new
|
|
515
|
+
matching entities land. Silent broken-incremental.
|
|
516
|
+
|
|
517
|
+
`test/scenarios/_harness.js` also auto-coalesces (replaces any
|
|
518
|
+
real `mikser-io-layouts/node_modules/mikser-io` directory with a
|
|
519
|
+
symlink to MIKSER_ROOT on every `setupFixture`) as a backstop —
|
|
520
|
+
the scenarios pass with or without the workspace, but the
|
|
521
|
+
workspace is the canonical setup. Same shape applies to any
|
|
522
|
+
future sibling that uses `queryContext` or other module-level
|
|
523
|
+
state from mikser-io.
|
|
524
|
+
|
|
525
|
+
For production consumers (someone `npm install mikser-io
|
|
526
|
+
mikser-io-layouts` in their own project), npm resolves both from
|
|
527
|
+
the consumer's own project tree — no duplication, no bug.
|
|
528
|
+
|
|
345
529
|
## Reference
|
|
346
530
|
|
|
347
531
|
- `documentation/architecture.md` — module map (audit before relying
|
|
@@ -350,5 +534,8 @@ There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
|
|
|
350
534
|
- `documentation/configuration.md` — config reference
|
|
351
535
|
- `documentation/api-reference.md` — public API
|
|
352
536
|
- `test/perf/` — render-pipeline perf rig
|
|
353
|
-
- Sibling repos: `mikser-io-
|
|
537
|
+
- Sibling repos: `mikser-io-layouts` (canonical SSG-flavor render-task
|
|
538
|
+
production policy), `mikser-io-mcp`, `mikser-io-vector`,
|
|
539
|
+
`mikser-io-schemas`, `mikser-io-forms`, `mikser-io-post-{mjml,pdf,email}`,
|
|
540
|
+
`mikser-io-render-{eta,liquid,markdown,metatext}`,
|
|
354
541
|
`mikser-io-sdk-{api,react,svelte,vue,vector}`, `mikser-io-example-blog`
|
package/README.md
CHANGED
|
@@ -50,6 +50,28 @@ Build mikser into the parts of your application that are content-shaped. Keep th
|
|
|
50
50
|
|
|
51
51
|
**Open source.** MIT-licensed, on GitHub, no telemetry, no auth wall, no SaaS dependency. What you see is what runs.
|
|
52
52
|
|
|
53
|
+
## Mix content from anywhere, ship to anywhere
|
|
54
|
+
|
|
55
|
+
A real content stack pulls from more than one system. Marketing copy lives in a CMS or a spreadsheet. Prices and stock live in an ERP. Hero images live in a DAM. Editorial pages live in markdown files in the repo. Most teams either pick one tool and contort the rest to fit it, or build sync services that copy everything into one database — and then more sync services when things drift out of sync.
|
|
56
|
+
|
|
57
|
+
Mikser is built around a different bet: **one queryable substrate that any source can pour into, and any frontend can read from.**
|
|
58
|
+
|
|
59
|
+
**Any source.** Provider plugins let you treat external systems as content sources. A Google Sheet, an ERP feed, a Drive folder, a Notion database, an HTTP webhook, the local repo's `.md` files — each becomes a stream of entities flowing into mikser's catalog with the same shape. The authoring tool doesn't change. The editorial workflow doesn't change. The team writing product copy in Google Sheets keeps writing product copy in Google Sheets — mikser just notices when they save.
|
|
60
|
+
|
|
61
|
+
**Cross-source composition through references.** Mikser tracks references between entities the way a graph database does. A product page can declare `$marketing` pointing at a row pulled from a spreadsheet, `$pricing` at a record pulled from your ERP, `$hero` at an asset pulled from your DAM. A single API call from your frontend asks for the product page *with* its referenced data — and gets the page, the marketing copy, the price, and the hero image merged into one response. One round trip, not three or four, and no consumer-side join logic to maintain.
|
|
62
|
+
|
|
63
|
+
**Live updates are uniform across every source.** Edit a cell in the spreadsheet, change a price in the ERP, replace an asset in the DAM — the pages depending on those entities update within seconds, in every frontend connected to mikser's live channel. You don't write per-source invalidation; mikser already knows which pages reference what.
|
|
64
|
+
|
|
65
|
+
**Your frontend is whatever you want.** Mikser exposes the catalog over HTTP (and over MCP, for AI agents). You query it from React, Vue, Svelte, SvelteKit, Next.js, an iOS app, a kiosk — anything that speaks HTTP. The framework SDKs (`mikser-io-sdk-react`, `mikser-io-sdk-vue`, `mikser-io-sdk-svelte`) make the calls feel native, but they're optional. Headless CMSes free you from the database lock. Frameworks like Astro free you from one kind of authoring lock. Mikser frees you from both at once: **any source on the input side, any framework on the output side.**
|
|
66
|
+
|
|
67
|
+
What this looks like in practice:
|
|
68
|
+
|
|
69
|
+
- Marketing edits product copy in Google Sheets, prices come from your ERP, both flow into the same product page in your storefront — changes appear within seconds, without your e-commerce team writing webhook handlers or cache-invalidation logic.
|
|
70
|
+
- A SaaS pulls customer-facing release notes from `release-notes/*.md` in the repo and feature flags from LaunchDarkly; both render in a dashboard built in the team's own React stack, not a framework somebody else picked.
|
|
71
|
+
- A magazine pulls editorial articles from local markdown, contributor bios from Notion, sponsorship info from Airtable, and renders the same article surface across web, email (MJML), and PDF — one catalog, one set of references, three outputs.
|
|
72
|
+
|
|
73
|
+
Adding a source is mechanical. See [`mikser-io-csv`](https://github.com/almero-digital-marketing/mikser-io-csv) (any HTTP-served CSV, with live polling), [`mikser-io-provider-gdrive`](https://github.com/almero-digital-marketing/mikser-io-provider-gdrive) (Google Drive), and the built-in `http` provider for the shape. If a system has an API, it can be a mikser source, usually in well under 100 lines.
|
|
74
|
+
|
|
53
75
|
## Built for AI-assisted development
|
|
54
76
|
|
|
55
77
|
Files-as-source isn't just a portability story — it makes the project unusually friendly to AI coding agents. There's a static-time half (the agent reads your tree the way it reads any repo — no DB connection, no schema upload, no sandboxed query layer to learn) and a runtime half (when the agent needs to write or render, mikser ships its own MCP server in core, so it talks to the live engine instead of a parallel REST shim you have to maintain).
|
|
@@ -150,6 +172,7 @@ The engine is what stays stable — the lifecycle, the catalog, the file-based c
|
|
|
150
172
|
|---|---|
|
|
151
173
|
| [`mikser-io-vector`](https://github.com/almero-digital-marketing/mikser-io-vector) | OpenAI embeddings + semantic search (sqlite-vec or pgvector) |
|
|
152
174
|
| [`mikser-io-schemas`](https://github.com/almero-digital-marketing/mikser-io-schemas) | Zod-backed entity validation + auto-generated TypeScript declarations for the SDK. Auto-detects `$`-keyed references and warns on broken ones — see [ADR-0007](./documentation/decisions/0007-references-declaration-and-expansion.md) |
|
|
175
|
+
| [`mikser-io-forms`](https://github.com/almero-digital-marketing/mikser-io-forms) | Public form-submission endpoints. POST → captcha + schema validation → write a document file plus uploaded files to disk; the `documents` / `files` plugins pick them up via their normal watch loop. Composes with `mikser-io-schemas` for schema-by-name; built-in captcha providers (Google v2/v3, hCaptcha, Turnstile) plus a custom-verify escape hatch |
|
|
153
176
|
| [`mikser-io-archive`](https://github.com/almero-digital-marketing/mikser-io-archive) | Persist matching entities to YAML — audit trail, versioned content history, downstream export |
|
|
154
177
|
| `mapper` | Run config-supplied transforms over matched entities each cycle (in-core, generic transformation layer) |
|
|
155
178
|
| [`mikser-io-live`](https://github.com/almero-digital-marketing/mikser-io-live) | Lightweight dev server with browser auto-refresh — pair with `--watch` for the classic save→reload loop |
|
package/index.js
CHANGED
|
@@ -28,7 +28,7 @@ export { documents } from './src/plugins/documents.js'
|
|
|
28
28
|
export { files } from './src/plugins/files.js'
|
|
29
29
|
export { frontMatter } from './src/plugins/front-matter.js'
|
|
30
30
|
export { json } from './src/plugins/json.js'
|
|
31
|
-
|
|
31
|
+
// layouts moved to sibling: import from 'mikser-io-layouts'
|
|
32
32
|
export { mapper } from './src/plugins/mapper.js'
|
|
33
33
|
export { observer } from './src/plugins/observer.js'
|
|
34
34
|
export { preview } from './src/plugins/preview.js'
|
package/package.json
CHANGED
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "mikser-io",
|
|
3
|
-
"version": "9.0.
|
|
3
|
+
"version": "9.0.1",
|
|
4
4
|
"description": "<p align=\"center\"> <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
|
|
5
5
|
"main": "index.js",
|
|
6
6
|
"exports": {
|
|
7
|
-
".":
|
|
7
|
+
".": "./index.js",
|
|
8
8
|
"./package.json": "./package.json",
|
|
9
|
-
"./*":
|
|
9
|
+
"./*": "./*"
|
|
10
10
|
},
|
|
11
11
|
"scripts": {
|
|
12
12
|
"debug": "node --no-warnings app.js --server --watch --working-folder test/fixture",
|
|
@@ -61,6 +61,7 @@
|
|
|
61
61
|
"express": "^5.2.1"
|
|
62
62
|
},
|
|
63
63
|
"devDependencies": {
|
|
64
|
+
"@ai-sdk/openai": "^3.0.71",
|
|
64
65
|
"fluent-ffmpeg": "^2.1.3",
|
|
65
66
|
"mikser-io-decap": "file:../mikser-io-decap",
|
|
66
67
|
"mikser-io-post-mjml": "file:../mikser-io-post-mjml",
|
package/src/engine.js
CHANGED
|
@@ -5,7 +5,7 @@ import { existsSync } from 'fs'
|
|
|
5
5
|
import _ from 'lodash'
|
|
6
6
|
import Piscina from 'piscina'
|
|
7
7
|
import runtime from './runtime.js'
|
|
8
|
-
import { onInitialize, onInitialized, onRender, onCancel, onCancelled, onFinalized, onLoaded, onBeforePostprocess, onPostprocess, postprocessEntities } from './lifecycle.js'
|
|
8
|
+
import { onInitialize, onInitialized, onLoad, onRender, onCancel, onCancelled, onFinalized, onLoaded, onBeforePostprocess, onPostprocess, postprocessEntities } from './lifecycle.js'
|
|
9
9
|
import { useJournal, updateEntry } from './journal.js'
|
|
10
10
|
import { globby } from 'globby'
|
|
11
11
|
import { OPERATION, TASKS } from './constants.js'
|
|
@@ -178,6 +178,39 @@ export async function setup(options) {
|
|
|
178
178
|
// bring-up logs come after the folder-info lines.
|
|
179
179
|
setupServer()
|
|
180
180
|
|
|
181
|
+
// Public URL resolution. CLI --url wins; otherwise read runtime.config.url
|
|
182
|
+
// populated by the user's mikser.config.js. Stamped on runtime.options.url
|
|
183
|
+
// (no trailing slash) for plugins to consume in their onLoaded.
|
|
184
|
+
//
|
|
185
|
+
// Plugins that need external reachability (webhook receivers, absolute
|
|
186
|
+
// links in emails, MCP preview URLs, forms share links) read this. The
|
|
187
|
+
// standard gating pattern is:
|
|
188
|
+
//
|
|
189
|
+
// const canPush = runtime.options.url?.startsWith('https://')
|
|
190
|
+
// if (canPush) registerWebhookAt(`${runtime.options.url}/api/X/webhook`)
|
|
191
|
+
// else setupPollingFallback()
|
|
192
|
+
//
|
|
193
|
+
// Runs at onLoad — after config.js fires its onLoad (which populates
|
|
194
|
+
// runtime.config) and before any plugin's onLoaded.
|
|
195
|
+
onLoad(async () => {
|
|
196
|
+
const logger = useLogger()
|
|
197
|
+
const cli = runtime.options.url
|
|
198
|
+
const cfg = runtime.config?.url
|
|
199
|
+
const raw = cli ?? cfg
|
|
200
|
+
if (raw == null || raw === '') {
|
|
201
|
+
runtime.options.url = undefined
|
|
202
|
+
return
|
|
203
|
+
}
|
|
204
|
+
try {
|
|
205
|
+
new URL(raw)
|
|
206
|
+
} catch (err) {
|
|
207
|
+
throw new Error(`Invalid --url / config.url value: ${raw} (${err.message})`)
|
|
208
|
+
}
|
|
209
|
+
runtime.options.url = String(raw).replace(/\/+$/, '')
|
|
210
|
+
logger.info('Public URL: %s%s', runtime.options.url,
|
|
211
|
+
runtime.options.url.startsWith('https://') ? ' (webhook-capable)' : ' (http; plugins will not enable push webhooks)')
|
|
212
|
+
})
|
|
213
|
+
|
|
181
214
|
onLoaded(async () => {
|
|
182
215
|
const logger = useLogger()
|
|
183
216
|
logger.debug(runtime.options, 'Mikser options')
|
|
@@ -479,15 +512,17 @@ export async function setup(options) {
|
|
|
479
512
|
|
|
480
513
|
const tasks = []
|
|
481
514
|
for await (const { entity, options, context, output } of useJournal('Queuing postprocess', [OPERATION.RENDER], signal)) {
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
//
|
|
489
|
-
//
|
|
490
|
-
|
|
515
|
+
// Read the postprocessor chain. layouts plugin carries
|
|
516
|
+
// it as `options.postprocessors` (array, normalized); the
|
|
517
|
+
// singular `options.postprocessor` is a back-compat alias
|
|
518
|
+
// for the head of the chain. Empty chain → no postprocess.
|
|
519
|
+
const chain = options.postprocessors ?? (options.postprocessor ? [options.postprocessor] : [])
|
|
520
|
+
if (output?.success && chain.length) {
|
|
521
|
+
// The FINAL destination extension is the last stage's
|
|
522
|
+
// `output:`. Intermediate stages' extensions are used
|
|
523
|
+
// only for scratch path naming inside src/postprocess.js.
|
|
524
|
+
const finalExt = await resolveOutputExt(chain[chain.length - 1])
|
|
525
|
+
const destination = changeExtension(entity.destination, finalExt)
|
|
491
526
|
|
|
492
527
|
tasks.push({
|
|
493
528
|
entity: {
|
|
@@ -496,7 +531,11 @@ export async function setup(options) {
|
|
|
496
531
|
destination
|
|
497
532
|
},
|
|
498
533
|
options: {
|
|
499
|
-
|
|
534
|
+
// Singular kept for back-compat with the worker
|
|
535
|
+
// serialization shape; chain is the source of
|
|
536
|
+
// truth.
|
|
537
|
+
postprocessor: chain[0],
|
|
538
|
+
postprocessors: chain,
|
|
500
539
|
tasks: options.tasks,
|
|
501
540
|
// When the originating render call passed
|
|
502
541
|
// `save: false`, the layouts plugin wrote the
|
package/src/lifecycle.js
CHANGED
|
@@ -300,17 +300,21 @@ export function onSync(name, callback) {
|
|
|
300
300
|
export function onValidate(operations, callback) {
|
|
301
301
|
const logger = useLogger()
|
|
302
302
|
runtime.validators.push(async (entry) => {
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
303
|
+
// Abstain explicitly when this validator isn't scoped to the
|
|
304
|
+
// entry's operation — returning undefined here means "pass"
|
|
305
|
+
// under runtime.validate's abstain-is-pass contract. The early
|
|
306
|
+
// return makes that intent legible instead of relying on a
|
|
307
|
+
// fall-through.
|
|
308
|
+
if (operations.indexOf(entry.operation) === -1) return
|
|
309
|
+
try {
|
|
310
|
+
const message = await callback(entry)
|
|
311
|
+
if (message) {
|
|
312
|
+
logger.warn('Validation problem: [%s] %s %s', entry.operation, entry.entity.name, message)
|
|
313
313
|
}
|
|
314
|
+
return true
|
|
315
|
+
} catch (err) {
|
|
316
|
+
logger.error('Validation error: [%s] %s %s', entry.operation, entry.entity.name, err.message)
|
|
317
|
+
return false
|
|
314
318
|
}
|
|
315
319
|
})
|
|
316
320
|
}
|
package/src/logger.js
CHANGED
|
@@ -37,6 +37,31 @@ import { onLoad } from './lifecycle.js'
|
|
|
37
37
|
// want visibility above info noise but aren't warnings.
|
|
38
38
|
const CUSTOM_LEVELS = { notice: 35 }
|
|
39
39
|
|
|
40
|
+
// Plugin-side log transport registry — paired with `addLogTransport`
|
|
41
|
+
// below. Two-state lifecycle:
|
|
42
|
+
//
|
|
43
|
+
// - Before createMikserLogger() runs (the plugin factory phase, when
|
|
44
|
+
// mikser.config.js is being import()'d and the factories embedded
|
|
45
|
+
// in `plugins: []` are calling addLogTransport synchronously), the
|
|
46
|
+
// entries land in pendingTransports. createMikserLogger drains it
|
|
47
|
+
// alongside the declarative runtime.config.logging.transports when
|
|
48
|
+
// it builds the multistream.
|
|
49
|
+
//
|
|
50
|
+
// - After createMikserLogger() has run, `currentStreams` and
|
|
51
|
+
// `currentLevel` are populated. Any later addLogTransport call
|
|
52
|
+
// (e.g. from a plugin's onLoaded hook, or a deferred integration)
|
|
53
|
+
// builds the new stream, pushes it onto currentStreams, and swaps
|
|
54
|
+
// runtime.engine.logger to a fresh pino instance backed by the
|
|
55
|
+
// updated multistream. useLogger() reads runtime.engine.logger
|
|
56
|
+
// fresh on each call, so the new transport starts receiving
|
|
57
|
+
// records immediately for every subsequent log call.
|
|
58
|
+
//
|
|
59
|
+
// The shape `{ level?, target, options? }` mirrors what
|
|
60
|
+
// runtime.config.logging.transports already accepts.
|
|
61
|
+
const pendingTransports = []
|
|
62
|
+
let currentStreams = null
|
|
63
|
+
let currentLevel = 'info'
|
|
64
|
+
|
|
40
65
|
// Per-level icons prepended via pino-pretty's messageFormat. Empty
|
|
41
66
|
// strings for levels we don't want to decorate — debug/trace are noisy
|
|
42
67
|
// enough already. messageFormat receives `log.level` as a number, so we
|
|
@@ -135,22 +160,26 @@ export function createMikserLogger(level = 'info') {
|
|
|
135
160
|
|
|
136
161
|
const streams = [{ level, stream: prettyStream }]
|
|
137
162
|
|
|
138
|
-
//
|
|
139
|
-
//
|
|
140
|
-
//
|
|
141
|
-
//
|
|
142
|
-
//
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
163
|
+
// Two sources of transports merged into one list:
|
|
164
|
+
//
|
|
165
|
+
// - runtime.config.logging.transports (declarative — user-config-
|
|
166
|
+
// driven). The historical surface; still works unchanged.
|
|
167
|
+
// - pendingTransports (plugin-side, drained here). Plugin factories
|
|
168
|
+
// that called addLogTransport at config-load time land here.
|
|
169
|
+
//
|
|
170
|
+
// Each entry is `{ level?, target, options? }` — same shape
|
|
171
|
+
// pino.transport() accepts. Transport workers that fail to load
|
|
172
|
+
// (missing dep, bad config) surface to stderr but don't crash the
|
|
173
|
+
// engine — the terminal stream stays alive.
|
|
174
|
+
const declared = runtime.config?.logging?.transports ?? []
|
|
175
|
+
for (const t of [...declared, ...pendingTransports]) {
|
|
176
|
+
const s = buildTransportStream(t, level)
|
|
177
|
+
if (s) streams.push(s)
|
|
153
178
|
}
|
|
179
|
+
pendingTransports.length = 0
|
|
180
|
+
|
|
181
|
+
currentStreams = streams
|
|
182
|
+
currentLevel = level
|
|
154
183
|
|
|
155
184
|
return pino(
|
|
156
185
|
{
|
|
@@ -161,6 +190,62 @@ export function createMikserLogger(level = 'info') {
|
|
|
161
190
|
)
|
|
162
191
|
}
|
|
163
192
|
|
|
193
|
+
// Construct a multistream entry for a transport descriptor. Returns
|
|
194
|
+
// null when the underlying pino.transport() call throws (missing
|
|
195
|
+
// dependency, bad options) — caller treats null as "skip".
|
|
196
|
+
function buildTransportStream(entry, defaultLevel) {
|
|
197
|
+
try {
|
|
198
|
+
return {
|
|
199
|
+
level: entry.level ?? defaultLevel,
|
|
200
|
+
stream: pino.transport({ target: entry.target, options: entry.options }),
|
|
201
|
+
}
|
|
202
|
+
} catch (err) {
|
|
203
|
+
process.stderr.write(
|
|
204
|
+
`Logger: failed to load transport "${entry.target}": ${err.message}\n`
|
|
205
|
+
)
|
|
206
|
+
return null
|
|
207
|
+
}
|
|
208
|
+
}
|
|
209
|
+
|
|
210
|
+
// Add a log transport from anywhere in mikser-io's lifecycle —
|
|
211
|
+
// typically from a plugin's factory (the canonical Better Stack /
|
|
212
|
+
// Datadog / Loki / Axiom / Sentry shape) or from a plugin's onLoaded
|
|
213
|
+
// hook for deferred / runtime-resolved integrations.
|
|
214
|
+
//
|
|
215
|
+
// Behavior depends on when this is called:
|
|
216
|
+
//
|
|
217
|
+
// - Before createMikserLogger() has run (factory phase during
|
|
218
|
+
// config.js's onLoad — plugin factories called inside the user's
|
|
219
|
+
// mikser.config.js evaluate here): the entry queues. The next
|
|
220
|
+
// createMikserLogger() call drains the queue.
|
|
221
|
+
//
|
|
222
|
+
// - After createMikserLogger() has run (plugin onLoaded, any later
|
|
223
|
+
// hook, runtime injection): the new pino.transport stream is
|
|
224
|
+
// built and pushed onto the live multistream, then runtime.engine
|
|
225
|
+
// .logger is swapped to a fresh pino backed by the updated list.
|
|
226
|
+
// useLogger() reads runtime.engine.logger on each call, so all
|
|
227
|
+
// subsequent log emissions reach the new transport.
|
|
228
|
+
//
|
|
229
|
+
// Returns true when the transport was queued or successfully added,
|
|
230
|
+
// false when transport stream construction failed (the load error
|
|
231
|
+
// already went to stderr via buildTransportStream).
|
|
232
|
+
export function addLogTransport(entry) {
|
|
233
|
+
if (currentStreams === null) {
|
|
234
|
+
pendingTransports.push(entry)
|
|
235
|
+
return true
|
|
236
|
+
}
|
|
237
|
+
const s = buildTransportStream(entry, currentLevel)
|
|
238
|
+
if (!s) return false
|
|
239
|
+
currentStreams.push(s)
|
|
240
|
+
if (runtime.engine) {
|
|
241
|
+
runtime.engine.logger = pino(
|
|
242
|
+
{ level: 'trace', customLevels: CUSTOM_LEVELS },
|
|
243
|
+
pino.multistream(currentStreams),
|
|
244
|
+
)
|
|
245
|
+
}
|
|
246
|
+
return true
|
|
247
|
+
}
|
|
248
|
+
|
|
164
249
|
// Replace the bootstrap logger (built by engine.setup() with the
|
|
165
250
|
// terminal-only stream) with one that includes any third-party
|
|
166
251
|
// transports from runtime.config.logging.transports. Runs at onLoad
|
|
@@ -176,7 +261,9 @@ export function createMikserLogger(level = 'info') {
|
|
|
176
261
|
// to closures that have already executed.
|
|
177
262
|
onLoad(() => {
|
|
178
263
|
if (!runtime.engine?.logger) return
|
|
179
|
-
|
|
264
|
+
const haveDeclared = (runtime.config?.logging?.transports?.length ?? 0) > 0
|
|
265
|
+
const havePending = pendingTransports.length > 0
|
|
266
|
+
if (!haveDeclared && !havePending) return
|
|
180
267
|
const level = runtime.engine.logger.level
|
|
181
268
|
runtime.engine.logger = createMikserLogger(level)
|
|
182
269
|
})
|