mikser-io 9.0.0 → 9.0.3

package/10.0-PLAN.md

@@ -37,7 +37,7 @@ When the working signals dominate over a sustained period of real use, v10 desig
 
 ## Why v10 (not 9.x)
 
-The 9.0 plan deliberately deferred mikser-to-mikser composition to v10. The reasoning is in `9.0-PLAN.md` under "Out of scope":
+The 9.0 design deliberately deferred mikser-to-mikser composition to v10. The reasoning:
 
 > Cross-instance concerns (discovery, peer identity, routing, cross-mikser refs, watch propagation, liveness, partition behavior, peer auth) belong to a distributed-systems substrate that iterates at "we hit a production partition bug" speed, not at engine-substrate speed. Folding any of it into `mikser-io` would re-fail ADR-0006 test #5 (release cadence) the moment it ships.
 

package/CLAUDE.md

@@ -10,22 +10,51 @@ paths. No migration markdown. No `task: pool` legacy aliasing for
 `task: inline`. Update READMEs and ADRs in place as source-of-truth
 changes; rewrite, don't supersede. The freedom is the point.
 
-**v9 has to pay out before v10 starts.** Not "ship v9 and start v10."
-**Use v9 in real daily work until the workflow pays out**, then
-v10. The v10 architecture (party-mikser-io) is more fun to design
-than v9 documentation is to write — the gate exists because that
-asymmetry kills projects. Concrete pass/fail signals in
-`10.0-PLAN.md` under "The gate." If a session brings up v10 design
-work, check the gate first; if v9 isn't paying out yet, the
-question to answer is "what about v9 isn't working in daily use?"
-— not "what should v10 look like?"
-
-**Position mikser by what it is, not by speed.** Hugo wins the speed
-race; mikser doesn't compete there. Position: AI-native, lifecycle-
-observable, files-as-source-of-truth, full-cycle introspection. Speed
-claim cap: "fast enough that watch-mode rebuilds feel instant."
-Honest range: 200–800 docs/sec depending on corpus size
-(see `test/perf/`).
+**v9 shipped. The gate now is daily-use payoff, not completion.**
+The engine, every sibling plugin, the agency template — all live.
+The next step isn't "finish v9," it's **use v9 in real daily work
+until the workflow earns the v10 layer**. The v10 architecture
+(party-mikser-io) is more fun to design than v9 dogfooding is to
+sustain — the gate exists because that asymmetry kills projects.
+Concrete pass/fail signals in `10.0-PLAN.md` under "The gate." If
+a session brings up v10 design work, check the gate first; if v9
+isn't paying out yet, the question to answer is "what about v9
+isn't working in daily use?" — not "what should v10 look like?"
+
+**mikser is a mixer.** The name is the architecture. It combines
+inputs (entities — from files, forms, queries, anywhere) through
+configurable rendering pipelines (renderer plugins + task-production
+policies) into outputs of any kind. Static-site generation is the
+canonical recipe — what mikser ships with and what most projects
+use today — but it's a recipe, not the definition. The same
+substrate runs:
+
+- HTML SSG via `layouts` + `renderHbs` (the recipe everything else
+  is currently shaped around)
+- PDFs via `renderHbs` + `post-pdf` (puppeteer-driven)
+- Emails via `renderHbs` → `post-mjml` → `post-email` (chain)
+- 3D renders by composing a hypothetical `renderBlender` against
+  `.blend` entities — same engine, different renderer + recipe
+- Video / audio / archives / data-export — same shape
+
+The engine has no opinion on what a renderer produces. `render.js`
+and `postprocess.js` are abstract worker entries; `manifest.js`
+tracks snapshots of *any* output; refs invalidation re-runs
+*any* render task. SSG is the dominant case in v9 because it's
+where the dogfooding lives, not because the architecture only
+fits there. When proposing changes, hold the broader view: a
+change that helps SSG but blocks the Blender/video/audio/archive
+cases is constraining the engine's actual reach.
+
+**Position mikser by what it is, not by speed.** Hugo wins the
+speed race for SSG specifically; mikser doesn't compete there
+(and "fastest mixer" isn't a useful claim because the bottleneck
+is the renderer, not the engine). Secondary attributes that
+follow from the mixer framing: AI-native, lifecycle-observable,
+files-as-source-of-truth, full-cycle introspection. Speed claim
+cap: "fast enough that watch-mode rebuilds feel instant." Honest
+range: 200–800 docs/sec on the SSG-flavor pipeline depending on
+corpus size (see `test/perf/`).
 
 **Direct critique preferred.** Skip "great question," skip "solid but,"
 skip softening. When something didn't pay off, say so. Match user
@@ -100,7 +129,16 @@ brevity.
   plugins to push onto), late-binding static mount + listen.
 - `logger.js` — pino + pino-pretty (inline) + gauge progress + custom
   Writable for progress coordination + `pino.multistream` for
-  third-party shipping (`runtime.config.logging.transports`).
+  third-party shipping. Two registration surfaces for transports:
+  - `runtime.config.logging.transports` — declarative; user config.
+  - `addLogTransport({ target, options?, level? })` — public export;
+    plugins call this from their factory (queues until logger build)
+    or any later hook (live-rebuilds the multistream + swaps
+    `runtime.engine.logger`). `useLogger()` reads the swap-target
+    fresh on each call so new transports start receiving records
+    immediately. Both surfaces compose; same `{level, target, options}`
+    shape. Enables Better Stack / Datadog / Loki / Axiom / Sentry as
+    standard sibling plugins without engine changes per vendor.
 - `utils.js` — shared pure helpers: `mimeForEntity`, `isLoopback`,
   `expandEntity`, `projectMeta`, `useCollection`, `useRenderer` (via
   render.js), `isTextEntity`, `readEntityContent`, `extractRefs`,
@@ -137,15 +175,52 @@ brevity.
   Strings produce a v9 migration error pointing at the new shape.
 - `manager.js` — file watching (chokidar) and cron scheduling.
 - `source.js` — `useSource` codifies the folder-of-files pattern.
+- `routes.js` — HTTP route registry. Plugins mount on
+  `runtime.options.app` directly; the Express router stack has the
+  paths but not the intent (loopback-only? streaming?). So plugins
+  declare each mount via `registerRoute({ path, plugin, reachability,
+  streaming })` as they make it. `reachability` is `public` | `token`
+  | `loopback`; `streaming` flags SSE/WS routes a facade must not
+  buffer. Inventory lives at `runtime.routes`; consumers (a Caddy/nginx
+  facade generator, healthcheck list, `mikser://routes` resource) read
+  it — none baked in. `registerRoute` also folds in the origin/location
+  URL building and the standard `"<label> mounted: <loc>
+  [<reachability>]"` boot log that api/preview/mcp/vector/forms/decap
+  were each copy-pasting. Pure inventory — takes no position on what to
+  do with the routes.
 - `constants.js` — `OPERATION` (CREATE/UPDATE/DELETE/RENDER/
   POSTPROCESS), `ACTION` (sync action types), `TASKS` (`INLINE`/
   `SERIAL`/`WORKER` — dispatch modes).
 
+## Canonical sibling plugins
+
+These ship as separate repos but most projects pull them in. They're
+not bundled because the engine is renderer-agnostic — each plugin
+encodes a particular recipe (SSG, MJML, PDF, email) rather than
+substrate.
+
+- **`mikser-io-layouts`** — the canonical SSG-flavor task-production
+  policy. Owns multi-match layout assignment (`meta.layout` /
+  `meta.layouts` dual key + `autoLayouts` peel ladder), per-layout
+  destination Handlebars templates, collision detection, postprocess
+  chain parsing (filename `name.html-mjml-email.hbs` → `postprocessors:
+  ['mjml','email']` + frontmatter `postprocessor`/`postprocessors`),
+  `inspect()` primitive at `runtime.options.layouts.inspect`. The
+  engine has no opinion on layouts; this plugin is one way of producing
+  render tasks. Other domains (3D rendering, video) would have their
+  own task-production plugins.
+- **`mikser-io-render-{hbs,liquid,eta,markdown,metatext}`** — renderer
+  plugins. `renderHbs` is bundled in core; the rest are siblings.
+- **`mikser-io-post-{mjml,pdf,email}`** — postprocessor plugins,
+  composable in chains (see `docs/rendering.md#postprocess`).
+- **`mikser-io-mcp`** — AI tooling surface (ADR-0006 test #5).
+- **`mikser-io-vector`** — vector index over the catalog.
+- **`mikser-io-forms`** — HTTP form receivers → entities.
+- **`mikser-io-schemas`** — TypeScript type generation from entity meta.
+
 ## Plugin map (`src/plugins/`)
 
 - `documents` — file→entity sync for the documents collection
-- `layouts` — layout matching + sitemap + `inspect()` primitive
-  (exposed at `runtime.options.layouts.inspect`)
 - `files` — file→entity sync for the files collection
 - `assets` — asset references and copy
 - `resources` — resource references
@@ -160,11 +235,87 @@ brevity.
 - `observer` / `mapper` / `validator` / `commands` / `shares` —
   utility plugins
 
+## Built-in content providers (`src/plugins/providers/`)
+
+The substrate-side `readEntityContent(entity)` dispatches by URI
+scheme. Two schemes are handled inline by built-in providers
+(no separate package required, ships with mikser-io):
+
+- **No scheme / plain path / `file://`** — filesystem read. Lives
+  inline in `src/utils.js readEntityContent` because the engine
+  already touches the filesystem everywhere.
+- **`http://` / `https://`** — `src/plugins/providers/http.js`.
+  Conditional GET with ETag + Last-Modified, in-memory response
+  cache, inflight coalescing on the same URL, binary mirror to
+  `runtime/http-cache/<sha-of-url>.<ext>`, operator-supplied
+  headers via `entity.meta.httpHeaders`, configurable timeout via
+  `entity.meta.httpTimeoutMs`. Test affordance:
+  `__resetHttpCacheForTests()` clears the module-level caches
+  between unit tests.
+
+Any other scheme (`gdrive://`, `notion://`, `s3://`, …) routes
+through the dynamic-import-by-name path — `mikser-io-provider-<scheme>`
+must be installed as a sibling package and export a top-level
+`read(entity)` function. See ADR-0010 / the gdrive provider for
+the convention.
+
+## Multi-emitter collections (sweep ownership)
+
+Collections like `documents` / `files` / `assets` aren't owned by a
+single plugin. The file source (via `useSource`) emits into them
+from its scanned folder, but other plugins legitimately emit into
+the same collection: `mikser-io-csv` fans CSV rows into `documents`,
+a gdrive sync would emit there too, an API endpoint can push into
+`files`. The catalog has no opinion on emitter identity — only on
+collection.
+
+The delete sweep MUST respect that. `sweepDeleted(collection,
+scanned, onDelete, ownerPrefix)` requires `ownerPrefix` (typically
+`absFolder` for file sources, `layoutsFolder` for layouts) and only
+considers entities whose `entity.uri` is rooted under that prefix.
+Foreign-emitter entities have a different `uri` shape:
+- CSV row entities: empty `uri` (synthetic — meta is the content)
+- HTTP-fetched CSV parent: `uri = 'https://…'`
+- gdrive-sourced docs: `uri = 'gdrive://…'`
+
+…and the LIKE clause excludes them. Without the scope, every cycle's
+file-source sweep silently wipes every co-collection emitter's
+entities. Throwing instead of silently sweeping the whole collection
+is intentional — this used to be a class-of-bugs landmine. If you're
+writing a new source-shaped plugin, the scoping is mandatory.
+
+Test coverage: `test/unit/source-sweep.test.js`.
+
 ## Naming conventions
 
 - **Engine state** at `runtime.<name>` — `runtime.refs`, `runtime.catalog`.
 - **Plugin surfaces** at `runtime.options.<plugin>` —
   `runtime.options.preview`, `runtime.options.layouts.inspect`.
+- **Reachability:** `runtime.options.url` is the engine's resolved
+  public URL (from `--url` or `config.url`, trailing slash stripped,
+  validated). Plugins that need external reachability — webhook
+  receivers, absolute links in emails, MCP preview URLs returned to
+  agents, forms share links — read this. Standard gating pattern:
+  ```js
+  const canPush = runtime.options.url?.startsWith('https://')
+  if (canPush) registerWebhookAt(`${runtime.options.url}/api/X/webhook`)
+  else         setupPollingFallback()
+  ```
+  `runtime.options.port` is the internal listener port (kept for
+  loopback/dev URL building); `runtime.options.url` is the external
+  origin. When in doubt: external = `url`, internal = `port`.
+- **Trust proxy default is `'loopback'`** (server.js), not Express's
+  `false`. The dominant deployment puts a same-host reverse proxy in
+  front, where every socket peer is 127.0.0.1; with no trust, mikser
+  reads every proxied request as loopback and the loopback-only gate
+  (api/mcp/forms/decap enforce `isLoopback(req.ip)`) inverts the
+  moment a facade is added. `'loopback'` honors `X-Forwarded-For` only
+  from a loopback peer — safe everywhere (a remote attacker's kernel-
+  set peer is never 127.0.0.1, so they can't forge a loopback
+  `req.ip`). Override via `config.server.trustProxy` — `'uniquelocal'`
+  for a sibling-container proxy, a CIDR for a specific subnet, `false`
+  to opt out. The loopback-enforcement and this default are a pair; a
+  facade that injects `X-Forwarded-For` (Caddy/nginx do) makes it work.
 - **Engine functions** as module-level exports from `mikser-io`:
   `import { queryEntities, subscribe, useRenderer, useCollection,
   readEntityContent, isTextEntity } from 'mikser-io'`.
@@ -238,6 +389,26 @@ brevity.
   plugin config moved entirely off `runtime.config.<plugin>`; it
   arrives as factory args, gets stashed on the descriptor, and is
   passed as `config` to `load`/`render`/`setup`/`postprocess`.
+- **0011** (implemented; proven against gpoint) — File and resource
+  entities expose deployed URLs. References to served files
+  (image/video/PDF) are `$`-keyed **served paths** (`/img/X.jpg`,
+  `/media/clip.mp4` — the path content authors, = the entity's
+  `meta.url`), resolving via a new `refFilter` `{ 'meta.url': … }`
+  clause (utils.js) backed by an indexed `meta_url` column
+  (catalog.js + the `INDEXED_COLUMNS` map in sift-to-sql.js; schema
+  9.0.1 → 9.0.2). No collection-prefixed ids leak into content.
+  Id-refs were tried and rejected — gpoint references content by
+  served path everywhere, and its `/media/**` `resources()` library
+  means the entity only exists *because* content references
+  `/media/…`. The `files` plugin stamps `meta.url`; the `resources`
+  plugin stamps its library's served location; the `assets` plugin
+  stamps `meta.presets` in `onProcessed` via the pure `presetUrl()`
+  helper. Expanding a ref then yields the served entity's URL set,
+  not a string to reconstruct; `lookupUrl` (render.js) is the sync
+  Handlebars helper resolving a ref to `meta.url` or a named preset.
+  Base-relative in the live catalog, absolute in static renders
+  (from `runtime.options.url`). Umbrella term is *served entity*
+  (file/resource/preset).
 
 ## MCP
 
@@ -326,6 +497,22 @@ There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
 - **New lifecycle hook?** Almost certainly no. Existing hooks cover
   all known patterns. If you think you need one, post the use case
   to ADR-0006 review.
+- **New content source (gdrive, notion, s3, github, …)?** Two
+  things ship together:
+  - A regular lifecycle plugin (the sync — emits entities into the
+    catalog, sets `entity.uri = '<scheme>://...'` so the dispatch
+    knows where to route reads).
+  - A named export `read(entity)` from the same package, which
+    must be named `mikser-io-provider-<scheme>` so the engine can
+    find it. `readEntityContent(entity)` parses the scheme out of
+    `entity.uri`, dynamic-imports the package, calls its `read`.
+    No registry, no descriptor — same package-name convention as
+    `mikser-io-render-*` and `mikser-io-post-*`.
+  Built-in: plain local paths (no scheme) and `file://` URIs are
+  read via `fs.readFile` directly — no provider package needed for
+  the canonical local-filesystem case. `entity.content` already
+  populated is a fast-path that skips the dispatch entirely (for
+  small remote docs eager-fetched at sync time).
 
 ## Test suites
 
@@ -342,13 +529,58 @@ There is **no `--mcp` CLI flag**. Activation is plugin-presence only.
   + clean-build timing). `SIZE=realistic` + entity count knobs
   documented in **Perf**.
 
+## Dev workspace
+
+The siblings live side-by-side under `/Users/dick/Projects/mikser/`
+and share an npm workspace declared at the parent's `package.json`:
+
+```json
+{
+  "private": true,
+  "workspaces": ["mikser-io", "mikser-io-*"]
+}
+```
+
+`npm install` at that root hoists everything: each sibling's
+`node_modules/mikser-io` (and any other cross-workspace dep)
+becomes a symlink to the working copy. The whole tree runs
+against ONE module instance of mikser-io.
+
+**This isn't optional ergonomics — it's correctness.** Plugins
+use `AsyncLocalStorage` (currently `queryContext` for sidecar
+findEntities tracking; the schema-version meta could grow more).
+Without workspace deduplication, npm 7+ auto-installs the peer
+dep into each sibling's own `node_modules`. Layouts'
+`import { queryContext } from 'mikser-io'` then resolves to its
+bundled copy — a different AsyncLocalStorage instance than the
+engine uses. Sidecar queries don't get tracked → no `query`
+edges in `mikser_snapshots.refClosure` →
+`manifest.queryAffected` returns empty → aggregate layouts
+(index pages, sitemaps, RSS feeds) never invalidate when new
+matching entities land. Silent broken-incremental.
+
+`test/scenarios/_harness.js` also auto-coalesces (replaces any
+real `mikser-io-layouts/node_modules/mikser-io` directory with a
+symlink to MIKSER_ROOT on every `setupFixture`) as a backstop —
+the scenarios pass with or without the workspace, but the
+workspace is the canonical setup. Same shape applies to any
+future sibling that uses `queryContext` or other module-level
+state from mikser-io.
+
+For production consumers (someone `npm install mikser-io
+mikser-io-layouts` in their own project), npm resolves both from
+the consumer's own project tree — no duplication, no bug.
+
 ## Reference
 
-- `documentation/architecture.md` — module map (audit before relying
+- `docs/architecture.md` — module map (audit before relying
   on specifics; may have drift)
-- `documentation/decisions/` — ADRs
-- `documentation/configuration.md` — config reference
-- `documentation/api-reference.md` — public API
+- `docs/decisions/` — ADRs
+- `docs/configuration.md` — config reference
+- `docs/api-reference.md` — public API
 - `test/perf/` — render-pipeline perf rig
-- Sibling repos: `mikser-io-mcp`, `mikser-io-vector`, `mikser-io-schemas`,
+- Sibling repos: `mikser-io-layouts` (canonical SSG-flavor render-task
+  production policy), `mikser-io-mcp`, `mikser-io-vector`,
+  `mikser-io-schemas`, `mikser-io-forms`, `mikser-io-post-{mjml,pdf,email}`,
+  `mikser-io-render-{eta,liquid,markdown,metatext}`,
   `mikser-io-sdk-{api,react,svelte,vue,vector}`, `mikser-io-example-blog`

package/README.md

@@ -2,17 +2,21 @@
   <img src="mikser-lockup-stacked.svg" alt="mikser" width="198" />
 </p>
 
+<p align="center"><strong>The AI-native content engine.</strong></p>
+
 # Mikser
 
-**Mikser is the content layer of your application.** Business logic, user accounts, transactions live in their own services; mikser handles the parts that *are* content — pages, docs, the published catalog, multi-format outputs. [Vue](https://github.com/almero-digital-marketing/mikser-io-sdk-vue), [React](https://github.com/almero-digital-marketing/mikser-io-sdk-react), and [Svelte](https://github.com/almero-digital-marketing/mikser-io-sdk-svelte) SDKs are the seam between them — same surface across all three (`useDocument`, `useDocuments`, multilingual `useHref`, live SSE updates), each in its framework's idiomatic shape.
+**Mikser is the AI-native content engine.** It mixes content from anywhere — markdown files, Google Sheets, your ERP, a CMS, any API — into one live catalog, links it together with references, and ships it to any frontend. Edit a price in your ERP or a cell in a spreadsheet, and every page that uses it updates within seconds — in whatever framework you built the site with.
+
+**Framework-agnostic on both ends.** Headless CMSes (Sanity, Contentful) free you from the database but lock authoring into their UI. Frameworks like Astro let you bring any frontend but lock you into the framework. Mikser frees both: **any source in, any frontend out.** Provider plugins turn external systems into content sources; the catalog is served over plain HTTP, with idiomatic [Vue](https://github.com/almero-digital-marketing/mikser-io-sdk-vue) / [React](https://github.com/almero-digital-marketing/mikser-io-sdk-react) / [Svelte](https://github.com/almero-digital-marketing/mikser-io-sdk-svelte) SDKs on top (`useDocument`, `useDocuments`, multilingual `useHref`, live SSE). It's also AI-native: agents read and write the same catalog over MCP using the same calls a frontend developer uses — there's no separate "AI API" to keep in sync.
 
-**An AI-native file-based content engine with live graph queries.** Content lives as plain text files you can read, search, and version-control like code. Mikser knows how those files reference each other — who an article's author is, which images a landing page uses, who else mentions a given product. When you fetch an article, you can pull its author and that author's organization along with it in one round-trip. When the author's bio changes, every page subscribed to a reference touching them updates live, without polling. AI agents talk to mikser through the same calls a frontend developer uses — there's no separate "AI API" to keep in sync. The whole graph is queryable from both sides at once.
+**References merge sources into one call.** Mikser tracks how entities reference each other — an article's author, the marketing copy and ERP price behind a product page, the images a landing page uses. Fetch the product and pull its marketing copy, price, and hero image along with it in one round-trip. Change any of them and every page subscribed to a reference touching it re-renders live, without polling — and mikser invalidates *exactly* the pages that depend on it, nothing more.
 
-Built for Node.js around a strict lifecycle and a composable plugin system. Every document, asset, and template flows through the same pipeline; plugins hook in at any phase. The same engine runs a single markdown blog and a multi-language publishing platform with PDF / email / AI-augmented asset pipelines — same lifecycle, more plugins.
+**Your content stays yours.** Source files live on disk as `.md` / `.yml` — diffable, version-controllable, portable on day one and year ten. No database lock-in, no proprietary export. And it's the content layer, not your whole backend: business logic, accounts, and transactions stay in their own services; mikser handles the part that's actually content — rendered to HTML, PDF, email, and other formats from the same source.
 
-It's MIT-licensed, runs on Node, has zero hosted dependencies, and the entire content tree it manages is a folder of `.md` and `.yml` files you can copy, diff, and version-control. **The portability promise is the architecture, not a feature.**
+Built for Node.js around a strict lifecycle and a composable plugin system: every document, asset, and template flows through the same pipeline, and plugins hook in at any phase. MIT-licensed, runs on Node, zero hosted dependencies. **The portability promise is the architecture, not a feature.**
 
-> **New to mikser?** Read the [Architecture Overview](./documentation/overview.md) — one document, end-to-end walkthrough of how a file becomes a deployed page across all twenty lifecycle phases. It's the doc most projects need first.
+> **New to mikser?** Read the [Architecture Overview](./docs/overview.md) — one document, end-to-end walkthrough of how a file becomes a deployed page across all twenty lifecycle phases. It's the doc most projects need first.
 
 ## Where it fits
 
@@ -50,6 +54,28 @@ Build mikser into the parts of your application that are content-shaped. Keep th
 
 **Open source.** MIT-licensed, on GitHub, no telemetry, no auth wall, no SaaS dependency. What you see is what runs.
 
+## Mix content from anywhere, ship to anywhere
+
+A real content stack pulls from more than one system. Marketing copy lives in a CMS or a spreadsheet. Prices and stock live in an ERP. Hero images live in a DAM. Editorial pages live in markdown files in the repo. Most teams either pick one tool and contort the rest to fit it, or build sync services that copy everything into one database — and then more sync services when things drift out of sync.
+
+Mikser is built around a different bet: **one queryable substrate that any source can pour into, and any frontend can read from.**
+
+**Any source.** Provider plugins let you treat external systems as content sources. A Google Sheet, an ERP feed, a Drive folder, a Notion database, an HTTP webhook, the local repo's `.md` files — each becomes a stream of entities flowing into mikser's catalog with the same shape. The authoring tool doesn't change. The editorial workflow doesn't change. The team writing product copy in Google Sheets keeps writing product copy in Google Sheets — mikser just notices when they save.
+
+**Cross-source composition through references.** Mikser tracks references between entities the way a graph database does. A product page can declare `$marketing` pointing at a row pulled from a spreadsheet, `$pricing` at a record pulled from your ERP, `$hero` at an asset pulled from your DAM. A single API call from your frontend asks for the product page *with* its referenced data — and gets the page, the marketing copy, the price, and the hero image merged into one response. One round trip, not three or four, and no consumer-side join logic to maintain.
+
+**Live updates are uniform across every source.** Edit a cell in the spreadsheet, change a price in the ERP, replace an asset in the DAM — the pages depending on those entities update within seconds, in every frontend connected to mikser's live channel. You don't write per-source invalidation; mikser already knows which pages reference what.
+
+**Your frontend is whatever you want.** Mikser exposes the catalog over HTTP (and over MCP, for AI agents). You query it from React, Vue, Svelte, SvelteKit, Next.js, an iOS app, a kiosk — anything that speaks HTTP. The framework SDKs (`mikser-io-sdk-react`, `mikser-io-sdk-vue`, `mikser-io-sdk-svelte`) make the calls feel native, but they're optional. Headless CMSes free you from the database lock. Frameworks like Astro free you from one kind of authoring lock. Mikser frees you from both at once: **any source on the input side, any framework on the output side.**
+
+What this looks like in practice:
+
+- Marketing edits product copy in Google Sheets, prices come from your ERP, both flow into the same product page in your storefront — changes appear within seconds, without your e-commerce team writing webhook handlers or cache-invalidation logic.
+- A SaaS pulls customer-facing release notes from `release-notes/*.md` in the repo and feature flags from LaunchDarkly; both render in a dashboard built in the team's own React stack, not a framework somebody else picked.
+- A magazine pulls editorial articles from local markdown, contributor bios from Notion, sponsorship info from Airtable, and renders the same article surface across web, email (MJML), and PDF — one catalog, one set of references, three outputs.
+
+Adding a source is mechanical. See [`mikser-io-csv`](https://github.com/almero-digital-marketing/mikser-io-csv) (any HTTP-served CSV, with live polling), [`mikser-io-provider-gdrive`](https://github.com/almero-digital-marketing/mikser-io-provider-gdrive) (Google Drive), and the built-in `http` provider for the shape. If a system has an API, it can be a mikser source, usually in well under 100 lines.
+
 ## Built for AI-assisted development
 
 Files-as-source isn't just a portability story — it makes the project unusually friendly to AI coding agents. There's a static-time half (the agent reads your tree the way it reads any repo — no DB connection, no schema upload, no sandboxed query layer to learn) and a runtime half (when the agent needs to write or render, mikser ships its own MCP server in core, so it talks to the live engine instead of a parallel REST shim you have to maintain).
@@ -141,7 +167,7 @@ The engine is what stays stable — the lifecycle, the catalog, the file-based c
 | Plugin | What it does |
 |---|---|
 | `data` | JSON snapshots of entities / context / catalog, written to disk for static serving |
-| `api` | REST endpoints with sift-backed queries, per-endpoint tokens, optional render, opt-in [per-query disk cache](./documentation/caching.md) for reverse-proxy failover |
+| `api` | REST endpoints with sift-backed queries, per-endpoint tokens, optional render, opt-in [per-query disk cache](./docs/caching.md) for reverse-proxy failover |
 | `preview` | In-memory render cache + `GET /preview/:filename` route. Companion to the `mikser_preview_render` MCP tool (in [`mikser-io-mcp`](https://github.com/almero-digital-marketing/mikser-io-mcp)) — transient render bytes served at a clickable URL, no filesystem footprint |
 
 **Integrations:**
@@ -149,7 +175,8 @@ The engine is what stays stable — the lifecycle, the catalog, the file-based c
 | Plugin | What it does |
 |---|---|
 | [`mikser-io-vector`](https://github.com/almero-digital-marketing/mikser-io-vector) | OpenAI embeddings + semantic search (sqlite-vec or pgvector) |
-| [`mikser-io-schemas`](https://github.com/almero-digital-marketing/mikser-io-schemas) | Zod-backed entity validation + auto-generated TypeScript declarations for the SDK. Auto-detects `$`-keyed references and warns on broken ones — see [ADR-0007](./documentation/decisions/0007-references-declaration-and-expansion.md) |
+| [`mikser-io-schemas`](https://github.com/almero-digital-marketing/mikser-io-schemas) | Zod-backed entity validation + auto-generated TypeScript declarations for the SDK. Auto-detects `$`-keyed references and warns on broken ones — see [ADR-0007](./docs/decisions/0007-references-declaration-and-expansion.md) |
+| [`mikser-io-forms`](https://github.com/almero-digital-marketing/mikser-io-forms) | Public form-submission endpoints. POST → captcha + schema validation → write a document file plus uploaded files to disk; the `documents` / `files` plugins pick them up via their normal watch loop. Composes with `mikser-io-schemas` for schema-by-name; built-in captcha providers (Google v2/v3, hCaptcha, Turnstile) plus a custom-verify escape hatch |
 | [`mikser-io-archive`](https://github.com/almero-digital-marketing/mikser-io-archive) | Persist matching entities to YAML — audit trail, versioned content history, downstream export |
 | `mapper` | Run config-supplied transforms over matched entities each cycle (in-core, generic transformation layer) |
 | [`mikser-io-live`](https://github.com/almero-digital-marketing/mikser-io-live) | Lightweight dev server with browser auto-refresh — pair with `--watch` for the classic save→reload loop |
@@ -194,13 +221,13 @@ npx mikser --watch      # incremental dev loop
 npx mikser --server     # build + serve at :3001
 ```
 
-For a working starter — config with a real plugin set, sample `documents/`, expected output — see [Getting Started](./documentation/getting-started.md). Or skip straight to "add mikser to this app" via the [Claude Code plugin](#built-for-ai-assisted-development) above.
+For a working starter — config with a real plugin set, sample `documents/`, expected output — see [Getting Started](./docs/getting-started.md). Or skip straight to "add mikser to this app" via the [Claude Code plugin](#built-for-ai-assisted-development) above.
 
 ## Core Concepts
 
 - **Lifecycle** — Processing runs through fixed phases: initialize → load → import → process → persist → render → finalize. Plugins hook into any phase.
 - **Entities** — Everything is an entity (document, file, layout, asset). Entities flow through the journal and are tracked in the catalog.
-- **References between entities** — A front-matter key starting with `$` (e.g. `$author: /authors/dick`) points at another entity. The engine knows the whole graph: templates can follow the links, the schemas plugin checks they resolve, and a single query can pull referenced entities along inline instead of one round trip per link. See [ADR-0007](./documentation/decisions/0007-references-declaration-and-expansion.md).
+- **References between entities** — A front-matter key starting with `$` (e.g. `$author: /authors/dick`) points at another entity. The engine knows the whole graph: templates can follow the links, the schemas plugin checks they resolve, and a single query can pull referenced entities along inline instead of one round trip per link. See [ADR-0007](./docs/decisions/0007-references-declaration-and-expansion.md).
 - **Plugins** — Functionality is delivered via plugins. Built-in plugins handle common sources (documents, files, layouts, assets). Custom plugins can be added to any project.
 - **Runtime Singleton** — A plain module-level object holds all global state and coordinates the lifecycle. The ES module cache guarantees every importer gets the same instance.
 - **Watch Mode** — In watch mode, file changes trigger incremental re-processing without restarting.
@@ -216,16 +243,16 @@ The shape mikser fits cleanly:
 - **Mixed-output publishing** — the same source document renders to HTML, PDF (via `post-pdf`), MJML email (via `post-mjml`), and JSON snapshots. One catalog, many output formats, all concurrent.
 - **Headless backends for static frontends** — pair the `api` plugin with `sdk-api` for SSE-driven live frontends; pair the `data` plugin output with any static host for pre-rendered consumption.
 
-The shape mikser **doesn't** fit cleanly: anything with non-technical content authors who can't or won't work with files, anything with non-content business logic at the core, anything needing multi-tenant / per-user auth. Those aren't bugs — they're outside the design envelope. See [`decisions/0001-content-layer-not-the-app.md`](./documentation/decisions/0001-content-layer-not-the-app.md) for the explicit scope decision.
+The shape mikser **doesn't** fit cleanly: anything with non-technical content authors who can't or won't work with files, anything with non-content business logic at the core, anything needing multi-tenant / per-user auth. Those aren't bugs — they're outside the design envelope. See [`decisions/0001-content-layer-not-the-app.md`](./docs/decisions/0001-content-layer-not-the-app.md) for the explicit scope decision.
 
 ## Engineering discipline
 
 What you get from how this project is built:
 
-- **Every load-bearing decision has an ADR.** The [`decisions/`](./documentation/decisions/) folder names which choices are structural — files-as-source, journal+catalog split, plugin-as-factory, when something goes in core vs. ships as a plugin — and explains what protects them. When you push against one, there's a written answer waiting instead of folklore.
+- **Every load-bearing decision has an ADR.** The [`decisions/`](./docs/decisions/) folder names which choices are structural — files-as-source, journal+catalog split, plugin-as-factory, when something goes in core vs. ships as a plugin — and explains what protects them. When you push against one, there's a written answer waiting instead of folklore.
 - **Engine stays small; capability ships in plugins.** The 15+ plugin ecosystem adds features without core changes, so your upgrade cost stays low. Probes like `decap` (a full third-party CMS mounted in ~150 lines, zero engine changes) are deliberate evidence the extension model holds where it counts.
 - **Builds are deterministic; no async middleware layer.** The journal is the only synchronization primitive — no event bus, no IoC container, no orchestrator running plugins in surprising order. The lifecycle is a list of named phases; "what ran when?" has an answer you can read off the source.
-- **The whole engine is one read.** The [Architecture Overview](./documentation/overview.md) walks the full pipeline top to bottom. Onboarding a new engineer is an afternoon, not a tour through fifteen reference docs.
+- **The whole engine is one read.** The [Architecture Overview](./docs/overview.md) walks the full pipeline top to bottom. Onboarding a new engineer is an afternoon, not a tour through fifteen reference docs.
 
 ## Mikser among static site generators
 
@@ -251,19 +278,19 @@ Mikser itself has a previous chapter: the [legacy 7.x line](https://github.com/a
 
 | Document                                              | Audience           | Description                                        |
 | ----------------------------------------------------- | ------------------ | -------------------------------------------------- |
-| [Architecture Overview](./documentation/overview.md)  | Everyone           | **Start here.** End-to-end walkthrough of how a file becomes a deployed page across all lifecycle phases. |
-| [Getting Started](./documentation/getting-started.md) | Users              | Installation, first project, basic usage           |
-| [Configuration](./documentation/configuration.md)     | Users              | All CLI options and config file reference          |
-| [Lifecycle](./documentation/lifecycle.md)             | Users & Developers | Complete lifecycle phases and hook system          |
-| [Plugins](./documentation/plugins.md)                 | Users & Developers | Built-in plugins, writing custom plugins, the assets / resources / AI pipeline |
-| [Entities](./documentation/entities.md)               | Users & Developers | Entity model, operations, journal, catalog         |
-| [Rendering](./documentation/rendering.md)             | Users & Developers | Render pipeline, render plugins, render modes      |
-| [Watch Mode](./documentation/watch-mode.md)           | Users              | File watching, scheduled tasks, incremental builds |
+| [Architecture Overview](./docs/overview.md)  | Everyone           | **Start here.** End-to-end walkthrough of how a file becomes a deployed page across all lifecycle phases. |
+| [Getting Started](./docs/getting-started.md) | Users              | Installation, first project, basic usage           |
+| [Configuration](./docs/configuration.md)     | Users              | All CLI options and config file reference          |
+| [Lifecycle](./docs/lifecycle.md)             | Users & Developers | Complete lifecycle phases and hook system          |
+| [Plugins](./docs/plugins.md)                 | Users & Developers | Built-in plugins, writing custom plugins, the assets / resources / AI pipeline |
+| [Entities](./docs/entities.md)               | Users & Developers | Entity model, operations, journal, catalog         |
+| [Rendering](./docs/rendering.md)             | Users & Developers | Render pipeline, render plugins, render modes      |
+| [Watch Mode](./docs/watch-mode.md)           | Users              | File watching, scheduled tasks, incremental builds |
 | [MCP](https://github.com/almero-digital-marketing/mikser-io-mcp#readme) | Users              | The `mikser-io-mcp` plugin — tool surface, `mikser://` resources, twelve worked AI-driven scenarios |
-| [Caching](./documentation/caching.md)                 | Users (production) | The `cache: true` disk cache + working nginx config for reverse-proxy failover |
-| [Architecture](./documentation/architecture.md)       | Developers         | Module-level reference — what's in each file       |
-| [API Reference](./documentation/api-reference.md)     | Developers         | Complete public API reference                      |
-| [Decisions (ADRs)](./documentation/decisions/)        | Developers         | Load-bearing architectural choices and what protects them |
+| [Caching](./docs/caching.md)                 | Users (production) | The `cache: true` disk cache + working nginx config for reverse-proxy failover |
+| [Architecture](./docs/architecture.md)       | Developers         | Module-level reference — what's in each file       |
+| [API Reference](./docs/api-reference.md)     | Developers         | Complete public API reference                      |
+| [Decisions (ADRs)](./docs/decisions/)        | Developers         | Load-bearing architectural choices and what protects them |
 
 ## License