mikser-io 9.50.2 → 9.57.0

package/CLAUDE.md

@@ -137,8 +137,15 @@ brevity.
   descriptors in `runtime.options.plugins` are projected to their
   `render-${name}` / `post-${name}` identifier so workers can
   resolve them via dynamic import.
-- `database/` — `createSqliteDatabase()`, `registerSchema()`,
-  `useDatabase()` (the `mikser_meta` table stamps schema_version).
+- `database/index.js` — the CACHE only. `createSqliteDatabase()`,
+  `registerSchema()`, `useDatabase()` (the `mikser_meta` table stamps
+  schema_version). Nothing durable lives here any more.
+- `database/durable.js` — the durable store. `registerMigrations()`,
+  `useDurableDatabase()` (a knex instance), `runMigrations()`,
+  `closeDurableDatabase()`. `adoptFromCache` carries tables out of a
+  pre-9.56 database, driven by the `durable_tables` record the old design
+  kept in `mikser_meta` — names alone would move any cache table that
+  happened to collide. `ensureIgnored` adds the file to `.gitignore`.
   `sift-to-sql.js` translates sift filters to SQL WHERE clauses
   against `INDEXED_COLUMNS`; un-pushed clauses fall through to
   JS-side sift. `query-context.js` is the AsyncLocalStorage that
@@ -150,6 +157,12 @@ brevity.
 - `server.js` — Express bring-up: CLI flags (`--server`, `--cors`,
   `--no-cors`), trust-proxy, CORS (with extensible header arrays for
   plugins to push onto), late-binding static mount + listen.
+- `report.js` — the `--json` build report. `warnings` is a VIEW of
+  `logger.warn`; `faults` is a view of `logger.error` **carrying a
+  `code`** — a subsystem declaring it cannot work, deduped by that code,
+  never cleared per cycle, and surfaced in `mikser_ping`. The log call is
+  the only registration for either; there is no `reportWarning()` or
+  `registerFault()` and there must not be.
 - `logger.js` — pino + pino-pretty (inline) + gauge progress + custom
   Writable for progress coordination + `pino.multistream` for
   third-party shipping. Two registration surfaces for transports:
@@ -434,15 +447,23 @@ Test coverage: `test/unit/source-sweep.test.js`.
 - **0008** — MCP-UI rendering + action delivery. Lives in
   `mikser-io-mcp/documentation/decisions/` (not core — moved with
   MCP).
-- **0009** — Sqlite is the engine's persistence substrate. Single
-  `runtime/mikser.sqlite` holds `mikser_entities` / `mikser_refs` /
-  `mikser_snapshots` / `mikser_journal` (+ `mikser_meta` for the
-  schema-version stamp). Sift→SQL pushdown + LRU for findById;
-  worker-side read-only sqlite for sync template helpers.
-  `registerSchema(name, sql)` + `useDatabase()` is the plugin-side
-  persistence pattern. Journal-on-sqlite (Phase 7) enables `--resume`;
-  auto-persist (Phase 9) means plugins mutate the yielded entity and
-  the journal writes back without an explicit `updateEntry` call.
+- **0009** — TWO databases, split by whether the data is DERIVED.
+  `runtime/mikser.sqlite` is the CACHE: sqlite + better-sqlite3, synchronous
+  (render workers need a sync handle for template helpers), holding
+  `mikser_entities` / `mikser_refs` / `mikser_snapshots` / `mikser_journal` /
+  `mikser_meta`. Deletable at any moment; a wipe is `unlink`.
+  `mikser.data.sqlite` at the WORKING-FOLDER ROOT is the DURABLE store: auth
+  grants, the change-set log — behind **knex**, async, main-thread only, so it
+  can point at Postgres via `config.database.durable`. Built by
+  `registerMigrations(owner, [{ name, up }])`, per owner, recorded in
+  `mikser_migrations`; `registerSchema(..., { durable: true })` now THROWS.
+  Outside `runtime/` because that folder exists to be deleted, and gitignored
+  on creation because it holds credentials. `runtime.start()` closes the knex
+  pool for a one-shot run or the process never exits.
+  Sift→SQL pushdown + LRU for findById on the cache side.
+  Journal-on-sqlite (Phase 7) enables `--resume`; auto-persist (Phase 9) means
+  plugins mutate the yielded entity and the journal writes back without an
+  explicit `updateEntry` call.
 - **0010** — Plugin bundles + factory-call form + inline options.
   Plugins are imported by name and called as factories;
   `plugins: []` carries factory returns, never strings.

package/docs/api-reference.md

@@ -62,9 +62,17 @@ logger.info('Hello %s', 'world')
 logger.debug({ data }, 'Debug message')
 logger.warn('Something might be wrong')
 logger.error('Something failed: %s', err.message)
+logger.error({ code: 'vector-index' }, 'The index is unreadable: %s', err.message)  // a fault
 logger.notice('Completion message')  // styled green in info mode
 ```
 
+Two levels are also read by the build report, and the log call is the only way
+to raise either. `logger.warn({ code, ...fields }, msg)` becomes a `warnings`
+entry for that cycle. `logger.error` **with a `code`** becomes a *fault* — a
+subsystem declaring it cannot work — which persists across cycles and shows up
+in `mikser_ping`. Uncoded error lines are ordinary errors and are not captured.
+See [Diagnostics → Faults](./diagnostics.md#faults).
+
 ---
 
 ## Runtime Singleton
@@ -844,6 +852,36 @@ AsyncLocalStorage than the engine's, queries record no edges, and index pages,
 sitemaps and feeds silently stop rebuilding. Production consumers resolve both
 from their own tree, so the problem is local to the dev workspace.
 
+## Inventory
+
+What this mikser is made of.
+
+### `inventory({ workingFolder } = {})`
+
+Every installed mikser package with its version, purpose and links, and
+`active: true` on the ones actually running.
+
+```js
+[
+  { name: 'mikser-io-drive', version: '0.14.0', active: true,
+    summary: 'WebDAV for mikser-io. Exposes working-folder directories as …',
+    homepage: '…', repository: 'https://github.com/…', npm: 'https://www.npmjs.com/package/…' },
+]
+```
+
+**Derived, never declared.** Every plugin already carries a description, a
+homepage and a repository in its own `package.json`, kept current because npm
+publishes from it. A second summary held elsewhere would be one more thing to
+drift — and the first time it drifted it would be describing a plugin that had
+changed underneath it.
+
+**Installed and active are different facts.** A package in `node_modules` that
+no config loads explains nothing about a site's behaviour, and an agent told
+otherwise looks for a feature that is not switched on. `active` is reported
+only where it can be established — a plugin that mounts a route names itself
+there, a renderer registers under a known name — and is absent rather than
+`false` where it cannot, because `false` would be a claim.
+
 ## Roles
 
 Enforcement needs only the flat capability list. Explaining a refusal needs the
@@ -855,7 +893,37 @@ indistinguishable from inside a session.
 | `describeAuthority({ capabilities, roles, catalogue, summaries })` | everything a session can say about its own authority |
 | `reachOf(capabilities)` | `{ writable, readOnly }` as collection names |
 | `actingRole(held, catalogue)` | which role is in force |
-| `otherRoles(held, catalogue, summaries)` | who to ask, and what they add |
+| `rolesIn(catalogue, { acting, summaries })` | every role and its reach, the acting one marked |
+| `registerCapability(capability, meaning)` | declare what a capability protects |
+| `capabilityMeaning(capability)` | what is known about one |
+
+### Capabilities describe themselves
+
+Core does not know what `drive:layouts` protects, where that folder is, or what
+it is for — the plugin enforcing it does. So the plugin declares it, the same
+way it declares a route or a schema:
+
+```js
+registerCapability('drive:documents', {
+    plugin: 'drive',
+    grants: 'read',                       // 'read' | 'write' | 'operate'
+    resource: {
+        kind: 'collection',
+        name: 'documents',
+        folder: 'documents',              // where it is on disk
+        summary: 'the words on the pages' // what it is FOR
+    },
+})
+```
+
+`reachOf` then describes a credential in terms of the SITE rather than of
+verbs — a collection, its folder and its purpose — which is what an agent needs
+to reason about where it is working, not merely whether a call will be refused.
+A capability nothing has declared lands in `also` rather than being dropped: a
+role described only by the part of it that maps to folders is not described.
+
+The `drive:<name>[:write]` convention is still recognised for deployments whose
+plugins predate the registry, so they keep their answer.
 | `explainRefusal({ capability, role, target, catalogue, summaries })` | the sentence an agent repeats |
 
 `readOnly` is the field that makes a refusal explainable, and it is more useful
@@ -867,6 +935,11 @@ capabilities cover the others — roles are normally written as widening tiers
 and `null` when none dominates, because the acting authority genuinely is the
 union and naming half of it would be a lie.
 
+`roles` lists every role, not only the ones the session lacks. One field has to
+serve two readers: someone deciding who to ask, and someone holding the widest
+role trying to see what exists at all — and a "roles you do not have" field
+tells the second one nothing, since for an admin it is always empty.
+
 > **Informational, permanently.** Naming the role that could do something is
 > what makes a handoff possible. There is no way to request one and none should
 > be added: a role is a decision about a person, taken by whoever configures the
@@ -1013,6 +1086,12 @@ Together they turn "write and guess" into one call that says what the edit
 invalidated. `currentCycle()` and `buildReport()` read the cycle in progress and
 the last completed report.
 
+`faults()` returns every fault raised since the process started — subsystems
+that have reported they cannot work — most recently seen first. Each entry
+carries its `code`, message, any fields from the log call, and `count` /
+`first` / `last`. Faults are raised by logging at error level with a `code`
+and nothing else; see [Diagnostics → Faults](./diagnostics.md#faults).
+
 ## Logging
 
 `addLogTransport({ target, options, level })` adds a pino transport from a

package/docs/decisions/0009-database-engine-substrate.md

@@ -56,9 +56,55 @@ the synchronization point.
 
 ## Decision
 
-### Single database file, schemas registered per subsystem
-
-One `runtime/mikser.sqlite` file. WAL mode + `synchronous=NORMAL` +
+### Two database files, schemas registered per subsystem
+
+Two files, split by whether the data is DERIVED.
+
+`runtime/mikser.sqlite` is the cache — catalog, refs, snapshots, journal.
+Per ADR-0002 the files on disk are the source of truth, so this can be
+deleted at any moment and costs a rebuild.
+
+`mikser.data.sqlite`, at the working-folder ROOT, is everything that is not
+derived: OAuth clients and refresh tokens, the change-set log. No file can
+reproduce it, so deleting it is data loss.
+
+They were one file, with a `durable: true` flag exempting tables from the
+wipe. That required a LIST of which tables were which, and the list was where
+the bugs lived — a config that did not load the owning plugin saw no durable
+tables and unlinked them; `--clear` had to be routed through the wipe rather
+than removing the file; the table names had to be parsed back out of DDL.
+Two files delete the list: a wipe is `unlink` again, and what must survive it
+is not in the file being unlinked. The durable file lives outside `runtime/`
+because that folder exists to be thrown away.
+
+The two are DIFFERENT ENGINES, not two files behind one API.
+
+The cache is locked to sqlite and a SYNCHRONOUS driver: render workers open
+their own read-only handle so template helpers like `lookupHref` can resolve a
+reference inline, and a promise cannot be awaited inside a Handlebars helper.
+That constraint is what killed the earlier pluggable-driver attempt.
+
+None of it reaches the durable store, which is read and written on the main
+thread only, by HTTP handlers and lifecycle hooks that are already async. So it
+sits behind **knex** and can point at Postgres instead of a local file — which
+is what several mikser processes sharing one sign-in and one change-set log
+would need. Splitting the files is what split that constraint apart.
+
+Durable tables are built by MIGRATIONS, not by an idempotent CREATE:
+`registerMigrations(owner, [{ name, up }])`, per owner, recorded in
+`mikser_migrations (owner, name, applied_at)`. Per owner because the tables
+belong to independently versioned npm packages, and installing a plugin next
+month must not reorder migrations that ran last year — which every
+directory-based migration runner assumes it can do. `CREATE TABLE IF NOT
+EXISTS` does nothing to a table that already exists, so a schema replayed every
+boot can never add a column, rename one, or backfill a value; that is invisible
+for a cache table and the permanent condition of a durable one.
+
+`registerSchema(name, sql, { durable: true })` now THROWS, pointing at
+`registerMigrations`. Accepting it would put the table in the file that gets
+deleted, and the first sign would be an operator asked to sign in again.
+
+For the cache: WAL mode + `synchronous=NORMAL` +
 `foreign_keys=ON`. Every persistent subsystem registers its schema via
 `registerSchema(name, sql)` at module-eval time; the database is
 opened once in `onInitialize` and applies all registered schemas

package/docs/diagnostics.md

@@ -25,6 +25,7 @@ engine source, the entry point is missing and belongs on this page.
 | What would break if I changed this file? | [`runtime.manifest`](#runtimemanifest) `affectedBy` |
 | I am an agent reading CLI output, not speaking MCP | [`--tools` / `--tool`](#the-two-agent-workflows) |
 | Did my schema validate anything at all? | [`schemas.names()`](#schemasnames--schemaslookup) |
+| A tool answered emptily — is it broken, or is there nothing to find? | [`faults`](#faults) |
 
 ## Command line
 
@@ -137,7 +138,7 @@ under this flag, so stdout parses whole.
 npx mikser --json | jq '.summary'
 ```
 
-Five buckets, and the distinction between them is the point:
+The buckets, and the distinction between them is the point:
 
 | Bucket | Meaning |
 | --- | --- |
@@ -146,6 +147,8 @@ Five buckets, and the distinction between them is the point:
 | `unchanged` | the render ran and produced bytes identical to what was already on disk |
 | `errors` | the render ran and **threw** — with `id`, `destination`, `error`, `layout` |
 | `gated` | a count — the source was unchanged, so no render was ever scheduled |
+| `warnings` | everything that went through `logger.warn` this cycle, with its `code` |
+| `faults` | subsystems that reported they **cannot work** — see [Faults](#faults) |
 
 Each report also carries `cycleId`, `startedAt` and `finishedAt`. Under
 `--watch` two consecutive reports are otherwise indistinguishable, so
@@ -441,16 +444,31 @@ denial-of-service knob rather than a diagnostic, and it stays on the CLI.
 
 ## The database
 
-Everything mikser knows lives in one SQLite file at
-`<workingFolder>/runtime/mikser.sqlite`. It is readable while a build
-runs (WAL mode) and it is often faster to ask it directly than to add
-logging.
+**Two files, and the difference between them is what is safe to delete.**
+
+| File | Holds | If you delete it |
+| --- | --- | --- |
+| `runtime/mikser.sqlite` | the derived cache — catalog, refs, snapshots, journal | costs a rebuild, nothing else (ADR-0002: your files are the source of truth) |
+| `mikser.data.sqlite` | everything **not** derived — sign-ins, the change-set log | data loss; no file can reproduce it |
+
+The durable one sits at the working-folder root rather than in `runtime/`
+on purpose. That folder exists to be thrown away — it is gitignored, and
+`rm -rf runtime` is a line in real deploy scripts — so anything kept
+inside it is one careless command from gone. `--clear` removes the cache
+and does not touch the other file.
+
+`mikser.data.sqlite` holds credentials, so mikser adds it to `.gitignore`
+the first time it creates it. A working folder is usually a git repo, and
+`mikser-io-git` runs `git add -A` over it.
+
+Both are readable while a build runs (WAL mode), and it is often faster to
+ask them directly than to add logging.
 
 ```bash
 sqlite3 runtime/mikser.sqlite
 ```
 
-Five tables:
+The cache holds five tables:
 
 | Table | Holds |
 | --- | --- |
@@ -460,6 +478,34 @@ Five tables:
 | `mikser_snapshots` | what was rendered, where, and from which inputs |
 | `mikser_meta` | schema version and config checksum, for cache invalidation |
 
+The durable store is behind **knex**, so it is the one part of mikser that can
+point at another engine — `config.database.durable` takes a path or a
+connection string. The cache cannot: render workers need a synchronous handle
+for template helpers, and nothing about that applies to a store only the main
+thread touches.
+
+Durable tables are built by migrations, per owning package:
+
+```js
+registerMigrations('grants', [
+    { name: '001-initial', up: (knex) => knex.schema.createTable('my_grants', (t) => {
+        t.string('id').primary()
+    }) },
+    { name: '002-last-used', up: (knex) => knex.schema.alterTable('my_grants', (t) => {
+        t.bigInteger('last_used_at')
+    }) },
+])
+```
+
+Applied once each and recorded in `mikser_migrations (owner, name, applied_at)`.
+Per owner because the tables belong to independently versioned packages —
+installing a plugin next month must not reorder migrations that ran last year.
+Names are permanent: renaming one makes it run again against a database that
+already has its effect. Append, never edit. There is no `down` — a durable
+store's rollback is a restore from backup.
+
+`registerSchema(..., { durable: true })` throws now and says so.
+
 ### `mikser_journal`
 
 The ordered record of what the engine did this cycle — one row per
@@ -657,6 +703,64 @@ content is clean.
 get, stats, config }` — the on-demand render cache, useful for asking
 what has been rendered outside a build.
 
+## Faults
+
+A **fault** is a subsystem saying it cannot do its job — as opposed to an
+error, which is one render that ran and threw.
+
+The difference matters because of how a broken subsystem fails. It does not
+raise; it answers. Search returns `[]`. The change-set log lists nothing.
+`affectedBy` reports that nothing is affected. Every one of those is
+byte-identical to the answer a *working* subsystem gives when there is
+genuinely nothing to report, which is why this class of bug is normally found
+by someone noticing the site is wrong days later.
+
+A fault is raised by logging at error level with a `code`:
+
+```js
+useLogger().error({ code: 'change-set-log' },
+    'The change-set log could not be written (%s). Writes still land on disk, '
+    + 'but they cannot be listed or undone until this is fixed.', err.message)
+```
+
+That log call is the *only* registration — the same contract as warnings,
+where `report.warnings` is a view of what went through `logger.warn`. There is
+no `registerFault()`, because a second way to raise one is a second thing to
+forget.
+
+The `code` is required, and is what separates a fault from an ordinary error
+line. It is the condition's identity, so forty occurrences are one fault with
+`count: 40` rather than forty faults. An uncoded `logger.error('Render error:
+%s', id)` is an event about one entity and already travels in `errors` with the
+entity attached. `logger.error(err, msg)` does not become a fault either — pino
+puts an errno under `err`, not at the top level, so a raw throw logged that way
+stays an event.
+
+Faults are recorded whether or not anyone asked for a report, and are **not**
+cleared between cycles. Both are deliberate: a subsystem that cannot work goes
+on not working after the cycle that noticed it ends, and the reader who most
+needs to know is usually not the operator watching the terminal — they saw the
+log line — but an agent connecting an hour later, for which the log is a
+channel it never reads.
+
+Where they surface:
+
+| Surface | Shape |
+| --- | --- |
+| `--json` | the `faults` array and `summary.faults` |
+| `mikser_ping` | a `faults` key, **absent** when there are none |
+| `faults()` | the same array, for a plugin or a REPL |
+
+Each entry carries the `code`, the message, any fields from the log call, and
+`count` / `first` / `last`. `last` is the field that says whether it is still
+happening: a fault seen once at boot and one firing every cycle read the same
+by presence alone.
+
+Nothing clears a fault. The condition is fixed by an operator, and the restart
+that follows is what clears it — an engine deciding on a subsystem's behalf
+that it has recovered would be inventing the one fact this surface exists to
+report honestly.
+
 ## When mikser is silent
 
 Silence is this engine's characteristic failure mode: a declaration that
@@ -677,6 +781,10 @@ surfaces that turn silence into a statement:
   part in cache invalidation, so a config change forces a rebuild; but a
   module the config *imports* does not. If you changed a helper the config
   pulls in, use `--force`.
+- **A tool that answers emptily because it is broken** — check
+  [`faults`](#faults) before reading an empty result as a fact about the
+  site. This is the one case where the answer and the failure are the same
+  bytes.
 - **A plugin that appears to do nothing** — `No plugins loaded` with a
   config present is a warning naming the file. A config that fails to
   load now exits non-zero rather than loading as empty.

package/index.js

@@ -3,6 +3,7 @@ export * as constants from './src/constants.js'
 export * from './src/utils.js'
 export * from './src/auth.js'
 export * from './src/roles.js'
+export * from './src/inventory.js'
 export * from './src/report.js'
 // The diagnostics behind --explain. Exported so a transport — the MCP tool
 // surface, the api plugin's routes — can serve the same structured report the
@@ -10,6 +11,11 @@ export * from './src/report.js'
 export * from './src/explain.js'
 export * from './src/lifecycle.js'
 export * from './src/database/index.js'
+// The durable store — registerMigrations / useDurableDatabase. A separate
+// module from the cache because it is a separate database with none of the
+// cache's constraints: main-thread only, so async, so knex, so portable to
+// another engine.
+export * from './src/database/durable.js'
 export * from './src/journal.js'
 export * from './src/catalog.js'
 export * from './src/search.js'

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "mikser-io",
-  "version": "9.50.2",
-  "description": "<p align=\"center\">   <img src=\"mikser-lockup-stacked.svg\" alt=\"mikser\" width=\"198\" /> </p>",
+  "version": "9.57.0",
+  "description": "A mixer for content: entities in, configurable render pipelines, outputs of any kind. Static sites are the canonical recipe, not the definition — the same engine renders PDFs, emails and whatever a renderer plugin produces. Files are the source of truth, every lifecycle phase is observable, and the build graph is queryable by an agent.",
   "main": "index.js",
   "exports": {
     ".": "./index.js",
@@ -44,6 +44,7 @@
     "handlebars": "^4.7.9",
     "hasha": "^7.0.0",
     "is-url": "^1.2.4",
+    "knex": "^3.3.0",
     "line-reader": "^0.4.0",
     "lodash": "^4.18.1",
     "mime-types": "^3.0.2",