@aotter/mantle 0.1.5-alpha.1 → 0.2.0-alpha.2

package/docs/handbook/concepts/runtime-and-adapters.md

@@ -1,184 +1,198 @@
 ---
-description: The sealed parse-link-compile-prepare-bind pipeline, Core versus optional products, and how Cloudflare, Bun, Vercel and IndexedDB differ.
+description: How createMantle boots the runtime from a storage adapter, how storage converges to the plan, the D1 and PostgreSQL dialects, and the fingerprint handshake.
 ---
-# Runtime pipeline and adapters
+# Runtime, Store and dialects
 
-Mantle has exactly one path from YAML to a running service, and one owner for every rule along it. This page walks that pipeline, then shows what changes and what stays fixed when you swap the host underneath it.
-
-## The sealed pipeline
-
-```txt
-ManifestSourceSet
-  -> parse + normalize -> ParsedManifestSet
-  -> link               -> LinkedManifestSet
-  -> compile            -> RuntimePlan
-  -> prepare deployment -> PreparedRevision
-  -> bind runtime       -> MantleRuntime
 ```
-
-Each stage's output can be constructed only by the stage that owns it, and a failed stage produces nothing the next stage can use. Parse, link and compile are pure and deterministic.
-
-| Stage | Owns | Rejects | Must not own |
-|---|---|---|---|
-| Parse + normalize | YAML syntax and alias limits, the closed four-atom shape, atom-local rules, behavior-affecting defaults, source metadata | Unknown keys, bad envelopes, unsupported JSON Schema keywords, invalid index and `uiSchema` shapes | Cross-atom references, handlers, storage, routes |
-| Link | Duplicate symbols, cross-atom references, guard graphs, translations, manifest-owned route and tool collisions | Unknown Schema or Procedure references, guard self-reference and chains, duplicate HTTP paths, MCP tool-name collisions | I/O, selected modules, handler availability |
-| Compile | Immutable lookup records, authorization plans, Trigger indices, Procedure descriptors, logical View plans, the semantic fingerprint | Nothing new; it projects an already valid graph | Connections, repositories, handlers, requests, templates, assets |
-| Prepare | Storage migrations, native Schema tables and indexes, prepared Views, handler availability, selected capability and reserved-route checks, the readiness revision | Missing handler refs, reserved-path conflicts, `View.spec.sql` on storage that does not declare the SQLite dialect | Re-interpreting YAML, executing requests |
-| Bind and invoke | Semantic ports, handler dispatch, parameter binding, centralized authorization, content, View, Procedure, Trigger and lifecycle operations | Invalid input, unauthorized callers, lifecycle transitions the state machine forbids | DDL, route mounting, assets, HTTP, session and cache policy |
-| Optional modules and adapters | Web and Admin composition; request, session, cache and platform translation | Whatever the platform itself rejects | Re-parsing, re-linking, or a second authorization stack |
-
-Nothing downstream may reinterpret raw manifests. That is what makes a rule verifiable exactly once, and why every rejection carries a diagnostic code instead of a guess. See [Diagnostic codes](../reference/diagnostics.md).
-
-## What `mantle generate` produces
-
-```sh
-pnpm exec mantle generate
-pnpm exec mantle generate --check
+src/index.ts (host entry) ─▶ mantle = createMantle(service, { plan, storage, schedules })
+                                 │ first request: createMantleRuntime → storage.prepare(plan)
+                                 ▼
+service.fetch(request, env, { runtime, waitUntil })
+  └ surfaces / your routes ─▶ runtime.invokeProcedure · runtime.store
+                                 └ Store: policy rewrite → dialect check → StoreExecutor
 ```
 
-`generate` validates and compiles the manifests directory, then writes one typed module at `.mantle/generated/mantle.ts` containing the sealed `plan` with its fingerprint, the handler types, and two entry points:
+## `createMantle`
 
 ```ts
-import { bindMantle, createMantle, plan } from "../.mantle/generated/mantle.js";
-
-// Eager: one preparation attempt, no caching and no retry.
-const mantle = await createMantle({ storage, handlers, ports });
-const notes = await mantle.views.publishedNotes();
-await mantle.entries.orders.createDraft({ data, authorId: user.id });
-
-// Or bind a runtime the host already assembled and owns the lifecycle of.
-const bound = bindMantle(runtime);
-await bound.runtime.archive.execute({ id, ctx });
+createMantle(service, { plan, storage: (env) => MantleStorageAdapter, schedules?, expectedFingerprint? })
+// → { fetch(request, env, ctx?), invokeSchedule(cron, scheduledTime, env, ctx?), runDeferredHook(message, env, ctx?) }
+// runDeferredHook replays an after-hook Invocation from your own queue; nothing in Mantle emits one
 ```
 
-Generated property names are deterministic lower-camel identifiers; calls keep the authored wire names internally, and a collision is an error (`CODEGEN_IDENTIFIER_COLLISION`). The emitter accepts either `emitMantleModule({ linked })` or `emitMantleModule({ plan })` for hosts that already compiled the plan; see [Typed queries](../guides/typed-queries.md). Code generation is a pure projection: it never caches, retries, mounts routes or owns host lifecycle, and the typed API keeps its raw `runtime` so it hides nothing. Skipping generation is valid — call `runtime.executeView({ view: "published-notes" })` by name.
-
-## Core, optional products, adapters
-
-| Layer | Package | Responsibility |
-|---|---|---|
-| Core | `@aotter/mantle-spec` | Sources, parse, normalize, link, introspection, code generation. No runtime or platform dependency. |
-| Core | `@aotter/mantle-runtime` | `RuntimePlan`, preparation contracts, semantic storage ports, `MantleRuntime`. No Web, Admin or platform dependency. |
-| Optional product | `@aotter/mantle-web` | Public HTML, Markdown mirrors, `llms.txt`, sitemap, SEO, preview, templates, path composition. Owns no routes. |
-| Optional product | `@aotter/mantle-admin` | Admin API orchestration, OAuth surfaces, the asset contract. |
-| Optional product | `@aotter/mantle-admin-ui` | The pre-built React Admin SPA artifact. |
-| Adapter | `@aotter/mantle-cloudflare`, `-bun`, `-vercel`, `-indexeddb` | Bind platform storage, lifecycle, request, session, cache and asset concerns to Core ports. |
-
-The umbrella `@aotter/mantle` installs Spec and Runtime only; every other subpath is an optional peer you install when you select it. Core does not reserve Admin paths or serve a UI when the module is absent.
-
-## Capability matrix
-
-| | Cloudflare | Bun | Vercel Functions | IndexedDB |
-|---|---|---|---|---|
-| Storage | D1 through the SQLite chain | Caller-owned `bun:sqlite` `Database` | Any injected `MantleStorageAdapter`; optional `/libsql` Turso driver | One application-owned IndexedDB database |
-| Public View REST | Yes | Yes | Yes | Not mounted; call the runtime directly |
-| HTTP Triggers | Yes | Yes | Yes | Not mounted; `runtime.invokeTrigger` |
-| Admin, Auth, OAuth | Yes | Absent | Absent | Absent |
-| MCP `/mcp`, `/mcp/staff` | Yes | Absent | Absent | Absent (WebMCP is a separate browser binding) |
-| Public web pages | Opt-in via `mountPublicRoutes` plus templates and a resolver | Absent | Absent | Absent |
-| Cache policy | Owned, applied at the final boundary | Host-owned | Host-owned | n/a |
-| `View.spec.sql` | Supported | Supported | Supported on a SQLite-family driver | Rejected at preparation |
-| Optional capabilities | R2 media, Queues deferred hooks, KV catalog cache | None | Platform `waitUntil` | None |
-| Host still owns | Application routes and frontend | `Bun.serve`, auth, CSRF, database shutdown | Web handler, auth, CSRF, route composition | Database naming, persistence requests, UI invalidation, sync |
-
-The Manifest does not change across that row. What changes is which surfaces exist to reach it.
-
-## Storage ports
-
-A storage adapter prepares one `RuntimePlan` into semantic ports. Three are required:
-
-| Port | Role |
+- `service` is `{ handlers, fetch(request, env, { runtime, waitUntil }) }`, a
+  WinterTC fetch the application owns. Mantle owns no route of its own: the
+  service decides what is mounted where.
+- The runtime boots lazily on the first call, once per isolate. A failed boot
+  is retried by the next request.
+- Each entry call binds its own `ctx.waitUntil` to the runtime facade handed
+  to the service. Handler calls, nested invocations and Store lifecycle hooks
+  retain work on that entry's context, including concurrent requests. Calls
+  without a context do not borrow another request's retainer.
+- `invokeSchedule` runs every enabled schedule Trigger whose cron is exactly
+  the given POSIX expression, as the system caller.
+- The host entry is a few lines the generator writes once
+  ([The service and its entry](../cloudflare/service-entry.md)); Core handles
+  no platform event types.
+
+`createMantleRuntime(args)` is the same boot without the service, for tests
+and custom hosts. The runtime exposes `plan`, `store`, `invokeProcedure`,
+`bootReport()` (`{ fingerprint, coreVersion }`) and, with site defaults,
+`site`.
+
+## Boot checks
+
+Boot refuses to serve when:
+
+- a plan `ref` has no handler (`HANDLER_NOT_REGISTERED`), or a handler is not
+  in the plan (`HANDLER_NOT_DECLARED`);
+- the plan was compiled for another dialect, or does not match
+  `expectedFingerprint` (`PLAN_FINGERPRINT_MISMATCH`);
+- an enabled schedule exists without `schedules: true` (`SCHEDULE_NOT_WIRED`);
+- storage has a change it cannot make (`STORAGE_CHANGE_BLOCKED`), or a table
+  Mantle would create already exists without being Mantle's
+  (`STORAGE_TABLE_NOT_OWNED`).
+
+## Storage converges to the plan
+
+There are no migration files for Schema tables. When the plan's fingerprint
+differs from the one the database last booted, boot reads the database's
+actual tables and indexes, compares them with the plan, and:
+
+| Difference | Boot |
 |---|---|
-| `MantleStorageAdapter` / `PreparedMantleStorage` | Prepares one plan into a revision: migrations, native Schema tables and indexes, prepared Views |
-| `EntryRepository` and `EntryReader` | Entry writes and reads |
-| `ViewQueryExecutor` | Executes compiled logical View plans |
-
-Two are optional and only when a feature needs them: `MediaStorage` for upload flows, and `DeferredHookDispatcher` for at-least-once `after_*` delivery. `DatabaseDriver` is not a portability contract — it is the reusable SQLite/D1 seam. A PostgreSQL, MongoDB or application-owned-table adapter implements the semantic ports directly rather than emulating D1.
-
-`PreparedMantleStorage.atomicEntries` is an optional semantic write capability.
-An adapter exposes it only when it can commit an ordered group of entry
-mutations all-or-nothing, including zero-row version/status failures. The
-SQLite implementation covers Cloudflare D1 and Bun; IndexedDB and custom
-adapters without that guarantee leave it absent. Ref Procedures use
-`ctx.writeAtomically` as described in [Procedure](../reference/procedure.md#atomic-entry-writes-in-a-ref-handler).
-
-Declarative Views compile to logical plans once, and preparation lowers those plans to native queries. `View.spec.sql` is the exception: it is explicitly SQLite-only in v0.1. Storage that does not declare that dialect rejects such a View at preparation with `VIEW_DIALECT_UNSUPPORTED`, before mutating any state. Mantle does not guess a translation and ships no universal query driver. A View that must run everywhere uses `from` with a filter AST; see [View](../reference/view.md).
-
-## Embedding each adapter
-
-Cloudflare, through the conventional facade:
-
-```ts
-import { createMantleWorker } from "@aotter/mantle/cloudflare";
-import { plan } from "../.mantle/generated/mantle.js";
-
-export default createMantleWorker({ plan });
-```
-
-Bun, with the server and SQLite handle staying yours:
-
-```ts
-import { Database } from "bun:sqlite";
-import { createBunMantle } from "@aotter/mantle-bun";  // experimental
-
-const database = new Database("app.sqlite");
-const mantle = createBunMantle({ plan, database, handlers });
-
-Bun.serve({
-  async fetch(request) {
-    return (await mantle.handle(request)) ?? new Response("not found", { status: 404 });
-  },
-});
-```
-
-Vercel Functions, with storage injected:
+| a missing table, field column, index, or unique index that builds | creates it (on SQLite, `STRICT` tables with their check and FTS triggers) |
+| a unique index that fails on existing rows, a column with another type, a changed index, a missing native column | refuses with `STORAGE_CHANGE_BLOCKED`, naming the change and a SQL hint |
+| a column or non-unique index the plan no longer declares | keeps it and warns; nothing is ever dropped |
+
+A renamed field is a new empty column beside the old one: copy the data
+yourself. A blocked change is the author's to make (SQL through the host's own
+tool, a data-copy Procedure, a dedupe), then boot checks again. With the same
+fingerprint boot reads nothing. `mantle generate --check --database <file>`
+prints the same diff against a local database without applying it.
+
+Mantle's own tables (`_mantle_*`) and its product tables (`site_config`,
+`media_assets`, `pending_media_uploads`) keep versioned migrations of their
+own. Identity `mantle` migrates Better Auth's tables through Better Auth.
+
+## The D1 dialect
+
+The SQL language is PostgreSQL syntax, pinned to PostgreSQL 18's parser. A
+**dialect** decides which of it an engine runs and how. The built-in dialect,
+`@aotter/mantle/d1`, runs on SQLite as D1 does: its compile side refuses what
+D1 cannot compute exactly, and its runtime side prints SQLite, encodes types
+and converges storage.
+
+- `date-time` (and `created_at` / `updated_at`) is stored as microseconds,
+  `date` as days, `numeric(p, s)` as an integer of the smallest unit, so
+  arithmetic is exact on D1. On the wire a date-time is an ISO string.
+- `checks` are enforced by triggers. `x-mantle-ref` is checked by
+  `mantle generate` and used by Admin; storage does not enforce it in 0.2.0.
+  `searchableFields` uses
+  an FTS5 trigram index; `format: geo` uses an R*Tree.
+- `date_trunc` and `extract` compute in the site time zone
+  (`d1Storage(db, { timeZone })`, default UTC).
+
+`mantle.config.json` names the dialect (`sqlite`, `postgres`, or another dialect package, which must pass
+`runStorageConformance` from `@aotter/mantle/testing`). A plan records its
+dialect, and boot refuses a plan compiled for another one.
+
+## The PostgreSQL dialect
+
+`"dialect": "postgres"` in `mantle.config.json` compiles the plan for
+PostgreSQL (13 or later). On Cloudflare, `mantle generate --dialect postgres`
+writes the preset over Hyperdrive; on any other platform, add `"host": "none"`
+and compose `createMantle` with `postgresStorage` yourself (ADR-0036). PostgreSQL
+is Mantle SQL's reference dialect (ADR-0037): it accepts D1's subset without
+SQLite's own `typeof`, `hex`, `json_extract`, `json_set`, `json_insert` and
+`json_remove` (write `x ->> '$.path'`), plus `WITH`, set operations, `LATERAL`,
+window frames, `FILTER`, jsonb operators and the rest of the
+[View reference](../reference/view.md)'s PostgreSQL table. A manifest that uses
+them no longer compiles for D1; one that does not moves between engines with a
+recompile.
+
+- Columns have native types: `timestamptz`, `date`, `numeric(p, s)`, `boolean`,
+  `jsonb`, `bigint` and `double precision`. Values are decoded by the type
+  PostgreSQL reports, so computed outputs (`now()`, `date_trunc`, a `CAST`) come
+  back as wire values too: ISO date-times, booleans, parsed JSON.
+- Every write batch is one `SERIALIZABLE` transaction, retried on a
+  serialization failure, so a guard such as `WHERE NOT EXISTS` holds under
+  concurrent writers as it does on SQLite's single writer.
+- `checks` are `CHECK` constraints added `NOT VALID`: they bind every later
+  write and leave older rows alone, like D1's triggers.
+- `json_each`, `->>` and `CAST(x AS bool)` read as they do on D1 through small
+  SQL functions Mantle creates (`_mantle_json_each`, `_mantle_jget`,
+  `_mantle_bool`). `json_group_array` and `json_group_object` order by value.
+- Results match D1 where Mantle decides them. Text columns use `COLLATE "C"`,
+  so they compare by code point. An `ORDER BY` key without `NULLS FIRST/LAST`
+  puts NULL first ascending, as Core pages every View. Values never depend on
+  the server's `DateStyle`, `IntervalStyle` or `TimeZone`. Elsewhere the meaning
+  is PostgreSQL's, where D1 differs:
+  - `LIKE` is case-sensitive.
+  - Division by zero is an error.
+  - `->>` returns text.
+  - `||` prints booleans and floats as PostgreSQL does.
+- `searchableFields` and `mantle.near()` scan without an index in 0.2.0, and
+  `mantle.search_rank()` counts occurrences rather than computing bm25.
+  Site settings and media are SQLite-only (D1, bun:sqlite).
+- `date_trunc` and `extract` compute in the site time zone
+  (`postgresStorage({ connect, timeZone })`, default UTC).
+- Every statement has a `statement_timeout` of 10 seconds, pinned in each
+  transaction (`statementTimeoutMs`; 0 is none). Storage convergence has none.
+  A statement past it fails with `RESOURCE_UNAVAILABLE` and writes nothing.
+- Connect as a role that owns the service's tables but is not a superuser and
+  holds no file or server privilege (`pg_read_server_files`,
+  `pg_execute_server_program`). Mantle's allowlist refuses such functions; the
+  role is the second line.
+
+## Storage adapters
+
+| Adapter | From | Driver |
+|---|---|---|
+| `d1Storage(env.DB, { timeZone?, site? })` | `@aotter/mantle/cloudflare` | Cloudflare D1, the one the preset uses |
+| `sqliteStorage(driver, { timeZone?, maxBindings?, site?, restrict? })` | `@aotter/mantle/d1` | any `DatabaseDriver`: `{ batch(statements) }`, all or nothing |
+| `postgresStorage({ connect, timeZone?, statementTimeoutMs?, restrict? })` | `@aotter/mantle/postgres` | `connect` opens one node-postgres (`pg`) client; Hyperdrive on Workers |
+| `bunPostgresStorage(sql, { timeZone?, statementTimeoutMs?, restrict? })` | `@aotter/mantle/bun` | a Bun.SQL pool with `prepare: false`; its role needs TEMPORARY (ADR-0038) |
+| `bunSqliteStorage(db, { timeZone?, site?, restrict? })` | `@aotter/mantle/bun` | a bun:sqlite `Database`; foreign keys are turned on |
 
-```ts
-import { SqliteMantleStorageAdapter } from "@aotter/mantle-runtime";
-import { createVercelMantle } from "@aotter/mantle-vercel";  // experimental
-import { LibsqlDatabaseDriver } from "@aotter/mantle-vercel/libsql";
-
-const mantle = createVercelMantle({
-  plan,
-  handlers,
-  storage: new SqliteMantleStorageAdapter(new LibsqlDatabaseDriver(client)),
-});
-```
+`restrict(plan, context)` returns refusals of its own, run after the dialect's
+on every program at runtime. It only narrows what runs, for an operator that
+runs other people's plans; a self-hosted service leaves it out (ADR-0037).
 
-Browser IndexedDB, with no HTTP transport at all:
+On Workers, PostgreSQL goes through Hyperdrive, which pools the connections, so
+`connect` opens a client per operation (a socket must not outlive its request):
 
 ```ts
-import { bootMantleRuntime } from "@aotter/mantle-runtime";
-import { IndexedDbMantleStorageAdapter } from "@aotter/mantle-indexeddb";
-
-const storage = new IndexedDbMantleStorageAdapter({ databaseName: "my-app" });
-const runtime = await bootMantleRuntime({ plan, storage, handlers });
-await runtime.invokeTrigger({ trigger: "rename-board-mcp", input, ctx });
+import pg from "pg";
+import { pgDatabaseDriver, pgPool, postgresStorage } from "@aotter/mantle/postgres";
+
+const connect = (env: Env) => async () => {
+  const client = new pg.Client({ connectionString: env.HYPERDRIVE.connectionString });
+  await client.connect();
+  return client;
+};
+// storage: (env) => postgresStorage({ connect: connect(env) })
+// identity: createMantleAuth({ database: pgPool(connect(env)), driver: pgDatabaseDriver(connect(env)), ... })
 ```
 
-`handle()` returns `null` for a path Mantle does not own, so the host keeps its own routes. Never treat a Vercel Function's filesystem or `/tmp` as durable state.
-
-## The adapter-boundary rule
-
-Platform bindings belong at the composition root only: the Worker entry, `createMantleWorker` options, the `bindings` hook and `wrangler.jsonc`. Procedure handlers receive them through `ctx.env`.
-
-Application code does not bypass Mantle's storage ports to write Schema tables, `site_config`, media, or Auth tables. Each Schema is a native table, but Mantle still owns its metadata columns, lifecycle checks, and optimistic concurrency. Use Manifests, runtime use cases, `runtime.entries`, `runtime.siteConfig`, generated `bindMantle(runtime)`, and Views instead. An application may own separate tables behind its own repository; that is different from writing around Core's invariants.
+The Worker needs `compatibility_flags: ["nodejs_compat"]` for `pg`. Mantle runs
+every read in a transaction, which Hyperdrive never answers from its cache.
+Better Auth's reads (sessions, roles) do not, so create the Hyperdrive config
+with caching disabled (`wrangler hyperdrive create … --caching-disabled`), or a
+revoked session can be accepted until the cache expires. `timeZone` must be an
+IANA name: PostgreSQL reads an offset such as `+08:00` with the opposite sign.
 
-If a normal feature cannot be expressed through a purpose-shaped surface, treat that as a gap in the abstraction rather than teaching the project Mantle's internals. Internals change between versions; the ports do not.
+Bun has a generated preset for both engines (`mantle generate --host bun`,
+ADR-0038); `bunDatabaseDriver` / `bunAuthDatabase` and `bunSqliteDriver` give
+Better Auth the same store. A `DatabaseDriver` is one method, so another engine
+(libSQL, for example) is a few lines of application code on `host: none`: the
+entry calls `createMantle` and its own server.
 
-## Related
+`site` turns on the site capability (site settings and media); see
+[Site defaults and site_config](../reference/site-config.md).
 
-- [The four atoms](./four-atoms.md) — what the pipeline is compiling.
-- [Bindings and primitives](../cloudflare/bindings.md) — the composition root in practice.
-- [Low-level composition](../cloudflare/low-level-composition.md) — assembling a Worker from the same public primitives.
-- [Project layout and the CLI loop](../start/project-and-cli.md) — where `generate` sits in the daily loop.
+## Testing a service
 
-## Source
-- [`docs/adr/0019-sealed-manifest-runtime-pipeline.md`](../../../docs/adr/0019-sealed-manifest-runtime-pipeline.md)
-- [`docs/adapter-guide.md`](../../../docs/adapter-guide.md)
-- [`packages/mantle/README.md`](../../../packages/mantle/README.md)
-- [`packages/mantle-runtime/README.md`](../../../packages/mantle-runtime/README.md)
-- [`packages/adapters/bun/README.md`](../../../packages/adapters/bun/README.md)
-- [`packages/adapters/vercel/README.md`](../../../packages/adapters/vercel/README.md)
-- [`packages/adapters/indexeddb/README.md`](../../../packages/adapters/indexeddb/README.md)
-- [`docs/skills/develop/SKILL.md`](../../skills/develop/SKILL.md)
+Run the service under `wrangler dev` against a fresh local D1 and drive it over
+HTTP, as the [reference service](../../examples/reference-service/README.md)'s
+`smoke.mjs` does.