@voltro/cli 0.33.0 → 0.35.0

package/templates/agent-docs/plugins/flags.md

@@ -109,6 +109,138 @@ GET /_voltro/inspect/plugins/flags/audit?flag=beta # one flag's history
 
 The `POST /toggle` body accepts an optional `actor` (the acting admin id) that is stored on the audit row. On the `memory` tier there is no durable audit (the endpoint returns an empty trail with a note).
 
+## Typed flags — `defineFlag()`
+
+A flag's VALUE had no type. `FlagVariant.value` is the `FlagVariantValue` union (`string | number | boolean | null | array | object`), so `{ name: 'big', value: 'lots' }` on a flag every reader treats as a number typechecked, and the mistake showed up at the call site as `NaN`.
+
+`defineFlag()` gives a flag a value **Schema**, and the flag is browser-safe by construction — ONE declaration, imported by `app.config.ts` and by the component that reads it.
+
+```ts
+// apps/api/lib/flags.ts
+import { Schema } from 'effect'
+import { defineFlag } from '@voltro/plugin-flags'
+
+export const checkoutButton = defineFlag({
+  key: 'checkout.button',
+  value: Schema.Literal('blue', 'green'),
+  default: 'blue',
+  variants: [{ name: 'control', value: 'blue' }, { name: 'green', value: 'green' }],
+})
+
+export const pageSize = defineFlag({
+  key: 'search.pageSize',
+  value: Schema.Number,
+  default: 20,
+// default: 'twenty',   ← Type 'string' is not assignable to type 'number'
+})
+```
+
+Register them with `flagsPlugin({ typedFlags: [checkoutButton, pageSize] })`. Keys share one namespace with `flags: { … }`; declaring a key in both is refused at construction.
+
+Read them typed on either side:
+
+```ts
+import { flagValue, flagVariant } from '@voltro/plugin-flags'
+const size: number = flagValue(ctx, pageSize)          // server
+const arm = flagVariant(ctx, checkoutButton)           // the served variant NAME, or null
+```
+
+```tsx
+import { useFlagValue } from '@voltro/plugin-flags/web'
+const colour = useFlagValue(checkoutButton)            // 'blue' | 'green'
+```
+
+### Which half a Schema reaches, precisely
+
+**Authored** values — `default`, every `variants[].value` — are checked by `tsc`. That is the headline and it is enforced by tests that fail `typecheck` if those lines ever start compiling.
+
+**Runtime** values cannot be. A postgres-tier override row, or a dashboard edit, is JSON long after `tsc` ran. So the same Schema is the runtime gate: an override whose variant values do not decode is **refused whole**, the code-declared definition stands, and the refusal is logged and shown in the dashboard panel. Not partially applied — dropping the one bad arm re-normalises the weights of the rest, silently reallocating every subject.
+
+Declaration also DECODES the authored default, which catches what a type cannot: `Schema.Int`'s TypeScript type is `number`, so `default: 20.5` typechecks and is a value the flag could never legally serve.
+
+## Dead-flag detection
+
+Every flag system accumulates flags nobody removes. `GET /_voltro/inspect/plugins/flags/list` carries a lifecycle report (rendered by the dashboard panel) built on two independent axes — and only some of it is a proof.
+
+**`shape`** is decided from the DEFINITION alone. No observation, no window:
+
+| shape | meaning |
+|---|---|
+| `constantOn` | `rollout` 100, no targeting, no variants, no live schedule — resolves `true` for everyone, forever |
+| `constantOff` | `enabled: false`, or `rollout: 0` |
+| `expired` | a `schedule.deactivateAt` that has passed — it can never be on again |
+| `notYetActive` | a future `activateAt` — pending, not dead |
+| `conditional` | genuinely selects between callers |
+
+**`usage`** is decided from observed evaluations, and exactly one state is a proof:
+
+| usage | meaning |
+|---|---|
+| `evaluated` | read server-side inside the threshold |
+| `stale` | reads EXIST in the window and the newest is older than the threshold. **Provable** |
+| `neverObserved` | no server-side read at all. Consistent with "dead" AND with "declared last Tuesday" — reported, never asserted, **never a removal candidate** |
+| `untracked` | nothing is recording |
+
+`removalCandidate` is set only by a proof: a constant/expired shape, or `stale`.
+
+### What it cannot see
+
+Shipped in the payload and rendered in the panel, not buried here:
+
+- **Reachability is not decided.** "Not evaluated since \<date>" is a measurement; "this code path is dead" is not decidable in general. A Black-Friday flag, a flag behind an admin route nobody visited this month, and a flag whose last call site was deleted are indistinguishable.
+- **Only SERVER-side reads count** — `isFlagEnabled` / `requireFlag` / `flagValue` / a `gatedBy` interception. `useFlag()` in the browser reads from the bulk set the server already sent, so the key never arrives as a named read. Bulk deliveries are recorded separately and never counted as use: one `useFlags()` poll evaluates the whole registry and would otherwise mark every flag in the app alive forever.
+- **The window is finite**, bounded by the observation table's retention. A flag last read BEFORE the window has no observation at all and reads `neverObserved` — exactly what a flag declared this morning reads.
+- On the day you turn tracking on, nothing has been observed, so every flag is `neverObserved` and nothing is proposed for removal. It is that SPLIT that prevents the day-one "everything is dead" report, not a coverage gate on top of it: such a gate is unreachable, because the observation that dates a stale flag is itself inside the window.
+
+### Tunables
+
+```ts
+flagsPlugin({
+  store: 'postgres',
+  usage: {
+    track: true,            // default: on with store:'postgres' (there is nowhere to write on memory)
+    flushIntervalMs: 300_000,   // the report resolves to a DAY, so a tighter interval buys nothing
+    staleAfterDays: 30,
+    retentionDays: 90,      // also VOLTRO_FLAG_USAGE_TTL_HOURS; the report's observation ceiling
+  },
+})
+```
+
+Observations land in `_voltro_feature_flag_usage`, one row per (flag, UTC day, source), retention-swept. `track: true` on the `memory` tier is refused at construction rather than silently observing nothing, and `staleAfterDays > retentionDays` is refused because staleness could then never be proven.
+
+## A flag can carry an experiment
+
+`defineExperiment` (`@voltro/runtime`) maintains standing A/B results as **live IVM aggregates recomputed per-write from CDC deltas** — real-time uplift with no batch pipeline. A flag can name one:
+
+```ts
+export const checkoutButton = defineFlag({
+  key: 'checkout.button',
+  value: Schema.Literal('blue', 'green'),
+  default: 'blue',
+  variants: [{ name: 'control', value: 'blue' }, { name: 'green', value: 'green' }],
+  experiment: 'checkout-colour',
+})
+
+// apps/api/experiments/checkoutColour.experiment.ts
+export default defineExperiment({
+  name: 'checkout-colour',
+  on: { table: 'orders' },
+  variantFrom: 'checkoutArm',                        // ← the arm the FLAG served
+  variants: [{ name: 'control' }, { name: 'green' }],
+  metric: { kind: 'conversionRate', column: 'completed' },
+})
+```
+
+Persist the served arm on the row you want to measure:
+
+```ts
+await ctx.store.insert('orders', { …, checkoutArm: flagVariant(ctx, checkoutButton) })
+```
+
+**Why `variantFrom` and not just a name.** The flag assigns by hashing FNV-1a over `${key}:variant`; an experiment in `subject` mode hashes over the EXPERIMENT name. Both are stable and uniform, and they are INDEPENDENT — roughly half the subjects served `green` land in the experiment's `control` arm. The uplift would be live, precise, and measuring a split nobody experienced. One assignment, persisted, read by the experiment.
+
+`flagsPlugin({ typedFlags, experiments })` **refuses to construct** when the link is wrong: a missing experiment, an experiment still in `subject` mode, an experiment-side holdout (carved AT assignment, which this experiment does not do), or arm names that do not match. Each of those is otherwise a wrong number rather than an error — a name mismatch shows up as a permanently-empty arm beside a permanently-full one, which reads as "the treatment has no effect".
+
 ## Three ways to use a flag
 
 **1. Declarative gate** — `gatedBy: { '<rpcTag>': '<flag>' }` (exact tag or `/regex/`). An off flag fails the call with typed `FlagDisabled` before the handler runs (merged into every procedure's wire-error union → typed on the client).

package/templates/agent-docs/plugins/governance.md

@@ -29,11 +29,11 @@ export default {
         { table: 'events', ttlMs: 90 * 86_400_000 },                                  // delete after 90 days
         { table: 'users', ttlMs: 365 * 86_400_000, action: 'anonymize', anonymizeFields: ['email', 'name'] },
       ],
-      // GDPR: which tables reference a subject + by which column.
+      // GDPR: DERIVE which tables hold a subject's data from the schema.
+      deriveSubjectScopes: { subjectTable: 'users' },
+      // …plus anything the schema cannot encode (see below).
       subjectScopes: [
-        { table: 'users', subjectField: 'id' },
-        { table: 'posts', subjectField: 'userId' },
-        { table: 'comments', subjectField: 'authorId' },
+        { table: 'audit_trail', subjectField: 'actorRef' },
       ],
       sweepIntervalMs: 3_600_000,   // default 1h
     }),
@@ -47,13 +47,111 @@ export default {
 
 ## GDPR — export + erasure (admin-gated)
 
-Two admin-only routes (guarded by `requireScope(ADMIN_SCOPE)`) walk `subjectScopes`:
+Two admin-only routes (guarded by `requireScope(ADMIN_SCOPE)`):
 
-- **`governance.export`** `{ subjectId }` → a portable bundle `{ [table]: rows[] }` of everything referencing the subject.
-- **`governance.erase`** `{ subjectId, mode? }` → deletes (or anonymises) the subject across every scope; returns an immutable `ErasureLogEntry` (`{ subjectId, at, mode, affected: [{ table, count }] }`).
+- **`governance.export`** `{ subjectId }` → a portable bundle `{ [table]: rows[] }` of everything belonging to the subject.
+- **`governance.erase`** `{ subjectId, mode? }` → deletes (or anonymises) the subject across every scope; returns an immutable `ErasureLogEntry` (`{ subjectId, at, mode, affected: [{ table, count }], truncated? }`).
 
 The same operations are available in-handler via `GovernanceService` (`exportSubject` / `eraseSubject`).
 
+Erasure runs **deepest-first** — children before parents — so a real foreign key
+neither refuses the delete nor cascades through rows the log entry never counted.
+
+### The subject scope is DERIVED from your schema
+
+A hand-written list of "every table holding this person's data" is wrong the day
+after someone adds a table — and that list *is* the compliance claim. So
+`deriveSubjectScopes: { subjectTable: 'users' }` walks the schema instead: the
+relations registry plus the `reference()` column graph, outward from the subject.
+
+That reaches rows a flat `{ table, subjectField }` entry cannot even express —
+`users → posts → comments` is two hops, so a comment on the subject's post is in
+the export without anyone listing `comments`.
+
+**Only CHILD edges are followed** — a table holding a reference to the subject's
+row. Never a parent or lookup edge, and a `manyToMany` follows the JUNCTION only.
+Getting that backwards would not be an over-broad export; it would be an erasure
+that walks from one member into their organisation and deletes everybody else's
+rows.
+
+| declaration | followed? |
+|---|---|
+| `many(posts, { foreignKey: 'authorId' })` | ✓ the target carries the FK |
+| `one(profile)` with the FK on the target | ✓ a 1:1 child |
+| `one(country, { sourceKey: 'countryId' })` | — a lookup this row points AT |
+| `manyToMany(orgs, { through: memberships })` | ✓ `memberships` only, never `orgs` |
+| a `reference()` column with no `relations()` block | ✓ found in the column graph |
+
+`subjectScopes` is still first-class and is **unioned on top**, never replaced.
+It is the only way to reach a link the schema does not encode: a subject id in a
+plain (non-`reference()`) column, a polymorphic `(ownerType, ownerId)` pair, an
+id inside JSON.
+
+### What the derivation cannot see — and says so
+
+A list of reachable tables, printed alone, reads as a completeness claim. So the
+blind spots ship in the same payload:
+
+```ts
+const gov = yield* GovernanceService
+const { paths, limitations } = gov.subjectGraph()
+```
+
+…and at `GET /_voltro/inspect/plugins/governance/subject-graph`. `limitations`
+names every table nothing links to the subject (`unreachable`), everything cut by
+the depth ceiling (`depth-truncated`, default 4 hops) and everything you excluded.
+Structurally outside the graph in every case: **object storage and uploaded
+files, external processors, log and metric sinks, backups, and any subject id
+embedded in JSON or free text.**
+
+The plugin also warns at boot if a derived scope reaches one table or fewer —
+that is what a typo'd `subjectTable` looks like, and it is otherwise
+indistinguishable from a working configuration until the first DSAR.
+
+### `voltro privacy`
+
+```bash
+voltro privacy scope --subject-table users        # the graph + its blind spots, OFFLINE
+voltro privacy scope --json                       # { reachable, unreachable, depthTruncated }
+voltro privacy export usr_123 --url https://api.example.com --out bundle.json
+voltro privacy erase  usr_123 --url https://api.example.com --confirm
+```
+
+`scope` needs no database and no running app — schema only, so it belongs in CI
+and in a PR review, where "does our erasure still reach every table" is a question
+somebody can still act on.
+
+`export` / `erase` deliberately go through the **running app's** admin-gated
+governance endpoint rather than opening their own connection. A CLI that erased
+directly would bypass your configured `anonymizeFields` and exclusions, bypass the
+erasure log (which is the compliance artefact, not a nicety), and work against a
+schema the deployed app may not be running. `erase` refuses without `--confirm`,
+without `--url`, and without an inspect credential.
+
+### Scale
+
+Every read is `WHERE <column> IN (<keys>)` on an indexed column, chunked at 500
+keys and memoised across paths that share a prefix — not a full table scan per
+scope. Per table, 50 000 rows is the ceiling; hitting it sets `truncated` on the
+erasure-log entry and exits non-zero from the CLI, because a short erasure
+presented as complete is exactly the failure this is built to prevent.
+
+### Crypto-shredding is NOT supported — and should not be faked
+
+"Erase a subject by destroying their key" needs a key **per subject**. The shipped
+cipher is one app-wide passphrase-derived key, so there is nothing subject-shaped
+to destroy: deleting it would make *every* subject's `.encrypted()` columns
+unreadable, which is an outage, not an erasure. Per-subject shredding needs
+envelope encryption — a DEK per subject, wrapped by a KEK, with every existing
+ciphertext re-wrapped — which is a re-architecture of the cipher rather than a
+mode of `eraseSubject`.
+
+What makes its absence cost less than it sounds: `.encrypted()` columns are
+decrypted transparently on read, so `delete` removes the ciphertext row and
+`anonymize` overwrites the ciphertext with a null. Both erase the data itself
+rather than the key guarding it. Crypto-shredding is an optimisation for erasure
+at rest across backups; it is not the only route to Art. 17.
+
 ## Consent ledger
 
 `governance.consent` `{ purpose, granted }` records the calling subject's decision; `governance.hasConsent` `{ purpose }` reads the latest (latest-write-wins per `(subject, purpose)`).

package/templates/agent-docs/plugins/multitenancy.md

@@ -1,6 +1,6 @@
 # Multi-tenancy
 
-> The tenant() schema mixin (auto-scope reads + auto-fill writes), the assertOwnTenant write-guard, and the typed TenantMismatch error.
+> The tenant() schema mixin (auto-scoped reads, auto-filled inserts, tenant-resolved keyed writes), the assertOwnTenant write-guard, and the typed TenantMismatch error.
 
 
 
@@ -9,7 +9,7 @@
 <!-- source: en/plugins/multitenancy.md -->
 ## Multi-tenancy
 
-_The tenant() schema mixin (auto-scope reads + auto-fill writes), the assertOwnTenant write-guard, and the typed TenantMismatch error._
+_The tenant() schema mixin (auto-scoped reads, auto-filled inserts, tenant-resolved keyed writes), the assertOwnTenant write-guard, and the typed TenantMismatch error._
 
 `@voltro/plugin-multitenancy` is the first-party multi-tenancy primitive. It has two surfaces: a **schema mixin** (`tenant()`) and a **write-time guard** (`assertOwnTenant` + the typed `TenantMismatch` error).
 
@@ -37,11 +37,13 @@ for projects that don't want the full barrel.
 What the runtime does for a `tenant()`-marked table:
 
 - **Reads are auto-scoped.** The runtime AND-merges `eq('tenantId', subject.tenantId)` into every subscription predicate against the table — tenant A never sees tenant B's rows, and a write in tenant A never wakes a subscription in tenant B.
-- **Writes are auto-filled.** On insert, `tenantId` is stamped from `ctx.request.subject.tenantId` when the caller didn't pass it explicitly. The wrapper never overrides a value the caller did pass.
+- **Inserts are auto-filled.** On insert, `tenantId` is stamped from `ctx.request.subject.tenantId` when the caller didn't pass it explicitly. The wrapper never overrides a value the caller did pass, and it refuses the insert outright when the subject has no tenant.
+- **Set-based writes are auto-scoped.** `updateMany` / `deleteMany` and the fluent `update(t).where(...)` / `delete(t).where(...)` builders get the same `eq('tenantId', …)` AND-merged onto their `WHERE`, so a tenant-blind predicate is confined rather than executed as written.
+- **Keyed-by-id writes are resolved inside the tenant.** `update(t, id, patch)`, `delete(t, id)`, `hardDelete(t, id)` and `patchJson(t, id, …)` address a row by primary key, so the runtime resolves that key within `subject.tenantId` first and fails with `TenantRowNotFound` (`@voltro/runtime`) when there is no such row there. The error is raised identically whether the row is missing or belongs to another tenant — reporting the two differently would let a caller probe for row ids in other tenants.
 
 ## The write-guard — `assertOwnTenant`
 
-Read isolation is automatic. **Writes are not** — a mutation receives raw, user-supplied input including `tenantId`, so without an explicit check a client authenticated as tenant A could submit `tenantId: 'B'` and land the row in B's data. Guard every tenant-scoped mutation:
+Isolation itself is enforced by the runtime on every path above. What `assertOwnTenant` covers is the one question the framework deliberately does not answer for you: a mutation whose input carries an explicit `tenantId` it intends to USE. Without a check, a client authenticated as tenant A could submit `tenantId: 'B'` and your handler would happily read that claim. Guard such a mutation:
 
 ```ts
 import { assertOwnTenant, TenantMismatch } from '@voltro/plugin-multitenancy'
@@ -54,6 +56,8 @@ const execute = async (input: { tenantId: string }, ctx) => {
 
 `assertOwnTenant(inputTenantId, subject)` throws `TenantMismatch` when `inputTenantId` doesn't equal the subject's `tenantId`. Anonymous subjects have no tenant scope at all, so the guard always throws for them — anonymous + tenant-scoped writes need an `apiKey` / `serviceAccount` subject instead.
 
+It checks a **claimed** `input.tenantId`, so a mutation whose input carries none never reaches it. That is why it is an early, typed convenience and not the boundary — the boundary is the store enforcement listed above.
+
 ## The typed error — `TenantMismatch`
 
 `TenantMismatch` is a `Schema.TaggedError` carrying `inputTenantId` + `subjectTenantId` (empty string for anonymous subjects). Declare it on the mutation's `error:` schema so the rpc layer surfaces the rejection typed. **Import it from the browser-safe `@voltro/plugin-multitenancy/guard` subpath in the descriptor (`*.mutation.ts`)** — the package root also re-exports the schema mixin, which pulls `@voltro/database` into the client rpcGroup bundle (a browser-safety violation):
@@ -65,6 +69,7 @@ import { Schema } from 'effect'
 
 export const createProject = defineMutation({
   name:   'projects.create',
+  guards: [{ scope: 'projects:write' }],   // WHO may call; TenantMismatch bounds WHICH tenant
   input:  Schema.Struct({ tenantId: Schema.String, name: Schema.String }),
   output: Schema.Struct({ id: Schema.String }),
   error:  TenantMismatch,

package/templates/agent-docs/plugins/presence.md

@@ -45,7 +45,16 @@ const Room = ({ channel }: { channel: string }) => {
 }
 ```
 
-`usePresence(channel, opts)` heartbeats on an interval (`heartbeatMs`, default 15s) and publishes per-member `meta` (status, cursor position, …). The roster is **push-driven** — `presence.list` is a reactive plugin query (`source: '_voltro_presence'`), so the framework pushes a fresh roster over the subscription transport on every roster change (heartbeat / leave / sweep / a peer departing) with NO client polling. `key` defaults to the subject id; pass an explicit `key` for anonymous members.
+`usePresence(channel, opts)` heartbeats on an interval (`heartbeatMs`, default 15s) and publishes per-member `meta` (status, cursor position, …). The roster is **push-driven** — `presence.list` is a reactive plugin query whose `source:` is a [reactivity channel](/docs/data/subscriptions#reactivity-channels), so the framework pushes a fresh roster over the subscription transport with NO client polling. `key` defaults to the subject id; pass an explicit `key` for anonymous members.
+
+A member is `{ key, meta }`. **It pushes only when the roster actually moves** — a join, a leave, a change to someone's `meta`, a sweep, a peer's delta, a peer's death. A heartbeat that repeats what the server already knows pushes nothing, which is what keeps a large steady room free.
+
+> **There is a second `usePresence`, and it is a different hook.**
+> [`@voltro/local-first/react`](/docs/local-first/overview#presence--awareness)
+> exports one for peer-to-peer *awareness* — `usePresence(roomId, self, { channel })`
+> → `{ presence, others, setPresence }` — carrying high-frequency cursor and
+> selection state over a pub/sub channel. This one is the server-backed roster.
+> Different packages, different signatures; pick by the question you are asking.
 
 ## Typing indicator
 
@@ -85,7 +94,9 @@ an event declared `delivery: 'latest'` rather than in presence metadata.
 
 ## Notes
 
-- The roster is push-driven: `presence.list` is a reactive plugin query (`source: '_voltro_presence'`), so the framework re-runs it and pushes deltas over the subscription transport on every roster change — no client polling. There is no such TABLE any more; the name survives purely as the reactivity key the framework routes on, which is why the roster push needed no second mechanism when presence moved into memory. Any plugin query route that names a `source` table becomes push-driven the same way.
+- The roster is push-driven: `presence.list` declares a [reactivity channel](/docs/data/subscriptions#reactivity-channels) as its `source:`, so the framework re-runs it and pushes deltas over the subscription transport — no client polling. There is **no table**, and there is no longer a table NAME either: presence used to declare `_voltro_presence` and never write to it, purely to own a name the reactivity layer would route on. If you have an existing `_voltro_presence`, it is empty and the upgrade does not drop it (the differ never plans a drop for a framework table no app declares) — remove it by hand when convenient.
+- **A repeat heartbeat does not push.** Measured on this machine at 2.7 µs per subscriber per publish, an unconditional push cost a steady room of N clients N² × that per heartbeat interval — ~107 ms of CPU per 15s at N=200, and a per-node ceiling around 750 subscribers that nothing in the app controlled. Publishing only on a real change removes that term entirely; what remains is linear in actual roster churn. Reproduce with `node --import tsx packages/plugin-presence/scripts/rosterFanoutBody.ts`.
+- **A member has no `lastSeen`.** It used to, and it was the owning replica's clock — "active 3 minutes ago" rendered from it is wrong by whatever the skew between two pods is. Use `meta` for anything you need to show.
 - A member counts as online for `timeoutMs` after its last heartbeat. The sweep runs every `timeoutMs / 3`, so a vanished member is gone within roughly 1.3× the window.
 - The sweep needs **no cluster coordination**, and that follows from the design rather than being a shortcut: every member is owned by exactly one replica and nobody else may touch it, so each replica sweeps its own and there is nothing to contend over. (The table version *did* need coordination — its rows were shared.)
 

package/templates/agent-docs/plugins/ratelimit.md

@@ -22,6 +22,15 @@ upgrade — never the individual calls. The Effect-native rpc interceptors are
 the only surface that sees each call **and** the resolved subject (needed for
 per-tenant limits).
 
+> **This plugin is opt-in, and nothing rate-limits your RPC surface until you add it.**
+> There is no built-in per-IP, per-API-key or per-tenant request cap — an app that
+> does not install and configure `@voltro/plugin-ratelimit` accepts calls as fast as
+> they arrive. The one throttle that *is* on by default is `plugin-auth`'s
+> [brute-force lockout](/docs/authentication/passwords#brute-force-lockout) on sign-in,
+> and it only covers credential attempts. Put a per-IP cap at your ingress as well —
+> see [production hardening](/docs/deployment/production-hardening); that layer holds
+> per-IP state across replicas, which an in-process limiter cannot.
+
 Live — `demo.limited` is capped at 5/min per visitor; the 6th call within a
 minute fails with the typed `RateLimited` error (carrying `retryAfterMs`):
 

package/templates/agent-docs/plugins/search.md

@@ -24,15 +24,19 @@ export default {
   name: 'api',
   plugins: [
     searchPlugin({
-      // backend: defaults to memory (dev). Swap for production:
-      // backend: typesenseBackend({ url, apiKey }) | meilisearchBackend({ ... }) | algoliaBackend({ ... })
+      // backend: defaults to memory — dev + tests only. Under NODE_ENV=production
+      // the memory backend REFUSES TO BOOT (see "The memory backend refuses to
+      // boot in production" below). Name a durable engine for a deployment:
+      // backend: { engine: 'typesense', url, apiKey } | { engine: 'meilisearch', … } | { engine: 'algolia', appId, apiKey }
       indexes: {
         posts: {
           index: 'posts',
           tenantField: 'tenantId',                       // index documents carry the tenant → query scopes by it
+          queryableFields: ['title', 'body', 'status'],  // optional: fields a caller may filter/facet/highlight on
           map: (row) => ({ id: String(row.id), title: String(row.title), body: String(row.body) }),
         },
       },
+      // allowedEngineParams: ['query_by'],               // optional: widen the engineParams allowlist (see below)
     }),
   ],
 }
@@ -42,7 +46,54 @@ export default {
 
 The plugin declares `onChangeEvent`. On every committed write to a configured table, `applyChange` maps the row via `map(row)` and upserts (insert/update) or removes (delete) the index document. No `*.subscribe.ts`, no manual indexing calls — the tap is the single sync path, and it runs under both `voltro dev` and `voltro serve`.
 
-For pre-existing rows, `backfillIndex(backend, spec, rows)` indexes the rows you supply (call it from a `*.startup.tsx` or a one-off script) — or use the **Reindex** button / `POST /reindex` inspect endpoint, which reads the table's current rows for you.
+For pre-existing rows, `backfillIndex(backend, spec, rows)` indexes the rows you supply (call it from a `*.startup.tsx` or a one-off script) — or use the **Reindex** button / `POST /reindex` inspect endpoint, which reads the table's current rows for you. The endpoint **streams** the table (keyset-paginated) and upserts one bounded page at a time — `sync.reindexBatchSize`, default 1000 rows — so memory stays flat no matter how large the table is. `backfillIndex` itself stays a plain array API for small explicit seeds.
+
+## When the engine is down
+
+The database commit has already happened by the time the tap runs, so a failed index write is **drift**: the row is in your database and missing from — or stale in — your index. That is not left to a log line.
+
+1. **Retry.** A failure the backend marks *transient* (network, engine unavailable, 5xx, rate limited) is retried inside the tap's own Effect with capped exponential backoff. A *permanent* failure — an unsupported query shape, a `map(row)` that throws on one row's shape — is **not** retried: repeating it cannot succeed, and it lands in step 2 immediately.
+2. **Record.** A change that outlives the retry is written to the framework-owned `_voltro_search_drift` table, one row per `(index, source row)`. Nothing is lost at that point: the entry lives in your own database — the thing that just committed successfully — while the search engine is what is down.
+3. **Repair.** A cluster-coordinated sweep re-reads each recorded row **from the database** and re-derives its document. It never replays the failed event, and that is what makes repair order-free and idempotent: a row updated three times during an outage converges in one pass, and a row deleted since the failure converges to a removal.
+
+The one case that is still a loss is the engine failing **and** the ledger write failing — and that one fails the tap loudly rather than reporting success.
+
+Every number here is yours to set (defaults shown):
+
+```ts
+searchPlugin({
+  indexes: { /* … */ },
+  sync: {
+    retries: 5,               // retries after the first attempt; transient failures only. 0 disables retry
+    retryBaseDelayMs: 200,    // first backoff — doubles per attempt
+    retryMaxDelayMs: 10_000,  // ceiling for any single wait
+    resyncIntervalMs: 60_000, // repair-sweep interval. 0 turns the sweep off (POST /resync still repairs on demand)
+    resyncBatchSize: 200,     // max ledger entries repaired per sweep
+    reindexBatchSize: 1000,   // rows per page for POST /reindex (streamed keyset walk — memory stays flat)
+    statsFlushIntervalMs: 5_000, // buffered sync-counter flush cadence. 0 = one durable write per event
+    statsFlushMaxBuffered: 1000, // flush early once this many counts are pending
+  },
+})
+```
+
+Drift is visible **without reading logs**:
+
+| Endpoint | Shows |
+|---|---|
+| `GET /_voltro/inspect/plugins/search/indexes` | per index: `dropped`, `pendingDrift`, `lastDriftAt`, `drifted` — plus the retry policy actually in force |
+| `GET …/search/drift` | the failing rows themselves, oldest first, with their last error and attempt count |
+| `POST …/search/resync` | runs a repair pass now (`{ scanned, repaired, failed }`) |
+
+An entry whose `attempts` keeps climbing is telling you something a retry cannot fix — a `map(row)` that throws on that row, a document the engine rejects. That is the case to look at by hand; everything else drains on its own.
+
+`_voltro_search_drift` is applied by the declarative differ on `voltro db apply` and on a `voltro dev` boot, on every dialect — there is nothing to migrate.
+
+### Under multiple replicas
+
+Index **writes** and index **counters** are deliberately treated differently when your store's change scope is `fleet` (every replica receives every change — postgres `LISTEN/NOTIFY` CDC, for instance):
+
+- **The write runs on every replica.** An `upsert`/`remove` of the same document is idempotent, so a duplicate costs write amplification — while electing a single writer would cost a *lost* update whenever that replica dies mid-change, and would import the leadership-gap window with it.
+- **The count runs everywhere too, but is aggregated as a maximum.** Each replica's stats row is already a fleet-wide count of the same changes, so `/indexes` takes the highest rather than the sum — the panel reports one sync per change, not one per replica. Under `local` scope only the replica that actually made the write counts (peers see an echo), and the rows are summed. Neither path needs a leader, so neither has a window in which counting stops.
 
 ## Querying
 
@@ -90,27 +141,130 @@ const res = await backend.query('posts', {
 - **facets** — per-value counts for the named fields (over the full matched set, before paging).
 - **highlight** — matched-term snippets per field, returned as `hit.highlights[field]`.
 - **fuzziness** — a max edit distance (`0` = exact) or `'auto'`.
-- **engineParams** — an escape-hatch bag forwarded verbatim into the underlying engine's search call. Ignored by the memory backend.
+- **engineParams** — the engine's **presentation-only** params (see the next section). Ignored by the memory backend.
 
 Native support degrades honestly: memory, Typesense, Meilisearch and Algolia all do filters/facets/highlighting; Typesense honors a numeric typo count (`num_typos` 0–2) while Meilisearch and Algolia only toggle typo tolerance on/off (`fuzziness: 0` disables it, other values keep their built-in tolerance). A backend op fails with a typed `SearchBackendError`.
 
+## What the server validates
+
+`search.query` is a public wire surface, and three of its inputs — the index name, the field names, and `engineParams` — become the search engine's **control plane**. The plugin validates all three server-side, before the engine sees them, and refuses with a typed error rather than answering a query it cannot scope:
+
+| Input | Rule | On violation |
+|---|---|---|
+| `index` | must be one of the indexes you declared in `searchPlugin({ indexes })` | `SearchIndexNotFound` |
+| `filters[].field`, `facets[]`, `highlight.fields[]` | a plain field path (`^[A-Za-z_][A-Za-z0-9_.]*$`), and — if the index declares `queryableFields` — one of those | `SearchFieldRejected` |
+| `engineParams` | only the engine's presentation-only keys (paging, ordering, typo tolerance, highlight shaping) | the key is dropped and logged |
+
+```ts
+import { SearchFieldRejected, SearchIndexNotFound } from '@voltro/plugin-search'
+```
+
+Why each one, because the reasoning is what tells you whether your own declaration is tight enough:
+
+- **An unknown index has no spec, therefore no `tenantField`, therefore no tenant clause.** Answering it would run the caller's query unscoped against whatever collection of that name exists on your engine — and a Typesense / Meilisearch / Algolia instance is usually shared with indexes this app never declared. It is refused, not answered empty.
+- **A field name is spliced into the engine's filter DSL.** Typesense's `filter_by` is one flat string that supports `||`, so a crafted name can re-group the boolean tree around the tenant clause appended after it. The identifier pattern is the floor every index gets; `queryableFields` narrows it further to what your UI actually needs. Your `tenantField` does **not** belong in that list — the tenant clause is injected after this check, by the server, and is never a caller's to name.
+- **`engineParams` is not a filter hatch.** It used to be merged last into the engine's params, so a caller could set `filter_by` (Typesense), `filter` (Meilisearch) or `facetFilters` (Algolia) and overwrite the tenant clause. Keys that could select a different document set — those, plus `query_by`, `restrictSearchableAttributes`, `preset`, `pinned_hits`, `enableRules`, … — are dropped. Express a filter as a `filters[]` clause instead: those are validated *and* ANDed with the tenant scope rather than replacing it.
+
+If your app genuinely needs one more engine key, widen the allowlist **server-side**, where it is a deliberate decision instead of a caller's:
+
+```ts
+searchPlugin({
+  indexes: { /* … */ },
+  allowedEngineParams: ['query_by'],   // this app trusts callers with this key
+})
+```
+
+Document-selecting keys stay refused even when listed there — that is the authority the tenant clause holds.
+
 ## Backends
 
 | Backend | Notes |
 |---|---|
-| `memoryBackend` (default) | Fully in-process; dev + tests. Not for production scale. |
+| `memoryBackend` (default) | Fully in-process; dev + tests. **Refuses to boot under `NODE_ENV=production`** — see below. |
 | `typesenseBackend` | Optional dep `typesense`. Lazy-loaded. |
 | `meilisearchBackend` | Optional dep `meilisearch`. Lazy-loaded. |
 | `algoliaBackend` | Optional dep `algoliasearch`. Lazy-loaded. |
 
 A backend is the `SearchBackend` interface (`upsert` / `remove` / `query`) — bring your own (OpenSearch, Elastic, …).
 
+### The memory backend refuses to boot in production
+
+This is a **refusal, not a warning, and not a scale caveat**. Under
+`NODE_ENV=production` on the in-memory backend, `onActivate` throws
+`SearchBackendNotDurable` and the process does not start:
+
+```text
+plugin-search refuses to boot in production on the in-memory backend.
+
+The memory index lives in THIS process's heap. Two consequences, both silent:
+  • every replica holds a different index, so a result depends on which replica served you;
+  • the index starts EMPTY after every restart/deploy, and nothing re-seeds it automatically.
+
+Configure a durable engine in app.config.ts:
+  searchPlugin({ backend: { engine: 'typesense',    url: …, apiKey: … }, indexes })
+  searchPlugin({ backend: { engine: 'meilisearch',  url: …, apiKey: … }, indexes })
+  searchPlugin({ backend: { engine: 'algolia',      appId: …, apiKey: … }, indexes })
+or pass your own `SearchBackend` implementation.
+```
+
+**Running one replica does not make it correct**, which is why the refusal is
+not conditional on detecting a cluster. Two independent things are wrong with an
+in-process index in a deployment and only one of them is the multi-replica
+story:
+
+1. **Per-process.** N replicas hold N divergent indexes. Which results you get
+   depends on which replica served the request — including *zero hits* for a
+   document that demonstrably exists.
+2. **Non-durable.** The index lives in the heap, so every restart and every
+   deploy starts EMPTY and nothing re-seeds it: `backfillIndex` is a function
+   your app calls, not something the plugin does at boot.
+
+(2) is what a single replica does not fix. It only removes one of the two ways
+the backend is wrong.
+
+Dev is **silent** — the memory backend is exactly right there, and a warning
+that fires on every `voltro dev` boot is a warning nobody reads.
+
+#### The escape hatch — `singleProcessMemoryIndex`
+
+If this deployment genuinely is ONE process that re-seeds its index at startup,
+say so and the boot proceeds:
+
+```ts
+searchPlugin({
+  singleProcessMemoryIndex: true,   // exactly one process, and it calls backfillIndex at startup
+  indexes: { /* … */ },
+})
+```
+
+Two things to be clear about before you reach for it:
+
+- **It is a claim about your topology, not a mute switch.** The plugin holds you
+  to both halves: exactly one process serves search, and a `*.startup.tsx`
+  calls `backfillIndex` for every index — because the index *is* empty after
+  each restart until something fills it. The boot logs a `note` at info
+  restating what you signed up for.
+- **The plugin checks the claim against reality.** When the instance-membership
+  registry reports that a peer replica joined, the plugin logs that the
+  declaration is now false — in *any* environment, because a peer announcing
+  itself is an observation rather than a guess about `NODE_ENV`:
+
+  ```text
+  search: replica "<id>" joined, but this app declared `singleProcessMemoryIndex: true`.
+  That declaration is now false: each replica has its own in-memory index, so search
+  results depend on which one serves the request. Configure a durable backend
+  (typesense / meilisearch / algolia) or run one process.
+  ```
+
+  Without the declaration, a peer joining while search is served from a heap
+  index warns too — same observation, different wording.
+
 ## Dashboard panel
 
-Both dashboards ship a **Search** panel (api apps): the configured indexes with per-index sync stats (docs synced/removed, last reindex) + the resolved backend, and a **Reindex** button per index that re-seeds it from the table's current rows (`backfillIndex`). Reindex gates on the `canReindexSearch` capability. Backed by `/_voltro/inspect/plugins/search/{indexes,reindex}`.
+Both dashboards ship a **Search** panel (api apps): the configured indexes with per-index sync stats (docs synced/removed/dropped, last reindex) + the resolved backend, per-index **drift badges** (`pendingDrift` / `drifted` / last drift time), the **repair queue itself** (the drift-ledger rows, oldest first, with attempt counts and the engine's last error), and two actions — **Reindex** per index (streams the table's current rows back in) and **Resync now** (repairs the drifted rows on demand). Reindex gates on the `canReindexSearch` capability, resync on `canResyncSearch`. Backed by `/_voltro/inspect/plugins/search/{indexes,drift,reindex,resync}`.
 
-The sync stats behind this panel are **durable and aggregated across replicas**. They live in a framework-owned `_voltro_search_stats` table (contributed via `extendSchema.tables`; the plugin declares `store:write`), one row per `(index, replica)`, each bumped with an atomic compare-and-set. `/indexes` sums every replica's row and takes the most-recent reindex — so the counts are truthful under multiple instances and survive a restart (they no longer reset with the process, and no replica under-reports the fleet total). Zero-infra dev/tests use an in-process stats store; `bindDataStore` swaps in the durable one at boot.
+The sync stats behind this panel are **durable and aggregated across replicas** — and counted **in memory first**: each replica buffers its per-event counts and flushes them to the stats table once per window (`sync.statsFlushIntervalMs`, default 5 s; early once `statsFlushMaxBuffered` counts are pending), so an indexed-table write never pays a per-write read+CAS against your primary. A graceful shutdown flushes the tail; a hard crash loses at most the current window of *counters* (never a change — the drift ledger, not these counters, is the durable record of what did not reach the engine). `GET /indexes` drains the buffer before reading, so the panel is always current. `statsFlushIntervalMs: 0` restores one durable write per event. They live in a framework-owned `_voltro_search_stats` table (contributed via `extendSchema.tables`; the plugin declares `store:write`), one row per `(index, replica)`, each bumped with an atomic compare-and-set. `/indexes` aggregates every replica's row — summed under `local` change scope, maxed under `fleet` (see [Under multiple replicas](#under-multiple-replicas)) — and takes the most-recent reindex, so the counts are truthful under multiple instances and survive a restart. Zero-infra dev/tests use an in-process stats store; `bindDataStore` swaps in the durable one at boot, along with the drift ledger.
 
 ## Permissions
 
-`store:changes:read` (the ChangeEvent tap) + `store:write` (the durable `_voltro_search_stats` counters) + `inspect:read` (dashboard panel) + `network:outbound:<host>` (for a remote backend).
+`store:changes:read` (the ChangeEvent tap) + `store:write` (the durable `_voltro_search_stats` counters and the `_voltro_search_drift` ledger) + `inspect:read` / `inspect:write` (dashboard panel, incl. reindex + resync) + `network:outbound:<host>` (for a remote backend).