@voltro/cli 0.31.0 → 0.33.0

package/templates/agent-docs/deployment.md

@@ -1168,13 +1168,38 @@ scaled. An operator's second pod died on `Connection timed out` for exactly
 this. `voltro serve` now prints the number and the arithmetic at boot:
 
 ```
-db pool: max=10 per replica (DB_MAX_CONNECTIONS) × 4 replicas = up to 40 connections.
+db pool: max=10 per replica (DB_MAX_CONNECTIONS) + 1 = 11 × 4 replicas = up to 44 connections.
+Check that against your database's limit. PLUS 1 outside the pool (CDC LISTEN consumer) —
+those do not come out of the pool budget, they come out of the DATABASE's.
 ```
 
 Set `REPLICA_COUNT` from your deployment (Helm: `{{ .Values.replicaCount }}`) and
 the line does the multiplication for you; without it the line still names the
 formula. `voltro dev` deliberately does not print it — one process, no replicas.
 
+**Some connections are not in the pool, and the count is per process.** A
+connection running `LISTEN` cannot be returned to a pool, so the driver opens a
+standalone one. There are three such places and a full deployment can hold all
+three:
+
+| Process | Connection | When |
+|---|---|---|
+| api `voltro serve` | CDC `LISTEN` consumer | `changeStrategy: 'cdc'` (the default on postgres) |
+| web `voltro start` | ISR invalidator `LISTEN` | any page declares `cacheInvalidatesOn` |
+| web `voltro start` | postgres ISR cache client | `SSR_CACHE=postgres` |
+
+The third is not a `LISTEN` at all, which is why counting `LISTEN` rows in
+`pg_stat_activity` undercounts. Each process prints its own number in the boot
+line above — including `No connections outside the pool in this process` when
+there are none, so "counted, zero" is distinguishable from "not counted".
+
+**And a rolling update needs the surge pod's connections too.** A budget sized
+for `replicaCount` is exactly full at steady state and short during every
+deploy: `maxSurge` adds a pod that opens a full pool of its own. If that pod
+cannot connect it never becomes ready, so the rollout does not complete and the
+cluster stays at the higher pod count — the deploy cannot free itself. Size for
+`(replicaCount + maxSurge) × (DB_MAX_CONNECTIONS + out-of-pool)`.
+
 **`POD_IP` is each replica's identity, not only a workflow setting.**
 
 ```yaml
@@ -1203,6 +1228,67 @@ operator's own guidance, and better than ours was: a value taken from the
 release namespace cannot be left out of one environment's config file, because
 there is no file to forget.
 
+### The framework's own background pollers
+
+Two framework tasks ride the coordinated scheduler and write a row into
+`_voltro_schedule_claims` on every tick they win: the workflow **admission
+drainer** and the offloaded-**inference dispatcher**.
+
+**Where a peer replica's write is visible here, they do not poll at all.** Each
+runs one tick at startup — not optional; it is what finds work a previous
+process left behind — and then stops until something arrives. The wake comes
+from the change events their queue tables already emit, which is the same
+mechanism the rest of the framework's reactivity runs on. Measured against a
+real Postgres on a deployment that never uses either queue: **2 claim rows in
+five minutes**, one per task, both written at boot.
+
+That "where" is the whole condition, and it is satisfied by Postgres
+LISTEN/NOTIFY or by a broadcast broker (Redis/NATS — which a multi-replica
+deployment already runs for cross-replica reactivity). Without either, a
+*remote* replica's enqueue produces no local event, so stopping would mean
+sleeping through it. There the tasks back off to a ceiling instead:
+
+```sh
+VOLTRO_POLL_CEILING_MS=30000   # how long an arrival can wait when NOTHING woke the task
+```
+
+Nothing is lost in that case either — the replica that enqueued always sees its
+own write inline and drains it itself. What the ceiling covers is the narrower
+case of a *crashed* writer's lease being reclaimed by someone else.
+
+A number worth knowing before you tune anything: on a two-replica deployment
+that had never enqueued into either queue, these two tasks accounted for **99.3 %
+of the claim ledger** — 2 506 rows an hour against 18 from the app's own eight
+schedules. A fixed interval has no way to learn a queue is empty. That is what
+changed; the ceiling is the fallback, not the fix.
+
+### Tuning the cadence
+
+The intervals are declarable, with defaults most apps never change:
+
+```ts
+// app.config.ts
+export default {
+  scheduling: {
+    admissionDrainMs: 1000,   // workflow admission drainer
+    inferenceTickMs:  250,    // offloaded-inference dispatcher
+    cancelSweepMs:    2000,   // cancelOn sweep
+    pollCeilingMs:    30000,  // idle ceiling, where nothing can wake a task
+  },
+}
+```
+
+Each has a matching env var — `VOLTRO_ADMISSION_DRAIN_MS`,
+`VOLTRO_INFERENCE_TICK_MS`, `VOLTRO_CANCEL_SWEEP_MS`, `VOLTRO_POLL_CEILING_MS` —
+which **overrides** the config field, the same way `VOLTRO_TENANT_ISOLATION`
+overrides `tenancy.isolation`. The config is what a project declares; the env var
+is what an operator changes on a running deployment without a rebuild.
+
+Lowering an interval does **not** make anything more responsive: an arrival
+already wakes the task at once. It only bounds the case where nothing announced
+the work. A value of `0` or a non-number is ignored rather than honoured — a
+zero interval would turn an idle task into a spin.
+
 ### Workflow failover across replicas
 
 On a **SQL store** (postgres / mysql / mariadb / mssql), durable workflows survive a replica crash: completed `step({...})` activities are checkpointed in the cluster journal, so when a replica dies mid-run, a **surviving replica takes over the run and continues it from the last completed step** — it replays the completed steps rather than re-running them. (On sqlite the engine is single-process — durable within one replica, no cross-replica failover.) Two requirements:

package/templates/agent-docs/plugins/audit.md

@@ -104,6 +104,30 @@ auditPlugin({ sink: 'datastore', redactInput: (e) => pick(e) })  // field-level
 
 **Why it is not driven by `.serverOnly()` / `.sensitive()`,** which is the obvious design: those markers live on TABLE COLUMNS, and this is a mutation's INPUT. A `changePassword({ oldPassword, newPassword })` has no column to consult, so a marker-driven default would cover exactly 0% of the case it exists for — while reading, to whoever configured it, like protection. (`.sensitive()` is also the [export axis](/docs/database/sensitivity), not "unsafe to log"; treating one as the other is the category error that page warns about.) Pass a function once you know your own inputs.
 
+### `redactOutcome` — what the call RETURNED
+
+**The default is `'all'`: `outcome.value` on success and `outcome.error` on failure are replaced by `{ __redacted: 'all' }`.** `kind`, `durationMs` and the error's `_tag` survive.
+
+```ts
+auditPlugin({ sink: 'datastore', redactOutcome: 'none' })              // the outcome verbatim
+auditPlugin({ sink: 'datastore', redactOutcome: (e) => summarise(e) }) // field-level control
+```
+
+**This is `redactInput`'s reasoning applied to the field it structurally cannot cover.** For a credential-ISSUING call the secret is never in the input:
+
+```ts
+apiKeys.createPersonalApiKey({ name, scopes })  // input: nothing sensitive
+  → { keyValue: '<the plaintext key>' }          // outcome: the whole point
+webhooks.create({ url, subscribedEvents })      // input: nothing sensitive
+  → { signingSecret: '<live secret>' }           // outcome: returned once, by design
+```
+
+The option that existed covers the field those calls leave empty. Found by a team running `voltro db scan-credentials` on the first release where it read the json blobs: nine rows of `_voltro_audit_log.outcome` matched, four of them `webhooks.create` carrying a live 64-character signing secret in full — with `redactInput: 'all'` and `redactSubject: 'metadata'` both already on.
+
+**The error's `_tag` survives on purpose.** A trail that records "something failed" without saying what is not a trail, and a tag is a schema-declared discriminant that structurally cannot be a secret. A `record` predicate still sees the live outcome, so a filter keyed on what a call returned keeps working — redaction applies to what is *stored*.
+
+> **A FUNCTION sink gets neither the table nor the retention sweep.** Both are gated on `sink` being the literal `'datastore'`, so a function that redacts and then delegates to `dataStoreAuditSink` still writes rows on a database where `_voltro_audit_log` already exists — while creating the table nowhere and arming the TTL nowhere. It works where you tested it and fails on the next fresh database. If you want the durable trail with different redaction, use `sink: 'datastore'` plus the three `redact*` options: those compose, the sink does not.
+
 ### `redactSubject` — what of the CALLER is kept
 
 **The default is `'metadata'`: `subject.metadata` is replaced by `{ __redacted: 'all' }`.** `type`, `id`, `tenantId` and `scopes` survive, which is everything the trail is actually read for.

package/templates/agent-docs/plugins.md

@@ -967,6 +967,56 @@ framework's ALREADY-OPEN handles so it never rebuilds them:
   one process needs no fan-out dedup). Returns a handle whose `stop()`
   cancels the task; the framework also stops every armed task at shutdown.
 
+  Each tick claims one row in `_voltro_schedule_claims`, so the interval you
+  pick is also a write rate: `250` is four rows per second, per task, fleet-wide.
+
+  **The interval is a floor, not a cadence — if you report idleness.** Return
+  `{ idle: true }` from the effect on a tick that found nothing to do, and the
+  runner backs off toward a 30 s ceiling (`VOLTRO_POLL_CEILING_MS`) instead of
+  ticking at your interval forever. Return nothing and the tick keeps its fixed
+  interval, which is the safe default for a task that cannot tell.
+
+  **Or stop it entirely.** Pass `{ disarmWhenIdle: true }` as a fourth argument
+  and an idle tick with nothing pending stops the timer altogether — the task
+  runs once at startup and then only when `wake()` says so. That is the
+  difference between a cheaper poller and no poller, and it is what the
+  framework's own two background tasks do.
+
+  Only pass it when an arrival is **guaranteed** to call `wake()`. That is a
+  claim about the deployment, not the task: with Postgres LISTEN/NOTIFY or a
+  broadcast broker every replica sees every write, so it holds. Without either,
+  a peer replica's enqueue produces no local event and a disarmed task would
+  sleep through it — the backoff ceiling is the correct choice there.
+
+  Two more fields make that affordable rather than a latency tax:
+
+  - `handle.wake()` runs a tick **now**. Wire it to whatever announces work —
+    a `store.onChange` on the queue table your plugin drains — and the work
+    starts on arrival rather than up to one interval later. Calls are coalesced
+    to at most one extra tick per interval, so calling it per row is fine.
+  - `{ idle: true, nextDueInMs }` caps the backoff. Return it when the task is
+    idle right now but already knows something is coming (a window closing, a
+    lease expiring); the next tick lands on that instant instead of on the
+    ceiling.
+
+```ts
+const outbox = ctx?.scheduleCoordinated('vendor.outbox', 1_000, async () => {
+  const sent = await drainOutbox()
+  return { idle: sent === 0 }
+}, { disarmWhenIdle: true })
+
+store.onChange((event) => {
+  if (event.table === 'vendor_outbox' && event.op === 'insert') outbox?.wake()
+})
+```
+
+  Why this exists: two framework tasks polled permanently-empty queues on a
+  consumer's deployment and wrote **2 506 claim rows an hour** between them,
+  against 18 from the app's own eight schedules. A fixed interval has no way to
+  learn that a queue is empty. Without a `wake()` source the ceiling is your
+  worst-case latency, so a task whose arrivals cannot announce themselves should
+  keep reporting nothing.
+
 ```ts
 import { definePlugin } from '@voltro/protocol'
 

package/templates/agent-docs/scheduling.md

@@ -346,7 +346,10 @@ replica B ─┼─ same bucket, same key ─┤
 replica C ─┘                          └─ PK conflict → stands down (no run row)
 ```
 
-- **Self-expiring.** The key includes the firing instant, so a crashed winner doesn't block the next firing — the next firing instant is a new key. Stale rows are pruned lazily.
+- **Self-expiring.** The key includes the firing instant, so a crashed winner doesn't block the next firing — the next firing instant is a new key.
+- **A claim does not outlive its bucket.** When a replica wins a claim it deletes that schedule's own older rows in the same pass, so the table's steady-state size is a small multiple of the number of schedules rather than a function of uptime. How far back it prunes scales with how far apart that caller's buckets are — a cron keeps roughly an hour of predecessors, a 250 ms coordinated task about a minute. The grace exists because deleting a claim too early is a **double fire**: a replica that is running late must still find the row that says its bucket was taken.
+- **Swept, on the scale it fills — on every dialect.** `_voltro_schedule_claims` gets a retention policy on both boot paths: rows older than **24 hours** by `claimedAt` are deleted, tunable with `VOLTRO_SCHEDULE_CLAIMS_TTL_HOURS`. The announcement at boot names it (`retention: N policy(ies) armed`). 24 h rather than the 30 days the framework's *history* tables get, because this is a lock ledger — a claim row answers a question about one firing instant, and nothing reads yesterday's. One row per (schedule, second-bucket) adds up faster than people expect: a deployment with a handful of sub-minute schedules measured 1 557 rows/hour, which a 30-day window would have let reach a million before the first row aged out. If you raise it, the number to reason about is the longest a replica may be paused and still be trusted not to re-fire a bucket it already lost. The sweep is the **backstop**, not the main bound — it is what cleans up after a schedule you renamed or deleted, which the per-schedule prune above can never revisit. (Through 0.32.0 the whole sweep was registered behind a postgres-only gate, so on the other four dialects it never ran and the boot said nothing about it. Fixed in 0.33.0.)
+- **Looked up by primary key.** The existence check the coordinator runs before its `INSERT` is a single-row read on `id`, which *is* the claim key — so the table's size does not enter the firing path.
 - **Pool-safe.** Unlike a session-level `pg_advisory_lock` (tied to a connection a pool may reassign), a claims row is durable and connection-independent.
 - **Fail-closed.** If the claims table is unreachable, the coordinator logs a warning and declines to fire rather than risk a double-fire. A missing run is recoverable via [backfill](/docs/scheduling/overlap-and-backfill); a double-fire (two charge emails) often isn't.
 

package/templates/agent-docs/security.md

@@ -95,7 +95,7 @@ voltro db scan-credentials
 voltro db scan-credentials --table my_events:actor
 ```
 
-Counts rows carrying a credential-shaped key — `token`, `secret`, `password`, `apikey`, `credential`, `privatekey` — plus any table you name with `--table <name>[:<column>]`. Exit code `1` on a hit, so CI can gate on it.
+Counts rows in which a credential-shaped **name** appears anywhere in the stored value — `token`, `secret`, `password`, `apikey`, `credential`, `privatekey` — plus any table you name with `--table <name>[:<column>]`. Exit code `1` on an unexplained hit, so CI can gate on it.
 
 By default it scans every column that can physically HOLD one:
 
@@ -106,6 +106,34 @@ By default it scans every column that can physically HOLD one:
 
 `input` and `outcome` are a procedure's arguments and result — the likeliest place of all for a token to arrive by accident, since nobody puts a credential on a subject on purpose and plenty of code passes one as an argument. `_voltro_row_history.data` is a full-row snapshot, so a credential column on *any* of your tables is in there, and it outlives deleting the source row.
 
+Every hit line names the needles that matched, with the number of rows each appears in:
+
+```
+✗  _voltro_audit_log.outcome — 69 of 149 row(s) contain a credential-shaped NAME (anywhere in the value)
+       matched (rows per needle, may overlap): token (61), secret (12)
+       examined 69 of 69 matched row(s): 61 as a JSON KEY, 8 only inside a redaction marker
+```
+
+**Those counts overlap and do not sum to the hit count** — a row holding both a token and a secret is counted by both needles. Without the breakdown the advice underneath (*purge them AND rotate the credentials*) is neither executable — purge what, rotate which — nor refutable: a column mentioning the word `token` in prose reads identically to one holding a live one.
+
+### Hits are explained, not just counted
+
+The predicate is a substring match over the whole serialized column, so a match is a match on a *name*, wherever it sits. The third line above is a bounded second pass that reads the matched rows back and says what actually matched — a JSON **key**, or only a **redaction marker**. Values are never printed and never logged.
+
+A redaction marker is the framework's own record that a credential was deliberately *not* stored: `_omitted` (`@voltro/plugin-versioning`, the column names left out of a row snapshot) and `__redacted` (`@voltro/plugin-audit`). A target whose every matched row is one of those is reported as explained, and does **not** fail CI:
+
+```
+~  _voltro_row_history.data — 69 of 149 row(s) contain a credential-shaped NAME (anywhere in the value)
+       matched (rows per needle, may overlap): token (69)
+       examined 69 of 69 matched row(s): 0 as a JSON KEY, 69 only inside a redaction marker
+       EXPLAINED — every match is `_omitted` / `__redacted`, the framework's own
+       record that a `.sensitive()` column was left OUT. Nothing to purge, nothing to rotate.
+```
+
+The bar for that is deliberately high: **every** matched row examined, **every** one of them a marker and nothing else. A target where the read-back was capped (at 500 rows), or where one row is a real key, or where one row is not parseable as JSON, stays a finding and still exits `1`. A partially-explained target is still a target.
+
+> **Why this exists.** A team with `.serverOnly().sensitive('secret')` columns and correctly-redacting plugins got 69 rows flagged, with *purge them AND rotate the credentials* underneath, against the framework's own proof that nothing was stored. The shape mattered more than the one key name: the more columns an app classifies correctly, the more markers it writes, and the redder the scan turns. A false-positive rate that rises with the care you take gets the tool muted — and a muted scan is worth less than none, because its silence still reads as evidence.
+
 > **If you ran this on 0.30.0, 0.30.1 or 0.30.2, run it again.** Those releases scanned `subjectId` on both tables — a flat opaque id that cannot hold a credential — and none of the blob columns above. The command ran cleanly, printed a scanned count beside a hit count, and exited `0` having never looked where credentials are. It came from fixing a crash: the default column had been the literal `subject` for every table, `_voltro_row_history` has no such column, and the fix replaced the name on *both* tables instead of the one that was wrong. A clean answer from a scan that looked in the wrong place is worse than the crash it replaced. `voltro update` prints this as a manual step on the way to 0.31.0.
 
 **It is a command and not a documented query on purpose.** The same check once shipped as SQL you were asked to run yourself, in its postgres spelling (`subject::text ILIKE '%token%'`). On MySQL/MariaDB the natural translation is a bare `LIKE` — and against the `utf8mb4_bin` collation the migrator emits for a `json()` column, `LIKE` is case-**sensitive**. So `'%token%'` does not match `jiraToken`, and almost every JSON key that carries a credential is camelCase. A team ran the translated query over 141 rows, got `0`, and nearly reported themselves clean; 117 of those rows held a working credential. Every dialect now casts to its own text type before lowering, in code you do not have to translate.
@@ -116,6 +144,8 @@ Run it on every environment. A development database is not a sample of productio
 
 If it finds something: purge the rows **and** rotate the credentials — assume anything written to a log table has been read — then move the credential off the Subject entirely with `connectionCredentials(...)`, which keeps it in the framework vault.
 
+If the finding is a credential column of your own sitting in plaintext, `.encrypted()` only protects rows written *after* you add it — [`voltro db encrypt-column`](/docs/database/sensitivity) converts the ones already there.
+
 ## Supply-chain assurance
 
 Every release passes automated supply-chain gates in CI before a single package is published: