@synapsor/runner 0.1.16 → 1.1.0

package/docs/release-policy.md

@@ -1,7 +1,10 @@
 # Release Policy
 
-Synapsor Runner `0.1.0` is the first stable local runner line. Use the stable
-package for normal installs:
+Synapsor Runner `0.1.0` was the first stable local runner line. Synapsor
+Runner `1.0.0` is the first production approval-loop semver line: batch apply,
+aggregate policy limits, verified operator identity, structured operations, and
+shared runtime-store deployment are part of the documented compatibility
+surface. Use the stable package for normal installs:
 
 ```bash
 npx -y -p @synapsor/runner synapsor-runner demo --quick
@@ -55,10 +58,58 @@ A stable `0.1.0` release should only be tagged after:
 - there are no known docs/code mismatches around transport, credentials,
   receipt tables, or handler expectations.
 
+## 1.0 Stability Gate
+
+Do not tag `1.0.0` only because the package is useful. `1.0.0` is the public
+semver contract for the Runner production approval loop, and it should be cut
+only after the following are true in the current tree and release artifacts:
+
+- batch apply can apply all approved proposals independently, reports
+  applied/conflict/skipped IDs, is safe to rerun through idempotency receipts,
+  and supports `--capability`, `--tenant`, and `--max`;
+- aggregate auto-approval limits are authored in DSL, represented in the
+  canonical contract spec, enforced as human-review fallback, persisted with
+  tripped-limit details, and visible in doctor/tools preview output;
+- approve/reject/apply can require a verified operator identity, enforce
+  contract reviewer roles and apply roles, and bind tamper-evident identity
+  records into the proposal ledger;
+- designed rejections and writeback outcomes emit structured logs, operational
+  counters are available per tenant and capability, and owner-only local store
+  permissions remain under test;
+- the documented public surfaces below have release-gate coverage from packed
+  artifacts, not only source-tree tests.
+
+After `1.0.0`, changes to the documented CLI, schema, contract, MCP result,
+writeback, approval, metrics, and replay surfaces must follow semantic
+versioning. Breaking changes require a new major version, except for security
+fixes that close a vulnerability while preserving the safest possible
+compatibility path.
+
+## 1.1 Fleet Gate
+
+Do not publish `1.1.0` until all 1.0 gates remain green and the local synthetic
+fleet verifier proves:
+
+- claim/context contradictions fail before serving;
+- two claim-bound Runners preserve tenant isolation and one active proposal;
+- shared fixed-window rate limits reject with `retry_after_ms`;
+- distinct verified reviewers satisfy canonical quorum and apply remains
+  blocked before `N/N`;
+- Postgres/MySQL pool acquisition and queue limits fail fast;
+- worker termination before write and after commit recovers without a duplicate
+  effect;
+- backup digest, clean restore, and archive-before-retention preserve active
+  records;
+- `/healthz`, `/readyz`, and separately authorized `/metrics` expose no secrets
+  or business object/principal labels.
+
+Run `corepack pnpm test:fleet`. Only homogeneous 1.1 fleet operation is part of
+this gate; mixed-minor operation must be tested before it is claimed.
+
 ## Stable Compatibility Promise
 
-Starting with `0.1.0`, Synapsor Runner keeps these public surfaces compatible
-through the `0.1.x` line unless a release note marks a deprecation first:
+Starting with `1.0.0`, Synapsor Runner keeps these public surfaces compatible
+through the `1.x` line unless a release note marks a deprecation first:
 
 - the `synapsor-runner` binary name and README quickstart commands;
 - `synapsor.runner.json` schema version `1` for documented fields;
@@ -73,8 +124,9 @@ through the `0.1.x` line unless a release note marks a deprecation first:
 Stable does not promise production SLA, hosted Cloud features, compliance
 certification, physical Postgres/MySQL branching, generic SQL writeback,
 generic multi-row writes, or compatibility for undocumented local SQLite
-internals. Local store migrations may happen inside `0.1.x`, but documented CLI
-inspection commands should remain the supported way to read the store.
+internals. Local store migrations may happen inside `1.x` and later minor
+versions, but documented CLI inspection commands should remain the supported way
+to read the store.
 
 Alpha users should pin an exact alpha version in package.json, CI, and MCP
 client snippets. Use `@alpha` only when intentionally testing the moving

package/docs/runner-config-reference.md

@@ -26,7 +26,7 @@ Unknown keys fail when `strict` is true (the default).
 | `mode` | Yes | `read_only`, `shadow`, `review`, or `cloud`. |
 | `result_format` | No | `1` legacy or `2` stable `ok/summary/data/proposal/error` envelope. Default `1`. |
 | `strict` | No | Reject unknown config keys. Default `true`. |
-| `storage` | Local | Local SQLite ledger. |
+| `storage` | Local | Local SQLite ledger plus optional shared Postgres mirror or runtime-store wiring. |
 | `sources` | Local | Named Postgres/MySQL source wiring. |
 | `trusted_context` | Conditional | Default trusted tenant/principal binding. |
 | `contexts` | No | Named trusted contexts referenced by capabilities/contracts. |
@@ -34,6 +34,10 @@ Unknown keys fail when `strict` is true (the default).
 | `capabilities` | Conditional | Embedded compatibility capabilities; may be `[]` with `contracts`. |
 | `policies` | No | Embedded reviewed policies; contract policies merge into the same catalog. |
 | `approvals` | No | Local approval overrides. |
+| `operator_identity` | No | Verified operator identity and apply-role wiring for approve/reject/apply. |
+| `session_auth` | HTTP claims | HS256 development or asymmetric RS256/ES256 session-token verification. |
+| `rate_limits` | No | Operational fixed-window limits; fleet-wide only with shared `runtime_store`. |
+| `metrics` | No | Separately authorized scrapeable HTTP metrics. Disabled by default. |
 | `executors` | No | App-owned writeback wiring. |
 | `cloud` | Cloud mode | Scoped Cloud adapter configuration. |
 
@@ -50,13 +54,54 @@ Unknown keys fail when `strict` is true (the default).
 ## Storage
 
 ```json
-{ "storage": { "sqlite_path": "./.synapsor/local.db" } }
+{
+  "storage": {
+    "sqlite_path": "./.synapsor/local.db",
+    "shared_postgres": {
+      "mode": "mirror",
+      "url_env": "SYNAPSOR_LEDGER_DATABASE_URL",
+      "schema": "synapsor_runner",
+      "lock_timeout_ms": 10000,
+      "max_entries": 10000
+    }
+  }
+}
 ```
 
 The ledger stores proposals, visible evidence copies, query audit, approvals,
 receipts, events, and replay. New POSIX stores are owner-only (`0600`). Protect
 the file like a database extract; see [Store Lifecycle](store-lifecycle.md).
 
+`storage.shared_postgres.mode = "mirror"` makes mutating CLI commands restore
+from the shared Postgres ledger before the local mutation and sync back after
+the command while holding a schema-scoped Postgres advisory lock. Mirror mode is
+a bounded operator handoff bridge; MCP serving still uses the local SQLite
+store.
+
+`storage.shared_postgres.mode = "runtime_store"` makes MCP serving use the
+shared Postgres ledger as the primary proposal/evidence/replay store. Runner
+creates a connection from the environment variable named by `url_env`, auto-runs
+the shared-ledger migration at startup, and uses a schema-scoped transaction
+advisory lock around runtime mutations. The URL is still referenced by
+environment variable name; never put the database URL in the config file.
+Bounded CLI approval/apply/worker commands bridge through the same Postgres
+ledger by restoring into a temporary local store under the Postgres advisory
+lock, running the existing mutation, and syncing back. Long-running worker
+processes repeat bounded drain cycles and release the advisory lock while idle.
+
+The bridge loads at most `max_entries` ledger records per operation. The default
+is 10,000; valid values are 100 through 100,000. It fails closed with
+`POSTGRES_RUNTIME_STORE_CAPACITY_EXCEEDED` above the bound. Reviewer CLI reads
+and the local UI use the same shared queue. This is a serialized small-fleet
+design, not an unbounded high-throughput primary store.
+
+`synapsor-runner doctor --config ./synapsor.runner.json` checks shared Postgres
+wiring when this block is present: the URL env var must be set, and the
+configured schema must already contain `ledger_entries`, `proposal_locks`,
+`worker_leases`, and `rate_limit_buckets`. Doctor reports env var names and
+table readiness only; it does
+not print database URLs or create the schema.
+
 ## Sources
 
 ```json
@@ -81,6 +126,13 @@ timeout. `ssl` carries adapter-specific reviewed SSL options when used.
 
 A contract's `SOURCE billing_postgres` must exactly match a `sources` key.
 
+Long-running servers reuse native driver pools. Optional `pool` keys are
+`max_connections` (default 10), `connection_timeout_ms` (3000),
+`idle_timeout_ms` (30000), `queue_timeout_ms` (5000), and `queue_limit`
+(default `max(10, max_connections * 4)`). Queue overflow returns
+`SOURCE_POOL_QUEUE_FULL`; acquisition timeout returns `SOURCE_POOL_TIMEOUT`.
+One-shot CLI commands close their pools.
+
 ## Trusted context
 
 ```json
@@ -100,6 +152,62 @@ Providers are `environment`, `static_dev`, `http_claims`, and `cloud_session`.
 Capabilities may reference a context by name. The model never receives tenant
 or principal as an overridable argument.
 
+For multi-tenant Streamable HTTP services, use `http_claims` plus signed
+session auth:
+
+```json
+{
+  "trusted_context": {
+    "provider": "http_claims",
+    "values": {
+      "tenant_id_key": "tenant_id",
+      "principal_key": "sub"
+    }
+  },
+  "session_auth": {
+    "provider": "jwt_hs256",
+    "secret_env": "SYNAPSOR_SESSION_JWT_SECRET",
+    "previous_secret_env": "SYNAPSOR_PREVIOUS_SESSION_JWT_SECRET",
+    "issuer": "https://identity.example",
+    "audience": "synapsor-runner"
+  }
+}
+```
+
+`previous_secret_env` is optional and is only for rotation windows. Runner tries
+the active secret first, then the previous secret. Existing MCP sessions remain
+bound to the exact token fingerprint, so clients cannot swap tenant/principal
+identity inside an initialized session.
+
+For networked deployments prefer asymmetric public-key verification:
+
+```json
+{
+  "session_auth": {
+    "provider": "jwt_asymmetric",
+    "algorithms": ["RS256"],
+    "jwks_url_env": "SYNAPSOR_SESSION_JWKS_URL",
+    "issuer": "https://identity.example",
+    "audience": "synapsor-runner",
+    "tenant_claim": "tenant_id",
+    "principal_claim": "sub",
+    "clock_skew_seconds": 30,
+    "jwks_cache_seconds": 600,
+    "jwks_cooldown_seconds": 30,
+    "fetch_timeout_ms": 3000,
+    "max_response_bytes": 1048576
+  }
+}
+```
+
+`jwt_asymmetric` requires an explicit `RS256`/`ES256` allowlist and exactly one
+public-key source: `jwks_url_env`, `public_key_env`, or `public_key_path`.
+JWKS selection uses `kid`; unknown keys get one controlled refresh. Fetches are
+timeout/size bounded and do not follow redirects. Private JWK fields are
+rejected. Every effective named context in an `http_claims` server must bind
+tenant and principal from claims; environment/static contradictions fail with
+`TRUSTED_CONTEXT_PROVIDER_CONFLICT` before serving.
+
 ## Contracts and embedded capabilities
 
 ```json
@@ -148,9 +256,104 @@ See [Writeback Executors](writeback-executors.md).
 
 `policies` is an array of reviewed policy objects. Local approval policies use
 `kind: "approval"` and numeric rules such as `{ "field": "amount_cents",
-"max": 2500 }`. `approvals.disable_auto_approval: true` disables local policy
-auto-approval without changing the reviewed contract. Approval never becomes an
-MCP tool.
+"max": 2500 }`. Canonical policies may also carry daily `count` or `total`
+limits scoped to `tenant_policy` or `tenant_policy_object`. These limits come
+from reviewed DSL `LIMIT` clauses, not an unreviewed local override. A tripped
+limit falls back to human review and is recorded in the ledger.
+`approvals.disable_auto_approval: true` disables local policy auto-approval
+without changing the reviewed contract. Approval never becomes an MCP tool.
+Portable `approval.required_approvals` defaults to `1` and accepts `1..10`.
+Distinct verified subjects fill quorum slots; one subject cannot count twice.
+A rejection is terminal. Policy auto-approval is deferred when quorum is
+greater than one.
+
+## Rate limits
+
+```json
+{
+  "rate_limits": {
+    "enabled": true,
+    "default": { "requests": 60, "window_seconds": 60 },
+    "capabilities": {
+      "billing.propose_refund": { "requests": 10, "window_seconds": 60 }
+    }
+  }
+}
+```
+
+Rules are fixed-window deployment controls keyed by trusted tenant plus
+reviewed capability. There is no separate burst allowance: all configured
+requests may arrive immediately, but the next request is rejected until the
+aligned window resets. They are process-local with SQLite and fleet-wide/atomic
+with shared `runtime_store`. `RATE_LIMITED` includes `retry_after_ms` to that
+reset boundary, records a counter, and creates no proposal. Rate limits are
+runtime wiring and never portable contract fields.
+
+## HTTP metrics
+
+```json
+{
+  "metrics": {
+    "enabled": true,
+    "token_env": "SYNAPSOR_METRICS_TOKEN"
+  }
+}
+```
+
+`/metrics` is disabled by default. Non-loopback exposure requires a separate
+metrics token; the MCP bearer does not authorize metrics. Labels are bounded to
+trusted tenant, capability, source, engine, and readiness component. Object
+IDs, principals, URLs, tokens, and raw errors are never labels.
+
+## Operator identity
+
+Local development compatibility uses an explicitly unverified environment
+identity:
+
+```json
+{
+  "operator_identity": {
+    "provider": "dev_env",
+    "actor_env": "SYNAPSOR_OPERATOR_ID",
+    "roles_env": "SYNAPSOR_OPERATOR_ROLES",
+    "apply_roles": ["writeback_operator"]
+  }
+}
+```
+
+Shared or production-like operation should register signed operator keys:
+
+```json
+{
+  "operator_identity": {
+    "provider": "signed_key",
+    "apply_roles": ["writeback_operator"],
+    "operators": {
+      "alice": {
+        "public_key_path": "./operators/alice.pub.pem",
+        "roles": ["support_reviewer", "writeback_operator"]
+      }
+    }
+  }
+}
+```
+
+The public key path resolves relative to `synapsor.runner.json`. Keep the
+private key outside the repository and invoke decisions with `--identity alice
+--identity-key /secure/path/alice.pem`. Runner verifies key possession and the
+contract's required role, then binds the exact proposal hash/version, action,
+subject, timestamp, roles, signature, and integrity hash into the approval
+ledger. `apply_roles` independently gates writeback. The local browser UI
+refuses approval/rejection in signed-key mode so it cannot bypass this check.
+
+For OIDC-style operator identity, use `provider: "jwt_oidc"` with the same
+asymmetric public-key options as session auth, plus one token source:
+`token_env`, `token_file_env`, or `token_stdin: true`. With stdin, pipe or paste
+one token and close stdin; combine it with `--yes` so confirmation does not
+compete for the stream. Also configure `roles_claim`, `subject_claim`, and an
+`attestation_secret_env` containing at least 32 bytes. Runner persists verified
+subject/roles/issuer/key ID and an attested decision proof, never the bearer
+token. Tokens must not be passed as command-line arguments.
 
 ## Cloud mode
 

package/docs/running-a-runner-fleet.md

@@ -0,0 +1,349 @@
+# Running A Small Runner Fleet
+
+Synapsor Runner 1.1 supports a bounded small-fleet deployment. It does not
+turn the OSS package into Synapsor Cloud and it does not promise an unbounded,
+multi-region, or SLA-backed control plane.
+
+```text
+TLS load balancer
+  -> N stateless Streamable HTTP Runners
+       -> shared Postgres runtime ledger
+       -> bounded Postgres/MySQL source pools
+  -> supervised writeback workers
+       -> restricted source writer or app-owned executor
+```
+
+The model still receives only reviewed semantic tools. Approval, apply,
+operator credentials, database credentials, and shared-ledger authority remain
+outside MCP.
+
+## Audit Before Deployment
+
+Audit the model-facing surface before connecting a production-like database:
+
+```bash
+npx -y @synapsor/runner audit --example dangerous-db-mcp
+synapsor-runner audit ./tools-list.json
+synapsor-runner audit https://mcp.example.com --bearer-env MCP_AUDIT_TOKEN
+```
+
+Treat generic SQL/query tools, model-controlled tenant/principal inputs, and
+model-facing approval/apply tools as blockers. Audit is a static review, not a
+security proof; follow it with `tools preview`, `doctor`, and the live fleet
+verification.
+
+## What 1.1 Guarantees
+
+With `storage.shared_postgres.mode = "runtime_store"`:
+
+- each mutation holds one schema-scoped Postgres advisory lock;
+- concurrent schema migration is serialized under that lock;
+- two instances cannot create two active proposals for the same canonical
+  tenant/capability/object tuple;
+- distinct verified reviewers append decisions without losing one another;
+- one worker claim/effect completes, conflicts, retries, or enters dead letter;
+- direct SQL and app-owned handler retries use durable source-side receipts;
+- fixed-window rate limits are atomic across instances and keyed from verified
+  tenant context plus reviewed capability;
+- CLI and local UI review reads use the same shared queue as MCP serving.
+
+The current implementation restores the shared ledger into a transient SQLite
+store while holding the advisory lock, performs one bounded operation, then
+syncs the result. `max_entries` defaults to 10,000 and may be set from 100 to
+100,000. Runner fails closed with `POSTGRES_RUNTIME_STORE_CAPACITY_EXCEEDED`
+rather than copying an unbounded ledger. This serialized design is appropriate
+for a small self-hosted fleet, not high-throughput horizontal scale.
+
+## Verified Local Topology
+
+The repository includes a synthetic Compose fixture:
+
+```bash
+docker compose --profile fleet \
+  -f examples/runner-fleet/docker-compose.yml up --build -d --wait
+```
+
+The stronger verification command is:
+
+```bash
+corepack pnpm test:fleet
+```
+
+Prerequisites: Node 22, pnpm through Corepack, Docker Engine, and Docker
+Compose. It needs no AWS, Cloud account, external identity provider, or
+internet service after container images and dependencies are available.
+
+Latest verified result: **2026-07-12 (America/Los_Angeles), PASS**. Assertions:
+
+```text
+two claim-bound Runners share one bounded Postgres ledger
+cross-tenant reads are denied
+one active proposal survives concurrent creation
+rate limits hold across instances
+two verified reviewers satisfy a 2-person quorum
+apply is blocked at 1/2 and succeeds at 2/2
+simultaneous reviewer processes preserve both decisions
+two competing workers create exactly one source effect
+worker termination before write recovers without an effect
+worker termination during an open write transaction rolls back and recovers
+worker termination after commit recovers as already-applied
+source-down, read-only-ledger, and timeout readiness failures recover without restart
+shared dead letters requeue or discard with receipts and events preserved
+Postgres and MySQL pool queues fail fast at configured bounds
+backup digest and clean restore match
+retention archives before delete and preserves active proposals
+```
+
+All source rows, identities, keys, tokens, and credentials are synthetic. The
+test tears down its volumes.
+
+## Claim-Bound Sessions
+
+Use asymmetric verification for a networked fleet:
+
+```json
+{
+  "trusted_context": {
+    "provider": "http_claims",
+    "values": {
+      "tenant_id_key": "tenant_id",
+      "principal_key": "sub"
+    }
+  },
+  "session_auth": {
+    "provider": "jwt_asymmetric",
+    "algorithms": ["RS256"],
+    "jwks_url_env": "SYNAPSOR_SESSION_JWKS_URL",
+    "issuer": "https://identity.example",
+    "audience": "synapsor-runner",
+    "tenant_claim": "tenant_id",
+    "principal_claim": "sub",
+    "clock_skew_seconds": 30,
+    "jwks_cache_seconds": 600,
+    "jwks_cooldown_seconds": 30,
+    "fetch_timeout_ms": 3000,
+    "max_response_bytes": 1048576
+  }
+}
+```
+
+Every served capability context must bind tenant and principal from
+`HTTP_CLAIM`. Runner rejects an environment-bound contract mixed into this
+server with `TRUSTED_CONTEXT_PROVIDER_CONFLICT` before tools are served.
+
+Use an HTTPS JWKS URL from an allowlisted identity system. Runner rejects
+redirects, oversized responses, unknown algorithms, `none`, missing/unknown
+`kid`, bad issuer/audience, expired/not-yet-valid tokens, and private JWK
+material. Unknown `kid` triggers at most one controlled refresh. For offline
+deployments, configure one public PEM through `public_key_env` or
+`public_key_path`; never place a private key in Runner config.
+
+## Ledger And Source Pools
+
+```json
+{
+  "storage": {
+    "shared_postgres": {
+      "mode": "runtime_store",
+      "url_env": "SYNAPSOR_LEDGER_DATABASE_URL",
+      "schema": "synapsor_runner",
+      "lock_timeout_ms": 5000,
+      "max_entries": 10000
+    }
+  },
+  "sources": {
+    "app_postgres": {
+      "engine": "postgres",
+      "read_url_env": "APP_POSTGRES_READ_URL",
+      "write_url_env": "APP_POSTGRES_WRITE_URL",
+      "statement_timeout_ms": 3000,
+      "pool": {
+        "max_connections": 4,
+        "connection_timeout_ms": 2000,
+        "idle_timeout_ms": 30000,
+        "queue_timeout_ms": 1000,
+        "queue_limit": 16
+      }
+    }
+  }
+}
+```
+
+Budget database connections across every replica and worker:
+
+```text
+replicas * sum(source max_connections) + readiness/admin headroom
+```
+
+Keep that below database role and instance limits. Queue overflow fails with
+`SOURCE_POOL_QUEUE_FULL`; acquisition timeout fails with
+`SOURCE_POOL_TIMEOUT`. Neither waits indefinitely or creates a proposal.
+
+Use separate roles:
+
+- source reader: `CONNECT`, schema `USAGE`, and `SELECT` only on reviewed
+  tables/views;
+- direct writer: reviewed column-level `UPDATE` plus source receipt-table
+  `SELECT/INSERT/UPDATE`;
+- ledger role: only the configured ledger schema and migration rights;
+- migration administrator: run migrations separately when Runner should not
+  own DDL authority.
+
+## Fleet-Wide Rate Limits
+
+```json
+{
+  "rate_limits": {
+    "enabled": true,
+    "default": { "requests": 60, "window_seconds": 60 },
+    "capabilities": {
+      "billing.propose_refund": { "requests": 10, "window_seconds": 60 }
+    }
+  }
+}
+```
+
+Semantics are fixed-window, keyed by a hash of verified tenant ID and
+capability. There is no separate burst allowance: a tenant may consume the
+entire configured count immediately, then must wait for the aligned window to
+reset. In local SQLite mode limits are process-local. In `runtime_store` mode
+they use an atomic shared Postgres bucket. Rejection returns `RATE_LIMITED` and
+`retry_after_ms` to the reset boundary, increments a metric, and creates no
+proposal.
+
+## Probes And Metrics
+
+- `GET /healthz`: unauthenticated cheap process liveness; no dependency
+  checks or configuration inventory.
+- `GET /readyz`: 200 only when catalog, required sources, authoritative ledger,
+  and required writeback/executor dependencies are ready; otherwise 503.
+- `GET /metrics`: disabled unless configured. A non-loopback bind requires a
+  separate metrics bearer token.
+
+```json
+{
+  "metrics": {
+    "enabled": true,
+    "token_env": "SYNAPSOR_METRICS_TOKEN"
+  }
+}
+```
+
+Do not use the model-facing MCP token as the metrics token. Metrics include
+controlled tenant/capability/source/component labels, never object IDs,
+principals, database URLs, tokens, or raw errors.
+
+## Reviewers, Quorum, And Workers
+
+Use `signed_key` or `jwt_oidc` operator identity. `dev_env` is explicitly
+unverified and is only for local fixtures. A capability may require multiple
+distinct reviewers in the canonical contract:
+
+```sql
+APPROVAL REQUIRED ROLE billing_lead
+REQUIRE 2 APPROVALS
+```
+
+The same verified subject cannot fill two slots. Policy auto-approval does not
+satisfy a multi-human quorum. Apply and workers reject the proposal until
+progress reaches `N/N`. A rejection is terminal and auditable.
+
+Use the shared queue explicitly:
+
+```bash
+synapsor-runner proposals list --config ./synapsor.runner.json
+synapsor-runner proposals show latest --config ./synapsor.runner.json
+synapsor-runner ui --config ./synapsor.runner.json
+synapsor-runner worker run --yes --config ./synapsor.runner.json
+```
+
+Dead-letter operations require verified operator identity:
+
+```bash
+synapsor-runner worker dead-letter list --config ./synapsor.runner.json
+synapsor-runner worker dead-letter show wrp_... --config ./synapsor.runner.json
+synapsor-runner worker dead-letter requeue wrp_... --retry-budget 3 --yes \
+  --config ./synapsor.runner.json --identity alice --identity-key /secure/alice.pem
+synapsor-runner worker dead-letter discard wrp_... --reason "closed" --yes \
+  --config ./synapsor.runner.json --identity alice --identity-key /secure/alice.pem
+```
+
+Requeue preserves history and refuses if a durable applied/already-applied
+receipt already proves the effect. Discard closes the queue item without
+deleting proposals, receipts, or events.
+
+## Backup, Restore, And Retention
+
+```bash
+synapsor-runner store shared-postgres backup \
+  --url-env SYNAPSOR_LEDGER_DATABASE_URL --schema synapsor_runner \
+  --output ./ledger-backup.json
+synapsor-runner store shared-postgres verify-backup --input ./ledger-backup.json
+synapsor-runner store shared-postgres restore-backup \
+  --input ./ledger-backup.json --url-env SYNAPSOR_LEDGER_DATABASE_URL \
+  --schema synapsor_runner_restore --yes
+```
+
+Restore only accepts an empty target schema and verifies the post-restore
+manifest digest. Store archives as sensitive database extracts with owner-only
+permissions.
+
+Retention is archive-before-delete:
+
+```bash
+synapsor-runner store shared-postgres retention --older-than 30d \
+  --url-env SYNAPSOR_LEDGER_DATABASE_URL --schema synapsor_runner --dry-run
+synapsor-runner store shared-postgres retention --older-than 30d \
+  --url-env SYNAPSOR_LEDGER_DATABASE_URL --schema synapsor_runner \
+  --output ./ledger-archive.json --yes
+```
+
+The archive is written and digest-verified before deletion. Pending review,
+approved, pending-worker, retry, failed, and dead-letter records are retained.
+The operation appends an immutable retention event.
+
+Restore runbook:
+
+1. Stop ingress and workers.
+2. Verify the backup digest.
+3. Restore into a clean schema.
+4. Run `store shared-postgres status` and `/readyz` against the restore.
+5. Inspect at least one proposal, receipt, and replay through CLI.
+6. Point one canary Runner at the restored schema.
+7. Resume ingress, then workers, and watch readiness/dead-letter metrics.
+
+## Shutdown, Rotation, And Upgrades
+
+Terminate with `SIGTERM`; Runner stops accepting work, closes HTTP sessions and
+source pools, and releases ledger locks. Give workers enough grace for the
+configured handler timeout. A hard kill is recoverable through leases and
+source-side idempotency receipts, but should remain an incident signal.
+
+Rotate JWKS keys by publishing the new `kid`, waiting at least the configured
+cache lifetime, switching issuers, then retiring the old key. Rotate metrics,
+handler, database, and ledger credentials independently through their named
+environment variables.
+
+Only homogeneous Runner 1.1 instances are verified by `test:fleet`. Mixed
+1.0/1.1 operation is not claimed. For rolling upgrades:
+
+1. back up and verify the ledger;
+2. run schema/config validation with the target version;
+3. drain one worker and remove one Runner from the load balancer;
+4. upgrade one canary, verify `/readyz`, MCP tools, proposal inspection, and
+   metrics;
+5. continue one instance at a time;
+6. upgrade workers after HTTP Runners.
+
+Rollback the binary only while contracts/config remain accepted by the prior
+minor. If the prior version does not understand a new canonical field or
+ledger entry, restore the verified pre-upgrade backup into a clean schema and
+roll back the full fleet. Never mix versions merely because both are `1.x`.
+
+## TLS Boundary
+
+Terminate public TLS or mTLS at a trusted load balancer, or configure Runner's
+tested TLS/mTLS options directly. Keep Runner-to-database TLS verification on,
+use private networking where possible, restrict `/metrics`, and never forward
+untrusted identity headers as claims. The bearer JWT itself, not a proxy-added
+tenant header, is the session authority.