@terreno/mcp 57.15.0 → 57.16.2

package/src/docs/versioned/57.15.0/implementationPlans/infra-mcp.md

@@ -0,0 +1,269 @@
+# Implementation Plan: Infrastructure MCP server (`@terreno/infra-mcp`)
+
+**Status:** Approved — unblocked; RBAC module is Complete ([rbac-permissions.md](rbac-permissions.md))
+**Roadmap issue:** https://github.com/TerrenoLabs/terreno/issues/1090
+**Branch:** TBD
+**Owner:** TBD
+**Created:** 2026-08-09
+**Research:** [infra-mcp-research.md](infra-mcp-research.md)
+
+## Goal
+
+A deployable, self-contained MCP server that puts privileged infrastructure tooling — GCP, Sentry, MongoDB, and later Expo/EAS and Vercel — behind a single MCP endpoint with per-user authentication, RBAC-driven read/write permission tiers, per-call write confirmation, and a full audit trail. Packaged as a reusable `@terreno/infra-mcp` workspace package targeting one-man shops through growing startups, dogfooded by our own Cloud Run deployment. Cloud agents stop receiving raw `GCP_SA_*` service-account env vars and instead call this server with scoped service tokens.
+
+## Non-Goals
+
+- Per-user vendor identity (on-behalf-of OAuth to GCP/Sentry/etc.) — the credential interface accepts a user context so this can be added later, but v1 executes with server-held credentials.
+- Proxying vendor MCP servers that require interactive browser OAuth (Vercel `mcp.vercel.com`, Expo `mcp.expo.dev`) — research showed these are allowlisted/interactive-only; those vendors get native tools instead.
+- A bespoke permission engine — permissioning is the `@terreno/api` RBAC module (`createAccess`), not custom code.
+- Rate limiting, cost tracking, tool versioning (future work).
+- Changes to the existing `@terreno/mcp` docs server (stays on SDK v1, unchanged).
+
+## Decisions
+
+| Question | Decision |
+|----------|----------|
+| Approach | **Option C (hybrid):** native curated tools first; proxied vendor-MCP adapters behind an explicit per-tool allow-list with mandatory read/write annotation |
+| Protocol | **MCP 2026-07-28 spec** via TS SDK v2 (`@modelcontextprotocol/server`), `createMcpHandler` with `legacy: "stateless"` so 2025-era clients keep working |
+| Permissioning | **`@terreno/api` RBAC module** — this IP is blocked behind that PR. Each integration is an RBAC resource with `read`/`write` actions; meta-permissions live on an `infra` resource |
+| Write safety | **Write tools require per-call confirmation (MCP elicitation).** A config option can disable confirmation per integration or globally, but it only takes effect for users holding `infra:writeUnconfirmed` — everyone else is still prompted |
+| Phase 1 scope | **Read-only:** GCP observability (logs/metrics/errors), Sentry, MongoDB |
+| GCP credentials | **Cloud Run default runtime service account only (ADC).** No injected SA key env vars anywhere; the `GCP_SA_*` pattern for cloud agents is retired. IAM on the runtime SA is the outer bound; the MCP's RBAC is the per-user gate |
+| Other credentials | Static tokens (Sentry auth token, Mongo connection strings, later `EXPO_TOKEN`/Vercel token) from Secret Manager/env |
+| User store & login | Mongo users via `@terreno/api` (`baseUserPlugin` + `rbacUserPlugin`); MCP clients authenticate via OAuth 2.1 using **`@better-auth/oauth-provider` + `@better-auth/mcp`** (already aligned with the 2026-07-28 authorization spec: CIMD, RFC 9728, `requireMcpAuth`) |
+| Non-human callers | Service tokens (Better Auth `apiKey` plugin if compatible, else a `ServiceToken` model) mapped to machine users with RBAC roles — this is how cloud agents call the server |
+| Packaging | New workspace package `infra-mcp/` publishing `@terreno/infra-mcp`; config-driven integration registry; deployed via a clone of the existing `mcp_service` terraform module |
+
+## Architecture
+
+```
+MCP client (Cursor, Claude Code, cloud agent)
+  | OAuth 2.1 bearer token (human) or service token (machine)
+  v
+POST /mcp  — SDK v2 createMcpHandler (2026-07-28 + legacy stateless)
+  |
+  1. Auth: Better Auth verifies token -> User (+ roles via rbacUserPlugin)
+  2. Authz: access.can({[integration]: [tier]}) using Mcp-Name header ->
+     integration/tool lookup; tools/list is filtered to the caller's grants
+  3. Write tools: elicitation confirmation (unless override + infra:writeUnconfirmed)
+  4. Dispatch: IntegrationAdapter (native handler | proxied MCP client)
+  5. Audit: InfraAuditLog record for every tools/call
+```
+
+### Integration adapter interface
+
+```typescript
+// infra-mcp/src/integrations/types.ts
+export interface InfraToolContext {
+  user: UserDocument;            // resolved caller (human or machine)
+  access: TerrenoAccess<InfraStatements>;
+  confirmWrite: (summary: string) => Promise<boolean>; // elicitation wrapper
+  logger: Logger;
+}
+
+export interface InfraTool {
+  name: string;                  // exposed as `${integration}_${name}`
+  tier: "read" | "write";       // maps to the RBAC action checked
+  description: string;
+  inputSchema: ZodType;          // zod 4 (SDK v2 Standard Schema requirement)
+  handler: (args: unknown, ctx: InfraToolContext) => Promise<ToolResult>;
+}
+
+export type IntegrationAdapter =
+  | {kind: "native"; name: string; tools: InfraTool[]}
+  | {
+      kind: "proxied";
+      name: string;
+      transport: {type: "stdio"; command: string[]} | {type: "http"; url: string; headers?: Record<string, string>};
+      // Deny-by-default: only listed upstream tools are exposed, and each
+      // MUST carry an explicit tier annotation.
+      allowedTools: Record<string, {tier: "read" | "write"; rename?: string}>;
+    };
+```
+
+Proxied adapters use `@modelcontextprotocol/client` (speaks both protocol eras; stateless — no per-upstream session). Stdio upstreams run as child processes inside the container.
+
+### RBAC statements
+
+```typescript
+// infra-mcp/src/access.ts — merged with terrenoStatements via createAccess
+export const infraStatements = {
+  ...terrenoStatements,
+  gcp: ["read", "write"],
+  sentry: ["read", "write"],
+  mongo: ["read", "write"],
+  expo: ["read", "write"],      // phase 3
+  vercel: ["read", "write"],    // phase 3
+  infra: ["viewAudit", "manageIntegrations", "writeUnconfirmed"],
+} as const;
+```
+
+- `read`/`write` are in the RBAC module's `READ_ACTIONS` semantics, so the shipped `auditor` role (read-only everywhere) works unmodified — that is the "log digging" role for free.
+- Default roles seeded on top of the RBAC defaults: `infra-readonly` (every integration `read`), `infra-operator` (`read` + `write`, still confirmed per call), `superadmin` from RBAC covers everything including `infra:writeUnconfirmed`.
+- Per-integration grants ("Sentry read only, no GCP") are just roles with narrower permission sets — no extra machinery.
+
+### Write confirmation flow
+
+1. Tool handler for a `write`-tier tool calls `ctx.confirmWrite(summary)` before mutating.
+2. `confirmWrite` sends an MCP **elicitation** request to the client ("Confirm: roll back Cloud Run service X to revision Y? yes/no") and resolves with the user's answer. Decline → tool returns a cancelled result; audit records `rejected`.
+3. Override: `config.integrations[name].confirmWrites: false` (or global `confirmWrites`) skips the elicitation **only when** `access.can({infra: ["writeUnconfirmed"]})` passes for the caller. Without the permission, the config flag is ignored and the prompt still fires.
+4. Clients that do not support elicitation (some non-interactive agents): write tools **fail closed** with an instructive error unless the caller holds `infra:writeUnconfirmed` and the override is on.
+
+### Configuration
+
+Static config (code/env, validated at boot with zod):
+
+```typescript
+// consuming app / our deployment: infra.config.ts
+export default defineInfraMcp({
+  integrations: {
+    gcp: {enabled: true, projectId: env("GCP_PROJECT")},           // ADC — no key
+    sentry: {enabled: true, authToken: secret("SENTRY_MCP_TOKEN"), org: "flourish"},
+    mongo: {enabled: true, readUri: secret("MONGO_READ_URI"), writeUri: secret("MONGO_WRITE_URI")},
+  },
+  confirmWrites: true, // global default; per-integration override allowed
+});
+```
+
+## Models
+
+All models live in `infra-mcp/src/types/models/` per convention; every field carries a `description`.
+
+### User
+
+`@terreno/api` user schema with `baseUserPlugin` + `rbacUserPlugin` (adds `roles: string[]`). Machine users (cloud agents) are ordinary users flagged `isServiceAccount: true`.
+
+### InfraAuditLog
+
+| Field | Type | Notes |
+|---|---|---|
+| `userId` | ObjectId ref User | caller (human or machine) |
+| `tool` | string | full exposed tool name |
+| `integration` | string | adapter name |
+| `tier` | "read" \| "write" | |
+| `args` | Mixed | redacted via per-tool redaction list before persist |
+| `status` | enum | `allowed`, `denied`, `confirmed`, `rejected`, `error` |
+| `error` | string? | |
+| `durationMs` | number | |
+| `created` | Date | plugin-managed |
+
+Indexed on `{userId, created}` and `{integration, created}`. Read surface gated by `infra:viewAudit`.
+
+### ServiceToken (only if Better Auth `apiKey` plugin is unsuitable)
+
+`name`, `hashedToken`, `userId`, `expiresAt?`, `lastUsedAt`, `created`. Verified in the same auth middleware path as bearer tokens.
+
+## APIs
+
+| Method | Path | Auth | Purpose |
+|---|---|---|---|
+| POST | `/mcp` | OAuth bearer / service token | MCP endpoint (2026-07-28 + legacy stateless) |
+| GET | `/.well-known/oauth-protected-resource` | — | RFC 9728 metadata (via `@better-auth/mcp`) |
+| * | `/oauth2/*`, `/api/auth/*` | — | Better Auth authorization server + login pages |
+| GET | `/health` | — | health check |
+| * | `/rbac/*` | RBAC perms | role CRUD/assignment (RBAC module's `rbacRouter`) |
+| * | `/admin/*` | `admin:access` | admin panel (`@terreno/admin-backend`): Users, InfraAuditLog (read-only), ServiceTokens |
+
+### Phase 1 tools (read tier)
+
+| Integration | Tools |
+|---|---|
+| gcp | `gcp_query_logs`, `gcp_list_log_names`, `gcp_query_metrics`, `gcp_list_error_groups`, `gcp_list_cloud_run_services`, `gcp_get_cloud_run_service` |
+| sentry | `sentry_search_issues`, `sentry_get_issue`, `sentry_list_issue_events`, `sentry_get_event` |
+| mongo | `mongo_list_collections`, `mongo_collection_schema`, `mongo_find`, `mongo_aggregate` (read-only stages; `$out`/`$merge` rejected; executed on the read-only connection) |
+
+GCP tools call the Cloud Logging/Monitoring/Error Reporting/Cloud Run Admin APIs via ADC. Sentry tools call the Sentry REST API. Mongo tools reuse the read-query/schema logic behind `mcp-server/src/local/tools/databaseQuery.ts`/`databaseSchema.ts` — extracted into a shared module both packages import rather than forked into infra-mcp — run against the configured read URI.
+
+### Phase 2 tools (write tier, all elicitation-confirmed)
+
+Initial minimal set: `sentry_resolve_issue`, `sentry_assign_issue`, `gcp_rollback_cloud_run_revision`, `gcp_update_cloud_run_traffic`, `mongo_update_one`, `mongo_delete_one` (write URI). Expansion is deliberate and per-PR.
+
+## Notifications
+
+None in v1. Future: optional Slack notifier (via `@terreno/api` notifiers) on write-tier tool execution.
+
+## UI
+
+No new frontend package. Admin surfaces reuse `@terreno/admin-backend` + `@terreno/admin-frontend`/`admin-spa`:
+
+- Users + role assignment (RBAC module's admin screens per its IP).
+- InfraAuditLog read-only table (`listFields`: created, user, tool, tier, status).
+- ServiceToken management (create shows the token once).
+
+## Phases
+
+### Phase 1 — Read-only core (the "log digging" MVP)
+
+- Package scaffold `infra-mcp/` (workspace, biome, tsconfig, bun test, catalog deps; zod 4 catalog entry).
+- SDK v2 `/mcp` endpoint (`createMcpHandler`, `legacy: "stateless"`, `toNodeHandler`) mounted on a TerrenoApp/Express server.
+- Better Auth OAuth 2.1 (`@better-auth/oauth-provider` + `@better-auth/mcp`): discovery, login, token verify middleware.
+- RBAC wiring: `infraStatements`, default roles, `access.can()` authz middleware keyed on `Mcp-Name`, filtered `tools/list`. **Gated on the RBAC PR.**
+- Native adapters: gcp (ADC), sentry, mongo — read tools above.
+- `InfraAuditLog` + audit middleware.
+- Terraform: clone `mcp_service` module (new service name, Artifact Registry, runtime SA with viewer/log IAM roles only for now).
+- Tests: authz matrix (role × tool), tool handlers against mocks/emulators, audit records.
+- **Deliverable:** a teammate signs in from Cursor/Claude Code, gets exactly the tools their role allows, digs logs across GCP/Sentry/Mongo.
+
+### Phase 2 — Write tier + machine callers
+
+- Elicitation `confirmWrite` wrapper; config override gated by `infra:writeUnconfirmed`; fail-closed for non-elicitation clients.
+- Initial write tools (list above); runtime SA IAM widened only for the specific write APIs.
+- Service tokens (Better Auth `apiKey` spike → fallback `ServiceToken` model); machine users with `infra-readonly` role.
+- Cloud agent migration: agents get `INFRA_MCP_URL` + service token instead of `GCP_SA_*`; update AGENTS.md/environment docs; retire the env-injected SA keys.
+- **Deliverable:** write actions work with per-call confirmation; cloud agents dig logs through the MCP with no raw GCP credentials.
+
+### Phase 3 — Proxied adapters + remaining vendors
+
+- `ProxiedAdapter` (MCP client, stdio + HTTP transports, allow-list with mandatory tier annotations, name prefixing).
+- First proxied integration: `@sentry/mcp-server` (stdio, static token) or `observability-mcp` — pick one as the reference implementation and keep the native tools as fallback.
+- Native Expo/EAS tools (`EXPO_TOKEN`: builds, updates, submissions status) and Vercel tools (API token: deployments, logs).
+- **Deliverable:** adding a vendor MCP is config + annotations, not code.
+
+### Phase 4 — OSS packaging
+
+- `defineInfraMcp` config reference docs, quickstart (one-man-shop path: tokens + deploy anywhere a container runs), website docs page.
+- Publish `@terreno/infra-mcp`; bootstrap/MCP-server docs updates; changelog + upgrade notes.
+- **Deliverable:** an external startup can deploy their own instance from docs alone.
+
+## Feature Flags & Migrations
+
+- **No data migrations** — all collections are new.
+- Rollout is additive: new package, new Cloud Run service. Nothing existing changes until Phase 2's cloud-agent credential switch, which is coordinated: keep `GCP_SA_*` injection until the MCP path is verified, then remove.
+- RBAC dependency: the RBAC module is Complete ([rbac-permissions.md](rbac-permissions.md)). Phase 1 wiring consumes `createAccess`, `rbacUserPlugin`, `rbacRouter`, default-role seeding.
+
+## Activity Log & User Updates
+
+`InfraAuditLog` records every `tools/call` (allowed, denied, confirmed, rejected, error) with user attribution — this is the compliance answer for server-held credentials, since vendor-side logs only show the service identity. RBAC role changes are covered by the RBAC module's `RbacAudit`.
+
+## Not Included / Future Work
+
+- Per-user vendor identity for the write tier (credential interface takes user context; add later).
+- Rate limiting / quotas per user or tier.
+- Slack notifications on write actions.
+- Proxying interactive-OAuth-only vendor MCPs (Vercel/Expo hosted) — revisit if they ship headless credentials.
+- Terraform-apply / IaC tools (high blast radius; needs its own design).
+- MCP Tasks extension for long-running operations (EAS builds) — Phase 3+ candidate.
+
+## Files to Create / Modify
+
+- `infra-mcp/` — new package: `src/index.ts` (server entry), `src/server.ts`, `src/access.ts`, `src/config.ts`, `src/auth.ts`, `src/audit.ts` + `src/models/infraAuditLog.ts`, `src/integrations/{types,registry,gcp/*,sentry/*,mongo/*,proxied}.ts`, `src/types/models/*`, tests.
+- `package.json` (root) — workspace entry, catalog: zod 4, `@modelcontextprotocol/server`, `@better-auth/oauth-provider`, `@better-auth/mcp`, Google API clients.
+- `terraform/main.tf`, `variables.tf`, `outputs.tf` — `infra_mcp_service` module instance + runtime SA IAM.
+- `.cursor/rules/infra-mcp/00-infra-mcp.mdc`, `AGENTS.md`/`CLAUDE.md` package lists, docs site page.
+- `ROADMAP.md` + `docs/explanation/roadmap-seed-issues.md` — roadmap entry and ready-to-paste tracking issue (done); `scripts/issueAreaLabels.ts` + bug-report package dropdown when the package lands.
+
+## Task List
+
+See [docs/tasks/infra-mcp.md](../tasks/infra-mcp.md).
+
+## Acceptance Criteria
+
+- [ ] An MCP client completes OAuth against the deployed server and lists only the tools the signed-in user's roles allow.
+- [ ] A user with only `auditor`/`infra-readonly` can query GCP logs, Sentry issues, and Mongo collections, and cannot see or call any write tool.
+- [ ] A write-tier tool call triggers an elicitation prompt; declining aborts with an audited `rejected` status.
+- [ ] With `confirmWrites: false` configured, confirmation is skipped **only** for callers holding `infra:writeUnconfirmed`; all others are still prompted.
+- [ ] Every tool call (including denials) produces an `InfraAuditLog` record with user attribution and redacted args.
+- [ ] The deployed service runs with only the Cloud Run default runtime SA — no SA key files or `GCP_SA_*` env vars anywhere in the stack.
+- [ ] A cloud agent authenticates with a service token and reads logs; the `GCP_SA_*` env vars are removed from agent environments.
+- [ ] A 2025-era MCP client (pre-2026 spec) connects successfully via the legacy stateless fallback.
+- [ ] `bun run lint`, package tests, and CI pass.

package/src/docs/versioned/57.15.0/implementationPlans/job-queues.md

@@ -0,0 +1,330 @@
+# Implementation Plan: Durable background jobs (`@terreno/jobs`)
+
+**Status:** Complete — 2026-09-10  
+**Branch:** `cursor/durable-background-jobs-803d`  
+**Owner:** —  
+**Created:** 2026-09-10  
+**Approved:** 2026-09-10  
+**Roadmap issue:** https://github.com/TerrenoLabs/terreno/issues/1188 (this feature **closes** that issue; implementation PRs use `Fixes #1188`)  
+**Task list:** [job-queues.md](../tasks/job-queues.md)  
+**Depends on:** —  
+**RTK deprecation flag:** Partial — admin screens use generated SDK / `useAdminApi`  
+**Program:** [B2B platform](b2b-platform-program.md)  
+
+## Goal
+
+Terreno apps can enqueue named jobs that survive process restarts: retries with
+backoff, delayed and recurring schedules, dead-lettering, and an admin operations
+UI. Persistence and the default worker are **Mongo**. Execution is a pluggable
+`JobRunner` so the same job definitions run in-process, in a standalone worker,
+on **GCP Cloud Tasks**, on **Vercel Queues**, or on a custom runner. Comms,
+webhooks, and billing stay on their current paths; this IP ships the queue they
+will enqueue onto later.
+
+## Non-Goals
+
+- BullMQ, Redis, or any Redis-backed queue.
+- Vercel Workflows (`"use workflow"` / `"use step"`).
+- Migrating `@terreno/comms` sends, Expo receipt polling, `WebhooksApp`
+  200-then-queue dispatch, or `@terreno/billing` sync onto the queue.
+- Replacing or merging `BackgroundTask` / admin **scripts**.
+- Changing in-process `cronjob()` behavior (hard-coded `America/Chicago`).
+- Live GCP or Vercel integration tests in CI.
+- Generic `modelRouter` CRUD as the operator UI.
+
+## Decisions
+
+| Question | Decision |
+|----------|----------|
+| Package | New workspace package **`@terreno/jobs`**. `JobsApp` implements `TerrenoPlugin`. |
+| Persistence | Mongo collections owned by this package. Default runner claims work from Mongo. |
+| `BackgroundTask` | Unchanged. Scripts stay in-process bookkeeping. Jobs are a separate `Job` model. |
+| v1 consumers | Framework + example only. Public `enqueue` is the seam later IPs call. |
+| Workers | **Both:** `jobs.startWorker()` in the API process **or** a dedicated worker process. Explicit opt-in; registering `JobsApp` does not start a poll loop. |
+| Schedules | Delayed `runAt` and recurring cron. Timezone is an **IANA** string on each schedule (default **`UTC`**). |
+| Cloud dispatch | **GCP Cloud Tasks** and **Vercel Queues** adapters (mocked-SDK tests). Custom `JobRunner` is a first-class type. |
+| Vercel product | **Queues** only. Job catalog and admin stay in Terreno. |
+| Execute HTTP | `POST {basePath}/execute` for GCP/custom HTTP push (GCP: OIDC). Vercel uses private `handleCallback` consumers → `executeJobById`. Mongo runner does **not** mount execute unless a cloud/custom runner needs it. |
+| Admin UI | Custom screen `jobs` (comms pattern): list/filter, detail/attempts, retry, DLQ requeue, cancel. Home widget. example-frontend + admin-spa. |
+| Existing `cronjob()` | Leave it. Docs call it process-local. Durable work uses `JobsApp`. |
+| Delivery | At-least-once. Handlers must be idempotent. Optional `idempotencyKey` unique with `name`. |
+| Payload | JSON-serializable only. Admin redacts by default (`redactPayload`). |
+| Recurring overlap | One in-flight instance per schedule `name`; skip the tick if still running. |
+
+## Architecture
+
+```
+@terreno/jobs
+  JobsApp (TerrenoPlugin)
+    define(name, {handler, retry, schedule?})
+    enqueue({name, payload, runAt?, idempotencyKey?})
+    startWorker()                    // Mongo poll / runner.start
+    adminContribution() → screen "jobs"
+    optional POST /jobs/execute      // cloud/custom HTTP
+
+JobRunner
+  enqueue(job) → dispatch
+  start?(signal) → local poll (Mongo)
+  stop?()
+
+MongoJobRunner     — claim via findOneAndUpdate; lock TTL
+GcpCloudTasksRunner — create HTTP task → POST /jobs/execute (OIDC)
+VercelQueuesRunner  — send(topic,{jobId}) → private handleCallback consumer → executeJobById
+Custom              — app-supplied JobRunner
+```
+
+```typescript
+import {JobsApp, MongoJobRunner, getJobsService} from "@terreno/jobs";
+
+const jobs = new JobsApp({
+  runner: new MongoJobRunner(),
+  timezone: "UTC",
+});
+
+jobs.define("send-welcome-email", {
+  handler: async (payload, ctx) => {
+    ctx.log.info("sending");
+    // work
+  },
+  retry: {maxAttempts: 5, backoffMs: 1000},
+});
+
+new TerrenoApp({userModel: User}).register(jobs);
+
+// Explicit — API process or a worker entrypoint
+await jobs.startWorker();
+
+await getJobsService().enqueue({
+  name: "send-welcome-email",
+  payload: {userId: "..."},
+});
+```
+
+Standalone worker (example-backend): same `JobsApp` definitions, `build()` without
+`listen`, then `startWorker()`. Do not start two poll loops against the same
+queue unless lock TTL and claim are in place (they are).
+
+### Claim (Mongo)
+
+`findOneAndUpdate` where `status` is `pending` or `scheduled`, `runAt <= now`,
+and `lockedAt` is missing or older than `lockTtlMs` (default 15 minutes). Set
+`status: "running"`, `lockedAt`, `lockedBy` (hostname + pid). On success, wrap
+`runWithRequestContext({jobId})`. On handler success → `completed`. On throw →
+increment `attemptCount`, set `lastError`, either `pending` with `runAt` backoff
+or `dead` when `attemptCount >= maxAttempts`.
+
+### Retry
+
+Exponential backoff from `retry.backoffMs` (default 1000) with jitter, cap
+`retry.maxBackoffMs` (default 15 minutes). `maxAttempts` default 5. Dead-letter
+status is **`dead`**. Admin **Retry** enqueues a new `Job` (`retriedFromId`);
+**Requeue DLQ** resets the same row to `pending` with `runAt: now` and clears
+the lock.
+
+### Recurring
+
+`define(name, {schedule: {cron, timezone?}})` upserts a `JobSchedule` row.
+The worker ticks due schedules and `enqueue`s a child `Job` with
+`scheduleId`. Timezone defaults to `JobsApp` `timezone` then `UTC`. Invalid cron
+fails `define` / startup, not silently.
+
+### `JobRunner`
+
+```typescript
+interface JobRunner {
+  readonly id: string;
+  enqueue(job: JobDocument): Promise<void>;
+  start?(options: {jobs: JobsApp; signal: AbortSignal}): Promise<void>;
+  stop?(): Promise<void>;
+}
+```
+
+Mongo `enqueue` only persists (already saved) then no-ops dispatch — the poller
+picks it up. Cloud runners persist then create a platform task whose HTTP body
+is `{jobId}` (not the payload). Execute loads the Mongo row, claims if needed,
+runs the handler.
+
+GCP and Vercel SDKs are **optional peers** on subpath exports
+(`@terreno/jobs/runners/gcpCloudTasks`, `@terreno/jobs/runners/vercelQueues`).
+Core `@terreno/jobs` has zero cloud SDKs. Tests inject fakes.
+
+### Execute HTTP
+
+Mounted only when `JobsApp` `mountExecuteRoute: true` or the runner requests it
+(GCP default true; Vercel uses private `handleCallback` consumers; Mongo default false).
+
+| Status | When |
+|--------|------|
+| 200 | Handler finished (success or recorded failure/dead) |
+| 401 | Missing/invalid OIDC or platform signature |
+| 404 | Unknown `jobId` |
+| 409 | Job already `completed` / `cancelled` (idempotent no-op 200 preferred for provider retries — **200** if terminal success; **409** only if still `running` under a live lock) |
+
+Do not skip rate limiting on this path. Do not add it to generated SDK consumer
+hooks (admin uses dedicated jobs routes). OpenAPI: document execute as
+internal/operator; omit from public SDK codegen if tagged internal.
+
+### Logging
+
+`createScopedLogger({prefix: "[Job]", labels: {jobName, jobId}})`. Never log
+full payloads or secrets. Correlation: `runWithRequestContext({jobId})`.
+
+## Models
+
+Five-type pattern, field `description`s, `createdUpdatedPlugin`. **No**
+`isDeletedPlugin` on live queue rows (use `cancelled`). Not `modelRouter`
+models.
+
+**Job** — `name` (string, indexed), `payload` (Mixed), `status` (`pending` \|
+`scheduled` \| `running` \| `completed` \| `failed` \| `cancelled` \| `dead`),
+`runAt` (date, indexed), `attemptCount` (number), `maxAttempts` (number),
+`lockedAt` / `lockedBy`, `lastError`, `attempts[]` (`{at, error, errorClass}`),
+`scheduleId` (optional ref), `idempotencyKey` (optional), `retriedFromId` /
+`retriedById`, `payloadRedacted` (boolean). Unique sparse index
+`(name, idempotencyKey)`. Compound index `(status, runAt)`.
+
+**JobSchedule** — `name` (unique), `cron`, `timezone`, `nextRunAt`, `enabled`,
+`handlerName` (= `name`).
+
+## APIs
+
+Default `basePath` `/jobs`. Admin routes require `admin:access` plus
+`admin:jobs` when `accessControl` is configured; legacy `user.admin` when not.
+
+| Method | Path | Purpose |
+|--------|------|---------|
+| GET | `/jobs` | Paginated list: `name`, `status`, `scheduleId`, date range, `q` |
+| GET | `/jobs/:id` | Detail: attempts, redacted payload, lock |
+| POST | `/jobs/:id/retry` | New job from payload; original `retriedById` |
+| POST | `/jobs/:id/requeue` | DLQ/`failed`/`cancelled` → `pending` now |
+| POST | `/jobs/:id/cancel` | `pending`/`scheduled`/`running` → `cancelled` (handler `ctx.signal`) |
+| GET | `/jobs/schedules` | Defined schedules + next run |
+| POST | `/jobs/schedules/:name/pause` | `enabled: false` |
+| POST | `/jobs/schedules/:name/resume` | `enabled: true` |
+| POST | `/jobs/execute` | Cloud/custom HTTP execute (not admin session) |
+
+Enqueue from app code only (`getJobsService().enqueue`). No public
+`POST /jobs` create for arbitrary names from the browser.
+
+## Notifications
+
+None. Failures are `Job` rows + `logger`. Downstream comms can `enqueue` later.
+
+## UI
+
+`JobsApp.adminContribution()`: custom screen `name: "jobs"`, display **Jobs**,
+icon `clock`. Built-in widget in `@terreno/admin-frontend` (`JOBS_ADMIN_WIDGETS`)
+so hosts do not re-map (comms pattern).
+
+| Piece | Role |
+|-------|------|
+| `JobsDashboardScreen` | Filters + table + pause/resume schedules |
+| `JobsJobDetail` | Attempts, error, retry / requeue / cancel |
+| Home widget `jobs` | Counts of `dead` + `running`; link to screen |
+
+example-frontend: `/admin/jobs`, `/admin/jobs/[id]`.  
+admin-spa: `/jobs`, `/jobs/[id]`.  
+`verify-ui-changes` on the example admin.
+
+RBAC: add `jobs` to `terrenoStatements.admin` (alongside `runScripts`).
+
+## Phases
+
+1. **Tracer:** package, `define`/`enqueue`, Mongo claim, `startWorker`, one example job.
+2. **Reliability:** retries, DLQ, lock TTL, cancel signal, delayed `runAt`, cron schedules.
+3. **Runners:** `JobRunner`, custom fake, GCP Cloud Tasks, Vercel Queues, execute HTTP.
+4. **Admin:** routes + UI + example + spa.
+5. **Docs + publish:** Diátaxis, changelog, CI, `publish-on-tag`, seed/positioning.
+
+## Feature Flags & Migrations
+
+New collections only. No user-data migration. Opt-in: apps that never register
+`JobsApp` are unchanged.
+
+## Activity Log & User Updates
+
+Each attempt is on `Job.attempts`. No end-user notifications.
+
+## Not Included / Future Work
+
+- Comms automatic scheduled retries (`comms-abstraction` / admin dashboard).
+- Webhook 200-then-queue (`inbound-webhooks`).
+- Billing Stripe handler offload (`billing-stripe`).
+- Vercel Workflows runner.
+- Redis/BullMQ runner.
+- Per-queue concurrency caps / rate limits beyond claim locks.
+- Streaming logs over sockets (poll admin GET).
+
+## Docs in this slice
+
+| Page | Change |
+|------|--------|
+| `docs/how-to/background-jobs.md` | **New.** Define, enqueue, worker vs API, Mongo vs GCP vs Vercel, execute auth, idempotency. |
+| `docs/reference/jobs.md` | **New.** Options, models, routes, runners, env. |
+| `docs/how-to/README.md` | Link the how-to. |
+| `docs/reference/api.md` | Contrast `cronjob()` / `wrapScript` as process-local. |
+| `docs/reference/environment-variables.md` | Jobs / Cloud Tasks / Vercel / execute audience. |
+| `docs/how-to/admin-import-prebuilt.md` | `JobsApp` screen `jobs`. |
+| `docs/explanation/roadmap-seed-issues.md` | IP + task URLs; `IP=job-queues`. |
+| `docs/explanation/positioning.md` | Celery/ActiveJob row → shipped via `@terreno/jobs` when Complete. |
+| `changelog/unreleased/job-queues.md` | Added. |
+| `.rulesync/rules/jobs/00-jobs.md` | Package rule; `bun run rules` / `skills:sync`. |
+| `.rulesync/skills/terreno-backend-api/SKILL.md` | Jobs vs scripts vs cron. |
+
+## Testing
+
+Bun + `@terreno/test` Mongo. Do not mock `@terreno/api` or models. Cloud adapters:
+inject fake Task/Queue clients. Execute route: signed vs unsigned. Claim: two
+workers cannot run the same job. Restart: `running` with expired lock is
+reclaimed. Recurring: timezone `America/New_York` vs `UTC` tick. Admin: 403
+without `admin:jobs`. UI: `verify-ui-changes`.
+
+## Files to Create / Modify
+
+- `jobs/` — new package (models, service, runners, routes, tests)
+- Root `package.json` workspace + `jobs:*` scripts
+- `api/src/rbac/statements.ts` — `admin: ["access", "runScripts", "viewBackgroundTasks", "jobs"]`
+- `admin-frontend` — jobs screens + widgets + tests
+- `admin-spa` — routes
+- `example-backend` — `JobsApp` + worker entry + demo job
+- `example-frontend` — admin routes + SDK regen if jobs admin paths are OpenAPI
+- `.github/workflows/jobs-ci.yml`, `publish-on-tag.yml`, `cd.yml` path filters
+- Docs/rules/changelog as in the table
+
+## Task List
+
+See [docs/tasks/job-queues.md](../tasks/job-queues.md).
+
+## Acceptance Criteria
+
+- [x] `enqueue` then kill the Node process; a new `startWorker()` runs the job.
+      Verified by bun test that simulates crash by leaving `pending` and starting
+      a second worker.
+- [x] Handler throw retries until `maxAttempts` then `status: "dead"`. Admin
+      requeue returns it to `pending` and the worker runs it. Verified by bun tests
+      + admin UI verification.
+- [x] Delayed `runAt` in the future is not claimed early; cron in a named IANA
+      timezone fires `nextRunAt` correctly. Verified by bun tests with frozen clocks
+      (Luxon).
+- [x] Two overlapping `startWorker()` loops claim a job once. Verified by bun test.
+- [x] Custom `JobRunner` receives `enqueue`; GCP and Vercel adapters create a
+      mocked platform task and `POST /jobs/execute` with valid auth runs the
+      handler, invalid auth 401. Verified by bun tests (no live cloud).
+- [x] Admin Jobs screen lists jobs, opens detail, retries, requeues DLQ, cancels.
+      Verified by `verify-ui-changes` artifacts.
+- [x] Apps without `JobsApp` behave as today (`cronjob`, scripts). Verified by
+      existing api/admin-backend tests remaining green.
+- [x] How-to + jobs reference match the shipped API. Verified in Roast;
+      `bun run website:build` on the docs task.
+
+## Named assumptions
+
+1. Handlers are registered in every process that executes jobs (API and worker).
+2. JSON payloads only; no Buffer/stream jobs in v1.
+3. Unique `(name, idempotencyKey)` makes duplicate enqueue a no-op returning the
+   existing row.
+4. `failed` is a transient row state between attempts if we surface it; terminal
+   exhaustion is `dead`. Pick may use `pending` between retries only — then omit
+   `failed` as terminal. Prefer **pending-with-future-`runAt`** between attempts
+   and **`dead`** at exhaustion so the admin filter is simple.
+5. Positioning Celery row updates when the IP is **Complete**, not while Draft.