experimental-a2 0.3.0 → 0.4.0

package/docs/guides/05-production.mdx

@@ -1,28 +1,29 @@
 ---
 title: Going to production
-description: Point the log at durable storage, add recovery for handler work, and know what to do when an event dead-letters.
+description: Point the store at durable storage, add a scheduler for recovery and timers, and know what to do when an event dead-letters.
 ---
 
 ## Two pieces
 
 Development needs zero setup: SQLite appears under `.a2/`, and tests run
-in memory. Every production server needs a durable log. Servers with handlers
-also need recovery. Neither choice changes your event contract.
+in memory. Every production server needs a durable store. Servers with handlers
+or scheduled events also need a scheduler. Neither choice changes your event
+contract.
 
-## 1. Choose a log
+## 1. Choose a store
 
-Production has no default log, on purpose. A server without one throws
-`LOG_NOT_CONFIGURED` at startup. A failed boot beats events written to a
+Production has no default store, on purpose. A server without one throws
+`STORE_NOT_CONFIGURED` at startup. A failed boot beats events written to a
 filesystem that evaporates.
 
 ```ts server/orders.ts
 import { createServer } from 'experimental-a2/server'
-import { postgres } from 'experimental-a2/log-postgres'
+import { postgres } from 'experimental-a2/store-postgres'
 import { orders } from '@/contracts'
 
 export const ordersServer = createServer({
   contract: orders,
-  log: postgres({ connectionString: process.env.DATABASE_URL }),
+  store: postgres({ connectionString: process.env.DATABASE_URL }),
   handlers: {
     /* ... */
   },
@@ -31,7 +32,7 @@ export const ordersServer = createServer({
 
 The Postgres backend uses real transactions; appends serialize per
 session on an advisory lock, while handler claims remain concurrent. The live
-stream polls the log with an
+stream polls the store with an
 activity-adaptive cadence: 25ms while a session is producing events
 (a token stream reads smoothly, not in clumps), backing off to 250ms
 when it goes quiet (a LISTEN/NOTIFY upgrade could still land without
@@ -40,7 +41,7 @@ box; transaction-mode poolers included, which is exactly why polling
 is the default. `pg` is an optional peer dependency; pass
 `connectionString`, or inject your own pool as `client`.
 
-Prefer Redis? `redis({ url })` from `experimental-a2/log-redis` stores each
+Prefer Redis? `redis({ url })` from `experimental-a2/store-redis` stores each
 session as a Redis Stream and streams push-natively: writes to watched
 sessions publish a disposable wake-up (sessions nobody watches cost no extra
 command), one shared subscriber connection per process serves every connected
@@ -49,51 +50,65 @@ Connections scale with your processes, not with your audience.
 `ioredis` is an optional peer dependency; pass `url`, or inject a client.
 Works on single instances and non-cluster providers such as Upstash, where
 durability is on by default. When only a REST API is available,
-`redisHttp({ url, token })` from `experimental-a2/log-redis-http` speaks the
+`redisHttp({ url, token })` from `experimental-a2/store-redis-http` speaks the
 same storage over `fetch`, holds no connections at all, and polls on the same
 adaptive cadence as Postgres.
 
-This configures storage for A2's session logs. It does not connect A2 to your
+This configures storage for A2's session histories. It does not connect A2 to your
 application tables or make them part of the append transaction. See
 [A2 and your database](/guides/application-data) for that boundary.
 
-## 2. Add recovery for handlers
+## 2. Add a scheduler for handlers and timers
 
-Recovery is what puts a clock on healing. `experimental-a2/recovery-vercel` rides
-Vercel Queues (`@vercel/queue` is a peer
-dependency). Same `server/orders.ts`, now with `recovery`:
+The scheduler puts a clock on recovery and future appends. Vercel Queues is the
+recommended default on Vercel. QStash is an alternative when you want HTTP
+delivery from separate infrastructure. Both adapters carry A2 recovery
+watchdogs and `session.schedule` tasks through the same route.
+
+### Vercel Queues (recommended)
+
+`experimental-a2/scheduler-vercel` rides Vercel Queues (`@vercel/queue` is a
+peer dependency). Same `server/orders.ts`, now with `scheduler`:
 
 ```ts
-// server/orders.ts, now with recovery:
+// server/orders.ts, now with a scheduler:
 import { createServer } from 'experimental-a2/server'
-import { postgres } from 'experimental-a2/log-postgres'
-import { vercelQueues } from 'experimental-a2/recovery-vercel'
+import { postgres } from 'experimental-a2/store-postgres'
+import { vercelQueues } from 'experimental-a2/scheduler-vercel'
 import { orders } from '@/contracts'
 
-export const recovery = vercelQueues()
+export const scheduler = vercelQueues()
 
 export const ordersServer = createServer({
   contract: orders,
-  log: postgres({ connectionString: process.env.DATABASE_URL }),
-  recovery,
+  store: postgres({ connectionString: process.env.DATABASE_URL }),
+  scheduler,
   handlers: {
     /* ... */
   },
 })
 ```
 
-One recovery instance is shared by every server, and mounted once:
+One scheduler instance is shared by every server on this route. The route
+derives that scheduler from the servers:
 
-```ts app/api/a2/recovery/route.ts
-import { recovery, ordersServer, billingServer } from '@/server'
+```ts app/api/a2/scheduler/route.ts
+import { schedulerHandler } from 'experimental-a2/http'
+import { ordersServer, billingServer } from '@/server'
 
-export const POST = recovery.handler(ordersServer, billingServer)
+export const POST = schedulerHandler(ordersServer, billingServer)
 ```
 
+`schedulerHandler(...)` checks the wiring synchronously when this module loads.
+Every argument must be an A2 server with a scheduler, every contract name must
+be unique, and every server must hold the exact same scheduler instance. A bad
+route fails during startup or a build that evaluates route modules, before it
+can accept a delivery.
+
 ```json vercel.json
 {
   "functions": {
-    "app/api/a2/recovery/route.ts": {
+    "app/api/a2/scheduler/route.ts": {
       "experimentalTriggers": [
         {
           "type": "queue/v2beta",
@@ -105,34 +120,247 @@ export const POST = recovery.handler(ordersServer, billingServer)
 }
 ```
 
-The trigger makes the route private. Only queue infrastructure can invoke
-it, so it needs no auth of its own.
-
-One route, one job. A top-level append that leaves or finds pending handler
-work starts a delayed, coalesced "drain this session" arm alongside its inline
-handler. The handler does not wait for the queue, while `append` joins the
-initial arm for up to two seconds before it returns. An unresponsive queue
-therefore cannot hold the append open indefinitely. Claim renewals arm another
-watchdog for just after the current window. Live handlers keep moving their
-per-event claims and the watchdog forward. A killed holder stops heartbeating,
-its claims expire, and the next watchdog retries those pending events. When
-Vercel exposes the
-function deadline, A2 caps the final claim window there so timeout recovery
-starts promptly. A failing handler keeps the current message and redelivers
-with backoff. Work outside its lane continues.
-
-Due times are rounded to one-second slots. Top-level arms, claim renewals, and
-racing deliveries targeting the same slot deduplicate into one queue message.
-Busy deliveries continue the current message's heartbeat-aligned slot series,
-so they do not create an independent stream of watchdog callbacks.
+The trigger is required. It is the Vercel Queues consumer integration and the
+route's protection boundary: Vercel makes the function private, and only queue
+infrastructure can invoke it. Do not expose the callback as an ordinary public
+route or add a second authentication scheme around it.
+
+The adapter pins recovery watchdogs and scheduled appends to the deployment
+that published them. This is the safe default. A recovery task always returns
+to code that understands the session work it armed.
+
+You can remove deployment affinity from scheduled appends only:
+
+```ts
+// server/orders.ts, when every consumer is compatible with pending timers:
+import { vercelQueues } from 'experimental-a2/scheduler-vercel'
+
+export const scheduler = vercelQueues({
+  scheduledAppends: 'deploymentless',
+})
+```
+
+`deploymentless` does not mean "latest production." It lets Vercel Queues
+choose an eligible deploymentless consumer for the topic. Keep task envelopes
+and event schemas compatible across every such consumer. Recovery watchdogs
+remain deployment-scoped in both modes. Scheduler task envelopes are
+versioned.
+
+Vercel Queues defaults to 24-hour message retention. The adapter sets retention
+to 7 days and accepts scheduled appends through 6 days, reserving the final day
+for delivery and retries. Recovery watchdogs are measured in seconds.
+
+### QStash
+
+`experimental-a2/scheduler-qstash` uses QStash HTTP delivery
+(`@upstash/qstash` 2.11.0 or newer is a peer dependency). The conventional
+callback URL is
+inferred in QStash development mode and on Vercel production and preview
+deployments:
+
+```ts
+// server/orders.ts, with QStash instead:
+import { createServer } from 'experimental-a2/server'
+import { postgres } from 'experimental-a2/store-postgres'
+import { qstash } from 'experimental-a2/scheduler-qstash'
+import { orders } from '@/contracts'
+
+export const scheduler = qstash()
+
+export const ordersServer = createServer({
+  contract: orders,
+  store: postgres({ connectionString: process.env.DATABASE_URL }),
+  scheduler,
+  handlers: {
+    /* ... */
+  },
+})
+```
+
+Mount `schedulerHandler(ordersServer, billingServer)` once with the same
+`app/api/a2/scheduler/route.ts` module shown above. Local Node.js development
+needs no QStash environment variables. With `@upstash/qstash` 2.11.0 or newer,
+A2 uses `http://localhost:${PORT ?? 3000}/api/a2/scheduler` and enables the
+SDK's development mode. On the first publish or verification, the lazy SDK
+transport downloads and starts its local server. `PORT` selects the application
+port. `QSTASH_DEV_PORT` optionally selects the QStash development server port
+and does not change the callback. This inferred local route sets SDK development
+mode directly; `QSTASH_DEV=false` does not disable it. Pass an explicit `url`
+to leave SDK mode under environment control.
+
+A process-less runtime cannot infer local development and needs an explicit
+`url`. An Edge runtime that exposes `process.env` can use the inferred localhost
+callback, but it cannot start the development server itself. Start the server
+from a Node.js instrumentation hook or run it manually. Configure the SDK with
+`QSTASH_DEV` or the manual `QSTASH_URL` and development credentials described
+in the [QStash local-development guide](https://upstash.com/docs/qstash/howto/local-development).
+
+A2 disables the SDK's development mode when `NODE_ENV=production`, even if
+`QSTASH_DEV` remains set. Production must use deployed QStash credentials and
+must not rely on the local development server.
+
+Production uses `VERCEL_PROJECT_PRODUCTION_URL`; preview uses
+`VERCEL_BRANCH_URL`. Both append the conventional `/api/a2/scheduler` path. An
+explicit `url` always wins. Pass one for another host or path; an application
+port that is not available through `PORT`; Docker networking; a public
+development tunnel; or a non-Vercel production deployment. Outside production,
+an explicit URL leaves SDK development mode under the SDK's normal environment
+configuration. This supports `QSTASH_DEV=true` and manually supplied
+`QSTASH_URL` plus credentials without changing the callback URL.
+
+Production and branch URLs are moving aliases. Pending QStash tasks can reach a
+later deployment on that alias, so keep scheduler task envelopes and event
+schemas compatible across deployments.
+
+The deployed route must be internet-reachable by QStash after any platform edge
+authentication. The local QStash server calls the inferred localhost route
+directly. A2 reads the raw request body and verifies `Upstash-Signature`, the
+callback URL, and the optional `Upstash-Region` before parsing the message or
+touching the store. Missing or invalid signatures return 401, so ordinary
+unsigned traffic cannot execute scheduler work.
+
+When A2 infers a Vercel preview callback, it opportunistically reads the current
+invocation's `x-vercel-oidc-token` on every publish. If the token is a parseable
+JWT that remains valid for at least 60 seconds after the task's first due time,
+the built-in transport forwards it as
+`x-vercel-trusted-oidc-idp-token` and asks QStash to redact that header from its
+dashboard and API. This lets near-term tasks enter a protected preview when its
+[Trusted Sources](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/trusted-sources)
+rule accepts the publishing deployment's identity. QStash still receives the
+credential so it can deliver it.
+
+This forwarding is best-effort first-delivery admission, not durable
+authentication. A missing, malformed, or near-expiry token leaves the publish
+unchanged so an unprotected preview does not regress. An explicit `url` opts out,
+and inferred production and local callbacks do not forward the token. QStash
+stores the static header and reuses it on retries, so a delayed task or retry can
+outlive the identity even if the token passed the first-due-time check.
+
+Callbacks that need durable access through Deployment Protection still require
+an explicit
+[automation bypass](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation)
+or a public ingress. Pass a dedicated bypass as a callback header when that is
+the right tradeoff for your project:
+
+```ts server/scheduler-qstash-protected.ts
+import { qstash } from 'experimental-a2/scheduler-qstash'
+
+const bypass = process.env.VERCEL_AUTOMATION_BYPASS_SECRET
+if (!bypass) throw new Error('Missing VERCEL_AUTOMATION_BYPASS_SECRET')
+
+export const scheduler = qstash({
+  headers: { 'x-vercel-protection-bypass': bypass },
+})
+```
+
+A2 asks QStash to redact every configured callback header from its dashboard
+and API. QStash still receives, stores, and delivers the original value. A
+rotation affects new messages only; already queued messages keep their captured
+header. A custom transport must deliver every `headers` entry and apply every
+`redact.header` name it receives.
+
+A separately implemented public Vercel production ingress can instead verify
+the QStash signature and relay the task at delivery time with its current
+Vercel OIDC token. That architecture requires an explicit
+production-to-preview Trusted Sources rule, a strict target allowlist, and a
+versioned routing envelope. The adapter does not implement that relay.
+`Upstash-Signature` remains the authentication boundary for scheduler work
+after any deployment-protection check.
+
+Servers configured with different scheduler instances use separate routes,
+even when both instances come from the same adapter. Each QStash instance needs
+its own callback URL, public when cloud QStash delivers to it, so additional
+instances pass `url` explicitly. Each Vercel Queues instance needs its own
+`topic` and a matching `experimentalTriggers` entry for that route. One route
+cannot safely choose a provider after a delivery arrives.
+
+By default, the adapter reads `QSTASH_TOKEN`,
+`QSTASH_CURRENT_SIGNING_KEY`, and `QSTASH_NEXT_SIGNING_KEY`. You can instead
+pass `token`, `currentSigningKey`, and `nextSigningKey` directly to `qstash()`.
+Pass a configured `@upstash/qstash` `Client` as `client` when you need to tune
+its API endpoint, publish-request retry policy, or telemetry. A2 continues to
+create the signature `Receiver` from the signing-key options. Use A2's
+`headers` option, not client-global headers, for callback credentials. Use the
+full `transport` override when publishing and verification both need custom
+behavior. An injected client or transport receives raw callback credentials
+and must honor A2's redaction request without logging them.
+The adapter reads at most 1 MiB per callback. Before an initial publish, it
+checks the normalized task with enough reserved room for the largest private
+store-outage lineage it may later need. Every accepted task can therefore carry
+that lineage under the same `maxBodyBytes` limit. Store-outage successors also
+check their actual body before publishing. Set `maxBodyBytes` to a larger plan
+limit when your scheduled event batches legitimately need it. Keep the same
+value on every deployment that can receive the callback URL. Lowering it during
+a rolling deployment can make the new deployment reject a task that the
+previous deployment accepted.
+
+QStash delivery is at least once. Stalled work publishes a fresh successor
+watchdog before acknowledging, so QStash's per-message retry budget cannot stop
+recovery before A2 reaches its own failure limit. A retryable store outage also
+hands recovery or timer delivery to a fresh message, but uses a slower outage
+backoff. The first successor waits 30 seconds. Each later generation waits four
+times longer, capped at one day. The generation stays in the signed message
+body and keeps increasing after the delay reaches the cap. Duplicate callbacks
+therefore converge on one successor generation even when they run at different
+times. Separate drain episodes use the first failing task's `dueAt` as part of
+their lineage. Append episodes use the contract, session, and stable task id.
+
+If the successor publish fails, the current delivery returns non-2xx and
+QStash retries it. The adapter omits `retries` by default, which lets QStash use
+its provider default. Pass `retries` to set an explicit non-negative value.
+Every delivery attempt is billable. Deterministic publish keys reduce duplicate
+watchdogs during QStash's documented 10-minute deduplication window. That is a
+cost optimization, not a correctness boundary. Durable store claims, attempt
+fencing, and idempotent handler side effects handle duplicate deliveries.
+
+No callback can publish a successor when QStash cannot reach the route at all.
+Monitor QStash's DLQ and redrive those messages after a prolonged endpoint or
+deployment outage.
+
+QStash currently caps a one-shot delay at 7 days on free plans and 1 year on
+usage-based plans; fixed plans have no fixed maximum. A2 recovery watchdogs
+are seconds apart. Scheduled appends must fit the active plan's limit. Leave at
+least one second of margin at a QStash plan maximum because `notBefore` uses
+whole Unix seconds.
+
+Both built-in adapters preserve the original error when a provider rejects a
+send. A recognized definite HTTP 4xx status or error is terminal, except 408,
+409, 425, and 429.
+Those four statuses, every 5xx response, network failures, and errors without a
+recognized status stay retryable because the send may be transient or
+ambiguous. The Vercel adapter also recognizes the SDK's name-only bad request,
+unauthorized, and forbidden errors. Its native duplicate-message error remains
+success. This classification lets durable A2 tools retry transient sends
+without retrying permanent credentials or request errors.
+
+One route, two task kinds. A top-level append that leaves or finds pending
+handler work sends a delayed, coalesced "drain this session" watchdog alongside
+its inline handler. The handler does not wait for the queue, while `append`
+joins the initial send for up to two seconds before it returns. An unresponsive
+queue therefore cannot hold the append open indefinitely. Claim renewals send
+another watchdog for just after the current window. Live handlers keep moving
+their per-event claims and the watchdog forward. A killed holder stops
+heartbeating, its claims expire, and the next watchdog retries those pending
+events. With Vercel Queues, A2 caps the final claim window at the platform
+function deadline when Vercel exposes it, so timeout recovery starts promptly.
+A failing handler keeps the current message and redelivers with backoff. Work
+outside its lane continues.
+
+Due times are rounded to one-second slots. Top-level sends, claim renewals, and
+racing deliveries targeting the same slot use the same deterministic publish
+key. Provider deduplication can fold them into one message. Busy deliveries
+continue the current message's heartbeat-aligned slot series, so they do not
+create an independent stream of watchdog callbacks.
 Events appended or returned by handlers ride their current execution window
-and add no recovery operation of their own.
+and add no scheduler operation of their own.
 
-The queue message carries the recovery state, and recovery consults the log.
+Drain tasks carry recovery state and consult the store. Timer tasks carry their
+fixed-id events until delivery, then enter the store through an ordinary
+top-level append.
 
 ### Dispatch in a fresh invocation
 
-Ordinary `append` starts handlers inline and uses recovery as the watchdog.
+Ordinary `append` starts handlers inline and uses the scheduler as the watchdog.
 Use `append.dispatch` when the current request should only persist the input
 and hand pending work directly to the queue:
 
@@ -155,6 +383,26 @@ If the send fails, the event may already be durable. Its explicit ID makes the
 retry idempotent. An existing drain for the same session can still claim the
 event first; dispatch controls the wakeup path, not worker affinity.
 
+### Schedule a future append
+
+`session.schedule(name, { delay: '30s' }, ...events)` uses this same adapter
+and route. The call validates the events, requires JSON payload values, and
+awaits provider acceptance. It does not write an event until delivery. The
+delivered events are ordinary root appends with `cause: null`.
+
+Use one stable name, timing, and event batch across retries. A root relative
+delay starts from each call, while a handler relative delay starts from its
+triggering event's durable `createdAt`. Use `{ at: Date }` when a retried root
+call needs one target time. [Timers and delays](/guides/timers) covers
+identity, stale-timer guards, and the full provider limits.
+
+:::warning
+Do not roll back to an A2 version without timer delivery while scheduled tasks
+are outstanding. New routes accept old recovery messages, but an old route can
+acknowledge a new append task without writing its events. Roll forward, or wait
+for those tasks to deliver or remove them through the provider first.
+:::
+
 ## 3. When an event dead-letters
 
 After ten caught handler failures, A2 stops retrying an event. It blocks later
@@ -184,7 +432,7 @@ import { orders } from '@/contracts'
 
 export const ordersServer = createServer({
   contract: orders,
-  // ...log, recovery, and handlers as above
+  // ...store, scheduler, and handlers as above
   telemetry: otel(),
 })
 ```
@@ -199,27 +447,28 @@ alert on when an event [needs manual
 resolution](#3-when-an-event-dead-letters). See the
 [API reference](/reference/api#a2otel) for the span catalogue.
 
-## Running without a queue
+## Running without a scheduler
 
-If none of a contract's event types have handlers, skip `recovery`. Those
-events settle in their append transaction, so they create no drains, claims,
-queue messages, or recovery callbacks. The server is a durable event log for
-history, reducers, and live sync. That is a complete production configuration,
-not degraded recovery, because there is no reaction to recover.
+If none of a contract's event types have handlers and the app never calls
+`session.schedule`, skip `scheduler`. Those events settle in their append
+transaction, so they create no drains, claims, queue messages, or scheduler
+callbacks. The server is a durable event log for history, reducers, and live
+sync. That is a complete production configuration, not degraded recovery,
+because there is no reaction or timer to deliver.
 
-With handlers, skipping `recovery` means the only wakeups are a top-level
+With handlers, skipping `scheduler` means the only wakeups are a top-level
 append or explicit `server.drain()`. Reads never wake the session.
 
 That's a real configuration, not a broken one. Fine for internal tools
 and low-stakes apps where "heals on the next write" is acceptable. But
 there's no clock in it: a session nobody wakes stays stuck until someone does.
-For production, configure recovery.
+For production, configure a scheduler.
 
 ## Checklist
 
 | Piece               | Done when                                                            |
 | ------------------- | -------------------------------------------------------------------- |
-| Log                 | `log: postgres(...)` on every server                                  |
-| Recovery            | for servers with handlers: one shared `vercelQueues()`, route, and trigger |
+| Store               | `store: postgres(...)` on every server                                |
+| Scheduler           | for servers with handlers or timers: one shared adapter, route, and provider config |
 | Idempotent handlers | external side effects take `event.id` as an idempotency key           |
 | Client split        | contracts/reducers isomorphic; only `experimental-a2/server` touches backends      |