experimental-a2 0.17.0 → 0.18.0

package/docs/guides/05-production.mdx

@@ -1,534 +0,0 @@
----
-title: Going to production
-description: Point the store at durable storage, add a scheduler for recovery and timers, and know what to do when an event dead-letters.
----
-
-## Two pieces
-
-Development needs zero setup: SQLite appears under `.a2/`, and tests run
-in memory. Every production server needs a durable store. Servers with handlers
-or scheduled events also need a scheduler. Neither choice changes your event
-contract.
-
-## 1. Choose a store
-
-Production has no default store, on purpose. A server without one throws
-`STORE_NOT_CONFIGURED` at startup. A failed boot beats events written to a
-filesystem that evaporates.
-
-```ts server/orders.ts
-import { createServer } from 'experimental-a2/server'
-import { postgres } from 'experimental-a2/store-postgres'
-import { orders } from '@/contracts'
-
-export const ordersServer = createServer({
-  contract: orders,
-  store: postgres({ connectionString: process.env.DATABASE_URL }),
-  handlers: {
-    /* ... */
-  },
-})
-```
-
-The Postgres backend uses real transactions; appends serialize per
-session on an advisory lock, while handler claims remain concurrent. The live
-stream polls the store with an
-activity-adaptive cadence: 25ms while a session is producing events
-(a token stream reads smoothly, not in clumps), backing off to two seconds
-when it goes quiet. Any Postgres works: Neon, Supabase, RDS, your own
-box; transaction-mode poolers included, which is exactly why polling
-is the default. `pg` is an optional peer dependency; pass
-`connectionString`, or inject your own pool as `client`.
-
-To use native Postgres notifications, pass `listenConnectionString` alongside
-`connectionString` (or `client`). It must point to the same database through a
-direct connection. Stores with matching connection strings, Postgres environment
-settings, and driver defaults share one query pool and one dedicated listener
-within the same JavaScript runtime, including across separately loaded copies
-of A2. Owned pools also share schema initialization for each schema revision.
-The listener handles every store's event and presence channels.
-
-On Vercel, pools created by A2 use the current request's `waitUntil` internally
-to let idle connections close before the instance suspends. Nearby requests
-reuse those connections. This adds no database queries and needs no setup.
-When you supply `client`, you manage its pool's platform lifecycle.
-
-Closing a store stops its own subscriptions. The listener closes when its last
-subscriber leaves; the query pool closes when its last store releases it.
-Injected clients are shared by object identity. Each store keeps its own clock,
-ID generator, cursors, and presence state. Different serverless instances still
-have separate pools and listeners, so the direct connection needs database
-connection headroom.
-
-Enabling this option installs transactional notification triggers on A2's event
-and presence tables. They also cover writes from instances that use only the
-pooled connection. The triggers remain installed when listeners disconnect.
-Notifications are delivered after commit; a reconnect catches up from the
-cursor. Native event and presence subscriptions also reconcile every ten
-seconds. Without `listenConnectionString`, the store uses adaptive polling.
-
-An idle stream stays connected and keeps receiving heartbeats. A new event
-returns polling to its fast cadence. A local `session.push()` refreshes an
-idle SSE stream immediately from its current cursor, even when the write and
-stream routes run in different invocations. WebSocket pushes wake the attached
-stream without reopening the socket. A stream that becomes idle while a push
-is pending, or before observing its acknowledgement, wakes again. An update
-from another client can take up to the idle interval plus query latency to
-arrive after a quiet period.
-
-Prefer Redis? `redis({ url })` from `experimental-a2/store-redis` stores each
-session as a Redis Stream and streams push-natively: every write publishes a
-disposable wake-up, one shared subscriber connection per process serves every
-connected viewer, and a safety re-read covers a lost wake-up within ten
-seconds. Connections scale with your processes, not with your audience. On
-providers that bill per command, such as Upstash, each command a script runs
-counts: appending an event with a handler and running it takes 24 commands,
-and an idle live stream takes 6 a minute.
-`ioredis` is an optional peer dependency; pass `url`, or inject a client.
-Works on single instances and non-cluster providers such as Upstash, where
-durability is on by default. When only a REST API is available,
-`redisHttp({ url, token })` from `experimental-a2/store-redis-http` speaks the
-same storage over `fetch`. Live events and presence use Upstash’s HTTP
-`SUBSCRIBE` endpoint (SSE), with the same URL and token. Each active channel
-holds one outbound HTTP subscription per store instance, shared by its local
-viewers and closed when the last viewer leaves. Writes publish wake-ups inside
-the same Redis script as the append, and each wake-up carries the new events so
-live streams deliver them without another read. Live streams trust those
-messages, so a Redis identity that can publish to the `:notify` channels can add
-or hide live events: grant publish access only to identities that can also
-write the store. Presence writes
-also publish their applied fields inside the script. A rejected notification
-does not fail the applied write. Idle event streams
-read once every ten seconds to recover missed notifications. Subscriptions reconnect automatically and catch up from
-the event cursor. Presence reads wait for subscription readiness; after a
-subscription reconnects, A2 re-reads the current presence fields, clears fields
-deleted during the gap, and may repeat their latest values. Updates received
-during that read take precedence. If the subscription endpoint returns 404, 405, or 501, A2
-checks ordinary Redis commands with `PING` and remembers that the store must
-poll instead. Events and presence then use adaptive polling, from 25ms to two
-seconds while idle. Authentication errors, timeouts, and other failures remain
-errors. No Redis client dependency or extra notifier setup is needed.
-
-This configures storage for A2's session histories. It does not connect A2 to your
-application tables or make them part of the append transaction. See
-[A2 and your database](/guides/application-data) for that boundary.
-
-## 2. Add a scheduler for handlers and timers
-
-The scheduler puts a clock on recovery and future appends. Vercel Queues is the
-recommended default on Vercel. QStash is an alternative when you want HTTP
-delivery from separate infrastructure. Both adapters carry A2 recovery
-watchdogs and `session.schedule` tasks through the same route.
-
-### Vercel Queues (recommended)
-
-`experimental-a2/scheduler-vercel` rides Vercel Queues (`@vercel/queue` is a
-peer dependency). Same `server/orders.ts`, now with `scheduler`:
-
-```ts
-// server/orders.ts, now with a scheduler:
-import { createServer } from 'experimental-a2/server'
-import { postgres } from 'experimental-a2/store-postgres'
-import { vercelQueues } from 'experimental-a2/scheduler-vercel'
-import { orders } from '@/contracts'
-
-export const scheduler = vercelQueues()
-
-export const ordersServer = createServer({
-  contract: orders,
-  store: postgres({ connectionString: process.env.DATABASE_URL }),
-  scheduler,
-  handlers: {
-    /* ... */
-  },
-})
-```
-
-One scheduler instance is shared by every server on this route. Mount its
-handler with every server it may deliver to:
-
-```ts app/api/a2/scheduler/route.ts
-import { billingServer, ordersServer, scheduler } from '@/server'
-
-export const POST = scheduler.handler(ordersServer, billingServer)
-```
-
-`scheduler.handler(...)` checks the server list synchronously when this module
-loads. It requires at least one drainable server and unique contract names. A
-bad route fails during startup or a build that evaluates route modules, before
-it can accept a delivery.
-
-```json vercel.json
-{
-  "functions": {
-    "app/api/a2/scheduler/route.ts": {
-      "experimentalTriggers": [
-        {
-          "type": "queue/v2beta",
-          "topic": "a2"
-        }
-      ]
-    }
-  }
-}
-```
-
-The trigger is required. It is the Vercel Queues consumer integration and the
-route's protection boundary: Vercel makes the function private, and only queue
-infrastructure can invoke it. Do not expose the callback as an ordinary public
-route or add a second authentication scheme around it.
-
-The adapter uses one queue client for recovery watchdogs, scheduled appends,
-and scheduled clears. Its `deploymentId` setting applies to sends, payload
-fetches, acknowledgements, and visibility changes. Omit it to use the publishing
-deployment, pass a deployment ID to target that deployment, or pass `null` for
-deploymentless delivery. Publishing and consuming clients must use the same
-deployment scope.
-
-Promoting a new deployment does not move pinned tasks. Older deployments keep
-receiving their own work until it is acknowledged or expires. Upgrading does
-not repair an older deployment's callback handler.
-
-With `deploymentId: null`, Vercel chooses an eligible deploymentless consumer;
-this does not select the latest production deployment. Every eligible consumer
-must use `deploymentId: null` and remain compatible with pending tasks and
-session work.
-
-Changing the setting does not move already queued tasks. Keep consumers for
-their original deployment scope until those tasks finish. The former
-`scheduledAppends: 'deploymentless'` option affected only timers; its replacement,
-`deploymentId: null`, also makes recovery deploymentless.
-
-Vercel Queues defaults to 24-hour message retention. The adapter sets retention
-to 7 days and accepts scheduled appends and clears through 6 days, reserving
-the final day
-for delivery and retries. Recovery watchdogs are measured in seconds.
-
-### QStash
-
-`experimental-a2/scheduler-qstash` uses QStash HTTP delivery
-(`@upstash/qstash` 2.11.0 or newer is a peer dependency). The conventional
-callback URL is
-inferred in QStash development mode and on Vercel production and preview
-deployments:
-
-```ts
-// server/orders.ts, with QStash instead:
-import { createServer } from 'experimental-a2/server'
-import { postgres } from 'experimental-a2/store-postgres'
-import { qstash } from 'experimental-a2/scheduler-qstash'
-import { orders } from '@/contracts'
-
-export const scheduler = qstash()
-
-export const ordersServer = createServer({
-  contract: orders,
-  store: postgres({ connectionString: process.env.DATABASE_URL }),
-  scheduler,
-  handlers: {
-    /* ... */
-  },
-})
-```
-
-Mount `scheduler.handler(ordersServer, billingServer)` once with the same
-`app/api/a2/scheduler/route.ts` module shown above. Local Node.js development
-needs no QStash environment variables. With `@upstash/qstash` 2.11.0 or newer,
-A2 uses `http://localhost:${PORT ?? 3000}/api/a2/scheduler` and enables the
-SDK's development mode. On the first publish or verification, the lazy SDK
-transport downloads and starts its local server. `PORT` selects the application
-port. `QSTASH_DEV_PORT` optionally selects the QStash development server port
-and does not change the callback. This inferred local route sets SDK development
-mode directly; `QSTASH_DEV=false` does not disable it. Pass an explicit `url`
-to leave SDK mode under environment control.
-
-A process-less runtime cannot infer local development and needs an explicit
-`url`. An Edge runtime that exposes `process.env` can use the inferred localhost
-callback, but it cannot start the development server itself. Start the server
-from a Node.js instrumentation hook or run it manually. Configure the SDK with
-`QSTASH_DEV` or the manual `QSTASH_URL` and development credentials described
-in the [QStash local-development guide](https://upstash.com/docs/qstash/howto/local-development).
-
-A2 disables the SDK's development mode when `NODE_ENV=production`, even if
-`QSTASH_DEV` remains set. Production must use deployed QStash credentials and
-must not rely on the local development server.
-
-Production uses `VERCEL_PROJECT_PRODUCTION_URL`; preview uses
-`VERCEL_BRANCH_URL`. Both append the conventional `/api/a2/scheduler` path. An
-explicit `url` always wins. Pass one for another host or path; an application
-port that is not available through `PORT`; Docker networking; a public
-development tunnel; or a non-Vercel production deployment. Outside production,
-an explicit URL leaves SDK development mode under the SDK's normal environment
-configuration. This supports `QSTASH_DEV=true` and manually supplied
-`QSTASH_URL` plus credentials without changing the callback URL.
-
-Production and branch URLs are moving aliases. Pending QStash tasks can reach a
-later deployment on that alias, so keep scheduler task envelopes and event
-schemas compatible across deployments.
-
-The deployed route must be internet-reachable by QStash after any platform edge
-authentication. The local QStash server calls the inferred localhost route
-directly. A2 reads the raw request body and verifies `Upstash-Signature`, the
-callback URL, and the optional `Upstash-Region` before parsing the message or
-touching the store. Missing or invalid signatures return 401, so ordinary
-unsigned traffic cannot execute scheduler work.
-
-When A2 infers a Vercel preview callback, it opportunistically reads the current
-invocation's `x-vercel-oidc-token` on every publish. If the token is a parseable
-JWT that remains valid for at least 60 seconds after the task's first due time,
-the built-in transport forwards it as
-`x-vercel-trusted-oidc-idp-token` and asks QStash to redact that header from its
-dashboard and API. This lets near-term tasks enter a protected preview when its
-[Trusted Sources](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/trusted-sources)
-rule accepts the publishing deployment's identity. QStash still receives the
-credential so it can deliver it.
-
-This forwarding is best-effort first-delivery admission, not durable
-authentication. A missing, malformed, or near-expiry token leaves the publish
-unchanged so an unprotected preview does not regress. An explicit `url` opts out,
-and inferred production and local callbacks do not forward the token. QStash
-stores the static header and reuses it on retries, so a delayed task or retry can
-outlive the identity even if the token passed the first-due-time check.
-
-Callbacks that need durable access through Deployment Protection still require
-an explicit
-[automation bypass](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation)
-or a public ingress. Pass a dedicated bypass as a callback header when that is
-the right tradeoff for your project:
-
-```ts server/scheduler-qstash-protected.ts
-import { qstash } from 'experimental-a2/scheduler-qstash'
-
-const bypass = process.env.VERCEL_AUTOMATION_BYPASS_SECRET
-if (!bypass) throw new Error('Missing VERCEL_AUTOMATION_BYPASS_SECRET')
-
-export const scheduler = qstash({
-  headers: { 'x-vercel-protection-bypass': bypass },
-})
-```
-
-A2 asks QStash to redact every configured callback header from its dashboard
-and API. QStash still receives, stores, and delivers the original value. A
-rotation affects new messages only; already queued messages keep their captured
-header. A custom transport must deliver every `headers` entry and apply every
-`redact.header` name it receives.
-
-A separately implemented public Vercel production ingress can instead verify
-the QStash signature and relay the task at delivery time with its current
-Vercel OIDC token. That architecture requires an explicit
-production-to-preview Trusted Sources rule, a strict target allowlist, and a
-versioned routing envelope. The adapter does not implement that relay.
-`Upstash-Signature` remains the authentication boundary for scheduler work
-after any deployment-protection check.
-
-Servers configured with different scheduler instances use separate routes,
-even when both instances come from the same adapter. Each QStash instance needs
-its own callback URL, public when cloud QStash delivers to it, so additional
-instances pass `url` explicitly. Each Vercel Queues instance needs its own
-`topic` and a matching `experimentalTriggers` entry for that route. One route
-cannot safely choose a provider after a delivery arrives.
-
-By default, the adapter reads `QSTASH_TOKEN`,
-`QSTASH_CURRENT_SIGNING_KEY`, and `QSTASH_NEXT_SIGNING_KEY`. You can instead
-pass `token`, `currentSigningKey`, and `nextSigningKey` directly to `qstash()`.
-Pass a configured `@upstash/qstash` `Client` as `client` when you need to tune
-its API endpoint, publish-request retry policy, or telemetry. A2 continues to
-create the signature `Receiver` from the signing-key options. Use A2's
-`headers` option, not client-global headers, for callback credentials. Use the
-full `transport` override when publishing and verification both need custom
-behavior. An injected client or transport receives raw callback credentials
-and must honor A2's redaction request without logging them.
-The adapter reads at most 1 MiB per callback. Before an initial publish, it
-checks the normalized task with enough reserved room for the largest private
-store-outage lineage it may later need. Every accepted task can therefore carry
-that lineage under the same `maxBodyBytes` limit. Store-outage successors also
-check their actual body before publishing. Set `maxBodyBytes` to a larger plan
-limit when your scheduled event batches legitimately need it. Keep the same
-value on every deployment that can receive the callback URL. Lowering it during
-a rolling deployment can make the new deployment reject a task that the
-previous deployment accepted.
-
-QStash delivery is at least once. Stalled work publishes a fresh successor
-watchdog before acknowledging, so QStash's per-message retry budget cannot stop
-recovery before A2 reaches its own failure limit. A retryable store outage also
-hands recovery or timer delivery to a fresh message, but uses a slower outage
-backoff. The first successor waits 30 seconds. Each later generation waits four
-times longer, capped at one day. The generation stays in the signed message
-body and keeps increasing after the delay reaches the cap. Duplicate callbacks
-therefore converge on one successor generation even when they run at different
-times. Separate drain episodes use the first failing task's `dueAt` as part of
-their lineage. Append episodes use the contract, session, and stable task id.
-
-If the successor publish fails, the current delivery returns non-2xx and
-QStash retries it. The adapter omits `retries` by default, which lets QStash use
-its provider default. Pass `retries` to set an explicit non-negative value.
-Every delivery attempt is billable. Deterministic publish keys reduce duplicate
-watchdogs during QStash's documented 10-minute deduplication window. That is a
-cost optimization, not a correctness boundary. Durable store claims, attempt
-fencing, and idempotent handler side effects handle duplicate deliveries.
-
-No callback can publish a successor when QStash cannot reach the route at all.
-Monitor QStash's DLQ and redrive those messages after a prolonged endpoint or
-deployment outage.
-
-QStash currently caps a one-shot delay at 7 days on free plans and 1 year on
-usage-based plans; fixed plans have no fixed maximum. A2 recovery watchdogs
-are seconds apart. Scheduled appends must fit the active plan's limit. Leave at
-least one second of margin at a QStash plan maximum because `notBefore` uses
-whole Unix seconds.
-
-Both built-in adapters preserve the original error when a provider rejects a
-send. A recognized definite HTTP 4xx status or error is terminal, except 408,
-409, 425, and 429.
-Those four statuses, every 5xx response, network failures, and errors without a
-recognized status stay retryable because the send may be transient or
-ambiguous. The Vercel adapter also recognizes the SDK's name-only bad request,
-unauthorized, and forbidden errors. Its native duplicate-message error remains
-success. This classification lets durable A2 tools retry transient sends
-without retrying permanent credentials or request errors.
-
-One route, two task kinds. A top-level append that leaves or finds pending
-handler work sends a delayed, coalesced "drain this session" watchdog alongside
-its inline handler. The handler does not wait for the queue, while `append`
-joins the initial send for up to two seconds before it returns. An unresponsive
-queue therefore cannot hold the append open indefinitely. Claim renewals send
-another watchdog for just after the current window. Live handlers keep moving
-their per-event claims and the watchdog forward. A killed holder stops
-heartbeating, its claims expire, and the next watchdog retries those pending
-events. With Vercel Queues, A2 caps the final claim window at the platform
-function deadline when Vercel exposes it, so timeout recovery starts promptly.
-A failing handler keeps the current message and redelivers with backoff. Work
-outside its lane continues.
-
-Due times are rounded to one-second slots. Top-level sends, claim renewals, and
-racing deliveries targeting the same slot use the same deterministic publish
-key. Provider deduplication can fold them into one message. Busy deliveries
-continue the current message's heartbeat-aligned slot series, so they do not
-create an independent stream of watchdog callbacks.
-Events appended or returned by handlers ride their current execution window
-and add no scheduler operation of their own.
-
-Drain tasks carry recovery state and consult the store. Timer tasks carry their
-fixed-id events until delivery, then enter the store through an ordinary
-top-level append.
-
-### Dispatch in a fresh invocation
-
-Ordinary `append` starts handlers inline and uses the scheduler as the watchdog.
-Use `append.dispatch` when the current request should only persist the input
-and hand pending work directly to the queue:
-
-```ts app/api/imports/route.ts
-import { ordersServer } from '@/server'
-
-export async function POST(request: Request) {
-  const { orderId, shopId, items } = await request.json()
-  await ordersServer.session(orderId).append.dispatch({
-    id: `created:${orderId}`,
-    type: 'created',
-    payload: { shopId, items },
-  })
-  return Response.json({ accepted: true }, { status: 202 })
-}
-```
-
-The call awaits the immediate queue send and does not start an inline drain.
-If the send fails, the event may already be durable. Its explicit ID makes the
-retry idempotent. An existing drain for the same session can still claim the
-event first; dispatch controls the wakeup path, not worker affinity.
-
-### Schedule a future append
-
-`session.schedule(name, { delay: '30s' }, ...events)` uses this same adapter
-and route. The call validates the events, requires JSON payload values, and
-awaits provider acceptance. It does not write an event until delivery. The
-delivered events are ordinary root appends with `cause: null`.
-
-Use one stable name, timing, and event batch across retries. A root relative
-delay starts from each call, while a handler relative delay starts from its
-triggering event's durable `createdAt`. Use `{ at: Date }` when a retried root
-call needs one target time. [Timers and delays](/guides/timers) covers
-identity, stale-timer guards, and the full provider limits.
-
-:::warning
-Do not roll back to an A2 version without timer delivery while scheduled tasks
-are outstanding. New routes accept old recovery messages, but an old route can
-acknowledge a new append task without writing its events. The same applies to
-scheduled clears and a version without them: an old Vercel Queues route keeps
-failing a clear task until retention ends, and an old QStash route rejects it
-as non-retryable, so QStash moves it to its dead-letter queue. Roll forward, or
-wait for those tasks to deliver or remove them through the provider first.
-:::
-
-## 3. When an event dead-letters
-
-After ten caught handler failures, A2 stops retrying an event. It blocks later
-events in the same lane; unlaned events and other lanes continue.
-[Durability](/concepts/durability#when-a-handler-keeps-failing) explains this
-boundary. Resolution is manual, and has
-exactly two shapes:
-
-- **Fix and retry.** Deploy the handler fix, clear the event's failure
-  marker.
-- **Skip.** Mark the event processed, accepting that its effects never
-  happened.
-
-Both are one-column updates on the events table.
-
-## Watch it in traces
-
-A2 takes an optional `telemetry`, an instrumentation hook that
-`experimental-a2/otel` implements over OpenTelemetry (`@opentelemetry/api` is a peer
-dependency):
-
-```ts
-// server/orders.ts, with telemetry:
-import { createServer } from 'experimental-a2/server'
-import { otel } from 'experimental-a2/otel'
-import { orders } from '@/contracts'
-
-export const ordersServer = createServer({
-  contract: orders,
-  // ...store, scheduler, and handlers as above
-  telemetry: otel(),
-})
-```
-
-With any OTel setup registered (on Vercel, `@vercel/otel`), the happy
-path lands in a single trace: the request → `a2.append` → the inline
-`a2.drain` → an `a2.event` span per handler → the appends those handlers
-make: the whole causal chain, visually. Handler failures mark their `a2.event`
-span with the exception, each dispatch reports `ctx.attempt` as
-`a2.event.attempt`, and `a2.event.outcome = dead_lettered` is the attribute to
-alert on when an event [needs manual
-resolution](#3-when-an-event-dead-letters). See the
-[API reference](/reference/api#a2otel) for the span catalogue.
-
-Agent servers use the same telemetry for `a2.ai.*` spans. Core and AI operations
-can nest and overlap, so their durations do not add up to separate layer totals.
-
-## Running without a scheduler
-
-If none of a contract's event types have handlers and the app never calls
-`session.schedule`, skip `scheduler`. Those events settle in their append
-transaction, so they create no drains, claims, queue messages, or scheduler
-callbacks. The server is a durable event log for history, reducers, and live
-sync. That is a complete production configuration, not degraded recovery,
-because there is no reaction or timer to deliver.
-
-With handlers, skipping `scheduler` means the only wakeups are a top-level
-append or explicit `server.drain()`. Reads never wake the session.
-
-That's a real configuration, not a broken one. Fine for internal tools
-and low-stakes apps where "heals on the next write" is acceptable. But
-there's no clock in it: a session nobody wakes stays stuck until someone does.
-For production, configure a scheduler.
-
-## Checklist
-
-| Piece               | Done when                                                            |
-| ------------------- | -------------------------------------------------------------------- |
-| Store               | `store: postgres(...)` on every server                                |
-| Scheduler           | for servers with handlers or timers: one shared adapter, route, and provider config |
-| Idempotent handlers | external side effects take `event.id` as an idempotency key           |
-| Client split        | contracts/reducers isomorphic; only `experimental-a2/server` touches backends      |