solid-objects 0.15.2 → 0.16.0

package/docs/api.md

@@ -45,6 +45,9 @@ authorization, capability boundaries, and release validation.
 - `Actor`: base class providing `ref()`, `actorId`, `currentMessage`,
   `observables()`, `reject()`, `emit()`, `transmit()`, `commitAction()`,
   `schedule()`, `sendTo()`, and protected lifecycle hooks.
+  `ScheduledOperationsFor` types the map `schedule()` returns, whose calls return a
+  `ReminderHandle`. `TransmittedOperationsFor` types the map `transmit()`
+  returns, whose calls return nothing.
 - `broadcastValue(value)`: mark an observable so its changed value enters the
   durable invalidation envelope.
 - `broadcastInvalidation(value)`: compare the real observable value but put
@@ -67,10 +70,16 @@ createdAtMs }` shape returned by `SolidObjectsRuntime.snapshotWithIncarnation`.
   `DestroyOptions`: the options for authorization, idempotency, time, and
   schedule that the reference methods use.
 
+`ScheduledReminder` is one armed reminder as an actor reads it, and
+`ReminderReader` is how a runtime supplies them.
+
 `ActorIntents`, `EffectIntent`, `CommitActionIntent`, `ReminderIntent`,
+`UnscheduleIntent`, `UnscheduleAllIntent`, `ReminderMutation`,
 `OutboundMessageIntent`, `ReminderOptions`, `OutboundMessageOptions`,
 `PayloadBroadcasts`, and `PayloadBroadcastValue` describe actor-declared
-transactional work and typed personalized projections.
+transactional work and typed personalized projections. `ReminderMutation` is the
+union of one scheduled reminder and the two cancellations, held in one list so
+they apply in the order the turn called them.
 
 `EffectFailurePayload<Arguments>`, `EffectSuccessPayload<Arguments, Result>`,
 and `SerializedError` describe effect callback messages. They are also exported
@@ -162,6 +171,130 @@ row per item. It also cannot strand an entry when the runtime coalesces an
 occurrence. Prefer it for a large queue of interchangeable items. Prefer `key`
 when one item needs an alarm that you can move on its own.
 
+#### Cancelling a reminder
+
+`schedule()` returns a `ReminderHandle` (`{ name: string }`) naming the alarm it
+armed. `unschedule()` cancels one alarm, by operation, by operation and key, or
+by that handle. `unscheduleAll()` cancels every key of one operation.
+
+```typescript
+const MONTH = 30 * 24 * 60 * 60 * 1000
+
+class Subscription extends Actor {
+  static override readonly actorType = "subscriptions"
+
+  status = "trialing"
+  renewal: ReminderHandle | null = null
+
+  startTrial(): void {
+    this.schedule({ at: new Date(Date.now() + (14 * MONTH) / 30) }).trialExpired()
+  }
+
+  convertToPaid(): void {
+    this.status = "active"
+    this.unschedule("trialExpired")
+    this.renewal = this.schedule({
+      at: new Date(Date.now() + MONTH),
+      everyMilliseconds: MONTH,
+    }).chargeRenewal()
+  }
+
+  cancelled(): void {
+    this.status = "cancelled"
+    if (this.renewal) this.unschedule(this.renewal)
+  }
+
+  trialExpired(): void {
+    this.status = "expired"
+  }
+
+  chargeRenewal(): void {}
+}
+```
+
+`this.unschedule(this.renewal)` and `this.unschedule("chargeRenewal")` cancel the
+same alarm. Prefer the handle when the actor already stored one, because it
+cannot drift from the name that armed the reminder.
+
+A keyed alarm cancels by the key that armed it, and `unscheduleAll()` cancels
+every key of one operation:
+
+```typescript
+class Shipment extends Actor {
+  static override readonly actorType = "shipments"
+
+  dispatch({ carrierIds }: { carrierIds: string[] }): void {
+    for (const carrierId of carrierIds) {
+      this.schedule({ at: new Date(Date.now() + MONTH / 30), key: carrierId }).chaseCarrier({
+        carrierId,
+      })
+    }
+  }
+
+  shipped({ carrierId }: { carrierId: string }): void {
+    this.unschedule("chaseCarrier", { key: carrierId })
+  }
+
+  stopChasing(): void {
+    this.unscheduleAll("chaseCarrier")
+  }
+
+  chaseCarrier(_options: { carrierId: string }): void {}
+}
+```
+
+`unschedule()` and `unscheduleAll()` refuse an operation the actor does not
+declare, with the `UnknownOperation` that `schedule()` already throws, so a typo
+fails the turn rather than cancelling nothing. A handle skips that check, because
+the `schedule()` call that produced it was already checked.
+
+#### Reading the schedule
+
+`reminder()` returns the armed alarm as a `ScheduledReminder`, or `undefined`.
+`reminders()` returns every key of one operation. Both are async, because an
+actor reads its own rows rather than holding them in memory:
+
+```typescript
+async nextChargeAt(): Promise<number | null> {
+  return (await this.reminder("chargeRenewal"))?.runAtMilliseconds ?? null
+}
+
+async pendingCarriers(): Promise<(string | null)[]> {
+  return (await this.reminders("chaseCarrier")).map((reminder) => reminder.key)
+}
+```
+
+A read applies the intents staged so far in the turn, so an actor that schedules
+and then reads sees what the commit will write, and one that cancels and then
+reads sees the alarm gone.
+
+`key` and `intervalMilliseconds` are `null` rather than `undefined` when absent,
+so a `ScheduledReminder` returns from an operation without a serialization error.
+
+Reading is available during a turn and from a snapshot projection, so an
+observable can report what is armed. `reminder()` and `reminders()` refuse an
+operation the actor does not declare, as `schedule()` and `unschedule()` do.
+
+A one-shot that already fired is not reported. Its row stays as `completed`, and
+an alarm that cannot fire again is not armed.
+
+`ScheduledReminder` carries no occurrence count. The SQL backends track one and
+Durable Objects does not, so it is left out rather than reported for one backend
+only.
+
+A cancellation is staged like a schedule, so it commits with the state change
+that decided it and a turn that throws cancels nothing. Both apply in the order
+the turn called them, so cancelling and then scheduling the same name leaves it
+armed at the new time.
+
+Cancelling an alarm that does not exist is not an error. `unschedule()` returns
+nothing, because it stages an intent rather than applying one, and an answer
+given at call time could be stale by the time the turn commits.
+
+A handle is a plain object, so it survives in actor state and still cancels
+after a deactivation. Passing a handle together with `key` is a `TypeError`,
+because the handle already names the key.
+
 ### Recovering abandoned effects
 
 `emit` returns an `EffectHandle` (`{ id: string }`) on every successful call. Save
@@ -358,9 +491,11 @@ effect results and the handler's declared argument/result types in agreement.
 ### Typed operation references
 
 `schedule` and `transmit` infer this actor's operation names and arguments, including
-inside actor methods and for inherited application operations. The returned
-`ScheduledOperationsFor<ActorType>` values return `void` and preserve required,
-optional, and zero-argument operation signatures. No non-null assertion is needed:
+inside actor methods and for inherited application operations. A scheduled operation
+call returns a `ReminderHandle`, through `ScheduledOperationsFor<ActorType>`, and a
+transmitted one returns `void`, through `TransmittedOperationsFor<ActorType>`. Both
+preserve required, optional, and zero-argument operation signatures. No non-null
+assertion is needed:
 
 ```typescript
 class ChatRun extends Actor {
@@ -415,7 +550,49 @@ runtime scheduling and transmission behavior are unchanged.
 Every manager below is available as a property on `SolidObjectsRuntime`; the
 class and result types are also exported for integration typing.
 
+- `runtime.findBy({ requestId })` and `reference.findBy({ idempotencyKey })`
+  rebuild a `MessageReference` for work whose reference a caller lost. A request
+  id is unique across the table, so the runtime answers it; an idempotency key
+  is unique per actor, so a reference supplies that scope.
+  `runtime.findBy({ reference, idempotencyKey })` is the explicit form. Naming
+  neither key, naming both, or naming an idempotency key without a reference
+  throws a `TypeError`. An absent row, an unregistered actor, and a caller the
+  policy refuses all return `undefined`.
+- An actor remembers the idempotency keys of its own last
+  `retainedIdempotencyKeys` finished turns, so `reference.findBy({
+idempotencyKey })` throws `MessagePruned` for a key the actor remembers and
+  whose message retention removed, and returns `undefined` for a key no caller
+  ever sent. An actor remembers the operation and original arguments beside each key, so the pruned
+  answer runs the same hook against the same operation and arguments a lookup of the
+  surviving row would, and a caller the policy refuses reads `undefined` for
+  both. `runtime.findBy({ requestId })` returns
+  `undefined` in both cases, because the runtime generates a request id and no
+  actor remembers one.
+- `retainedIdempotencyKeysBytes` bounds the serialized memory as well, because
+  an idempotency key has no length limit and the memory outlives the message
+  row. An actor drops its oldest keys until the list fits, so a key long enough
+  to fill the limit by itself is never remembered and its lookup answers
+  `undefined` rather than throwing.
+- Remembered arguments count toward the serialized memory limit and remain until
+  the entry is evicted or the instance is removed. Entries from older versions
+  that lack arguments return absence after pruning because their original
+  authorization cannot be reproduced.
+- `messageReference.outcome()` returns an `Outcome`: the status, the result, an
+  `ErrorRecord` for a dead message, a `RejectionRecord` for a rejected one, and
+  the attempt count.
 - `runtime.deadLetters` / `DeadLetterManager`: `all()` and idempotent `retry()`.
+- `runtime.deadLetters.effects` and `runtime.deadLetters.broadcasts`: a
+  `DeadLetterScope` for one `DeadLetterKind`. `all()` lists its dead rows as
+  `DeadRow` values, `retry(id)` returns one to pending, and `redrive(options)`
+  moves a whole scope. `RedriveOptions` names `actorType`, `failedAfter`, and
+  `limit`, which become the `RedriveFilters` the task records. `UnknownDeadRow`
+  reports an id that does not exist.
+- `runtime.redrives` / `RedriveManager`: `find(id)`, `all({ status })`,
+  `cancel(id)`, and `advance()`, which moves one bounded batch. A `RedriveTask`
+  carries its id, kind, filters, `RedriveStatus`, `moved`, `remaining`,
+  `startedAt`, `finishedAt`, and its own `cancel()`. `RedriveScheduler` is the
+  component that advances tasks inside `runtime.run()`. `UnknownRedrive` and
+  `RedriveNotStarted` report a missing task and a task that could not open.
 - `runtime.reminders` / `ReminderManager`: cursor-paginated `all()` and
   idempotent paused-alarm `resume()`.
 - `runtime.processes` / `ProcessManager`: immutable role `all()` and stale-owner
@@ -529,13 +706,32 @@ wait. `WakeUpWatch.wait()` returns `true` for a notification and `false` for a
 timeout or cancellation. A legacy `void` result remains accepted and preserves
 the fast polling cadence.
 
+`configuration.wakeUp` takes a name or an adapter. `WakeUpSetting` is that
+union, `WakeUpName` is one of `WAKE_UP_NAMES`, and the default is
+`"automatic"`. Selection prefers `SOLID_OBJECTS_REDIS_URL`, then PostgreSQL
+notifications, then polling. An unknown name throws rather than polls.
+
+`selectWakeUp(options)` runs that choice and returns a `SelectedWakeUp`, which
+pairs the adapter with a `WakeUpCapability`. `WakeUpSelectionOptions` names the
+inputs: the setting, the database, the idle polling interval, a logger, and an
+optional Redis URL and probe timeout. `WakeUpCapability` reports
+`WakeUpAdapterName`, whether the adapter crosses processes, the measured floor
+in milliseconds, and the reason. `runtime.wakeUpAdapter()` and
+`runtime.wakeUpCapability()` return the choice, which is made once per runtime.
+
+An adapter may declare its own `defaultCapability`, which is what it reports
+about itself. Selection keeps what a configured adapter declares rather than
+assume, and `runtime.wakeUpCapability()` reports what selection installed. `NotificationWakeUpAdapter` adds
+`channelFor(role)`, which a database that offers a notification channel
+provides through `database.wakeUp(options)`.
+
 ### Errors
 
 The root exports `SolidObjectsError` and its supported subclasses:
 
 - policy and caller outcomes: `Unauthorized`, `Rejected`, `ActorDestroyed`,
-  `SyncEnqueueTimeout`, `SyncTimeout`, `SyncInsideTransaction`, and
-  `MessageFailed`;
+  `SyncEnqueueTimeout`, `SyncTimeout`, `SyncInsideTransaction`,
+  `MessagePruned`, and `MessageFailed`;
 - admission and payload failures: `MailboxFull`, `InvalidPayload`,
   `PayloadTooLarge`, `IdempotencyConflict`, `InvalidPayloadBroadcast`, and
   `UnknownPayloadBroadcast`;

package/docs/cloudflare.md

@@ -124,6 +124,7 @@ replay current committed projections, not every event missed while offline.
 | Actor-scoped dead letters and reminder administration              | Supported                              |
 | `commitAction`, shared application SQL transactions                | Unsupported                            |
 | Global repository, reconciliation, process controls, SQL dashboard | Unsupported                            |
+| `findBy({ requestId })` without a reference                        | Unsupported; name the actor            |
 | Process-local `runtime.realtime.connect()` / server `ref.live`     | Unsupported; use browser subscriptions |
 | SQL-to-Durable-Objects data migration                              | Not provided                           |
 

package/docs/configuration.md

@@ -49,19 +49,19 @@ as an instruction to divide the actor. See
 
 ## Runtime roles and supervision
 
-| Option                                      |                  Default | Contract                                                 |
-| ------------------------------------------- | -----------------------: | -------------------------------------------------------- |
-| `workerCount`                               |                      `1` | Non-negative actor workers.                              |
-| `effectWorkerCount`                         |                      `1` | Non-negative effect workers.                             |
-| `broadcastWorkerCount`                      |                      `1` | Non-negative broadcast workers when realtime is enabled. |
-| `reminderSchedulerCount`                    |                      `1` | Non-negative reminder schedulers.                        |
-| `processHeartbeatIntervalMilliseconds`      |                 `15_000` | Positive persisted heartbeat cadence.                    |
-| `processAliveThresholdMilliseconds`         |                 `60_000` | Positive age after which an owner is stale.              |
-| `shutdownTimeoutMilliseconds`               |                 `15_000` | Positive shared graceful-shutdown budget.                |
-| `supervisorRestartDelayMilliseconds`        |                    `100` | Positive initial failed-role replacement delay.          |
-| `supervisorMaximumRestartDelayMilliseconds` |                 `10_000` | Positive cap no smaller than the initial delay.          |
-| `wakeUp`                                    | `InProcessWakeUpAdapter` | Adapter implementing `watch`, `notify`, and `close`.     |
-| `logger`                                    |          console methods | Structured `debug`, `info`, `warn`, and `error` sink.    |
+| Option                                      |         Default | Contract                                                          |
+| ------------------------------------------- | --------------: | ----------------------------------------------------------------- |
+| `workerCount`                               |             `1` | Non-negative actor workers.                                       |
+| `effectWorkerCount`                         |             `1` | Non-negative effect workers.                                      |
+| `broadcastWorkerCount`                      |             `1` | Non-negative broadcast workers when realtime is enabled.          |
+| `reminderSchedulerCount`                    |             `1` | Non-negative reminder schedulers.                                 |
+| `processHeartbeatIntervalMilliseconds`      |        `15_000` | Positive persisted heartbeat cadence.                             |
+| `processAliveThresholdMilliseconds`         |        `60_000` | Positive age after which an owner is stale.                       |
+| `shutdownTimeoutMilliseconds`               |        `15_000` | Positive shared graceful-shutdown budget.                         |
+| `supervisorRestartDelayMilliseconds`        |           `100` | Positive initial failed-role replacement delay.                   |
+| `supervisorMaximumRestartDelayMilliseconds` |        `10_000` | Positive cap no smaller than the initial delay.                   |
+| `wakeUp`                                    |   `"automatic"` | A name or an adapter implementing `watch`, `notify`, and `close`. |
+| `logger`                                    | console methods | Structured `debug`, `info`, `warn`, and `error` sink.             |
 
 Counts may be zero, but the complete configuration must leave at least one
 runtime role enabled. Broadcast workers are started only when `broadcast` or
@@ -78,15 +78,20 @@ polling remains the correctness path.
 
 ## Retention and cleanup
 
-| Option                                   |     Default | Contract                                                         |
-| ---------------------------------------- | ----------: | ---------------------------------------------------------------- |
-| `retentionIntervalMilliseconds`          | `3_600_000` | Non-negative automatic history-pruning cadence; `0` disables it. |
-| `deadProcessCleanupIntervalMilliseconds` |    `60_000` | Non-negative stale-owner recovery cadence; `0` disables it.      |
-| `messageRetentionMilliseconds`           |     30 days | Positive default completed-message retention.                    |
-| `messageRetentionByActorType`            |        `{}` | Positive per-type message-retention overrides.                   |
-| `instanceRetentionByActorType`           |        `{}` | Positive per-type instance-expiration opt-ins.                   |
-| `processRetentionMilliseconds`           |      7 days | Positive stopped-process retention.                              |
-| `pruneBatchSize`                         |     `1_000` | Positive integer maximum rows rechecked per pruning transaction. |
+| Option                                   |           Default | Contract                                                         |
+| ---------------------------------------- | ----------------: | ---------------------------------------------------------------- |
+| `retentionIntervalMilliseconds`          |       `3_600_000` | Non-negative automatic history-pruning cadence; `0` disables it. |
+| `deadProcessCleanupIntervalMilliseconds` |          `60_000` | Non-negative stale-owner recovery cadence; `0` disables it.      |
+| `messageRetentionMilliseconds`           |           30 days | Positive default completed-message retention.                    |
+| `messageRetentionByActorType`            |              `{}` | Positive per-type message-retention overrides.                   |
+| `instanceRetentionByActorType`           |              `{}` | Positive per-type instance-expiration opt-ins.                   |
+| `processRetentionMilliseconds`           |            7 days | Positive stopped-process retention.                              |
+| `pruneBatchSize`                         |           `1_000` | Positive integer maximum rows rechecked per pruning transaction. |
+| `retainedIdempotencyKeys`                |              `64` | Positive integer idempotency keys an actor remembers.            |
+| `retainedIdempotencyKeysBytes`           |          `16_384` | Positive integer serialized size those keys must fit.            |
+| `redriveBatchSize`                       |             `100` | Positive integer rows a redrive moves per transaction.           |
+| `redriveBatchPauseMilliseconds`          |              `50` | Non-negative pause between redrive batches.                      |
+| `administrationIdentity`                 | `String(context)` | Names the operator recorded in an administration event.          |
 
 Automatic retention prunes messages and stopped process records. Actor
 instance expiration remains an explicit administration action even when a type
@@ -158,6 +163,31 @@ short capped backoff only when no synchronous deadline is active.
 | `applicationName`               |          `"solid-objects"` |
 | `onPoolError`                   | structured `console.error` |
 
+`wakeUp` takes a name or an adapter, as `config.cache_store` does in Rails.
+`"automatic"` is the default. It prefers `SOLID_OBJECTS_REDIS_URL`, then
+PostgreSQL notifications, then polling. `"in_process"` opts out, `"postgresql"`
+and `"redis"` force one, and an unknown name throws rather than polls quietly.
+
+A requested adapter that the environment cannot provide polls instead and says
+so. `"postgresql"` on a database with no notification channel, `"redis"` without
+`SOLID_OBJECTS_REDIS_URL`, and a Redis URL without the `redis` package each log
+`solid_objects.wake_up.unavailable` once and report the reason in the
+capability. Only a name that does not exist is refused, because a typo cannot be
+honoured at all.
+
+Selection runs once per runtime, on first use. `runtime.wakeUpCapability()`
+reports what it chose, whether that choice crosses processes, its measured
+floor in milliseconds, and why. The doctor reports the same record, and the
+polling-only warning fires on what was installed rather than on whether a
+setting was set. An adapter that declares its own `defaultCapability` keeps it, so a
+configured `InProcessWakeUpAdapter` still warns.
+
+Automatic selection proves the PostgreSQL path before it chooses it. It listens
+on a probe channel, sends one `NOTIFY` from a second connection, and waits up to
+two seconds for it to arrive. A probe that does not deliver falls back to
+polling and warns once, because `LISTEN` does not survive a transaction pooler
+such as PgBouncer.
+
 `database.wakeUp(options)` creates a dedicated notification adapter using the
 same connection string. Its options are `channelPrefix = "solid_objects"`,
 `applicationName = "solid-objects-wake-up"`, and `onListenerError`. Use a

package/docs/correctness.md

@@ -5,6 +5,14 @@
 - Delivery is ordered per actor identity and at least once.
 - Different identities may execute concurrently.
 - Sequence allocation and durable enqueue are one transaction.
+- An actor reads its own schedule. A read applies the intents staged so far in
+  the turn, so it agrees with what the commit will write rather than with what
+  the turn began with.
+- A reminder can be cancelled. A cancellation commits with the state change that
+  decided it, and applies in the order the turn called it. A cancellation cannot
+  recall an occurrence the scheduler already turned into a message. It does
+  pre-empt one the scheduler claimed but has not yet enqueued, and the scheduler
+  treats that as ordinary work rather than a failure.
 - Concurrent callers that create the same actor produce one instance row and
   distinct sequences. The mailbox locks that row by its primary key, so MySQL
   does not upgrade a shared lock and the enqueue does not deadlock.

package/docs/dashboard.md

@@ -141,6 +141,15 @@ Dead-letter retry calls `runtime.deadLetters.retry()`. It keeps the durable
 idempotency and the actor-operation validation of that method. If the runtime
 refuses a retry, the detail page shows it with status 422.
 
+Dead effects and broadcasts have the same API, which the dashboard does not yet
+surface. `runtime.deadLetters.effects` and `runtime.deadLetters.broadcasts` read
+and retry their own kind, and `redrive` moves a whole scope as a durable task.
+See [Operations](operations.md) for both.
+
+Every retry and every redrive transition writes one row to
+`solid_objects_administration_events`, holding the action, the kind, the
+subject, and the identity that asked for it.
+
 `HEAD /` performs only a schema reachability query and creates no CSRF session
 state. Use it for liveness checks instead of polling the full dashboard.
 

package/docs/fit.md

@@ -43,7 +43,9 @@ history, and no actor-state migration contract.
   limiter, see [Solid Objects Pro](https://solidobjects.pro/): grouped commits
   coalesce concurrent writes into one insert, and ephemeral operations keep
   loss-tolerant calls out of the durable journal. It ships for the Rails gem
-  today, and the Node build is in development.
+  today, and the Node build is in development. Distributed per-actor rate
+  limits, global admission control, and cache-capacity eviction are answered
+  there rather than in this runtime, and are not open roadmap items here.
 - Durable workflow replay across named steps is more important than a mutable
   object with ordered operations. The runtime redelivers an ordered message and
   retries it. It does not replay a function from a step log.

package/docs/operations.md

@@ -14,16 +14,25 @@ work and wake-up notifications reset the role to the fast interval. Actor
 workers clamp the ceiling to `leaseRenewalIntervalMilliseconds` while they may
 hold cached activations.
 
-The default generation-based wake-up adapter interrupts waits for new actor
+The runtime selects a wake-up adapter on first use. It prefers
+`SOLID_OBJECTS_REDIS_URL`, then PostgreSQL notifications, then polling.
+The generation-based in-process adapter interrupts waits for new actor
 messages, effects, reminders, and broadcasts in the same Node process. It does
-not cross a process boundary. When live processes share the database without a
-configured adapter, the runtime logs
-`solid_objects.polling_only_cross_process_wake_up` once. Use PostgreSQL
-notifications or optional Redis Pub/Sub when separate processes need prompt
-delivery. Without one of them, newly committed work can wait for the current
-idle polling interval. The runtime isolates notification errors and logs them by
+not cross a process boundary. When live processes share the database and the
+installed adapter does not cross processes, the runtime logs
+`solid_objects.polling_only_cross_process_wake_up` once. Without a
+cross-process adapter, newly committed work can wait for the current idle
+polling interval. The runtime isolates notification errors and logs them by
 role and error class. The committed work does not fail.
 
+`runtime.wakeUpCapability()` and the `wakeUp` doctor check report the adapter
+that is installed, whether it crosses processes, its measured floor, and why it
+was chosen. On PostgreSQL, selection first proves the path: it listens on a
+probe channel, notifies it from a second connection, and waits for the
+notification. A probe that does not arrive logs
+`solid_objects.wake_up.pooled_session` once and falls back to polling, because
+`LISTEN` does not survive a transaction pooler such as PgBouncer.
+
 The warning excludes process rows with the current hostname and host process ID.
 It can therefore appear during a rolling deployment or restart overlap when an
 older and newer process briefly share the same database. A process that stopped
@@ -218,3 +227,91 @@ be resumed.
 Scheduling an existing actor operation for a different time emits
 `solid_objects.reminder.replaced` after the fenced actor commit. Scheduling it
 for the same time emits nothing.
+
+## Dead letters, retry, and redrive
+
+A message that exhausts its attempts becomes a dead letter. An effect or a
+broadcast that exhausts its attempts stays in its own table with
+`status = 'dead'`. All three are read and retried through one receiver, which
+carries the kind:
+
+```ts
+await runtime.deadLetters.all({ authorizationContext })
+await runtime.deadLetters.retry(deadLetterId, { authorizationContext })
+
+await runtime.deadLetters.effects.all({ authorizationContext })
+await runtime.deadLetters.effects.retry(effectId, { authorizationContext })
+await runtime.deadLetters.broadcasts.retry(broadcastId, { authorizationContext })
+```
+
+An effect or broadcast retry returns the row to pending with a zero attempt
+count, no claim, and immediate availability, and keeps its id, so a handler that
+deduplicates on the effect id still sees the same key. An effect is
+at-least-once by contract, so a retried effect can run twice.
+
+Retry acts only on a dead row. A row that is pending, processing, or completed
+comes back unchanged, so pressing a button twice cannot double-enqueue and
+cannot take a row away from a worker that holds it.
+
+An incident produces dead rows in the hundreds, so a scope also answers
+`redrive`:
+
+```ts
+const task = await runtime.deadLetters.effects.redrive({
+  actorType: "payments",
+  failedAfter: new Date(Date.now() - 6 * 60 * 60 * 1000),
+  limit: 5_000,
+  authorizationContext,
+})
+
+await task.cancel({ authorizationContext })
+```
+
+`redrive` returns at once. The task is durable, and `runtime.run()` advances one
+bounded batch per pass, so a redrive of thousands of rows never holds a
+transaction longer than one batch. `redriveBatchSize` defaults to 100 and
+`redriveBatchPauseMilliseconds` to 50.
+
+A redrive moves the rows that were already dead when it started. A row that
+fails again lands back in the same scope, and without that bound a task whose
+handler is still broken would move it forever.
+
+A redrive is idempotent over its scope and its filters. Starting the same one
+while it runs returns the running task rather than a second one, which a
+dashboard button an operator can press twice needs. A different scope or a
+different filter starts its own task, and the same scope can be redriven again
+once the first task finishes.
+
+Read tasks back with `runtime.redrives`:
+
+```ts
+await runtime.redrives.find(task.id, { authorizationContext })
+await runtime.redrives.all({ status: "running", authorizationContext })
+```
+
+A running task reports what is left to move rather than a stored estimate,
+because rows die and are retried while it runs.
+
+Retry, redrive, and cancel each go through `authorizeAdministration` under their
+own resource name: `dead_letters`, `effect_dead_letters`,
+`broadcast_dead_letters`, and `redrives`. Every retry and every task transition
+writes one row to `solid_objects_administration_events`, holding the action, the
+kind, the subject, the identity, and when it happened. The identity comes from
+`administrationIdentity`, which receives the authorization context the caller
+passed and defaults to its `String` form.
+
+An event records an authorized press, not a state transition. Pressing retry
+twice writes two rows, because an operator did two things and a log that shows
+one cannot answer who pressed what. The row the event names carries the outcome.
+A refused caller writes nothing, and a retry that raises after the lookup writes
+nothing, because the event shares the transaction with the work. The redrive
+transitions are different: `redrive.start`, `redrive.finish`, and
+`redrive.cancel` are written only when the task actually changes.
+
+The Durable Objects engine keeps its own message and outbox tables inside each
+object, so these scopes and redrive cover the SQL backends. `deadLetters` there
+reports its own dead rows as it did before.
+
+Automatic redrive on a schedule is deliberately absent. A dead row means a
+person decided something, and these APIs give that person an alternative to an
+`UPDATE` against a runtime table.