durability 4.2.0 → 5.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +340 -12
- package/dist/dev.d.ts +22 -0
- package/dist/dev.js +149 -0
- package/dist/durability-alarms.d.ts +16 -3
- package/dist/durability-alarms.js +21 -7
- package/dist/durability.d.ts +31 -6
- package/dist/durability.js +42 -16
- package/dist/errors.d.ts +31 -0
- package/dist/errors.js +45 -0
- package/dist/experimental.d.ts +5 -0
- package/dist/experimental.js +3 -0
- package/dist/fanout.d.ts +6 -0
- package/dist/fanout.js +20 -9
- package/dist/index.d.ts +4 -2
- package/dist/index.js +3 -2
- package/dist/lifecycle.d.ts +6 -1
- package/dist/lifecycle.js +3 -2
- package/dist/log.js +61 -16
- package/dist/memoize.d.ts +34 -0
- package/dist/memoize.js +136 -0
- package/dist/migrations.d.ts +64 -2
- package/dist/migrations.js +179 -34
- package/dist/outbox.d.ts +13 -0
- package/dist/outbox.js +78 -30
- package/dist/scheduler.d.ts +9 -0
- package/dist/scheduler.js +53 -11
- package/dist/semaphore-leases.js +15 -9
- package/dist/semaphore-queue-store.d.ts +12 -2
- package/dist/semaphore-queue-store.js +32 -5
- package/dist/semaphore-queue.js +109 -43
- package/dist/semaphore-server.js +7 -9
- package/dist/socket.js +19 -10
- package/dist/storage.d.ts +1 -0
- package/dist/storage.js +1 -1
- package/dist/topology-client.d.ts +40 -0
- package/dist/topology-client.js +180 -0
- package/dist/topology-snapshot.d.ts +70 -0
- package/dist/topology-snapshot.js +48 -0
- package/dist/topology.d.ts +94 -0
- package/dist/topology.js +468 -0
- package/dist/tracked-source.d.ts +2 -0
- package/dist/tracker.js +2 -0
- package/package.json +3 -2
package/README.md
CHANGED
|
@@ -2,18 +2,59 @@
|
|
|
2
2
|
|
|
3
3
|
Composable capabilities for application-owned Cloudflare Durable Objects:
|
|
4
4
|
|
|
5
|
-
- **`Durability`** —
|
|
5
|
+
- **`Durability`** — at-least-once operations with stable idempotency keys, retries, and result lookup.
|
|
6
6
|
- **`DurabilityAlarms`** — named logical alarms with retries and stable idempotency keys.
|
|
7
7
|
- **`DurabilityFanout`** — durable batch acceptance and independent delivery to static consumer or storage targets.
|
|
8
8
|
- **`DurabilityOutbox`** _(experimental, `durability/experimental`)_ — keyed latest-state publication: stage the newest value for a key with the state it describes, and it is republished until acknowledged.
|
|
9
9
|
- **`DurabilitySemaphore`** _(experimental, `durability/experimental/semaphore`)_ — one concurrency limit shared by many client Durable Objects, allocated in arrival-ordered rounds with cumulative claim/release accounting.
|
|
10
|
+
- **`DurabilityTopology`** _(experimental, `durability/experimental`)_ — membership of shard groups held by one coordinator, published to a read store and pushed to every member as the latest value of each state channel, with a read-only cached client.
|
|
10
11
|
- **`DurabilityRouting`** — a DO-side load observer and client-side shard selector (`durability/routing`), supporting location hints and caller transforms.
|
|
11
12
|
- **`DurabilitySocket`** — typed WebSocket RPC and replayable event push from a Durable Object to browsers (`durability/socket`), with a dependency-free browser client (`durability/client`).
|
|
12
13
|
- **`DurabilityTracker`** _(experimental, `durability/experimental`)_ — lists work across capabilities and custom sources, as RPC handles that can be waited on or as plain records.
|
|
13
14
|
|
|
14
15
|
Durable capabilities own their storage and migrations. Routing observations are in-memory and advisory. When an object composes durable capabilities, one `DurabilityScheduler` shares its physical alarm and concurrency pool. No queue factory, generated DO class, or special consumer superclass is required.
|
|
15
16
|
|
|
16
|
-
Durability is not an event-sourcing, replay-log, or workflow-orchestration engine. Operation methods return `Promise<void>` after registration commits; they do not wait for the handler result. Failed work is retried from later alarms. Only SQLite-backed Durable Object classes are supported.
|
|
17
|
+
Durability is not an event-sourcing, replay-log, or workflow-orchestration engine: [memoized](#memoize) results only last while one operation or alarm occurrence is retried, and are deleted when it settles. Operation methods return `Promise<void>` after registration commits; they do not wait for the handler result. Failed work is retried from later alarms. Only SQLite-backed Durable Object classes are supported.
|
|
18
|
+
|
|
19
|
+
## Delivery and safety contract
|
|
20
|
+
|
|
21
|
+
Durability executes work **at least once** with a stable idempotency key per operation or named alarm. "Effectively once" is a business outcome you earn by making the receiver deduplicate that key or reconcile an uncertain prior outcome; it is never an unconditional platform guarantee. Read this before copying an example that mutates an external system.
|
|
22
|
+
|
|
23
|
+
| Concern | Guarantee |
|
|
24
|
+
| -------------------------- | ---------------------------------------------------------------------------- |
|
|
25
|
+
| Registration / scheduling | Exactly once — the durable record commits atomically or not at all. |
|
|
26
|
+
| Local completion record | Deduplicated by the stable key, so retries never double-record a result. |
|
|
27
|
+
| Handler delivery | At least once — a handler may run again after a crash, timeout, or eviction. |
|
|
28
|
+
| External side effect | At least once — the library cannot close a cross-system transaction. |
|
|
29
|
+
| "Effectively once" outcome | Your responsibility — requires an idempotent or reconciling receiver. |
|
|
30
|
+
|
|
31
|
+
The unavoidable window: a handler asks an external system to commit an effect, the system commits it, then the acknowledgement is lost or the object stops before the local completion record commits. Durability retries the same stable key. Pass that key (`DurableCall.id`, or an alarm handler's `idempotencyKey`) to the external system so it can collapse the duplicate. An HTTP success — or failure — is not proof of the committed outcome after a lost acknowledgement.
|
|
32
|
+
|
|
33
|
+
The stable key is the call `id` you choose (or an alarm name). Namespace it per logical operation (`resize:${imageId}`), keep it stable across retries, and never reuse one key for different payloads: an ID stays bound to its operation and payload for the life of the record. A receiver-side idempotency table turns at-least-once delivery into an effectively-once effect by keying committed effects on it:
|
|
34
|
+
|
|
35
|
+
```ts
|
|
36
|
+
// In the receiving Durable Object, keyed on the caller's stable id.
|
|
37
|
+
async chargeOnce(key: string, amountCents: number) {
|
|
38
|
+
const prior = this.ctx.storage.sql
|
|
39
|
+
.exec<{ result: string }>('SELECT result FROM charges WHERE key = ?', key)
|
|
40
|
+
.toArray();
|
|
41
|
+
if (prior.length > 0) {
|
|
42
|
+
return JSON.parse(prior[0]!.result); // replay: return the first committed outcome
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
const result = await this.gateway.charge(amountCents);
|
|
46
|
+
this.ctx.storage.sql.exec(
|
|
47
|
+
'INSERT INTO charges (key, result) VALUES (?, ?)',
|
|
48
|
+
key,
|
|
49
|
+
JSON.stringify(result)
|
|
50
|
+
);
|
|
51
|
+
return result;
|
|
52
|
+
}
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
The sketch assumes one call per key at a time; the gateway `await` lets other requests interleave, so concurrent calls for the same key need a pending row or the gateway's own idempotency key.
|
|
56
|
+
|
|
57
|
+
When a receiver can neither deduplicate the key nor reconcile the prior outcome, do not retry the effect automatically: compensate (a saga) or require a human decision instead.
|
|
17
58
|
|
|
18
59
|
## Migrating to v3
|
|
19
60
|
|
|
@@ -127,7 +168,7 @@ Scheduling a name again replaces its pending occurrence. If that name is already
|
|
|
127
168
|
|
|
128
169
|
Named alarms inside the scheduler's `alarmMinDelayMs` window start from an in-memory timer at their requested time. The option accepts 1 to 15 seconds and defaults to 15 seconds; the scheduler never sets the physical alarm sooner than that delay; the physical alarm remains as a durable fallback for restarts. If concurrency capacity is full when the timer fires, execution waits in the shared FIFO queue. The scheduling method resolves after persistence and never waits for the handler to finish.
|
|
129
170
|
|
|
130
|
-
Each occurrence receives a new internal generation. Its `idempotencyKey` stays stable across retries while `attempt`
|
|
171
|
+
Each occurrence receives a new internal generation. Its `idempotencyKey` stays stable across retries, while `attempt` records which attempt the handler is running. The ordinal advances each time an attempt is claimed and is retry context, not a unique per-handler-entry token (see [Identity and attempt numbers](#identity-and-attempt-numbers)). A successful handler removes only the occurrence it executed, so it cannot delete a replacement scheduled while it was running.
|
|
131
172
|
|
|
132
173
|
A timeout aborts the handler's `signal` and is terminal by default because the external outcome is unknown. Set `retryTimeouts: true` only when the handler's side effects use the idempotency key or reconcile their outcome before retrying. Named alarms remain at-least-once across eviction or restart.
|
|
133
174
|
|
|
@@ -166,6 +207,36 @@ export class ImageJobs extends DurableObject<Env> {
|
|
|
166
207
|
|
|
167
208
|
A helper configured with `context` instead of `scheduler` creates a private scheduler and accepts the scheduler options (`alarmConcurrency`, `alarmMinDelayMs`, `alarmHandoffMs`) in the same object. Two helpers on the same object must share one scheduler; otherwise each would reconcile the physical alarm against only its own records.
|
|
168
209
|
|
|
210
|
+
## Memoize
|
|
211
|
+
|
|
212
|
+
A retried attempt runs the whole handler again. Wrap work that must not repeat in `memoize(key, callback)`, which operation handlers, named alarm handlers, and [fanout messages](#independent-delivery) all receive. Once a key's callback succeeds, its result is persisted and every later call with that key returns it instead of running the callback:
|
|
213
|
+
|
|
214
|
+
```ts
|
|
215
|
+
const deliver = async ({ payload, memoize }: DurableCall<Batch>) => {
|
|
216
|
+
const targets = await memoize('targets', () => lookUpTargets(payload));
|
|
217
|
+
const results = await Promise.allSettled(
|
|
218
|
+
targets.map((target) =>
|
|
219
|
+
memoize(`send:${target.id}`, () => send(target, payload.events))
|
|
220
|
+
)
|
|
221
|
+
);
|
|
222
|
+
const failed = results.filter((result) => result.status === 'rejected');
|
|
223
|
+
if (failed.length > 0) {
|
|
224
|
+
throw new AggregateError(failed.map((result) => result.reason));
|
|
225
|
+
}
|
|
226
|
+
};
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
Here a retry sends only to the targets that failed, and does not look them up again.
|
|
230
|
+
|
|
231
|
+
- **Keys.** A key is a non-empty string that identifies one piece of work within the operation or alarm occurrence. Calling a key again returns its result, whether the call comes from a later attempt, later in the same attempt, or from inside another key's callback. A call for a key whose callback is still running rejects instead of waiting, including one made from that callback, which would otherwise wait on itself; await the first call instead.
|
|
232
|
+
- **Results.** A callback that throws persists nothing; the next call with that key runs it again. Results must be JSON-serializable, and `memoize` returns the persisted form even on the first run, so a replay sees the same value. A result that cannot be serialized fails the work with `DurableMemoSerializationError` and no retry.
|
|
233
|
+
- **Ownership.** Only the attempt that owns the record can run or persist a callback. Once its alarm is rescheduled or its record purged, a `memoize` call rejects without running, and one already running rejects instead of writing.
|
|
234
|
+
- **Limits.** One operation, alarm occurrence, or fanout delivery may memoize at most `maxMemosPerRecord` (1,000) keys, and each serialized result may be at most `maxMemoBytes` (1 MiB) of UTF-8. A call that would exceed either fails the work with `DurableMemoLimitError` and no retry; a key that would exceed the count is rejected before its callback runs.
|
|
235
|
+
- **Cleanup.** Memoized results are deleted in the transaction that settles the operation (completed or failed), the alarm occurrence, or the fanout delivery (acked or dead-lettered), when a named alarm is rescheduled, and by `purgeBefore`. Nothing is kept once the work is done.
|
|
236
|
+
- **Tracking.** Pending operations and alarms list their memoized keys as `state.memoized` in [`DurabilityTracker`](#tracking-work-experimental) records, in the order they were saved. Results themselves are not exposed.
|
|
237
|
+
|
|
238
|
+
Memoizing does not change delivery semantics. A process can stop after a callback's side effect succeeds but before its result commits, so the callback can run again; keep its side effects idempotent.
|
|
239
|
+
|
|
169
240
|
## Storage
|
|
170
241
|
|
|
171
242
|
Durability stores its records in the object's SQLite database. Configure the Durable Object class with `new_sqlite_classes`:
|
|
@@ -176,7 +247,7 @@ Durability stores its records in the object's SQLite database. Configure the Dur
|
|
|
176
247
|
}
|
|
177
248
|
```
|
|
178
249
|
|
|
179
|
-
`Durability` creates and migrates `durability_calls`; `DurabilityAlarms` does the same for `durability_alarms`. Both histories are tracked in the shared `durability_migrations` table and only the constructed helpers' tables exist.
|
|
250
|
+
`Durability` creates and migrates `durability_calls`; `DurabilityAlarms` does the same for `durability_alarms`. Both also create `durability_memos`, which they share. All histories are tracked in the shared `durability_migrations` table and only the constructed helpers' tables exist. Reverting either helper's schema leaves `durability_memos` in place for the other.
|
|
180
251
|
|
|
181
252
|
### Explicit migrations
|
|
182
253
|
|
|
@@ -194,6 +265,32 @@ Passing a name moves that capability's schema to exactly that version. Each batc
|
|
|
194
265
|
|
|
195
266
|
Every history row stores its migration's `down` script. If a Durable Object later runs an older `durability` version, that version reverts the migrations it does not recognize using the stored scripts, so downgrades are safe without code changes. To guard against deploying a very old version by mistake, at most two unknown migrations are reverted; beyond that, construction throws. The `durability_migrations` table is dropped once no durability history remains.
|
|
196
267
|
|
|
268
|
+
### Application schemas
|
|
269
|
+
|
|
270
|
+
`migrate` gives an application's own tables the same guarantees. Its history lives in a separate table (`schema_migrations` unless `table` is set), so it never mixes with durability's:
|
|
271
|
+
|
|
272
|
+
```ts
|
|
273
|
+
import { DurableObject } from 'cloudflare:workers';
|
|
274
|
+
import { migrate, type DurableMigrations } from 'durability';
|
|
275
|
+
|
|
276
|
+
const migrations: DurableMigrations = [
|
|
277
|
+
{
|
|
278
|
+
name: '0001_create_jobs',
|
|
279
|
+
up: 'CREATE TABLE jobs (id TEXT PRIMARY KEY);',
|
|
280
|
+
down: 'DROP TABLE jobs;',
|
|
281
|
+
},
|
|
282
|
+
];
|
|
283
|
+
|
|
284
|
+
export class Jobs extends DurableObject {
|
|
285
|
+
constructor(ctx: DurableObjectState, env: Env) {
|
|
286
|
+
super(ctx, env);
|
|
287
|
+
migrate(ctx.storage, migrations);
|
|
288
|
+
}
|
|
289
|
+
}
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
Pass `target` to move to a named migration, or `null` to revert every migration and drop the history table.
|
|
293
|
+
|
|
197
294
|
## Retries, timeouts, and terminal failures
|
|
198
295
|
|
|
199
296
|
Attempts use exponential backoff with equal jitter, stop after five attempts, and time out after five minutes by default. The retry `delay` function fully controls scheduling and can be overridden globally or per method on either helper:
|
|
@@ -225,14 +322,48 @@ Each attempt receives its own `AbortSignal`. A timed-out operation follows the n
|
|
|
225
322
|
|
|
226
323
|
Throw `NonRetryableError` to move work directly to `failed` without another attempt. Errors named `NonRetryableError` from other packages, such as `cloudflare:workflows`, are recognized too.
|
|
227
324
|
|
|
325
|
+
## Identity and attempt numbers
|
|
326
|
+
|
|
327
|
+
A handler receives several identifiers; only some are stable:
|
|
328
|
+
|
|
329
|
+
- **Operation `id`** (operations) and **`idempotencyKey`** (named alarms) are the logical identity. They are stable across every attempt and retry, so they are the only safe deduplication keys for downstream services.
|
|
330
|
+
- **`operation` / `name`** is the handler the work was registered under.
|
|
331
|
+
- **`attempt`** is the one-based ordinal of the current attempt, claimed in a durable write before the handler runs. It is retry/logging context, not an execution identity.
|
|
332
|
+
- **Named-alarm generation** is an internal token minted per scheduled occurrence; it backs `idempotencyKey` and is not exposed directly.
|
|
333
|
+
|
|
334
|
+
`attempt` is operational context for retry policies and logging, not a record of what the handler did. It is not strictly unique per handler entry: an entry whose claim write never commits can see the same number again on recovery. Cloudflare's output gates hold that entry's outbound fetches and RPCs until its storage writes are confirmed and discard them if the write is lost, so a replayed ordinal never corresponds to an escaped external effect — but it is still not a reliable "what already happened" signal.
|
|
335
|
+
|
|
336
|
+
That is why `attempt === 1` is not a durable "first run" gate. Attempt 1 can fail or time out before or after its effect, and retries carry a higher ordinal, so gating on the number either skips the effect forever once it increments or cannot tell whether the effect already ran. Gate once-only work on durable state keyed by the stable `id` instead, preferably in the authoritative receiver:
|
|
337
|
+
|
|
338
|
+
```ts
|
|
339
|
+
// Wrong: attempt is not a record of whether the effect ran. If attempt 1
|
|
340
|
+
// fails after charging, the retry (attempt 2) skips the charge; if it fails
|
|
341
|
+
// before, nothing distinguishes that from success.
|
|
342
|
+
if (attempt === 1) {
|
|
343
|
+
await chargeCard(payload);
|
|
344
|
+
}
|
|
345
|
+
|
|
346
|
+
// Right: the receiver dedupes on the stable id, independent of attempt.
|
|
347
|
+
await this.env.BILLING.chargeOnce(id, payload);
|
|
348
|
+
```
|
|
349
|
+
|
|
228
350
|
## Delivery semantics and concurrency
|
|
229
351
|
|
|
230
|
-
Work is delivered at least once. A process can stop after a side effect succeeds but before its completion record commits, so handlers should be idempotent and pass the call ID or alarm `idempotencyKey` to external services.
|
|
352
|
+
Work is delivered at least once, as the [delivery and safety contract](#delivery-and-safety-contract) describes. A process can stop after a side effect succeeds but before its completion record commits, so handlers should be idempotent and pass the call ID or alarm `idempotencyKey` to external services.
|
|
231
353
|
|
|
232
354
|
`alarmConcurrency` defaults to 10 and is one FIFO limit shared by eager operation handlers, timer-driven work, and physical-alarm work attached to the same scheduler. Newly registered operations start eagerly only when a permit is immediately available. Work due before the physical-alarm minimum enters the same queue when its in-memory timer fires.
|
|
233
355
|
|
|
234
356
|
Completion records are retained indefinitely so IDs remain deduplicated. The scheduler owns the Durable Object's alarm; compose unrelated scheduled work as named alarms instead of replacing the physical alarm.
|
|
235
357
|
|
|
358
|
+
## Load envelope
|
|
359
|
+
|
|
360
|
+
`pnpm bench:load` runs an opt-in Workerd characterization suite (`bench/`, excluded from `pnpm test`) that drains large backlogs, a zero-delay retry storm, and a mixed operation plus named-alarm workload, emitting one machine-readable `BENCH {...}` JSON line per scenario. The numbers below come from that suite on `compatibility_date` `2026-07-07` with the default `alarmConcurrency` of 10; treat them as a tested envelope on one machine, not a platform capacity promise.
|
|
361
|
+
|
|
362
|
+
- A live object drains its backlog eagerly and through in-memory timers, so 100, 1,000, and 10,000 simultaneously due operations all complete with no lost calls, no extra terminal transitions, and nothing left pending. The physical alarm is a post-eviction fallback, not the live drain path.
|
|
363
|
+
- Each retained completed row costs roughly 200–260 bytes plus the serialized payload and result; 10,000 empty-payload operations occupy about 2 MB of `databaseSize`. Rows are never reclaimed automatically, so storage grows linearly until `purgeBefore` runs.
|
|
364
|
+
- `alarmConcurrency` is one shared budget: a mixed backlog of overlapping operations and a named alarm never exceeds 10 concurrent handlers in total, rather than 10 per category.
|
|
365
|
+
- A zero-delay retry storm performs exactly one attempt per retry with no extra work and settles every operation.
|
|
366
|
+
|
|
236
367
|
## Destructive retention purge
|
|
237
368
|
|
|
238
369
|
`purgeBefore(timestamp)` on either helper deletes that helper's records created strictly before `timestamp`, regardless of status, and returns the count. Records created exactly at the cutoff are retained.
|
|
@@ -258,16 +389,66 @@ const durability = new Durability({
|
|
|
258
389
|
|
|
259
390
|
The hook is best-effort and non-blocking, is not a durable journal, and may duplicate or drop events across crashes. Hook failures are reported to `console.error` and never change queue state. Returned promises are attached to `waitUntil` when the context exposes it.
|
|
260
391
|
|
|
261
|
-
|
|
392
|
+
Events are built to be safe to forward to shared telemetry sinks: they carry structural fields (entity kind, operation or alarm name, idempotency `id`, `generation`, `attempt`, `durationMs`, `nextAttemptAt`) and a normalized `error` of `{ name, message }` with no stack, but **no payloads and no handler results by default**. The `id` is the raw idempotency key you chose, so hash or drop it before exporting to a metrics backend if it is high-cardinality or sensitive. Because an event is emitted per attempt, retry, and purge, treat these as counters and durations rather than an audit trail.
|
|
393
|
+
|
|
394
|
+
To include a `completed` settlement's handler `result` — for example to bridge results to a WebSocket or cache — opt in with `includeResults: true`. Enable it only when every attached hook is trusted with result content, since results may contain sensitive data:
|
|
395
|
+
|
|
396
|
+
```ts
|
|
397
|
+
const durability = new Durability({
|
|
398
|
+
context: this.ctx,
|
|
399
|
+
handlers,
|
|
400
|
+
includeResults: true,
|
|
401
|
+
onLifecycleEvent: (event) => recordDurabilityMetric(event),
|
|
402
|
+
});
|
|
403
|
+
```
|
|
404
|
+
|
|
405
|
+
Further hooks attach after construction with `lifecycle(hook)`, which returns a function that detaches it. `Durability` hooks are typed per operation: a `completed` settlement carries the handler's `result` only when `includeResults` is enabled, so the field is typed as optional.
|
|
262
406
|
|
|
263
407
|
```ts
|
|
264
408
|
const detach = durability.lifecycle((event) => {
|
|
265
|
-
if (
|
|
409
|
+
if (
|
|
410
|
+
event.type === 'attempt_settled' &&
|
|
411
|
+
event.outcome === 'completed' &&
|
|
412
|
+
event.result !== undefined
|
|
413
|
+
) {
|
|
266
414
|
publish(event.operation, event.result);
|
|
267
415
|
}
|
|
268
416
|
});
|
|
269
417
|
```
|
|
270
418
|
|
|
419
|
+
## Dev mode tracing
|
|
420
|
+
|
|
421
|
+
Define `__DURABILITY_DEV__` as `true` at build time to log every operation the package performs. Each entry is a structured `console.debug` object whose `event` starts with `durability.dev.`:
|
|
422
|
+
|
|
423
|
+
| `event` | Logged for |
|
|
424
|
+
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
425
|
+
| `durability.dev.call` | Each public method call on every helper, including the experimental outbox and semaphores, generated operation, alarm, and socket outbound methods, `memoize`, and fanout `ack`, `retry`, and `deadLetter`, with its arguments. |
|
|
426
|
+
| `durability.dev.call.return` / `.call.throw` | The call's result or serialized error, linked to the call by `call`. |
|
|
427
|
+
| `durability.dev.storage` | Each storage method: transactions and alarm reads and writes, including those inside `transaction`. |
|
|
428
|
+
| `durability.dev.storage.return` / `.storage.throw` | The storage call's result or serialized error, linked by `call`. |
|
|
429
|
+
| `durability.dev.sql` | Each SQL statement, whitespace-compacted, with its bindings. |
|
|
430
|
+
| `durability.dev.lifecycle` | Every lifecycle event, whether or not a hook is attached. |
|
|
431
|
+
| `durability.dev.scheduler.early_alarm` | An in-memory timer that runs due work before the physical alarm. |
|
|
432
|
+
| `durability.dev.socket.send` / `.socket.close` | Each frame sent to a browser socket, and each socket closed for a protocol error. |
|
|
433
|
+
|
|
434
|
+
Every entry carries `seq`, a process-wide counter that orders entries across interleaved async work, and `at`, the `Date.now()` reading when it was logged. A deterministic simulation that controls time, randomness, and scheduling gets a replayable transcript by collecting the entries in order, for example with `vi.spyOn(console, 'debug')`.
|
|
435
|
+
|
|
436
|
+
Set the flag with the bundler's `define`:
|
|
437
|
+
|
|
438
|
+
```jsonc
|
|
439
|
+
// wrangler.jsonc
|
|
440
|
+
{ "define": { "__DURABILITY_DEV__": "true" } }
|
|
441
|
+
```
|
|
442
|
+
|
|
443
|
+
```ts
|
|
444
|
+
// vite.config.ts or vitest.config.ts
|
|
445
|
+
export default defineConfig({
|
|
446
|
+
define: { __DURABILITY_DEV__: 'true' },
|
|
447
|
+
});
|
|
448
|
+
```
|
|
449
|
+
|
|
450
|
+
Leave it undefined or define it as `false` in production. With `false`, the bundler drops the tracing code entirely; left undefined, each traced call costs one branch. Tracing logs payloads, results, and SQL bindings verbatim, so never enable it where those contain secrets. In this repository, `DURABILITY_DEV=1 pnpm --filter durability test` runs the test suite in dev mode.
|
|
451
|
+
|
|
271
452
|
## Browser sockets
|
|
272
453
|
|
|
273
454
|
`DurabilitySocket` lets browsers call a Durable Object and receive events from it over one hibernatable WebSocket. `inbound` handlers become callable methods on the client; `outbound` events become methods on the socket that push to every subscribed connection. Nothing outside `inbound` is reachable, and no result is broadcast unless an outbound method sends it.
|
|
@@ -301,6 +482,8 @@ export class TodoList extends DurableObject<Env> {
|
|
|
301
482
|
return payload;
|
|
302
483
|
},
|
|
303
484
|
},
|
|
485
|
+
// Lifecycle events omit handler results unless this is set.
|
|
486
|
+
includeResults: true,
|
|
304
487
|
});
|
|
305
488
|
|
|
306
489
|
readonly socket = new DurabilitySocket({
|
|
@@ -340,7 +523,11 @@ export class TodoList extends DurableObject<Env> {
|
|
|
340
523
|
super(ctx, env);
|
|
341
524
|
// Push a durable operation's result to every subscriber once it has committed.
|
|
342
525
|
this.durability.lifecycle((e) => {
|
|
343
|
-
if (
|
|
526
|
+
if (
|
|
527
|
+
e.type === 'attempt_settled' &&
|
|
528
|
+
e.outcome === 'completed' &&
|
|
529
|
+
e.result !== undefined
|
|
530
|
+
) {
|
|
344
531
|
this.socket.created(e.result);
|
|
345
532
|
}
|
|
346
533
|
});
|
|
@@ -451,7 +638,7 @@ All three methods take the same query: `{ source, name, status, limit, cursor }`
|
|
|
451
638
|
- `get(source, id)` returns one such handle, or `undefined`.
|
|
452
639
|
- `records(query)` returns `{ records, cursor }` with the same data and no functions.
|
|
453
640
|
|
|
454
|
-
Narrowing on `source` and `name` narrows `state.result` to that handler's result type.
|
|
641
|
+
Narrowing on `source` and `name` narrows `state.result` to that handler's result type. A pending operation or alarm also lists `state.memoized`, the keys its next attempt will [reuse](#memoize).
|
|
455
642
|
|
|
456
643
|
### Handles over RPC
|
|
457
644
|
|
|
@@ -591,7 +778,7 @@ new DurabilitySemaphore({
|
|
|
591
778
|
- **`under`** — a reserve inside the limit is kept for leases; grants may use at most `limit - reserve`. Still never exceeds the limit, and leases have guaranteed headroom. The reserve never takes the last permit: with a reserve at or above the limit (including after lowering the limit), grants keep one, so groups excluded from leases and clients without lease support still make progress.
|
|
592
779
|
- **`over`** — leases come from a separate budget above the limit (the original gatekeeper behaviour); the effective ceiling is `limit + extra`.
|
|
593
780
|
|
|
594
|
-
Leases are issued only to clients that report `leases` (the queue always does), to groups without their own limit unless `limitedGroups: 'withinGroupLimit'`, and only to a client that holds no undrawn permits for that group. Each round shares the remaining lease budget equally among candidates, oldest demand first, up to `permitsPerLease`; with fewer permits left than candidates, the oldest get one each. When a round leaves demand unserved, the holder marks its replies `contended`, and each client returns the undrawn permits of leases in groups where it has nothing queued, so an idle lease cannot keep other clients waiting. A client that stops talking to the holder keeps its lease until `ttlMs`. Reports (`drawn`, `released` high-water marks) heartbeat a lease;
|
|
781
|
+
Leases are issued only to clients that report `leases` (the queue always does), to groups without their own limit unless `limitedGroups: 'withinGroupLimit'`, and only to a client that holds no undrawn permits for that group. Each round shares the remaining lease budget equally among candidates, oldest demand first, up to `permitsPerLease`; with fewer permits left than candidates, the oldest get one each. When a round leaves demand unserved, the holder marks its replies `contended`, and each client returns the undrawn permits of leases in groups where it has nothing queued, so an idle lease cannot keep other clients waiting. A client that stops talking to the holder keeps its lease until `ttlMs`. Reports (`drawn`, `released` high-water marks) heartbeat a lease, and the queue reports a lease with permits in use at half its lifetime even when nothing else is happening, so long-running work keeps its permits counted; a lease unreported for `ttlMs` is retired rather than forgotten: its client may be cut off from the holder with work still running, so, like a claimed grant permit, its footprint stays counted until the client reports what it drew and released. The queue keeps reporting a stopped lease until the holder settles it. A report for a lease the holder never knew is answered retired and its live footprint charged against that round. On the client, a drawn permit is one-shot: releasing the item returns footprint to the holder but never makes the permit drawable again. Lowering a group's eligibility, or the limit, retires leases on the next round. A retired lease stops drawing at once but stays counted in full until its client confirms how much it drew, since it may have drawn after its last report.
|
|
595
782
|
|
|
596
783
|
### Queueing work in a client Durable Object
|
|
597
784
|
|
|
@@ -753,7 +940,7 @@ First deliveries are ordered by enqueue sequence per shard and target. Waiting r
|
|
|
753
940
|
|
|
754
941
|
Set `batchDelayMs` to hold first deliveries for up to that many milliseconds, so messages enqueued together reach the consumer as one batch instead of one call each. Each first attempt is rounded up to the next multiple of `batchDelayMs`, so messages enqueued within the same window become due at the same time. It delays delivery, not acceptance: `enqueue` still returns once the batch commits. Retries keep their policy delay.
|
|
755
942
|
|
|
756
|
-
Consumer callbacks may forward the batch to any ordinary WorkerEntrypoint RPC method. Messages include `id`, `deliveryId`, `target`, `body`, `attempt`, `enqueuedAt`, and callable settlement capabilities:
|
|
943
|
+
Consumer callbacks may forward the batch to any ordinary WorkerEntrypoint RPC method. Messages include `id`, `deliveryId`, `target`, `body`, `attempt`, `enqueuedAt`, a [`memoize`](#memoize) scoped to that message and target, so a retried delivery reuses work an earlier attempt finished, and callable settlement capabilities:
|
|
757
944
|
|
|
758
945
|
- `ack()` settles this target only.
|
|
759
946
|
- `retry(10)` schedules eligibility after 10 ms, overriding the policy delay. It is not a real-time delivery guarantee.
|
|
@@ -876,6 +1063,147 @@ export class Account extends DurableObject<Env> {
|
|
|
876
1063
|
|
|
877
1064
|
`DurabilityOutbox.migrate(this.ctx, null)` removes the outbox schema.
|
|
878
1065
|
|
|
1066
|
+
## Shard topology (experimental)
|
|
1067
|
+
|
|
1068
|
+
`DurabilityTopology` is for work spread over several Durable Objects (shards) whose membership changes over time, where something else needs to reach them: a router picking a shard for a write, a reader querying every shard, or a coordinator pushing configuration to all of them. It lives in `durability/experimental`, so pin an exact version.
|
|
1069
|
+
|
|
1070
|
+
One coordinator Durable Object holds each group. It is the only thing that can change or publish that group. Readers never call it: they read the snapshot it published.
|
|
1071
|
+
|
|
1072
|
+
```ts
|
|
1073
|
+
import {
|
|
1074
|
+
DurabilityTopology,
|
|
1075
|
+
type TopologyStateUpdate,
|
|
1076
|
+
} from 'durability/experimental';
|
|
1077
|
+
|
|
1078
|
+
export class Coordinator extends DurableObject<Env> {
|
|
1079
|
+
private readonly scheduler = new DurabilityScheduler({ context: this.ctx });
|
|
1080
|
+
private readonly topology = new DurabilityTopology({
|
|
1081
|
+
scheduler: this.scheduler,
|
|
1082
|
+
publish: {
|
|
1083
|
+
write: (snapshot, encoded) =>
|
|
1084
|
+
this.env.TOPOLOGY.put(`topology/${snapshot.group}`, encoded),
|
|
1085
|
+
delete: (group) => this.env.TOPOLOGY.delete(`topology/${group}`),
|
|
1086
|
+
},
|
|
1087
|
+
states: {
|
|
1088
|
+
limits: {
|
|
1089
|
+
deliver: (member, update: TopologyStateUpdate<Limits>) =>
|
|
1090
|
+
this.env.SHARDS.getByName(member.id).applyLimits(update),
|
|
1091
|
+
},
|
|
1092
|
+
},
|
|
1093
|
+
});
|
|
1094
|
+
|
|
1095
|
+
createTenant(tenant: string, limits: Limits) {
|
|
1096
|
+
return this.topology.update(tenant, () => ({
|
|
1097
|
+
members: [0, 1, 2].map((n) => ({
|
|
1098
|
+
id: `${tenant}-${n}`,
|
|
1099
|
+
state: 'active' as const,
|
|
1100
|
+
})),
|
|
1101
|
+
states: { limits },
|
|
1102
|
+
}));
|
|
1103
|
+
}
|
|
1104
|
+
|
|
1105
|
+
retireShard(tenant: string, shard: string) {
|
|
1106
|
+
return this.topology.update(tenant, ({ members }) => ({
|
|
1107
|
+
members: [
|
|
1108
|
+
...members.map((m) =>
|
|
1109
|
+
m.id === shard ? { ...m, state: 'draining' as const } : m
|
|
1110
|
+
),
|
|
1111
|
+
{ id: `${tenant}-${members.length}`, state: 'active' },
|
|
1112
|
+
],
|
|
1113
|
+
}));
|
|
1114
|
+
}
|
|
1115
|
+
|
|
1116
|
+
deleteTenant(tenant: string) {
|
|
1117
|
+
return this.topology.delete(tenant);
|
|
1118
|
+
}
|
|
1119
|
+
|
|
1120
|
+
alarm(info?: AlarmInvocationInfo) {
|
|
1121
|
+
return this.scheduler.alarm(info);
|
|
1122
|
+
}
|
|
1123
|
+
}
|
|
1124
|
+
```
|
|
1125
|
+
|
|
1126
|
+
The coordinator has three methods:
|
|
1127
|
+
|
|
1128
|
+
- `update(group, next)` runs `next(current)` inside one storage transaction and commits its draft, staging publication and deliveries in the same transaction. `current` holds the members and the current state values. The draft can set `members`, `states`, or both, and `next` must return synchronously, so nothing interleaves between reading and writing. Membership changes bump the group's `version`; unchanged drafts do not. After the commit, one publish is attempted: `published: false` means the snapshot is not yet final in the store, because the write failed or an older write may still land over it. The change is committed either way, and the outbox keeps going until it is. An empty member list is rejected; use `delete`.
|
|
1129
|
+
- `delete(group)` drops the members and state values, cancels pending syncs, and asks the publisher to delete the snapshot. Versions keep counting, so a recreated group is newer than anything a reader cached.
|
|
1130
|
+
- `snapshot(group)` reads the group inside the coordinator. It is not a lookup path for other Workers.
|
|
1131
|
+
|
|
1132
|
+
An `active` member is in the write set and the read set. A `draining` member is only in the read set: it receives no new work but still holds data, so queries must still reach it. Members leave the read set only when a draft omits them.
|
|
1133
|
+
|
|
1134
|
+
### When to use it
|
|
1135
|
+
|
|
1136
|
+
Topology is for when something needs the list of shards. A shard that only does work when a caller reaches it, and only talks to a coordinator whose name it can derive, does not need one: create it on demand.
|
|
1137
|
+
|
|
1138
|
+
| Need | Use |
|
|
1139
|
+
| ---------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
|
1140
|
+
| A shard does work when a caller sends it something | Nothing: address it by name, for example with `DurabilityRouting` |
|
|
1141
|
+
| A shard asks a central object for something it needs, such as permits or capacity | `DurabilitySemaphore`: shards pull, and the holder learns about them from their requests |
|
|
1142
|
+
| A caller must pick among a group's current shards | Topology membership |
|
|
1143
|
+
| A reader must reach every shard that may hold data | Topology membership, including `draining` members |
|
|
1144
|
+
| Every shard must hold the same data before any caller reaches it, such as schedules it runs by itself or configuration it enforces | A topology state channel |
|
|
1145
|
+
|
|
1146
|
+
A state channel sends one value to every member in its role. Values that differ per shard, or that must be accounted for in both directions (granted, used, returned), do not fit it; that is the semaphore's model.
|
|
1147
|
+
|
|
1148
|
+
A shard created on demand can still join its group, so readers can find it later: its first request can call the coordinator, which adds it with one `update`. Only do this when a reader needs it.
|
|
1149
|
+
|
|
1150
|
+
### Publishing
|
|
1151
|
+
|
|
1152
|
+
`publish.write` receives the snapshot and an `encoded` string. Store `encoded` as is; it is the only format the client reads, and no encoder is exported, so only the coordinator can produce one. It carries the coordinator's Durable Object ID, because versions only compare within one coordinator: if a group moves to another coordinator, its versions start over, and readers accept the new snapshot. Publication runs on a `DurabilityOutbox` channel (`durability.topology`) keyed by group, so it shares the outbox's guarantees: only the newest snapshot of a group is pending, publish calls for the channel run one at a time, and attempts never stop (`retries.delay` controls backoff). The outbox expects sinks to keep the highest version, which a plain `KV.put` cannot, so the topology closes that gap itself, durably:
|
|
1153
|
+
|
|
1154
|
+
- Every write is recorded in storage before it starts and forgotten once it settles.
|
|
1155
|
+
- A record that outlives its write, because the write timed out or the object was evicted mid-write, means an older snapshot may still land. A newer write is then not acknowledged until that record's landing deadline has passed, and is written again at that point.
|
|
1156
|
+
- An abandoned write that settles while the object is still running restages the latest snapshot immediately.
|
|
1157
|
+
|
|
1158
|
+
The landing deadline is the write's start plus `attemptTimeoutMs` plus five minutes. A write that lands later than that can still overwrite a newer snapshot until the next change; only a sink that keeps the highest version avoids that bound. `attemptTimeoutMs` bounds one write.
|
|
1159
|
+
|
|
1160
|
+
### State channels
|
|
1161
|
+
|
|
1162
|
+
Each entry in `states` is a channel whose latest value every member converges to. `members: 'write'` (the default) reaches active members; `'read'` reaches draining members too. Setting a value delivers it to every member in the role. A member that enters the role receives the current value in the same transaction that added it, so a new shard never starts with an old value. A member that leaves the role has its pending delivery discarded; one already in flight may still arrive. Each channel is its own outbox channel (`durability.topology.<name>`) keyed by group and member, so every member is retried independently; `attemptTimeoutMs` bounds one delivery, and a member that hangs holds up the rest of its batch only until then. Setting a channel to the value it already has delivers nothing.
|
|
1163
|
+
|
|
1164
|
+
Only the latest value is delivered, and a timed-out delivery may still arrive after a newer one. `version` is the outbox version, which only increases. Apply an update only when it is newer than the last one applied, in the same transaction:
|
|
1165
|
+
|
|
1166
|
+
```ts
|
|
1167
|
+
applyLimits(update: TopologyStateUpdate<Limits>) {
|
|
1168
|
+
this.ctx.storage.transactionSync(() => {
|
|
1169
|
+
const applied = this.ctx.storage.kv.get<number>('limits:version') ?? 0;
|
|
1170
|
+
if (update.version <= applied) {
|
|
1171
|
+
return;
|
|
1172
|
+
}
|
|
1173
|
+
this.ctx.storage.kv.put('limits', update.value);
|
|
1174
|
+
this.ctx.storage.kv.put('limits:version', update.version);
|
|
1175
|
+
});
|
|
1176
|
+
}
|
|
1177
|
+
```
|
|
1178
|
+
|
|
1179
|
+
`onLifecycleEvent` receives the outbox events of those channels: `channel` names the publication or state channel, and `id` is the group, or `[group, member]` as JSON for a delivery.
|
|
1180
|
+
|
|
1181
|
+
### Reading
|
|
1182
|
+
|
|
1183
|
+
Stateless Workers read through a client. It has one method:
|
|
1184
|
+
|
|
1185
|
+
```ts
|
|
1186
|
+
const topology = DurabilityTopology.client({
|
|
1187
|
+
read: (tenant) => env.TOPOLOGY.get(`topology/${tenant}`),
|
|
1188
|
+
});
|
|
1189
|
+
|
|
1190
|
+
const { members } = await topology.get(tenant);
|
|
1191
|
+
const writable = members.filter((member) => member.state === 'active');
|
|
1192
|
+
const shard = writable[Math.floor(Math.random() * writable.length)];
|
|
1193
|
+
```
|
|
1194
|
+
|
|
1195
|
+
Routing, querying every member, and resolving stubs are ordinary code over `members`.
|
|
1196
|
+
|
|
1197
|
+
The client caches each group in the isolate for `ttlMs` (default 30 seconds, jittered by ±20%), shares one read between concurrent callers, and never replaces a snapshot with an older version from the same coordinator. A missing or failed read is remembered for `negativeTtlMs` (default 5 seconds). Reads time out after `readTimeoutMs` (default 1 second).
|
|
1198
|
+
|
|
1199
|
+
When the published copy is missing, invalid, slow, or the read throws, `get` throws `TopologyUnavailableError` with a `reason`. Every fallback is opt-in, and they run in this order:
|
|
1200
|
+
|
|
1201
|
+
1. `staleIfErrorMs` serves this isolate's last good snapshot for that long after it expired. It never applies to a missing group.
|
|
1202
|
+
2. `seed(group)` derives members locally, without I/O.
|
|
1203
|
+
3. `fallback(group, reason)` is application code and is not cached.
|
|
1204
|
+
|
|
1205
|
+
The client never contacts the coordinator. Wiring `fallback` to it moves every reader onto one Durable Object at the moment the read store is struggling, so prefer failing or serving stale. `onFallback` reports each fallback and its source.
|
|
1206
|
+
|
|
879
1207
|
## Retained log
|
|
880
1208
|
|
|
881
1209
|
Fanout deletes a record once every target settles it, which is what a work queue wants. A log wants the opposite: records are **retained** so a consumer can rewind, a new consumer can start from the beginning, and several consumers can read the same history at different speeds. `DurabilityLog` is that second shape, and it attaches to the same scheduler.
|
|
@@ -962,7 +1290,7 @@ type LogColdStorage<Body> = {
|
|
|
962
1290
|
|
|
963
1291
|
External storage expires on its own schedule, so mirror a lifecycle rule with `forgetColdBefore(offset)` to drop stale index entries. Returning `undefined` from `read` is also honoured: a segment the index outlived surfaces as truncation rather than a silent gap.
|
|
964
1292
|
|
|
965
|
-
Retention still **ignores cursors**, exactly as a log should: a consumer slower than everything readable loses records. It learns loudly rather than silently skipping, because reading below the oldest readable offset throws `LogTruncatedError` carrying `requestedOffset` and `oldestOffset`. Compare `bounds()` against a cursor to detect the risk before it happens.
|
|
1293
|
+
Retention still **ignores cursors**, exactly as a log should: a consumer slower than everything readable loses records. It learns loudly rather than silently skipping, because reading below the oldest readable offset throws `LogTruncatedError` carrying `requestedOffset` and `oldestOffset`. Leased consumers learn the same way: when a range awaiting redelivery (nacked or expired) was trimmed, `lease()` throws `LogTruncatedError` once, drops the trimmed offsets from the range, and the next call carries on from what is still retained. A leased consumer's cursor moves past offsets trimmed before anyone leased them. Compare `bounds()` against a cursor to detect the risk before it happens.
|
|
966
1294
|
|
|
967
1295
|
### Leases for parallel consumers
|
|
968
1296
|
|
package/dist/dev.d.ts
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Logs one dev-mode trace entry and returns its sequence number. The sequence
|
|
3
|
+
* is process-wide and monotonic so interleaved async work can be reordered
|
|
4
|
+
* and correlated when replaying a simulation.
|
|
5
|
+
*/
|
|
6
|
+
export declare const trace: (operation: string, details?: Record<string, unknown>) => number;
|
|
7
|
+
/**
|
|
8
|
+
* Wraps a public function so dev mode logs its arguments and outcome. `details`
|
|
9
|
+
* identifies the receiver when the arguments alone do not.
|
|
10
|
+
*/
|
|
11
|
+
export declare const traceFunction: <Args extends unknown[], Result>(method: string, fn: (...args: Args) => Result, details?: Record<string, unknown>) => ((...args: Args) => Result);
|
|
12
|
+
/** Wraps every method on a class prototype with {@link traceFunction}. */
|
|
13
|
+
export declare const traceMethods: (prototype: object, owner: string) => void;
|
|
14
|
+
/** Logs each lifecycle event before `emit` hands it to the consumer's hooks. */
|
|
15
|
+
export declare const traceLifecycle: <Event extends Record<string, unknown>>(emit: (event: Event) => void) => ((event: Event) => void);
|
|
16
|
+
/**
|
|
17
|
+
* In dev mode, returns a view of Durable Object storage that logs every SQL
|
|
18
|
+
* statement, transaction, and alarm read or write. Outside dev mode, returns
|
|
19
|
+
* the storage unchanged. Idempotent, so helpers sharing one storage log each
|
|
20
|
+
* call once.
|
|
21
|
+
*/
|
|
22
|
+
export declare const traceStorage: <Storage extends object>(storage: Storage) => Storage;
|