durability 5.0.0 → 5.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +98 -0
- package/dist/durability-alarms.d.ts +10 -1
- package/dist/durability-alarms.js +33 -3
- package/dist/durability.js +6 -1
- package/dist/errors.d.ts +15 -0
- package/dist/errors.js +24 -0
- package/dist/experimental.d.ts +3 -0
- package/dist/experimental.js +3 -0
- package/dist/fanout.js +2 -4
- package/dist/index.d.ts +1 -1
- package/dist/migrations.d.ts +10 -1
- package/dist/migrations.js +44 -1
- package/dist/outbox.js +2 -3
- package/dist/schedule.d.ts +157 -0
- package/dist/schedule.js +651 -0
- package/dist/scheduler.d.ts +51 -1
- package/dist/scheduler.js +135 -24
- package/dist/semaphore-queue-store.js +23 -19
- package/dist/storage.d.ts +5 -2
- package/dist/storage.js +12 -6
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -375,6 +375,104 @@ const removedAlarms = await this.alarms.purgeBefore(retentionCutoff);
|
|
|
375
375
|
|
|
376
376
|
The purge runs in one transaction. It best-effort aborts matching active handlers, but cannot force an abort-ignoring handler to stop. Operation IDs can be reused immediately; a late old generation cannot overwrite the replacement record.
|
|
377
377
|
|
|
378
|
+
## Experimental options
|
|
379
|
+
|
|
380
|
+
These options, methods, and capabilities are opt-in: none takes effect unless you set, call, or construct it, so leaving them alone keeps every behavior described above. Their names and behavior may change in a minor release. Scheduler options also apply to the private scheduler of a helper configured with `context`.
|
|
381
|
+
|
|
382
|
+
```ts
|
|
383
|
+
const scheduler = new DurabilityScheduler({
|
|
384
|
+
context: this.ctx,
|
|
385
|
+
liveTimers: true,
|
|
386
|
+
});
|
|
387
|
+
|
|
388
|
+
const alarms = new DurabilityAlarms({
|
|
389
|
+
scheduler,
|
|
390
|
+
handlers: { tick: async () => this.enqueueDueRuns() },
|
|
391
|
+
});
|
|
392
|
+
|
|
393
|
+
await alarms.tick.cancel();
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
### `liveTimers` (scheduler)
|
|
397
|
+
|
|
398
|
+
`liveTimers?: boolean`. Due work starts from an in-memory timer at its exact scheduled time, however far beyond `alarmMinDelayMs`, and an alarm pass skips records that are already running instead of waiting for them, so a long handler no longer delays newly due work. The physical alarm stays armed at the usual bounded delay as the eviction fallback, and the handoff before the alarm wall-time limit is unchanged.
|
|
399
|
+
|
|
400
|
+
Gives up: every settled attempt runs one more reconciliation transaction to re-arm the timer, and timers do not survive eviction, so after one the work runs only when the physical alarm fires. Attempts interrupted by the eviction are counted as usual, so an operation with `maxAttempts: 1` fails instead of running again.
|
|
401
|
+
|
|
402
|
+
### `cancel` (named alarm method)
|
|
403
|
+
|
|
404
|
+
`alarms.<name>.cancel(): Promise<boolean>`. This deletes the alarm's pending occurrence and its memos and reconciles the physical alarm in one transaction, then aborts the `signal` of a running attempt of it. That attempt's settlement can no longer retry it. It resolves `true` when a pending occurrence was deleted. Failed occurrences are left for `purgeBefore`.
|
|
405
|
+
|
|
406
|
+
Gives up: cancelling does not stop later scheduling. A call to the scheduling method after `cancel`, including one from the running handler before it returns, schedules a new occurrence.
|
|
407
|
+
|
|
408
|
+
### `onAlarmRetry` (scheduler)
|
|
409
|
+
|
|
410
|
+
`onAlarmRetry?: (info: AlarmRetryInfo) => void | Promise<void>`, where `info` is `{ error: { name, message }, retryInMs, retryCount?, isRetry?, objectId? }`. It is called each time an alarm run fails and the scheduler re-arms its own fallback alarm (see [When a run cannot finish](#when-a-run-cannot-finish)), so the error can go to an external service. `retryInMs` is `alarmMinDelayMs`; `retryCount` and `isRetry` come from the platform's `AlarmInvocationInfo` when the invocation carried one, and `objectId` from the context.
|
|
411
|
+
|
|
412
|
+
The `durability.alarm.failed` report still happens. The hook is best effort: it runs after the fallback is armed, a returned promise is passed to `waitUntil`, and a throw or rejection is only reported as `durability.alarm.retry_hook_failed`, so it cannot change scheduling. It is not called when even the fallback cannot be armed, because the failure then reaches the platform.
|
|
413
|
+
|
|
414
|
+
```ts
|
|
415
|
+
const scheduler = new DurabilityScheduler({
|
|
416
|
+
context: this.ctx,
|
|
417
|
+
onAlarmRetry: (info) => this.env.ERRORS.send(info),
|
|
418
|
+
});
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
### `DurabilitySchedule`
|
|
422
|
+
|
|
423
|
+
`DurabilitySchedule` is exported from `durability/experimental`. It runs one recurring schedule per Durable Object: each tick starts a run, runs overlap, every run is tracked on its own, and its outcome is reported once and then forgotten. It replaces a `DurabilityAlarms` tick that registers a `Durability` operation per run, which needs retained records to deduplicate and keeps them until purged.
|
|
424
|
+
|
|
425
|
+
```ts
|
|
426
|
+
import { DurableObject } from 'cloudflare:workers';
|
|
427
|
+
import { DurabilityScheduler } from 'durability';
|
|
428
|
+
import { DurabilitySchedule } from 'durability/experimental';
|
|
429
|
+
import { nextCronTime } from './cron';
|
|
430
|
+
|
|
431
|
+
export class Cron extends DurableObject<Env> {
|
|
432
|
+
private readonly scheduler = new DurabilityScheduler({ context: this.ctx });
|
|
433
|
+
private readonly schedule = new DurabilitySchedule({
|
|
434
|
+
scheduler: this.scheduler,
|
|
435
|
+
// Durability ships no cron parser: return the first tick after `after`.
|
|
436
|
+
nextOccurrence: (expression, after) => nextCronTime(expression, after),
|
|
437
|
+
run: ({ scheduledTime, idempotencyKey, signal }) =>
|
|
438
|
+
this.env.TARGET.fire({ scheduledTime, idempotencyKey }, { signal }),
|
|
439
|
+
onOutcome: (run, outcome) => this.env.TARGET.report(run, outcome),
|
|
440
|
+
maxConcurrentRuns: 20,
|
|
441
|
+
runTimeoutMs: 15 * 60_000,
|
|
442
|
+
});
|
|
443
|
+
|
|
444
|
+
configure(expression: string, version: number) {
|
|
445
|
+
// Spread objects sharing an expression over the first 30 seconds.
|
|
446
|
+
return this.schedule.set({
|
|
447
|
+
expression,
|
|
448
|
+
version,
|
|
449
|
+
jitterMs: { maxMs: 30_000 },
|
|
450
|
+
});
|
|
451
|
+
}
|
|
452
|
+
|
|
453
|
+
remove(version: number) {
|
|
454
|
+
return this.schedule.cancel(version);
|
|
455
|
+
}
|
|
456
|
+
|
|
457
|
+
override alarm(info?: AlarmInvocationInfo) {
|
|
458
|
+
return this.scheduler.alarm(info);
|
|
459
|
+
}
|
|
460
|
+
}
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
- **Versions.** `set({ expression, version })` stores the schedule and its next tick only when `version` is higher than the stored one, and resolves `true` when it did. A lower version is ignored. The same version changes nothing but reconciles the physical alarm, so resending it repairs a lost alarm. The expression is opaque; only `nextOccurrence(expression, after)` reads it, and it must return a safe integer after `after`, or `undefined` to stop ticking. `get()` returns `{ expression, version, jitterMs, nextAt }`, where `nextAt` is the next tick's nominal time, or `undefined` when no schedule is live.
|
|
464
|
+
- **Jitter.** `set({ expression, version, jitterMs })` delays every tick by one fixed offset, so many objects sharing a popular expression do not all fire at the same instant. `jitterMs` is either a non-negative integer, the offset itself, or `{ maxMs }`, which picks an offset uniformly in `[0, maxMs]` when the version is stored. It defaults to 0, and both forms are at most 2^31 - 1 ms. The offset is stored in the schedule's row, so it costs no write per run, and resending the same version keeps it; only a new version picks a new one. A tick's nominal time `scheduledTime` is what `nextOccurrence` returned, and it fires at `dueAt`, which is `scheduledTime + jitterMs`. `idempotencyKey` derives from `scheduledTime`, and the next tick is always searched from nominal times, so the offset never drifts. Keep `jitterMs` below the expression's interval; durability cannot know the interval of an opaque expression. A larger offset is allowed: every tick still fires once, in nominal order, each `jitterMs` after its nominal time, even when that is after the next tick's nominal time.
|
|
465
|
+
- **Cancel.** `cancel(version?)` stops future ticks and resolves `true` when a schedule was live. Runs already due or running still finish and report. With a version, the cancel is ignored when the stored version is higher, and is otherwise kept as a one-row tombstone so a delayed `set` of that version or lower cannot revive the schedule.
|
|
466
|
+
- **One tick, one run.** A tick inserts its run in the transaction that advances the stored next tick, so a replayed or duplicated alarm finds the cursor already moved and cannot fire the tick again; deduplication needs no finished-run records. When ticks were missed, for example while the object was down, they collapse into one run for the earliest missed time, and the next tick is the first whose `dueAt` is after now. A `nextOccurrence` that throws or returns an invalid time rolls the tick back and fails the alarm run, which is retried at `alarmMinDelayMs` and reported through `onAlarmRetry`.
|
|
467
|
+
- **Runs.** `run({ scheduledTime, dueAt, idempotencyKey, signal })` starts outside the alarm pass that recorded it, so a long run never delays later ticks. `idempotencyKey` is `<objectId>:<scheduledTime>`. At most `maxConcurrentRuns` (default 10) are in flight; later runs wait durably in tick order. This limit is separate from the scheduler's `alarmConcurrency`. A run that throws, or exceeds `runTimeoutMs` (default 15 minutes, which aborts its `signal`), is `errored`. A run is attempted once and never retried. A timed-out run keeps its slot until its handler actually settles.
|
|
468
|
+
- **Keeping the object alive.** While runs or outcome deliveries are in flight, the alarm pass stays pending, so an alarm invocation stays open and holds the object in memory; it fires the next ticks and outcome retries itself, at their exact times. Before the wall-time limit the scheduler hands off at `alarmHandoffMs`, and the next alarm joins the same loop. A tick started by an in-memory timer is joined by the next physical alarm. Other helpers on the same scheduler still run on each alarm pass, but work of theirs that becomes due during the hold waits for it unless the scheduler uses `liveTimers`.
|
|
469
|
+
- **Interrupted runs.** A run still marked running when no live instance holds it, for example after the object was evicted, reset, or redeployed mid-run, is reported as `errored` with `ScheduledRunInterruptedError` on the next alarm, and is not run again. While a run or outcome delivery is in flight, the physical alarm stays on the next tick, which is its eviction fallback, so an interrupted run is reported by the next tick at the latest, or within `alarmMinDelayMs` when no tick is left.
|
|
470
|
+
- **Outcomes.** `onOutcome(run, outcome)` receives `{ scheduledTime, dueAt, idempotencyKey, attempt }` and `{ status: 'complete' }` or `{ status: 'errored', error: { name, message } }`. The outcome is persisted before the first delivery, so delivery is at least once with the same outcome, even across a crash. Returning acknowledges it and deletes the run's row; nothing about the run is kept afterwards. Throwing, or exceeding `outcomeTimeoutMs` (default 60 seconds), retries it with `outcomeRetries.delay(attempt)` (default jittered exponential capped at 5 minutes) up to `outcomeRetries.maxAttempts` (default 10) deliveries. After the last, the row is deleted, `durability.schedule.outcome_dead_lettered` is reported, and a `dead_lettered` lifecycle event is emitted.
|
|
471
|
+
- **Storage.** Two rowid tables with no secondary indexes: `durability_schedule` holds the schedule's one row and `durability_schedule_runs` holds one small row per run from its tick until its outcome is delivered. A run writes four rows: the tick's cursor update and run insert, the outcome update, and the delete on delivery. It also arms the physical alarm once, for the next tick, and each `setAlarm()` is billed as a row written. A run that waits for a slot writes one more row, and each failed delivery one more.
|
|
472
|
+
- **Lifecycle events.** `onLifecycleEvent` receives `scheduled_run` events keyed by `idempotencyKey`: `due` (with `queued`), `started`, `settled` (with `outcome`, `error`, and `durationMs`), `outcome_retry`, and `dead_lettered`. Like every lifecycle hook, it is best effort.
|
|
473
|
+
|
|
474
|
+
One schedule attaches per scheduler; attaching a second throws. Construct it once per object instance, as with every helper. `DurabilitySchedule.migrate(this.ctx, null)` removes its schema.
|
|
475
|
+
|
|
378
476
|
## Lifecycle metrics hook
|
|
379
477
|
|
|
380
478
|
Each helper accepts `onLifecycleEvent`, which receives compact events for registration or scheduling, attempt start and settlement, pre-invocation attempt exhaustion, and purges:
|
|
@@ -59,7 +59,16 @@ export type DurabilityAlarmEvent = Extract<DurabilityLifecycleEvent, {
|
|
|
59
59
|
type: 'purged';
|
|
60
60
|
}>;
|
|
61
61
|
/** Schedules (or replaces) the named alarm's pending occurrence. */
|
|
62
|
-
export type DurableNamedAlarm = (scheduledTime: number) => Promise<void
|
|
62
|
+
export type DurableNamedAlarm = ((scheduledTime: number) => Promise<void>) & {
|
|
63
|
+
/**
|
|
64
|
+
* Experimental. Deletes the alarm's pending occurrence and reconciles the
|
|
65
|
+
* physical alarm, then aborts a running attempt of it. That attempt can no
|
|
66
|
+
* longer retry when it settles, but calling the scheduling method again,
|
|
67
|
+
* including from that handler, schedules a new occurrence. Resolves whether
|
|
68
|
+
* a pending occurrence was deleted.
|
|
69
|
+
*/
|
|
70
|
+
cancel: () => Promise<boolean>;
|
|
71
|
+
};
|
|
63
72
|
declare class DurabilityAlarmsCore<Names extends string> {
|
|
64
73
|
#private;
|
|
65
74
|
constructor(config: DurabilityAlarmsConfig<Names>);
|
|
@@ -9,7 +9,7 @@ var __classPrivateFieldGet = (this && this.__classPrivateFieldGet) || function (
|
|
|
9
9
|
if (typeof state === "function" ? receiver !== state || !f : !state.has(receiver)) throw new TypeError("Cannot read private member from an object whose class did not declare it");
|
|
10
10
|
return kind === "m" ? f : kind === "a" ? f.call(receiver) : f ? f.value : state.get(receiver);
|
|
11
11
|
};
|
|
12
|
-
var _DurabilityAlarmsCore_instances, _DurabilityAlarmsCore_engine, _DurabilityAlarmsCore_handlers, _DurabilityAlarmsCore_policy, _DurabilityAlarmsCore_lifecycle, _DurabilityAlarmsCore_emit, _DurabilityAlarmsCore_active, _DurabilityAlarmsCore_trackedSource, _DurabilityAlarmsCore_runDue, _DurabilityAlarmsCore_schedule, _DurabilityAlarmsCore_execute;
|
|
12
|
+
var _DurabilityAlarmsCore_instances, _DurabilityAlarmsCore_engine, _DurabilityAlarmsCore_handlers, _DurabilityAlarmsCore_policy, _DurabilityAlarmsCore_lifecycle, _DurabilityAlarmsCore_emit, _DurabilityAlarmsCore_active, _DurabilityAlarmsCore_trackedSource, _DurabilityAlarmsCore_runDue, _DurabilityAlarmsCore_schedule, _DurabilityAlarmsCore_cancel, _DurabilityAlarmsCore_execute;
|
|
13
13
|
import { z } from 'zod';
|
|
14
14
|
import { traceFunction, traceMethods } from './dev.js';
|
|
15
15
|
import { DurableAlarmTimeoutError, DurableAttemptsExhaustedError, isErrorInstance, NonRetryableError, } from './errors.js';
|
|
@@ -85,8 +85,11 @@ class DurabilityAlarmsCore {
|
|
|
85
85
|
if (reserved.has(name)) {
|
|
86
86
|
throw new Error(`"${name}" is reserved by DurabilityAlarms`);
|
|
87
87
|
}
|
|
88
|
+
const method = Object.assign(traceFunction(`DurabilityAlarms.${name}`, (scheduledTime) => __classPrivateFieldGet(this, _DurabilityAlarmsCore_instances, "m", _DurabilityAlarmsCore_schedule).call(this, name, scheduledTime)), {
|
|
89
|
+
cancel: traceFunction(`DurabilityAlarms.${name}.cancel`, () => __classPrivateFieldGet(this, _DurabilityAlarmsCore_instances, "m", _DurabilityAlarmsCore_cancel).call(this, name)),
|
|
90
|
+
});
|
|
88
91
|
Object.defineProperty(this, name, {
|
|
89
|
-
value:
|
|
92
|
+
value: method,
|
|
90
93
|
enumerable: true,
|
|
91
94
|
});
|
|
92
95
|
}
|
|
@@ -174,8 +177,13 @@ _DurabilityAlarmsCore_engine = new WeakMap(), _DurabilityAlarmsCore_handlers = n
|
|
|
174
177
|
ORDER BY next_attempt_at ASC
|
|
175
178
|
LIMIT 100`, now)
|
|
176
179
|
.toArray());
|
|
180
|
+
// With liveTimers a running occurrence re-arms itself when it settles, so
|
|
181
|
+
// skipping it keeps a long handler from blocking a concurrent alarm pass.
|
|
182
|
+
const pending = __classPrivateFieldGet(this, _DurabilityAlarmsCore_engine, "f").liveTimers
|
|
183
|
+
? due.filter((row) => !__classPrivateFieldGet(this, _DurabilityAlarmsCore_active, "f").has(row.name))
|
|
184
|
+
: due;
|
|
177
185
|
const failures = [];
|
|
178
|
-
await __classPrivateFieldGet(this, _DurabilityAlarmsCore_engine, "f").runConcurrent(
|
|
186
|
+
await __classPrivateFieldGet(this, _DurabilityAlarmsCore_engine, "f").runConcurrent(pending, async (row) => {
|
|
179
187
|
try {
|
|
180
188
|
await __classPrivateFieldGet(this, _DurabilityAlarmsCore_instances, "m", _DurabilityAlarmsCore_execute).call(this, row, info);
|
|
181
189
|
}
|
|
@@ -221,6 +229,28 @@ _DurabilityAlarmsCore_engine = new WeakMap(), _DurabilityAlarmsCore_handlers = n
|
|
|
221
229
|
attempt: 0,
|
|
222
230
|
scheduledTime,
|
|
223
231
|
});
|
|
232
|
+
}, _DurabilityAlarmsCore_cancel = async function _DurabilityAlarmsCore_cancel(name) {
|
|
233
|
+
const cancelled = await __classPrivateFieldGet(this, _DurabilityAlarmsCore_engine, "f").transaction((sql) => {
|
|
234
|
+
const removed = sql
|
|
235
|
+
.exec(`DELETE FROM durability_alarms
|
|
236
|
+
WHERE name = ? AND status = 'pending'
|
|
237
|
+
RETURNING name`, name)
|
|
238
|
+
.toArray().length > 0;
|
|
239
|
+
if (removed) {
|
|
240
|
+
deleteMemos(sql, alarmsTable, name);
|
|
241
|
+
}
|
|
242
|
+
return removed;
|
|
243
|
+
});
|
|
244
|
+
if (!cancelled) {
|
|
245
|
+
return false;
|
|
246
|
+
}
|
|
247
|
+
// Every settlement write is conditional on the deleted row, so a running
|
|
248
|
+
// attempt can no longer retry; abort it as purging does.
|
|
249
|
+
__classPrivateFieldGet(this, _DurabilityAlarmsCore_active, "f")
|
|
250
|
+
.get(name)
|
|
251
|
+
?.controller?.abort(new Error(`Durable alarm "${name}" was cancelled`));
|
|
252
|
+
__classPrivateFieldGet(this, _DurabilityAlarmsCore_engine, "f").notify(alarmsTable, name);
|
|
253
|
+
return true;
|
|
224
254
|
}, _DurabilityAlarmsCore_execute = function _DurabilityAlarmsCore_execute(row, platform) {
|
|
225
255
|
const running = __classPrivateFieldGet(this, _DurabilityAlarmsCore_active, "f").get(row.name);
|
|
226
256
|
if (running) {
|
package/dist/durability.js
CHANGED
|
@@ -195,7 +195,12 @@ _DurabilityCore_engine = new WeakMap(), _DurabilityCore_handlers = new WeakMap()
|
|
|
195
195
|
ORDER BY next_attempt_at ASC
|
|
196
196
|
LIMIT 100`, now)
|
|
197
197
|
.toArray());
|
|
198
|
-
|
|
198
|
+
// With liveTimers an active operation reschedules itself when it settles, so
|
|
199
|
+
// skipping it keeps a long handler from blocking a concurrent alarm pass.
|
|
200
|
+
const pending = __classPrivateFieldGet(this, _DurabilityCore_engine, "f").liveTimers
|
|
201
|
+
? due.filter(({ id }) => !__classPrivateFieldGet(this, _DurabilityCore_active, "f").has(id))
|
|
202
|
+
: due;
|
|
203
|
+
await __classPrivateFieldGet(this, _DurabilityCore_engine, "f").runConcurrent(pending, async ({ id }) => {
|
|
199
204
|
try {
|
|
200
205
|
await __classPrivateFieldGet(this, _DurabilityCore_instances, "m", _DurabilityCore_execute).call(this, id);
|
|
201
206
|
}
|
package/dist/errors.d.ts
CHANGED
|
@@ -85,6 +85,21 @@ export declare class FanoutTimeoutError extends Error {
|
|
|
85
85
|
export declare class OutboxTimeoutError extends Error {
|
|
86
86
|
constructor(timeoutMs: number);
|
|
87
87
|
}
|
|
88
|
+
/** Outcome error of a scheduled run that exceeded `runTimeoutMs`. */
|
|
89
|
+
export declare class ScheduledRunTimeoutError extends Error {
|
|
90
|
+
constructor(timeoutMs: number);
|
|
91
|
+
}
|
|
92
|
+
/**
|
|
93
|
+
* Outcome error of a scheduled run that was still marked running when no live
|
|
94
|
+
* instance was executing it, for example after its object was evicted or reset.
|
|
95
|
+
*/
|
|
96
|
+
export declare class ScheduledRunInterruptedError extends Error {
|
|
97
|
+
constructor();
|
|
98
|
+
}
|
|
99
|
+
/** Error recorded when one `onOutcome` delivery exceeds `outcomeTimeoutMs`. */
|
|
100
|
+
export declare class ScheduledOutcomeTimeoutError extends Error {
|
|
101
|
+
constructor(timeoutMs: number);
|
|
102
|
+
}
|
|
88
103
|
/** A definite rejection before the batch was durably accepted. */
|
|
89
104
|
export declare class FanoutEnqueueError extends Error {
|
|
90
105
|
constructor(message: string);
|
package/dist/errors.js
CHANGED
|
@@ -135,6 +135,30 @@ export class OutboxTimeoutError extends Error {
|
|
|
135
135
|
this.name = 'OutboxTimeoutError';
|
|
136
136
|
}
|
|
137
137
|
}
|
|
138
|
+
/** Outcome error of a scheduled run that exceeded `runTimeoutMs`. */
|
|
139
|
+
export class ScheduledRunTimeoutError extends Error {
|
|
140
|
+
constructor(timeoutMs) {
|
|
141
|
+
super(`Scheduled run timed out after ${timeoutMs}ms`);
|
|
142
|
+
this.name = 'ScheduledRunTimeoutError';
|
|
143
|
+
}
|
|
144
|
+
}
|
|
145
|
+
/**
|
|
146
|
+
* Outcome error of a scheduled run that was still marked running when no live
|
|
147
|
+
* instance was executing it, for example after its object was evicted or reset.
|
|
148
|
+
*/
|
|
149
|
+
export class ScheduledRunInterruptedError extends Error {
|
|
150
|
+
constructor() {
|
|
151
|
+
super('Scheduled run was interrupted before it reported an outcome');
|
|
152
|
+
this.name = 'ScheduledRunInterruptedError';
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
/** Error recorded when one `onOutcome` delivery exceeds `outcomeTimeoutMs`. */
|
|
156
|
+
export class ScheduledOutcomeTimeoutError extends Error {
|
|
157
|
+
constructor(timeoutMs) {
|
|
158
|
+
super(`Scheduled run outcome delivery timed out after ${timeoutMs}ms`);
|
|
159
|
+
this.name = 'ScheduledOutcomeTimeoutError';
|
|
160
|
+
}
|
|
161
|
+
}
|
|
138
162
|
/** A definite rejection before the batch was durably accepted. */
|
|
139
163
|
export class FanoutEnqueueError extends Error {
|
|
140
164
|
constructor(message) {
|
package/dist/experimental.d.ts
CHANGED
|
@@ -14,3 +14,6 @@ export type { TopologyClient, TopologyClientConfig, TopologySource, TopologyView
|
|
|
14
14
|
export type { TopologyMember, TopologyMemberInput, TopologyMemberState, TopologySnapshot, } from './topology-snapshot.js';
|
|
15
15
|
export { TopologyError, TopologyUnavailableError, type TopologyFallbackReason, } from './errors.js';
|
|
16
16
|
export { durabilityTopologyMigrations } from './migrations.js';
|
|
17
|
+
export { DurabilitySchedule, type DurabilityScheduleConfig, type DurabilityScheduleEvent, type ScheduleDefinition, type ScheduledOutcomeDelivery, type ScheduledRun, type ScheduledRunContext, type ScheduledRunOutcome, type ScheduleState, } from './schedule.js';
|
|
18
|
+
export { ScheduledOutcomeTimeoutError, ScheduledRunInterruptedError, ScheduledRunTimeoutError, } from './errors.js';
|
|
19
|
+
export { durabilityScheduleMigrations } from './migrations.js';
|
package/dist/experimental.js
CHANGED
|
@@ -11,3 +11,6 @@ export { durabilityOutboxMigrations } from './migrations.js';
|
|
|
11
11
|
export { DurabilityTopology, } from './topology.js';
|
|
12
12
|
export { TopologyError, TopologyUnavailableError, } from './errors.js';
|
|
13
13
|
export { durabilityTopologyMigrations } from './migrations.js';
|
|
14
|
+
export { DurabilitySchedule, } from './schedule.js';
|
|
15
|
+
export { ScheduledOutcomeTimeoutError, ScheduledRunInterruptedError, ScheduledRunTimeoutError, } from './errors.js';
|
|
16
|
+
export { durabilityScheduleMigrations } from './migrations.js';
|
package/dist/fanout.js
CHANGED
|
@@ -338,13 +338,11 @@ _DurabilityFanout_engine = new WeakMap(), _DurabilityFanout_routing = new WeakMa
|
|
|
338
338
|
exhausted.push({ row, message });
|
|
339
339
|
continue;
|
|
340
340
|
}
|
|
341
|
-
const attempt = claimAttempt(sql, deliveriesTable, row.id, row.generation_id, policy.maxAttempts);
|
|
341
|
+
const attempt = claimAttempt(sql, deliveriesTable, row.id, row.generation_id, policy.maxAttempts, deadline);
|
|
342
342
|
if (attempt === undefined) {
|
|
343
343
|
continue;
|
|
344
344
|
}
|
|
345
|
-
|
|
346
|
-
updateAttempt(sql, deliveriesTable, attemptOf(claim.row), 'next_attempt_at = ?', deadline);
|
|
347
|
-
claimed.push(claim);
|
|
345
|
+
claimed.push({ row: { ...row, attempt }, message });
|
|
348
346
|
}
|
|
349
347
|
return { claims: claimed, terminal: exhausted };
|
|
350
348
|
});
|
package/dist/index.d.ts
CHANGED
|
@@ -5,7 +5,7 @@ export { DurabilityFanout, type DurabilityFanoutConfig, type DurabilityFanoutEve
|
|
|
5
5
|
export { DurabilityLog, maxLogBytes, type DurabilityLogConfig, type LogAppendInput, type LogColdStorage, type LogLease, type LogLeaseOptions, type LogPage, type LogReadOptions, type LogRecord, type LogRetention, type LogSegment, type LogBounds, } from './log.js';
|
|
6
6
|
export { RoutingLoad, type EnqueueResult, type LoadSnapshot, type ProcessingLoad, } from './load.js';
|
|
7
7
|
export type { MessageWrite, StoredMessage } from './stored-message.js';
|
|
8
|
-
export { DurabilityScheduler, type DurabilityContext, type DurabilitySchedulerConfig, type DurabilitySchedulerOptions, type SchedulerAttachment, } from './scheduler.js';
|
|
8
|
+
export { DurabilityScheduler, type AlarmRetryInfo, type DurabilityContext, type DurabilitySchedulerConfig, type DurabilitySchedulerOptions, type SchedulerAttachment, } from './scheduler.js';
|
|
9
9
|
export { DuplicateDurableCallError, DuplicateDurablePayloadError, DurableAlarmTimeoutError, DurableAttemptsExhaustedError, DurableAttemptTimeoutError, DurableResultSerializationError, DurableRetryPolicyError, DurableMemoLimitError, DurableMemoSerializationError, NonRetryableError, FanoutTimeoutError, FanoutSettlementError, FanoutEnqueueError, LogAppendError, LogLeaseError, LogTruncatedError, RoutingError, } from './errors.js';
|
|
10
10
|
export type { DurabilityLifecycleEvent, LifecycleHook } from './lifecycle.js';
|
|
11
11
|
export { durabilityNamedAlarmMigrations, durabilityOperationMigrations, durabilityFanoutMigrations, durabilityLogLeaseMigrations, durabilityLogMigrations, durabilityMemoMigrations, migrate, type DurabilityMigrationResult, type MigrateOptions, type MigrationCapability, } from './migrations.js';
|
package/dist/migrations.d.ts
CHANGED
|
@@ -102,6 +102,15 @@ export declare const durabilityTopologyMigrations: {
|
|
|
102
102
|
up: string;
|
|
103
103
|
down: string;
|
|
104
104
|
}[];
|
|
105
|
+
/**
|
|
106
|
+
* Ordered SQLite migrations for the experimental schedule. Applied
|
|
107
|
+
* automatically on construction.
|
|
108
|
+
*/
|
|
109
|
+
export declare const durabilityScheduleMigrations: {
|
|
110
|
+
name: string;
|
|
111
|
+
up: string;
|
|
112
|
+
down: string;
|
|
113
|
+
}[];
|
|
105
114
|
/** Migration names changed by one migrate call. */
|
|
106
115
|
export type DurabilityMigrationResult = {
|
|
107
116
|
/** Migrations applied in ascending order. */
|
|
@@ -109,7 +118,7 @@ export type DurabilityMigrationResult = {
|
|
|
109
118
|
/** Migrations reverted in descending order. */
|
|
110
119
|
rolledBack: string[];
|
|
111
120
|
};
|
|
112
|
-
export type MigrationCapability = 'operations' | 'namedAlarms' | 'memos' | 'fanout' | 'socket' | 'log' | 'logLeases' | 'outbox' | 'semaphore' | 'semaphoreQueue' | 'topology';
|
|
121
|
+
export type MigrationCapability = 'operations' | 'namedAlarms' | 'memos' | 'fanout' | 'socket' | 'log' | 'logLeases' | 'outbox' | 'semaphore' | 'semaphoreQueue' | 'topology' | 'schedule';
|
|
113
122
|
type MigrationStorage = Pick<DurableObjectStorage, 'sql' | 'transactionSync'>;
|
|
114
123
|
/**
|
|
115
124
|
* Whether a capability's history already lists exactly these migrations with
|
package/dist/migrations.js
CHANGED
|
@@ -570,6 +570,49 @@ export const durabilityTopologyMigrations = [
|
|
|
570
570
|
`,
|
|
571
571
|
},
|
|
572
572
|
];
|
|
573
|
+
/**
|
|
574
|
+
* Ordered SQLite migrations for the experimental schedule. Applied
|
|
575
|
+
* automatically on construction.
|
|
576
|
+
*/
|
|
577
|
+
export const durabilityScheduleMigrations = [
|
|
578
|
+
{
|
|
579
|
+
name: 'durability_0013_create_schedule',
|
|
580
|
+
// INTEGER PRIMARY KEY aliases the rowid, so each table is a single b-tree
|
|
581
|
+
// with no secondary index for per-run writes to maintain.
|
|
582
|
+
up: `
|
|
583
|
+
CREATE TABLE IF NOT EXISTS durability_schedule (
|
|
584
|
+
id INTEGER PRIMARY KEY CHECK (id = 1),
|
|
585
|
+
version INTEGER NOT NULL,
|
|
586
|
+
expression TEXT,
|
|
587
|
+
next_at INTEGER
|
|
588
|
+
);
|
|
589
|
+
CREATE TABLE IF NOT EXISTS durability_schedule_runs (
|
|
590
|
+
scheduled_time INTEGER PRIMARY KEY,
|
|
591
|
+
state TEXT NOT NULL
|
|
592
|
+
CHECK (state IN ('queued', 'running', 'complete', 'errored')),
|
|
593
|
+
next_at INTEGER NOT NULL,
|
|
594
|
+
attempt INTEGER NOT NULL DEFAULT 0,
|
|
595
|
+
error TEXT
|
|
596
|
+
);
|
|
597
|
+
`,
|
|
598
|
+
down: `
|
|
599
|
+
DROP TABLE IF EXISTS durability_schedule_runs;
|
|
600
|
+
DROP TABLE IF EXISTS durability_schedule;
|
|
601
|
+
`,
|
|
602
|
+
},
|
|
603
|
+
{
|
|
604
|
+
name: 'durability_0014_add_schedule_jitter',
|
|
605
|
+
up: `
|
|
606
|
+
ALTER TABLE durability_schedule ADD COLUMN jitter_ms INTEGER NOT NULL DEFAULT 0;
|
|
607
|
+
ALTER TABLE durability_schedule_runs ADD COLUMN due_at INTEGER;
|
|
608
|
+
UPDATE durability_schedule_runs SET due_at = scheduled_time;
|
|
609
|
+
`,
|
|
610
|
+
down: `
|
|
611
|
+
ALTER TABLE durability_schedule_runs DROP COLUMN due_at;
|
|
612
|
+
ALTER TABLE durability_schedule DROP COLUMN jitter_ms;
|
|
613
|
+
`,
|
|
614
|
+
},
|
|
615
|
+
];
|
|
573
616
|
/**
|
|
574
617
|
* Newer library versions may leave migrations this one does not know about.
|
|
575
618
|
* On downgrade their stored `down` scripts are run, but only this many, so an
|
|
@@ -720,7 +763,7 @@ const migrateScope = (storage, scope, migrations, target) => {
|
|
|
720
763
|
.filter(({ name }) => !applied.has(name));
|
|
721
764
|
const toRefresh = migrations
|
|
722
765
|
.slice(0, targetIndex + 1)
|
|
723
|
-
.filter(({ name, down }) => applied.get(name)?.down !== down);
|
|
766
|
+
.filter(({ name, down }) => applied.has(name) && applied.get(name)?.down !== down);
|
|
724
767
|
storage.transactionSync(() => {
|
|
725
768
|
for (const migration of toRevert) {
|
|
726
769
|
runDown(storage.sql, scope.table, migration.name, migration.down);
|
package/dist/outbox.js
CHANGED
|
@@ -288,12 +288,11 @@ _DurabilityOutbox_engine = new WeakMap(), _DurabilityOutbox_channel = new WeakMa
|
|
|
288
288
|
const deadline = Date.now() + __classPrivateFieldGet(this, _DurabilityOutbox_attemptTimeoutMs, "f");
|
|
289
289
|
const claimed = [];
|
|
290
290
|
for (const row of due) {
|
|
291
|
-
|
|
291
|
+
// A crash mid-publication leaves the claim due again at the deadline.
|
|
292
|
+
const attempt = claimAttempt(sql, outboxTable, row.id, row.generation_id, Number.MAX_SAFE_INTEGER, deadline);
|
|
292
293
|
if (attempt === undefined) {
|
|
293
294
|
continue;
|
|
294
295
|
}
|
|
295
|
-
// A crash mid-publication leaves the claim due again at the deadline.
|
|
296
|
-
updateAttempt(sql, outboxTable, { key: row.id, generation: row.generation_id, attempt }, 'next_attempt_at = ?', deadline);
|
|
297
296
|
claimed.push({ ...row, attempt });
|
|
298
297
|
}
|
|
299
298
|
return claimed;
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
import { type SerializedError } from './errors.js';
|
|
2
|
+
import { type DurabilityMigrationResult } from './migrations.js';
|
|
3
|
+
import { type DurabilityRetryOptions } from './policy.js';
|
|
4
|
+
import { type SchedulerAttachment } from './scheduler.js';
|
|
5
|
+
/** A schedule as passed to {@link DurabilitySchedule.set}. */
|
|
6
|
+
export type ScheduleDefinition = {
|
|
7
|
+
/** Opaque to durability: only `nextOccurrence` interprets it. */
|
|
8
|
+
expression: string;
|
|
9
|
+
/**
|
|
10
|
+
* Orders updates: `set` stores only a version higher than the stored one,
|
|
11
|
+
* and the same version only re-arms a lost alarm.
|
|
12
|
+
*/
|
|
13
|
+
version: number;
|
|
14
|
+
/**
|
|
15
|
+
* Delays every tick by one fixed offset after its nominal time, to spread
|
|
16
|
+
* objects that share an expression. A number is the offset; `{ maxMs }`
|
|
17
|
+
* picks one uniformly in `[0, maxMs]` when the version is stored, so
|
|
18
|
+
* resending that version keeps it. Defaults to 0, at most 2^31 - 1. Keep
|
|
19
|
+
* it below the expression's interval.
|
|
20
|
+
*/
|
|
21
|
+
jitterMs?: number | {
|
|
22
|
+
maxMs: number;
|
|
23
|
+
};
|
|
24
|
+
};
|
|
25
|
+
/** The stored schedule returned by {@link DurabilitySchedule.get}. */
|
|
26
|
+
export type ScheduleState = {
|
|
27
|
+
expression: string;
|
|
28
|
+
version: number;
|
|
29
|
+
/** The stored offset, including one picked from `{ maxMs }`. */
|
|
30
|
+
jitterMs: number;
|
|
31
|
+
/**
|
|
32
|
+
* The next tick's nominal time, or `undefined` once `nextOccurrence`
|
|
33
|
+
* returned none. It fires at `nextAt + jitterMs`.
|
|
34
|
+
*/
|
|
35
|
+
nextAt: number | undefined;
|
|
36
|
+
};
|
|
37
|
+
/** One tick's run. */
|
|
38
|
+
export type ScheduledRun = {
|
|
39
|
+
/** The tick's nominal time, as `nextOccurrence` returned it. */
|
|
40
|
+
scheduledTime: number;
|
|
41
|
+
/**
|
|
42
|
+
* When the tick fired: `scheduledTime` plus the schedule's `jitterMs`. A run
|
|
43
|
+
* that waited for a slot starts later.
|
|
44
|
+
*/
|
|
45
|
+
dueAt: number;
|
|
46
|
+
/**
|
|
47
|
+
* `<objectId>:<scheduledTime>`, or the bare scheduled time when the context
|
|
48
|
+
* has no object ID. A tick fires at most once, so the key is unique per run.
|
|
49
|
+
*/
|
|
50
|
+
idempotencyKey: string;
|
|
51
|
+
};
|
|
52
|
+
/** What `run` receives. */
|
|
53
|
+
export type ScheduledRunContext = ScheduledRun & {
|
|
54
|
+
/** Aborted when the run exceeds `runTimeoutMs`. */
|
|
55
|
+
signal: AbortSignal;
|
|
56
|
+
};
|
|
57
|
+
/** What a run ended with, as passed to `onOutcome`. */
|
|
58
|
+
export type ScheduledRunOutcome = {
|
|
59
|
+
status: 'complete';
|
|
60
|
+
} | {
|
|
61
|
+
status: 'errored';
|
|
62
|
+
error: SerializedError;
|
|
63
|
+
};
|
|
64
|
+
/** The run an `onOutcome` call reports. */
|
|
65
|
+
export type ScheduledOutcomeDelivery = ScheduledRun & {
|
|
66
|
+
/** One-based delivery attempt of this outcome. */
|
|
67
|
+
attempt: number;
|
|
68
|
+
};
|
|
69
|
+
type ScheduledRunEntity = {
|
|
70
|
+
entityKind: 'scheduled_run';
|
|
71
|
+
/** The run's idempotency key. */
|
|
72
|
+
id: string;
|
|
73
|
+
scheduledTime: number;
|
|
74
|
+
timestamp: number;
|
|
75
|
+
};
|
|
76
|
+
/** Best-effort lifecycle signal for one scheduled run. */
|
|
77
|
+
export type DurabilityScheduleEvent = ScheduledRunEntity & ({
|
|
78
|
+
/** A tick recorded the run; `queued` when no run slot was free. */
|
|
79
|
+
type: 'due';
|
|
80
|
+
queued: boolean;
|
|
81
|
+
} | {
|
|
82
|
+
type: 'started';
|
|
83
|
+
} | {
|
|
84
|
+
/**
|
|
85
|
+
* The outcome was recorded. An interrupted run has no `durationMs` and
|
|
86
|
+
* a `ScheduledRunInterruptedError`.
|
|
87
|
+
*/
|
|
88
|
+
type: 'settled';
|
|
89
|
+
outcome: ScheduledRunOutcome['status'];
|
|
90
|
+
error?: SerializedError;
|
|
91
|
+
durationMs?: number;
|
|
92
|
+
} | {
|
|
93
|
+
type: 'outcome_retry';
|
|
94
|
+
attempt: number;
|
|
95
|
+
error: SerializedError;
|
|
96
|
+
nextAttemptAt: number;
|
|
97
|
+
} | {
|
|
98
|
+
/** `onOutcome` failed `maxAttempts` times; the run's row is deleted. */
|
|
99
|
+
type: 'dead_lettered';
|
|
100
|
+
attempt: number;
|
|
101
|
+
error: SerializedError;
|
|
102
|
+
});
|
|
103
|
+
export type DurabilityScheduleConfig = SchedulerAttachment & {
|
|
104
|
+
/**
|
|
105
|
+
* Returns the first tick strictly after `after`, or `undefined` for none.
|
|
106
|
+
* Must be synchronous and deterministic; durability ships no cron parser.
|
|
107
|
+
*/
|
|
108
|
+
nextOccurrence(expression: string, after: number): number | undefined;
|
|
109
|
+
/** The work of one tick. Throwing or timing out makes the run `errored`. */
|
|
110
|
+
run(run: ScheduledRunContext): unknown;
|
|
111
|
+
/**
|
|
112
|
+
* Receives each run's outcome at least once. Returning acknowledges it and
|
|
113
|
+
* deletes the run's row; throwing retries it with `outcomeRetries`.
|
|
114
|
+
*/
|
|
115
|
+
onOutcome(run: ScheduledOutcomeDelivery, outcome: ScheduledRunOutcome): unknown;
|
|
116
|
+
/** Runs in flight at once; later runs wait durably in order. Defaults to 10. */
|
|
117
|
+
maxConcurrentRuns?: number;
|
|
118
|
+
/** Maximum duration of one run. Defaults to 15 minutes. */
|
|
119
|
+
runTimeoutMs?: number;
|
|
120
|
+
/** Maximum duration of one `onOutcome` call. Defaults to 60 seconds. */
|
|
121
|
+
outcomeTimeoutMs?: number;
|
|
122
|
+
/** `onOutcome` retries. `maxAttempts` defaults to 10, counting the first. */
|
|
123
|
+
outcomeRetries?: DurabilityRetryOptions;
|
|
124
|
+
onLifecycleEvent?: (event: DurabilityScheduleEvent) => void | Promise<void>;
|
|
125
|
+
};
|
|
126
|
+
/**
|
|
127
|
+
* Experimental. One recurring schedule per Durable Object whose ticks start
|
|
128
|
+
* individually tracked, overlapping runs and report each run's outcome once,
|
|
129
|
+
* keeping a run's row only until its outcome is delivered.
|
|
130
|
+
*
|
|
131
|
+
* A tick inserts its run in the transaction that advances the stored next
|
|
132
|
+
* tick, so a tick fires at most once without retaining finished runs. Missed
|
|
133
|
+
* ticks collapse into one run for the earliest missed time.
|
|
134
|
+
*/
|
|
135
|
+
export declare class DurabilitySchedule {
|
|
136
|
+
#private;
|
|
137
|
+
constructor(config: DurabilityScheduleConfig);
|
|
138
|
+
static migrate(context: Pick<DurableObjectState, 'storage'>, target?: string | null): DurabilityMigrationResult;
|
|
139
|
+
alarm(info?: AlarmInvocationInfo): Promise<void>;
|
|
140
|
+
/** The stored schedule, or `undefined` when none is set or it was cancelled. */
|
|
141
|
+
get(): ScheduleState | undefined;
|
|
142
|
+
/**
|
|
143
|
+
* Stores the schedule and its next tick when `version` is higher than the
|
|
144
|
+
* stored one, and resolves `true` when it did. A lower version is ignored; the
|
|
145
|
+
* same version changes nothing, keeping its stored jitter, but re-arms the
|
|
146
|
+
* alarm if it was lost.
|
|
147
|
+
*/
|
|
148
|
+
set(definition: ScheduleDefinition): Promise<boolean>;
|
|
149
|
+
/**
|
|
150
|
+
* Stops future ticks and resolves `true` when a schedule was live. Runs
|
|
151
|
+
* already due or running still finish and report. With `version`, the
|
|
152
|
+
* cancellation is ignored when the stored version is higher, and is
|
|
153
|
+
* otherwise recorded so a later `set` of that version or lower is ignored.
|
|
154
|
+
*/
|
|
155
|
+
cancel(version?: number): Promise<boolean>;
|
|
156
|
+
}
|
|
157
|
+
export {};
|