@structure-ai/jobs 0.0.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Ligerian Labs
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,67 @@
1
+ # @structure-ai/jobs
2
+
3
+ Delayed and recurring jobs on PostgreSQL. Jobs are named, schema-typed handlers registered at boot; the queue is a pair of tables dispatched with `SELECT … FOR UPDATE SKIP LOCKED`, heartbeat leases, at-least-once delivery, bounded jittered retries, and dead letters. Workers drain gracefully through the `@structure-ai/runtime` Shutdown coordinator, and every execution carries the scheduling site's correlation id.
4
+
5
+ ## Quick start
6
+
7
+ ```ts
8
+ import { defineJob, jobsReadinessCheck, layer, workerLayer, Scheduler } from "@structure-ai/jobs";
9
+ import { Effect, Layer, Schema } from "effect";
10
+
11
+ const SendDigest = defineJob(
12
+ {
13
+ name: "digest.send",
14
+ payloadSchema: Schema.parseJson(Schema.Struct({ userId: Schema.String })),
15
+ },
16
+ (payload) => sendDigest(payload.userId),
17
+ { maxAttempts: 5 },
18
+ );
19
+
20
+ const Jobs = layer({ url: process.env.DATABASE_URL }); // PgClient + migrate + Scheduler
21
+ const Worker = workerLayer({ role: "all" }); // drains on shutdown
22
+
23
+ // Scheduling (from commands, HTTP handlers, anywhere):
24
+ // yield* scheduler.schedule(SendDigest, { userId }, { delay: "5 minutes" });
25
+ // yield* scheduler.recur(SendDigest, { userId }, { cron: "0 9 * * mon-fri", timezone: "Europe/Paris" });
26
+ ```
27
+
28
+ ## Handlers, scheduling, cancellation
29
+
30
+ ```ts
31
+ const scheduler = yield* Scheduler;
32
+
33
+ yield* scheduler.register(SendDigest); // named handlers, idempotent per name
34
+ const id = yield* scheduler.schedule(SendDigest, { userId: "u-1" }, { delay: "1 hour" });
35
+ yield* scheduler.recur(SendDigest, { userId: "u-1" }, {
36
+ cron: "*/5 * * * *", // 5-field cron: lists, ranges, steps, day names
37
+ timezone: "UTC", // IANA names, DST-aware
38
+ scheduleKey: "digest-u-1", // one row per key; recur replaces it
39
+ });
40
+ yield* scheduler.cancel(id);
41
+ ```
42
+
43
+ Payloads are `Schema<P, string>` (usually `Schema.parseJson(...)`) — stored as text, decoded before each execution; a payload that no longer decodes dead-letters immediately.
44
+
45
+ ## Delivery semantics
46
+
47
+ - **At-least-once.** Handlers must be idempotent or dedupe (e.g. through `@structure-ai/eventsourcing`'s Inbox). A lease that expires (worker crash, GC pause) makes the run reclaimable — the job may execute twice.
48
+ - **Retries.** Handlers fail with `{ reason, classification }`: `transient` failures retry with exponential jittered backoff (1s base, 5min cap) up to `maxAttempts` (default 5); `permanent` failures dead-letter on the spot.
49
+ - **Dead letters.** Exhausted or permanently failed jobs move to `jobs_dead_letters` with their attempts, last error (bounded), and correlation id — inspectable with plain SQL.
50
+ - **Cron missed-run policy: skip.** The next occurrence is computed strictly after `max(scheduled, now)` — downtime never produces a catch-up burst.
51
+ - **`JobContext.atLeastOnce: true`** is carried into every execution as standing guidance.
52
+
53
+ ## Worker lifecycle and roles
54
+
55
+ `workerLayer({ role })` follows the platform's `SERVICE_ROLE` convention: `api` processes only schedule (the layer logs and does nothing), `worker`/`all` fork the dispatch loop. The loop registers a `Shutdown` finalizer: on shutdown it stops claiming, waits for in-flight handlers (bounded by the coordinator's finalizer timeout), then exits — no in-flight job is killed. `jobsSettings` (`@structure-ai/config`) maps `SERVICE_ROLE`, `JOBS_POLL_INTERVAL`, `JOBS_BATCH_SIZE`, `JOBS_LEASE`, `JOBS_TABLE_PREFIX`.
56
+
57
+ ## Observability
58
+
59
+ Per-job metrics under bounded, handler-derived names: `job_<name>_calls_total` / `_errors_total` / `_duration_ms` (via `Metrics.track`), plus `jobs_dispatched_total`, `jobs_succeeded_total`, `jobs_retried_total`, `jobs_dead_lettered_total` and the `jobs_queue_depth` gauge. Structured logs per scheduling and per attempt carry job id, name, attempt, classification, bounded reason, and the correlation id captured at the scheduling site (`Correlation.within` at dispatch). `jobsReadinessCheck(scheduler, { maxDepth, maxLagMillis })` reports queue depth/lag into `/health/ready`.
60
+
61
+ ## Errors
62
+
63
+ `UnknownJob`, `InvalidJobPayload` (permanent), `JobQueueError` (transient), `InvalidCronExpression` (permanent, lists every problem). All classified per the framework taxonomy.
64
+
65
+ ## Schema
66
+
67
+ Two tables created by the idempotent `migrate` (own prefix, `@structure-ai/migrations`-compatible DDL): `jobs_queue` (status `queued|running`, `run_at`, `cron_expr`, `cron_timezone`, `attempt`, `max_attempts`, `lease_expires_at`, `last_error`, correlation fields) with dispatch and lease indexes, and `jobs_dead_letters`.
package/package.json ADDED
@@ -0,0 +1,38 @@
1
+ {
2
+ "name": "@structure-ai/jobs",
3
+ "version": "0.0.10",
4
+ "description": "Delayed and recurring jobs on PostgreSQL: named schema-typed handlers, SKIP LOCKED dispatch with heartbeat leases, cron scheduling with timezones, at-least-once delivery, dead letters, graceful drain, per-job metrics.",
5
+ "license": "MIT",
6
+ "type": "module",
7
+ "repository": {
8
+ "type": "git",
9
+ "url": "git+https://github.com/Ligerian-labs/structure.git",
10
+ "directory": "packages/jobs"
11
+ },
12
+ "exports": {
13
+ ".": "./src/index.ts"
14
+ },
15
+ "files": [
16
+ "src",
17
+ "README.md"
18
+ ],
19
+ "publishConfig": {
20
+ "access": "public"
21
+ },
22
+ "scripts": {
23
+ "typecheck": "tsc --noEmit",
24
+ "test": "bun test"
25
+ },
26
+ "dependencies": {
27
+ "@structure-ai/config": "0.0.10",
28
+ "@structure-ai/observability": "0.0.10",
29
+ "@structure-ai/runtime": "0.0.10",
30
+ "@effect/sql": "^0.52.1",
31
+ "@effect/sql-pg": "^0.53.0",
32
+ "effect": "^3.22.1"
33
+ },
34
+ "devDependencies": {
35
+ "@types/bun": "^1.3.14",
36
+ "typescript": "^5.9.2"
37
+ }
38
+ }
package/src/cron.ts ADDED
@@ -0,0 +1,295 @@
1
+ import { Data, Effect } from "effect";
2
+
3
+ /** A cron expression is invalid (unknown field, out of range, bad syntax). */
4
+ export class InvalidCronExpression extends Data.TaggedError("InvalidCronExpression")<{
5
+ readonly expression: string;
6
+ readonly problems: ReadonlyArray<string>;
7
+ }> {
8
+ readonly classification: "permanent" = "permanent";
9
+ override get message(): string {
10
+ return `invalid cron expression "${this.expression}": ${this.problems.join("; ")}`;
11
+ }
12
+ }
13
+
14
+ /** A 5-field cron schedule: minute hour day-of-month month day-of-week. */
15
+ export interface CronFields {
16
+ readonly minutes: ReadonlyArray<number>;
17
+ readonly hours: ReadonlyArray<number>;
18
+ readonly daysOfMonth: ReadonlyArray<number>;
19
+ readonly months: ReadonlyArray<number>;
20
+ readonly daysOfWeek: ReadonlyArray<number>;
21
+ /** `true` when day-of-month is unrestricted (`*`) — matters when dow is restricted. */
22
+ readonly domRestricted: boolean;
23
+ readonly dowRestricted: boolean;
24
+ }
25
+
26
+ const RANGES: ReadonlyArray<[string, number, number]> = [
27
+ ["minute", 0, 59],
28
+ ["hour", 0, 23],
29
+ ["day-of-month", 1, 31],
30
+ ["month", 1, 12],
31
+ ["day-of-week", 0, 6],
32
+ ];
33
+
34
+ const DAY_NAMES: Readonly<Record<string, number>> = {
35
+ sun: 0,
36
+ mon: 1,
37
+ tue: 2,
38
+ wed: 3,
39
+ thu: 4,
40
+ fri: 5,
41
+ sat: 6,
42
+ };
43
+
44
+ const parseField = (
45
+ field: string,
46
+ name: string,
47
+ min: number,
48
+ max: number,
49
+ ): Effect.Effect<ReadonlyArray<number>, string> =>
50
+ Effect.gen(function* () {
51
+ const values = new Set<number>();
52
+ for (const part of field.split(",")) {
53
+ const stepMatch = /^(\*|\d+-\d+|\d+)(?:\/(\d+))?$/u.exec(part.trim());
54
+ if (stepMatch === null) {
55
+ return yield* Effect.fail(`${name}: cannot parse "${part}"`);
56
+ }
57
+ const rangePart = stepMatch[1] ?? "";
58
+ const step = stepMatch[2] === undefined ? 1 : Number(stepMatch[2]);
59
+ if (!Number.isInteger(step) || step < 1) {
60
+ return yield* Effect.fail(`${name}: step must be a positive integer`);
61
+ }
62
+ let from = min;
63
+ let to = max;
64
+ if (rangePart !== "*") {
65
+ if (rangePart.includes("-")) {
66
+ const [rawFrom, rawTo] = rangePart.split("-");
67
+ from = Number(rawFrom);
68
+ to = Number(rawTo);
69
+ } else {
70
+ from = Number(rangePart);
71
+ to = rangePart.includes("/") ? max : from;
72
+ }
73
+ }
74
+ if (!Number.isInteger(from) || !Number.isInteger(to) || from < min || to > max || from > to) {
75
+ return yield* Effect.fail(`${name}: values must be within ${min}-${max}`);
76
+ }
77
+ for (let value = from; value <= to; value += step) values.add(value);
78
+ }
79
+ return [...values].sort((a, b) => a - b);
80
+ });
81
+
82
+ /**
83
+ * Parses a standard 5-field cron expression: `*`, lists (`1,15`), ranges
84
+ * (`1-5`), steps (every-10th `0/10`, range-stepped `5-40/5`). Day-of-week:
85
+ * 0 and 7 are Sunday.
86
+ */
87
+ export const parseCron = (expression: string): Effect.Effect<CronFields, InvalidCronExpression> =>
88
+ Effect.gen(function* () {
89
+ const parts = expression.trim().split(/\s+/u);
90
+ if (parts.length !== 5) {
91
+ return yield* new InvalidCronExpression({
92
+ expression,
93
+ problems: [
94
+ `expected 5 fields (minute hour day-of-month month day-of-week), got ${parts.length}`,
95
+ ],
96
+ });
97
+ }
98
+ const problems: Array<string> = [];
99
+ const parsed: Array<ReadonlyArray<number>> = [];
100
+ for (let index = 0; index < parts.length; index++) {
101
+ const [name, min, max] = RANGES[index] ?? ["field", 0, 0];
102
+ const part = parts[index] ?? "*";
103
+ const field = part
104
+ .replace(/^7$/u, "0")
105
+ .replace(/\b(sun|mon|tue|wed|thu|fri|sat)\b/giu, (name) =>
106
+ String(DAY_NAMES[name.toLowerCase()] ?? name),
107
+ );
108
+ const result = yield* parseField(field, name, min, max).pipe(Effect.either);
109
+ if (result._tag === "Left") {
110
+ problems.push(result.left);
111
+ parsed.push([]);
112
+ } else {
113
+ parsed.push(result.right);
114
+ }
115
+ }
116
+ if (problems.length > 0) {
117
+ return yield* new InvalidCronExpression({ expression, problems });
118
+ }
119
+ const [minutes, hours, daysOfMonth, months, daysOfWeek] = parsed as [
120
+ ReadonlyArray<number>,
121
+ ReadonlyArray<number>,
122
+ ReadonlyArray<number>,
123
+ ReadonlyArray<number>,
124
+ ReadonlyArray<number>,
125
+ ];
126
+ return {
127
+ minutes,
128
+ hours,
129
+ daysOfMonth,
130
+ months,
131
+ daysOfWeek,
132
+ domRestricted: daysOfMonth.length !== 31,
133
+ dowRestricted: daysOfWeek.length !== 7,
134
+ };
135
+ });
136
+
137
+ const DAYS = ["sun", "mon", "tue", "wed", "thu", "fri", "sat"] as const;
138
+
139
+ /** Human-readable summary, e.g. `at 09:30 every mon-fri`. */
140
+ export const describeCron = (fields: CronFields): string => {
141
+ const time =
142
+ fields.hours.length === 24 && fields.minutes.length === 60
143
+ ? "every minute"
144
+ : fields.hours
145
+ .flatMap((hour) =>
146
+ fields.minutes.map(
147
+ (minute) => `${String(hour).padStart(2, "0")}:${String(minute).padStart(2, "0")}`,
148
+ ),
149
+ )
150
+ .join(", ");
151
+ const days = fields.dowRestricted
152
+ ? fields.daysOfWeek.map((day) => DAYS[day] ?? String(day)).join(", ")
153
+ : "every day";
154
+ const months = fields.months.length === 12 ? "" : ` in months ${fields.months.join(",")}`;
155
+ const dom = fields.domRestricted ? ` on days ${fields.daysOfMonth.join(",")}` : "";
156
+ return `at ${time} ${days}${dom}${months}`;
157
+ };
158
+
159
+ interface WallClock {
160
+ readonly year: number;
161
+ readonly month: number; // 1-12
162
+ readonly day: number;
163
+ readonly hour: number;
164
+ readonly minute: number;
165
+ readonly weekday: number; // 0=Sunday
166
+ }
167
+
168
+ const wallClockFormatter = (timeZone: string): Intl.DateTimeFormat => {
169
+ try {
170
+ return new Intl.DateTimeFormat("en-US", {
171
+ timeZone,
172
+ hour12: false,
173
+ year: "numeric",
174
+ month: "2-digit",
175
+ day: "2-digit",
176
+ hour: "2-digit",
177
+ minute: "2-digit",
178
+ weekday: "short",
179
+ });
180
+ } catch {
181
+ return new Intl.DateTimeFormat("en-US", {
182
+ timeZone: "UTC",
183
+ hour12: false,
184
+ year: "numeric",
185
+ month: "2-digit",
186
+ day: "2-digit",
187
+ hour: "2-digit",
188
+ minute: "2-digit",
189
+ weekday: "short",
190
+ });
191
+ }
192
+ };
193
+
194
+ const weekdayIndex: Readonly<Record<string, number>> = {
195
+ Sun: 0,
196
+ Mon: 1,
197
+ Tue: 2,
198
+ Wed: 3,
199
+ Thu: 4,
200
+ Fri: 5,
201
+ Sat: 6,
202
+ };
203
+
204
+ const wallClockOf = (instant: Date, formatter: Intl.DateTimeFormat): WallClock => {
205
+ const parts = formatter.formatToParts(instant);
206
+ const pick = (type: string): number => {
207
+ const value = parts.find((part) => part.type === type)?.value ?? "0";
208
+ return Number(value);
209
+ };
210
+ const weekday = weekdayIndex[parts.find((part) => part.type === "weekday")?.value ?? "Sun"] ?? 0;
211
+ const hour = pick("hour") % 24;
212
+ return {
213
+ year: pick("year"),
214
+ month: pick("month"),
215
+ day: pick("day"),
216
+ hour,
217
+ minute: pick("minute"),
218
+ weekday,
219
+ };
220
+ };
221
+
222
+ /** UTC instant for a wall-clock time in `timeZone` (two-pass DST offset). */
223
+ const instantFromWallClock = (
224
+ clock: Omit<WallClock, "weekday">,
225
+ formatter: Intl.DateTimeFormat,
226
+ ): Date => {
227
+ const guess = Date.UTC(clock.year, clock.month - 1, clock.day, clock.hour, clock.minute, 0, 0);
228
+ const wallAtGuess = wallClockOf(new Date(guess), formatter);
229
+ const wallAsUtc = Date.UTC(
230
+ wallAtGuess.year,
231
+ wallAtGuess.month - 1,
232
+ wallAtGuess.day,
233
+ wallAtGuess.hour,
234
+ wallAtGuess.minute,
235
+ 0,
236
+ 0,
237
+ );
238
+ const offset = wallAsUtc - guess;
239
+ return new Date(guess - offset);
240
+ };
241
+
242
+ const dayMatches = (clock: WallClock, fields: CronFields): boolean => {
243
+ if (!fields.months.includes(clock.month)) return false;
244
+ const domOk = fields.daysOfMonth.includes(clock.day);
245
+ const dowOk = fields.daysOfWeek.includes(clock.weekday);
246
+ // POSIX cron: when both dom and dow are restricted, either may match.
247
+ if (fields.domRestricted && fields.dowRestricted) return domOk || dowOk;
248
+ if (fields.domRestricted) return domOk;
249
+ if (fields.dowRestricted) return dowOk;
250
+ return true;
251
+ };
252
+
253
+ const SEARCH_LIMIT_DAYS = 400;
254
+
255
+ /**
256
+ * The next fire time strictly after `after`, in the expression's timezone
257
+ * (default UTC). Missed occurrences are skipped — the scheduler never
258
+ * bursts to catch up on time that already passed while it was down.
259
+ */
260
+ export const nextRun = (after: Date, fields: CronFields, timeZone = "UTC"): Date | undefined => {
261
+ const formatter = wallClockFormatter(timeZone);
262
+ const start = wallClockOf(after, formatter);
263
+ for (let dayOffset = 0; dayOffset <= SEARCH_LIMIT_DAYS; dayOffset++) {
264
+ // Walk whole days: from the minute after `after` on day 0, then full days.
265
+ const base = instantFromWallClock(
266
+ {
267
+ year: start.year,
268
+ month: start.month,
269
+ day: start.day,
270
+ hour: 0,
271
+ minute: 0,
272
+ },
273
+ formatter,
274
+ );
275
+ const dayStart = new Date(base.getTime() + dayOffset * 86_400_000);
276
+ const dayClock = wallClockOf(dayStart, formatter);
277
+ if (!dayMatches(dayClock, fields)) continue;
278
+ for (const hour of fields.hours) {
279
+ for (const minute of fields.minutes) {
280
+ const candidate = instantFromWallClock(
281
+ {
282
+ year: dayClock.year,
283
+ month: dayClock.month,
284
+ day: dayClock.day,
285
+ hour,
286
+ minute,
287
+ },
288
+ formatter,
289
+ );
290
+ if (candidate.getTime() > after.getTime()) return candidate;
291
+ }
292
+ }
293
+ }
294
+ return undefined;
295
+ };
package/src/index.ts ADDED
@@ -0,0 +1,40 @@
1
+ /**
2
+ * `@structure-ai/jobs` — delayed and recurring jobs on PostgreSQL: named
3
+ * schema-typed handlers, SKIP LOCKED dispatch with heartbeat leases, cron
4
+ * scheduling with timezone support, at-least-once delivery, bounded jittered
5
+ * retries with dead letters, graceful drain through the Shutdown
6
+ * coordinator, per-job metrics and correlation propagation.
7
+ */
8
+
9
+ export {
10
+ type CronFields,
11
+ describeCron,
12
+ InvalidCronExpression,
13
+ nextRun,
14
+ parseCron,
15
+ } from "./cron.js";
16
+ export { type JobsLayerOptions, layer, workerLayer } from "./layer.js";
17
+ export { type JobsReadinessOptions, jobsReadinessCheck } from "./readiness.js";
18
+ export {
19
+ backoffMillis,
20
+ defineJob,
21
+ InvalidJobPayload,
22
+ type JobContext,
23
+ type JobFailure,
24
+ type JobHandler,
25
+ type JobId,
26
+ JobQueueError,
27
+ type JobRef,
28
+ jobFailure,
29
+ makeScheduler,
30
+ type RecurOptions,
31
+ type ScheduleOptions,
32
+ Scheduler,
33
+ type SchedulerOptions,
34
+ type SchedulerService,
35
+ schedulerLayer,
36
+ UnknownJob,
37
+ type WorkerOptions,
38
+ } from "./scheduler.js";
39
+ export { type AdapterOptions, migrate, type TableNames, tableNames } from "./schema.js";
40
+ export { jobsSettings } from "./settings.js";
package/src/layer.ts ADDED
@@ -0,0 +1,69 @@
1
+ import type * as SqlClient from "@effect/sql/SqlClient";
2
+ import type { SqlError } from "@effect/sql/SqlError";
3
+ import { PgClient } from "@effect/sql-pg";
4
+ import type { Shutdown } from "@structure-ai/runtime";
5
+ import { Effect, Layer, Redacted } from "effect";
6
+ import {
7
+ Scheduler,
8
+ type SchedulerOptions,
9
+ schedulerLayer,
10
+ type WorkerOptions,
11
+ } from "./scheduler.js";
12
+ import { migrate } from "./schema.js";
13
+
14
+ /** All-in-one layer configuration. */
15
+ export interface JobsLayerOptions extends SchedulerOptions {
16
+ /**
17
+ * Postgres connection URL. Defaults to `DATABASE_URL`; when neither is
18
+ * set, the client falls back to libpq defaults.
19
+ */
20
+ readonly url?: string;
21
+ readonly maxConnections?: number;
22
+ readonly applicationName?: string;
23
+ }
24
+
25
+ /**
26
+ * Everything in one layer: a `PgClient` (from `options.url` or
27
+ * `DATABASE_URL`), the schema migration at build, and the `Scheduler`
28
+ * service. The client is exposed for the app's own queries.
29
+ */
30
+ export const layer = (
31
+ options?: JobsLayerOptions,
32
+ ): Layer.Layer<Scheduler | PgClient.PgClient | SqlClient.SqlClient, SqlError> => {
33
+ const url = options?.url ?? process.env.DATABASE_URL;
34
+ const client = PgClient.layer({
35
+ ...(url !== undefined ? { url: Redacted.make(url) } : {}),
36
+ ...(options?.maxConnections !== undefined ? { maxConnections: options.maxConnections } : {}),
37
+ ...(options?.applicationName !== undefined ? { applicationName: options.applicationName } : {}),
38
+ });
39
+ const migrated = Layer.effectDiscard(migrate(options)).pipe(Layer.provideMerge(client));
40
+ return schedulerLayer(options).pipe(Layer.provideMerge(migrated));
41
+ };
42
+
43
+ /**
44
+ * Worker layer: role-aware boot. `api` starts nothing (the process only
45
+ * schedules); `worker` and `all` fork the dispatch loop inside a scope and
46
+ * register a `Shutdown` finalizer that drains it — no in-flight job is
47
+ * killed on shutdown. Compose on top of {@link layer}.
48
+ *
49
+ * ```ts
50
+ * const Jobs = layer({ url: process.env.DATABASE_URL });
51
+ * const Worker = workerLayer({ role: process.env.SERVICE_ROLE as "api" });
52
+ * // Layer.provide(Worker, Jobs) — or merge both into the runtime.
53
+ * ```
54
+ */
55
+ export const workerLayer = (
56
+ options?: WorkerOptions,
57
+ ): Layer.Layer<never, never, Scheduler | Shutdown> =>
58
+ Layer.scopedDiscard(
59
+ Effect.gen(function* () {
60
+ const scheduler = yield* Scheduler;
61
+ if ((options?.role ?? "all") === "api") {
62
+ yield* Effect.logInfo("jobs worker not started (api role)").pipe(
63
+ Effect.annotateLogs({ jobsRole: "api" }),
64
+ );
65
+ return;
66
+ }
67
+ yield* Effect.forkScoped(scheduler.runWorker(options));
68
+ }),
69
+ );
@@ -0,0 +1,30 @@
1
+ import type { ReadinessCheck } from "@structure-ai/runtime";
2
+ import { Effect } from "effect";
3
+ import type { SchedulerService } from "./scheduler.js";
4
+
5
+ export interface JobsReadinessOptions {
6
+ /** Ready fails when the queue holds more than this many jobs. */
7
+ readonly maxDepth?: number;
8
+ /** Ready fails when the oldest due job has waited longer than this. */
9
+ readonly maxLagMillis?: number;
10
+ }
11
+
12
+ /**
13
+ * Readiness check reporting queue health: depth and lag of due work. A
14
+ * process whose scheduler cannot keep up (or whose queue is unreachable)
15
+ * reports not-ready so orchestrators stop routing to it. Defaults are
16
+ * generous: 10_000 depth, 5 minutes of lag.
17
+ */
18
+ export const jobsReadinessCheck = (
19
+ scheduler: SchedulerService,
20
+ options: JobsReadinessOptions = {},
21
+ ): ReadinessCheck => ({
22
+ name: "jobs",
23
+ run: Effect.gen(function* () {
24
+ const depth = yield* scheduler.depth.pipe(Effect.orElseSucceed(() => Number.POSITIVE_INFINITY));
25
+ const lag = yield* scheduler.lagMillis.pipe(
26
+ Effect.orElseSucceed(() => Number.POSITIVE_INFINITY),
27
+ );
28
+ return depth <= (options.maxDepth ?? 10_000) && lag <= (options.maxLagMillis ?? 5 * 60_000);
29
+ }),
30
+ });
@@ -0,0 +1,616 @@
1
+ import * as SqlClient from "@effect/sql/SqlClient";
2
+ import { Correlation, Metrics } from "@structure-ai/observability";
3
+ import { Shutdown } from "@structure-ai/runtime";
4
+ import {
5
+ Context,
6
+ Data,
7
+ Duration,
8
+ Effect,
9
+ Fiber,
10
+ Layer,
11
+ Metric,
12
+ Schema as S,
13
+ Schedule,
14
+ } from "effect";
15
+ import { InvalidCronExpression, nextRun, parseCron } from "./cron.js";
16
+ import { type AdapterOptions, tableNames } from "./schema.js";
17
+
18
+ // --- errors ---------------------------------------------------------------------
19
+
20
+ /** The job reference is not registered in this process. */
21
+ export class UnknownJob extends Data.TaggedError("UnknownJob")<{
22
+ readonly jobName: string;
23
+ }> {
24
+ readonly classification: "permanent" = "permanent";
25
+ override get message(): string {
26
+ return `job "${this.jobName}" is not registered`;
27
+ }
28
+ }
29
+
30
+ /** The payload failed schema decoding. */
31
+ export class InvalidJobPayload extends Data.TaggedError("InvalidJobPayload")<{
32
+ readonly jobName: string;
33
+ readonly reason: string;
34
+ }> {
35
+ readonly classification: "permanent" = "permanent";
36
+ override get message(): string {
37
+ return `payload for job "${this.jobName}" is invalid`;
38
+ }
39
+ }
40
+
41
+ /** Scheduling failed against the queue. Transient. */
42
+ export class JobQueueError extends Data.TaggedError("JobQueueError")<{
43
+ readonly operation: string;
44
+ readonly cause?: unknown;
45
+ }> {
46
+ readonly classification: "transient" = "transient";
47
+ override get message(): string {
48
+ return `job queue failed during ${this.operation}`;
49
+ }
50
+ }
51
+
52
+ /** Storage shape a handler returns to steer retry behavior. */
53
+ export interface JobFailure {
54
+ readonly reason: string;
55
+ readonly classification: "transient" | "permanent";
56
+ }
57
+
58
+ // --- definitions ------------------------------------------------------------------
59
+
60
+ /** Minimal reference for scheduling: a name plus its payload codec schema. */
61
+ export interface JobRef<P> {
62
+ readonly name: string;
63
+ /** Schema between the payload type and its stored JSON text. */
64
+ readonly payloadSchema: S.Schema<P, string>;
65
+ }
66
+
67
+ export interface JobContext {
68
+ readonly jobId: string;
69
+ readonly attempt: number;
70
+ readonly scheduledFor: Date;
71
+ /** Delivery is at-least-once: handlers must be idempotent or dedupe. */
72
+ readonly atLeastOnce: true;
73
+ }
74
+
75
+ /** A named handler registered at boot; the scheduler dispatches into it. */
76
+ export interface JobHandler<P> extends JobRef<P> {
77
+ readonly handle: (payload: P, context: JobContext) => Effect.Effect<void, JobFailure>;
78
+ /** Overrides the default of 5 attempts (transient failures only). */
79
+ readonly maxAttempts?: number;
80
+ }
81
+
82
+ /** Type-erased handler as the scheduler stores it internally. */
83
+ export interface StoredJobHandler {
84
+ readonly name: string;
85
+ readonly payloadSchema: S.Schema<unknown, string>;
86
+ readonly handle: (payload: unknown, context: JobContext) => Effect.Effect<void, JobFailure>;
87
+ readonly maxAttempts?: number;
88
+ }
89
+
90
+ export const defineJob = <P>(
91
+ ref: JobRef<P>,
92
+ handle: JobHandler<P>["handle"],
93
+ options?: { readonly maxAttempts?: number },
94
+ ): JobHandler<P> => ({
95
+ ...ref,
96
+ handle,
97
+ ...(options?.maxAttempts === undefined ? {} : { maxAttempts: options.maxAttempts }),
98
+ });
99
+
100
+ // --- scheduler service --------------------------------------------------------------
101
+
102
+ export type JobId = string;
103
+
104
+ export interface ScheduleOptions {
105
+ /** Delay before the job becomes due. Default: immediately. */
106
+ readonly delay?: Duration.DurationInput;
107
+ /** Overrides the scheduling site's correlation id. */
108
+ readonly correlationId?: string;
109
+ }
110
+
111
+ export interface RecurOptions {
112
+ /** 5-field cron expression, evaluated in `timezone` (default UTC). */
113
+ readonly cron: string;
114
+ readonly timezone?: string;
115
+ /**
116
+ * Stable id for the recurring row: one row per key, `recur` replaces it.
117
+ * Defaults to a fresh uuid (a second call schedules a second row).
118
+ */
119
+ readonly scheduleKey?: string;
120
+ readonly correlationId?: string;
121
+ }
122
+
123
+ export interface WorkerOptions {
124
+ /** Idle poll interval. Default 1s. */
125
+ readonly pollInterval?: Duration.DurationInput;
126
+ /** Claims per poll. Default 10. */
127
+ readonly batchSize?: number;
128
+ /** Lease held while a handler runs; expiry makes the row reclaimable. Default 60s. */
129
+ readonly lease?: Duration.DurationInput;
130
+ /**
131
+ * Role-aware boot: `api` runs no worker (scheduling only), `worker` and
132
+ * `all` do. Default `all`.
133
+ */
134
+ readonly role?: "api" | "worker" | "all";
135
+ }
136
+
137
+ interface QueueRow {
138
+ readonly id: string;
139
+ readonly job_name: string;
140
+ readonly payload: string;
141
+ readonly attempt: number;
142
+ readonly max_attempts: number;
143
+ readonly cron_expr: string | null;
144
+ readonly cron_timezone: string | null;
145
+ readonly correlation_id: string | null;
146
+ readonly run_at: Date | string;
147
+ }
148
+
149
+ export interface SchedulerService {
150
+ /** Registers a named handler. Idempotent per name (last registration wins). */
151
+ readonly register: <P>(handler: JobHandler<P>) => Effect.Effect<void>;
152
+ readonly schedule: <P>(
153
+ job: JobRef<P>,
154
+ payload: P,
155
+ options?: ScheduleOptions,
156
+ ) => Effect.Effect<JobId, InvalidJobPayload | JobQueueError>;
157
+ readonly recur: <P>(
158
+ job: JobRef<P>,
159
+ payload: P,
160
+ options: RecurOptions,
161
+ ) => Effect.Effect<JobId, InvalidCronExpression | InvalidJobPayload | JobQueueError>;
162
+ readonly cancel: (jobId: JobId) => Effect.Effect<void, JobQueueError>;
163
+ /** Queue depth: rows not yet completed (queued + running). */
164
+ readonly depth: Effect.Effect<number, JobQueueError>;
165
+ /** Lag: milliseconds since the oldest due-but-incomplete run. */
166
+ readonly lagMillis: Effect.Effect<number, JobQueueError>;
167
+ /** Names of registered handlers, for boot-time wiring checks. */
168
+ readonly registeredJobs: () => ReadonlyArray<string>;
169
+ /**
170
+ * Runs the worker loop until the Shutdown coordinator triggers, then
171
+ * drains: no in-flight job is interrupted. Claims with
172
+ * `FOR UPDATE SKIP LOCKED`, heartbeats its lease, retries transient
173
+ * failures with jittered backoff, and dead-letters permanent failures and
174
+ * exhausted attempts.
175
+ */
176
+ readonly runWorker: (options?: WorkerOptions) => Effect.Effect<void, never, Shutdown>;
177
+ }
178
+
179
+ export class Scheduler extends Context.Tag("@structure-ai/jobs/Scheduler")<
180
+ Scheduler,
181
+ SchedulerService
182
+ >() {}
183
+
184
+ export interface SchedulerOptions extends AdapterOptions {
185
+ /** Injectable clock for deterministic tests. */
186
+ readonly now?: () => Date;
187
+ readonly random?: () => number;
188
+ }
189
+
190
+ const runAtOf = (row: QueueRow): Date =>
191
+ row.run_at instanceof Date ? row.run_at : new Date(row.run_at);
192
+
193
+ /** Backoff for attempt N (1-based): exponential, capped, ±50% jitter. */
194
+ export const backoffMillis = (
195
+ attempt: number,
196
+ random: () => number,
197
+ baseMillis = 1_000,
198
+ capMillis = 5 * 60_000,
199
+ ): number => {
200
+ const exponential = Math.min(baseMillis * 2 ** (attempt - 1), capMillis);
201
+ return Math.round(exponential * (0.5 + random()));
202
+ };
203
+
204
+ export const makeScheduler = (
205
+ options: SchedulerOptions = {},
206
+ ): Effect.Effect<SchedulerService, never, SqlClient.SqlClient> =>
207
+ Effect.gen(function* () {
208
+ const sql = yield* SqlClient.SqlClient;
209
+ const tables = tableNames(options);
210
+ const now = options.now ?? (() => new Date());
211
+ const random = options.random ?? Math.random;
212
+ const handlers = new Map<string, StoredJobHandler>();
213
+
214
+ const dispatched = Metric.counter("jobs_dispatched_total", { incremental: true });
215
+ const succeeded = Metric.counter("jobs_succeeded_total", { incremental: true });
216
+ const retried = Metric.counter("jobs_retried_total", { incremental: true });
217
+ const deadLettered = Metric.counter("jobs_dead_lettered_total", { incremental: true });
218
+ const boundaries = new Map<string, Metrics.BoundaryMetrics>();
219
+ const boundaryFor = (jobName: string): Metrics.BoundaryMetrics => {
220
+ const existing = boundaries.get(jobName);
221
+ if (existing !== undefined) return existing;
222
+ const created = Metrics.boundary(`job_${jobName}`);
223
+ boundaries.set(jobName, created);
224
+ return created;
225
+ };
226
+
227
+ const queueError = (operation: string, cause: unknown): JobQueueError =>
228
+ new JobQueueError({ operation, cause });
229
+
230
+ const encodePayload = <P>(schema: S.Schema<P, string>, payload: P) =>
231
+ S.encodeUnknown(schema)(payload);
232
+
233
+ const maxAttemptsFor = (jobName: string): number => handlers.get(jobName)?.maxAttempts ?? 5;
234
+
235
+ const deadLetter = (row: QueueRow, reason: string): Effect.Effect<void, JobQueueError> =>
236
+ Effect.gen(function* () {
237
+ yield* sql`
238
+ INSERT INTO ${sql(tables.deadLetters)}
239
+ (id, job_name, payload, attempts, last_error, correlation_id, dead_at)
240
+ VALUES
241
+ (${crypto.randomUUID()}, ${row.job_name}, ${row.payload}, ${row.attempt},
242
+ ${reason.slice(0, 2_048)}, ${row.correlation_id}, ${now().toISOString()})
243
+ `;
244
+ yield* sql`DELETE FROM ${sql(tables.queue)} WHERE id = ${row.id}`;
245
+ yield* Metric.increment(deadLettered);
246
+ yield* Effect.logError("job dead-lettered").pipe(
247
+ Effect.annotateLogs({
248
+ jobId: row.id,
249
+ jobName: row.job_name,
250
+ jobAttempts: row.attempt,
251
+ jobReason: reason.slice(0, 256),
252
+ }),
253
+ );
254
+ }).pipe(Effect.mapError((cause) => queueError("dead-letter", cause)));
255
+
256
+ const completeSuccess = (row: QueueRow): Effect.Effect<void, JobQueueError> =>
257
+ Effect.gen(function* () {
258
+ if (row.cron_expr === null) {
259
+ yield* sql`DELETE FROM ${sql(tables.queue)} WHERE id = ${row.id}`;
260
+ return;
261
+ }
262
+ const fields = yield* parseCron(row.cron_expr);
263
+ const timezone = row.cron_timezone ?? "UTC";
264
+ // Missed-run policy: skip. The next fire is strictly after
265
+ // max(scheduled time, now) — downtime never produces a catch-up burst.
266
+ const next = nextRun(
267
+ new Date(Math.max(runAtOf(row).getTime(), now().getTime())),
268
+ fields,
269
+ timezone,
270
+ );
271
+ if (next === undefined) {
272
+ yield* sql`DELETE FROM ${sql(tables.queue)} WHERE id = ${row.id}`;
273
+ return;
274
+ }
275
+ yield* sql`
276
+ UPDATE ${sql(tables.queue)}
277
+ SET status = 'queued', run_at = ${next.toISOString()}, attempt = 0,
278
+ lease_expires_at = NULL, updated_at = ${now().toISOString()}
279
+ WHERE id = ${row.id}
280
+ `;
281
+ }).pipe(Effect.mapError((cause) => queueError("complete-success", cause)));
282
+
283
+ const rescheduleRetry = (row: QueueRow, reason: string): Effect.Effect<void, JobQueueError> =>
284
+ Effect.gen(function* () {
285
+ const backoff = backoffMillis(row.attempt, random);
286
+ yield* sql`
287
+ UPDATE ${sql(tables.queue)}
288
+ SET status = 'queued', run_at = ${new Date(now().getTime() + backoff).toISOString()},
289
+ lease_expires_at = NULL, last_error = ${reason.slice(0, 2_048)},
290
+ updated_at = ${now().toISOString()}
291
+ WHERE id = ${row.id}
292
+ `;
293
+ yield* Metric.increment(retried);
294
+ yield* Effect.logWarning("job attempt failed, retrying with backoff").pipe(
295
+ Effect.annotateLogs({
296
+ jobId: row.id,
297
+ jobName: row.job_name,
298
+ jobAttempt: row.attempt,
299
+ jobBackoffMillis: backoff,
300
+ jobReason: reason.slice(0, 256),
301
+ }),
302
+ );
303
+ }).pipe(Effect.mapError((cause) => queueError("reschedule-retry", cause)));
304
+
305
+ const claim = (
306
+ batchSize: number,
307
+ leaseMillis: number,
308
+ ): Effect.Effect<ReadonlyArray<QueueRow>, JobQueueError> => {
309
+ const timestamp = now();
310
+ const leaseUntil = new Date(timestamp.getTime() + leaseMillis);
311
+ return sql<QueueRow>`
312
+ WITH picked AS (
313
+ SELECT id FROM ${sql(tables.queue)}
314
+ WHERE (status = 'queued' AND run_at <= ${timestamp.toISOString()})
315
+ OR (status = 'running' AND lease_expires_at <= ${timestamp.toISOString()})
316
+ ORDER BY run_at
317
+ LIMIT ${batchSize}
318
+ FOR UPDATE SKIP LOCKED
319
+ )
320
+ UPDATE ${sql(tables.queue)} queue
321
+ SET status = 'running', attempt = queue.attempt + 1,
322
+ lease_expires_at = ${leaseUntil.toISOString()},
323
+ updated_at = ${timestamp.toISOString()}
324
+ FROM picked
325
+ WHERE queue.id = picked.id
326
+ RETURNING queue.id, queue.job_name, queue.payload, queue.attempt,
327
+ queue.max_attempts, queue.cron_expr, queue.cron_timezone,
328
+ queue.correlation_id, queue.run_at
329
+ `.pipe(
330
+ Effect.mapError((cause) => queueError("claim", cause)),
331
+ Effect.tap((rows) =>
332
+ rows.length === 0 ? Effect.void : Metric.incrementBy(dispatched, rows.length),
333
+ ),
334
+ );
335
+ };
336
+
337
+ const execute = (row: QueueRow, leaseMillis: number): Effect.Effect<void> => {
338
+ const handler: StoredJobHandler | undefined = handlers.get(row.job_name);
339
+ const context: JobContext = {
340
+ jobId: row.id,
341
+ attempt: row.attempt,
342
+ scheduledFor: runAtOf(row),
343
+ atLeastOnce: true,
344
+ };
345
+ const correlation = Correlation.within({
346
+ ...(row.correlation_id === null ? {} : { correlationId: row.correlation_id }),
347
+ causationId: row.id,
348
+ });
349
+
350
+ const heartbeat: Effect.Effect<void> = Effect.gen(function* () {
351
+ const until = new Date(now().getTime() + leaseMillis);
352
+ yield* sql`
353
+ UPDATE ${sql(tables.queue)}
354
+ SET lease_expires_at = ${until.toISOString()}
355
+ WHERE id = ${row.id} AND status = 'running'
356
+ `.pipe(Effect.asVoid);
357
+ }).pipe(
358
+ Effect.asVoid,
359
+ Effect.repeat(Schedule.spaced(`${Math.max(1, Math.floor(leaseMillis / 3))} millis`)),
360
+ Effect.asVoid,
361
+ Effect.catchAllCause(() => Effect.void),
362
+ );
363
+
364
+ const runOutcome: Effect.Effect<void> = Effect.gen(function* () {
365
+ if (handler === undefined) {
366
+ yield* Effect.logError("job dispatched with no registered handler").pipe(
367
+ Effect.annotateLogs({ jobId: row.id, jobName: row.job_name }),
368
+ );
369
+ yield* deadLetter(row, "unknown-job").pipe(Effect.orDie);
370
+ return;
371
+ }
372
+ const decoded = yield* S.decodeUnknown(handler.payloadSchema)(row.payload).pipe(
373
+ Effect.either,
374
+ );
375
+ if (decoded._tag === "Left") {
376
+ yield* deadLetter(row, `invalid-payload: ${String(decoded.left).slice(0, 128)}`).pipe(
377
+ Effect.orDie,
378
+ );
379
+ return;
380
+ }
381
+ const failure = yield* handler
382
+ .handle(decoded.right, context)
383
+ .pipe(Metrics.track(`job_${row.job_name}`, boundaryFor(row.job_name)), Effect.either);
384
+ if (failure._tag === "Right") {
385
+ yield* Metric.increment(succeeded);
386
+ yield* completeSuccess(row).pipe(Effect.orDie);
387
+ return;
388
+ }
389
+ const error = failure.left;
390
+ yield* Effect.logWarning("job attempt failed").pipe(
391
+ Effect.annotateLogs({
392
+ jobId: row.id,
393
+ jobName: row.job_name,
394
+ jobAttempt: row.attempt,
395
+ jobClassification: error.classification,
396
+ jobReason: error.reason.slice(0, 256),
397
+ }),
398
+ );
399
+ const exhausted = row.attempt >= (row.max_attempts || maxAttemptsFor(row.job_name));
400
+ if (error.classification === "permanent" || exhausted) {
401
+ yield* deadLetter(row, error.reason).pipe(Effect.orDie);
402
+ return;
403
+ }
404
+ yield* rescheduleRetry(row, error.reason).pipe(Effect.orDie);
405
+ });
406
+
407
+ return Effect.gen(function* () {
408
+ const heartbeatFiber = yield* Effect.fork(
409
+ heartbeat.pipe(Effect.catchAllCause(() => Effect.void)),
410
+ );
411
+ yield* runOutcome.pipe(
412
+ Effect.ensuring(
413
+ Fiber.interrupt(heartbeatFiber).pipe(
414
+ Effect.catchAllCause(() => Effect.void),
415
+ Effect.asVoid,
416
+ ),
417
+ ),
418
+ correlation,
419
+ );
420
+ });
421
+ };
422
+
423
+ const runWorker = (workerOptions: WorkerOptions = {}): Effect.Effect<void, never, Shutdown> =>
424
+ Effect.gen(function* () {
425
+ const shutdown = yield* Shutdown;
426
+ const pollMillis =
427
+ workerOptions.pollInterval === undefined
428
+ ? 1_000
429
+ : Duration.toMillis(Duration.decode(workerOptions.pollInterval));
430
+ const batchSize = workerOptions.batchSize ?? 10;
431
+ const leaseMillis =
432
+ workerOptions.lease === undefined
433
+ ? 60_000
434
+ : Duration.toMillis(Duration.decode(workerOptions.lease));
435
+
436
+ const inflight = new Set<Fiber.RuntimeFiber<void, unknown>>();
437
+ let stopRequested = false;
438
+
439
+ yield* shutdown.onShutdown(
440
+ "jobs-worker",
441
+ Effect.sync(() => {
442
+ stopRequested = true;
443
+ }).pipe(
444
+ Effect.zipRight(Effect.logInfo("jobs worker draining")),
445
+ Effect.zipRight(
446
+ Effect.whileLoop({
447
+ while: () => inflight.size > 0,
448
+ body: () => Effect.sleep("10 millis"),
449
+ step: () => undefined,
450
+ }),
451
+ ),
452
+ ),
453
+ );
454
+
455
+ const loop: Effect.Effect<void> = Effect.whileLoop({
456
+ while: () => !stopRequested,
457
+ body: () =>
458
+ Effect.gen(function* () {
459
+ const shuttingDown = yield* shutdown.isShuttingDown;
460
+ if (shuttingDown) stopRequested = true;
461
+ if (stopRequested) return;
462
+ const rows = yield* Effect.orDie(claim(batchSize, leaseMillis));
463
+ if (rows.length === 0) {
464
+ yield* Effect.sleep(pollMillis);
465
+ return;
466
+ }
467
+ for (const row of rows) {
468
+ const fiber = yield* Effect.fork(execute(row, leaseMillis));
469
+ inflight.add(fiber);
470
+ void fiber.addObserver(() => inflight.delete(fiber));
471
+ }
472
+ }),
473
+ step: () => undefined,
474
+ });
475
+ yield* loop;
476
+ // Graceful drain: wait for in-flight handlers to finish.
477
+ yield* Effect.whileLoop({
478
+ while: () => inflight.size > 0,
479
+ body: () => Effect.sleep("10 millis"),
480
+ step: () => undefined,
481
+ });
482
+ yield* Effect.logInfo("jobs worker drained").pipe(
483
+ Effect.annotateLogs({ drained: inflight.size }),
484
+ );
485
+ });
486
+
487
+ const service: SchedulerService = {
488
+ register: <P>(handler: JobHandler<P>) =>
489
+ Effect.sync(() => {
490
+ handlers.set(handler.name, handler as unknown as StoredJobHandler);
491
+ }),
492
+ schedule: (job, payload, scheduleOptions) =>
493
+ Effect.gen(function* () {
494
+ const encoded = yield* encodePayload(job.payloadSchema, payload).pipe(
495
+ Effect.mapError(
496
+ (cause): InvalidJobPayload =>
497
+ new InvalidJobPayload({ jobName: job.name, reason: String(cause) }),
498
+ ),
499
+ );
500
+ const jobId = crypto.randomUUID();
501
+ const correlation =
502
+ scheduleOptions?.correlationId ??
503
+ (yield* Effect.map(Correlation.current, (context) => context.correlationId ?? null));
504
+ const fireAt =
505
+ scheduleOptions?.delay === undefined
506
+ ? now()
507
+ : new Date(
508
+ now().getTime() + Duration.toMillis(Duration.decode(scheduleOptions.delay)),
509
+ );
510
+ yield* sql`
511
+ INSERT INTO ${sql(tables.queue)}
512
+ (id, job_name, payload, status, run_at, attempt, max_attempts,
513
+ correlation_id, created_at, updated_at)
514
+ VALUES
515
+ (${jobId}, ${job.name}, ${encoded}, 'queued', ${fireAt.toISOString()}, 0,
516
+ ${maxAttemptsFor(job.name)}, ${correlation},
517
+ ${now().toISOString()}, ${now().toISOString()})
518
+ `.pipe(Effect.mapError((cause) => queueError("schedule", cause)));
519
+ yield* Effect.logInfo("job scheduled").pipe(
520
+ Effect.annotateLogs({ jobId, jobName: job.name, jobRunAt: fireAt.toISOString() }),
521
+ );
522
+ return jobId;
523
+ }),
524
+ recur: (job, payload, recurOptions) =>
525
+ Effect.gen(function* () {
526
+ const fields = yield* parseCron(recurOptions.cron);
527
+ const encoded = yield* encodePayload(job.payloadSchema, payload).pipe(
528
+ Effect.mapError(
529
+ (cause): InvalidJobPayload =>
530
+ new InvalidJobPayload({ jobName: job.name, reason: String(cause) }),
531
+ ),
532
+ );
533
+ const first = nextRun(now(), fields, recurOptions.timezone ?? "UTC");
534
+ if (first === undefined) {
535
+ return yield* new InvalidCronExpression({
536
+ expression: recurOptions.cron,
537
+ problems: ["never fires"],
538
+ });
539
+ }
540
+ const jobId = recurOptions.scheduleKey ?? crypto.randomUUID();
541
+ const correlation =
542
+ recurOptions.correlationId ??
543
+ (yield* Effect.map(Correlation.current, (context) => context.correlationId ?? null));
544
+ yield* sql`
545
+ INSERT INTO ${sql(tables.queue)}
546
+ (id, job_name, payload, status, run_at, cron_expr, cron_timezone, attempt,
547
+ max_attempts, correlation_id, created_at, updated_at)
548
+ VALUES
549
+ (${jobId}, ${job.name}, ${encoded}, 'queued', ${first.toISOString()},
550
+ ${recurOptions.cron}, ${recurOptions.timezone ?? "UTC"}, 0,
551
+ ${maxAttemptsFor(job.name)}, ${correlation},
552
+ ${now().toISOString()}, ${now().toISOString()})
553
+ ON CONFLICT (id) DO UPDATE SET
554
+ payload = excluded.payload,
555
+ run_at = excluded.run_at,
556
+ cron_expr = excluded.cron_expr,
557
+ cron_timezone = excluded.cron_timezone,
558
+ status = 'queued',
559
+ attempt = 0,
560
+ lease_expires_at = NULL,
561
+ correlation_id = excluded.correlation_id,
562
+ updated_at = excluded.updated_at
563
+ `.pipe(Effect.mapError((cause) => queueError("recur", cause)));
564
+ yield* Effect.logInfo("recurring job scheduled").pipe(
565
+ Effect.annotateLogs({
566
+ jobId,
567
+ jobName: job.name,
568
+ jobCron: recurOptions.cron,
569
+ jobNextRun: first.toISOString(),
570
+ }),
571
+ );
572
+ return jobId;
573
+ }),
574
+ cancel: (jobId) =>
575
+ Effect.asVoid(
576
+ sql`DELETE FROM ${sql(tables.queue)} WHERE id = ${jobId}`.pipe(
577
+ Effect.mapError((cause) => queueError("cancel", cause)),
578
+ ),
579
+ ),
580
+ depth: Effect.map(
581
+ sql<{ readonly count: string }>`
582
+ SELECT COUNT(*)::text AS count FROM ${sql(tables.queue)}
583
+ `.pipe(Effect.mapError((cause) => queueError("depth", cause))),
584
+ (rows) => Number(rows[0]?.count ?? "0"),
585
+ ),
586
+ lagMillis: Effect.map(
587
+ sql<{ readonly lag: string | null }>`
588
+ SELECT (EXTRACT(EPOCH FROM (${now().toISOString()}::timestamptz - MIN(run_at))) * 1000)::text AS lag
589
+ FROM ${sql(tables.queue)}
590
+ WHERE status = 'queued' AND run_at <= ${now().toISOString()}
591
+ `.pipe(Effect.mapError((cause) => queueError("lag", cause))),
592
+ (rows) => {
593
+ const lag = rows[0]?.lag;
594
+ return lag === null || lag === undefined ? 0 : Math.max(0, Number(lag));
595
+ },
596
+ ),
597
+ registeredJobs: () => [...handlers.keys()],
598
+ runWorker,
599
+ };
600
+ return service;
601
+ });
602
+
603
+ /** `Scheduler` layer over an existing `SqlClient` (schema must exist). */
604
+ export const schedulerLayer = (
605
+ options?: SchedulerOptions,
606
+ ): Layer.Layer<Scheduler, never, SqlClient.SqlClient> =>
607
+ Layer.effect(Scheduler, makeScheduler(options));
608
+
609
+ /** Decoded failure helper for tests and app-level handlers. */
610
+ export const jobFailure = (
611
+ reason: string,
612
+ classification: JobFailure["classification"] = "transient",
613
+ ): JobFailure => ({
614
+ reason,
615
+ classification,
616
+ });
package/src/schema.ts ADDED
@@ -0,0 +1,72 @@
1
+ import * as SqlClient from "@effect/sql/SqlClient";
2
+ import { Effect } from "effect";
3
+
4
+ export interface AdapterOptions {
5
+ /** Table name prefix. Defaults to `jobs_`. */
6
+ readonly tablePrefix?: string;
7
+ }
8
+
9
+ export interface TableNames {
10
+ readonly queue: string;
11
+ readonly deadLetters: string;
12
+ }
13
+
14
+ export const tableNames = (options: AdapterOptions = {}): TableNames => {
15
+ const prefix = options.tablePrefix ?? "jobs_";
16
+ return {
17
+ queue: `${prefix}queue`,
18
+ deadLetters: `${prefix}dead_letters`,
19
+ };
20
+ };
21
+
22
+ /**
23
+ * Creates the jobs schema in one transaction, `@structure-ai/migrations`
24
+ * style (idempotent DDL a designated migrator can run at boot). The queue
25
+ * index covers the dispatch predicate (`status = queued AND run_at <= now`
26
+ * plus lease-expiry reclaims).
27
+ */
28
+ export const migrate = (
29
+ options: AdapterOptions = {},
30
+ ): Effect.Effect<void, never, SqlClient.SqlClient> =>
31
+ Effect.gen(function* () {
32
+ const sql = yield* SqlClient.SqlClient;
33
+ const tables = tableNames(options);
34
+ yield* sql`
35
+ CREATE TABLE IF NOT EXISTS ${sql(tables.queue)} (
36
+ id TEXT PRIMARY KEY,
37
+ job_name TEXT NOT NULL,
38
+ payload TEXT NOT NULL,
39
+ status TEXT NOT NULL CHECK (status IN ('queued', 'running')),
40
+ run_at TIMESTAMPTZ NOT NULL,
41
+ cron_expr TEXT,
42
+ cron_timezone TEXT,
43
+ attempt INTEGER NOT NULL DEFAULT 0,
44
+ max_attempts INTEGER NOT NULL DEFAULT 5,
45
+ lease_expires_at TIMESTAMPTZ,
46
+ last_error TEXT,
47
+ correlation_id TEXT,
48
+ causation_id TEXT,
49
+ created_at TIMESTAMPTZ NOT NULL,
50
+ updated_at TIMESTAMPTZ NOT NULL
51
+ )
52
+ `;
53
+ yield* sql`
54
+ CREATE INDEX IF NOT EXISTS ${sql(`${tables.queue}_dispatch_idx`)}
55
+ ON ${sql(tables.queue)} (status, run_at)
56
+ `;
57
+ yield* sql`
58
+ CREATE INDEX IF NOT EXISTS ${sql(`${tables.queue}_lease_idx`)}
59
+ ON ${sql(tables.queue)} (status, lease_expires_at)
60
+ `;
61
+ yield* sql`
62
+ CREATE TABLE IF NOT EXISTS ${sql(tables.deadLetters)} (
63
+ id TEXT PRIMARY KEY,
64
+ job_name TEXT NOT NULL,
65
+ payload TEXT NOT NULL,
66
+ attempts INTEGER NOT NULL,
67
+ last_error TEXT,
68
+ correlation_id TEXT,
69
+ dead_at TIMESTAMPTZ NOT NULL
70
+ )
71
+ `;
72
+ }).pipe(Effect.orDie);
@@ -0,0 +1,30 @@
1
+ import { Settings } from "@structure-ai/config";
2
+ import { Duration } from "effect";
3
+
4
+ /**
5
+ * Standard jobs settings. `SERVICE_ROLE` follows the platform convention:
6
+ * `api` processes schedule but never run jobs, `worker` processes only run
7
+ * them, `all` does both.
8
+ */
9
+ export const jobsSettings = Settings.struct({
10
+ role: Settings.literal("SERVICE_ROLE", ["api", "worker", "all"], {
11
+ description: "process role: api schedules only, worker/all also dispatch jobs",
12
+ default: "all",
13
+ }),
14
+ pollInterval: Settings.duration("JOBS_POLL_INTERVAL", {
15
+ description: "worker idle poll interval",
16
+ default: Duration.seconds(1),
17
+ }),
18
+ batchSize: Settings.int("JOBS_BATCH_SIZE", {
19
+ description: "jobs claimed per poll",
20
+ default: 10,
21
+ }),
22
+ lease: Settings.duration("JOBS_LEASE", {
23
+ description: "dispatch lease duration before a running job becomes reclaimable",
24
+ default: Duration.seconds(60),
25
+ }),
26
+ tablePrefix: Settings.string("JOBS_TABLE_PREFIX", {
27
+ description: "jobs table name prefix",
28
+ default: "jobs_",
29
+ }),
30
+ });