@sigil-dev/plugin-jobs 0.9.29 → 0.9.30

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,102 +1,81 @@
1
1
  # @sigil-dev/plugin-jobs
2
2
 
3
- Durable background jobs for [Grimoire](https://sigil.cane1712.dev/docs): enqueue, retry, recurring schedules, and a `sigil jobs` CLI.
3
+ Durable background jobs for [Grimoire](https://sigil.cane1712.dev/docs): one job per file, typed payloads, retries, recurring jobs, and a `sigil jobs` CLI.
4
4
 
5
5
  Opt-in. Grimoire core knows nothing about it: not installed means no queue, no table, no worker.
6
6
 
7
7
  ## Setup
8
8
 
9
- The queue lives in a database `@sigil-dev/plugin-sql` opened — `"db"` unless
10
- you say otherwise — so there is nothing to connect:
11
-
12
- ```ts
13
- // src/lib/jobs.ts
14
- import { jobs } from "@sigil-dev/plugin-jobs";
15
-
16
- export const queue = jobs({
17
- inlineWorker: true, // run the worker in the server; see "Running it"
18
- handlers: {
19
- "send-welcome": async (p: { email: string }) => sendEmail(p.email),
20
- "rankings.snapshot": async () => snapshotRankings(),
21
- },
22
- schedules: {
23
- "rankings.snapshot": { every: "5m" },
24
- },
25
- });
9
+ ```sh
10
+ sigil add jobs # installs, adds jobs() to sigil.config.ts
11
+ sigil db generate add_jobs && sigil db migrate
12
+ sigil g job send-welcome # src/jobs/send-welcome.ts
26
13
  ```
27
14
 
28
- ```ts
29
- // sigil.config.ts
30
- import { adapter } from "@sigil-dev/plugin-sql/bun";
31
- import { queue } from "./src/lib/jobs.ts";
15
+ By hand, it is one line:
32
16
 
33
- export default {
34
- plugins: [adapter(process.env.DATABASE_URL, { maxTotal: 12 }), queue],
35
- };
17
+ ```ts
18
+ // sigil.config.ts — the queue's table joins your schema through tome
19
+ plugins: [tome(), jobs()]
36
20
  ```
37
21
 
38
- `database: "userDb"` puts the queue in another plugin-sql database. Prefer
39
- passing the adapter itself � `database: userDb`, where
40
- `const userDb = adapter({ key: "userDb", url, optional: true })`. A typo is then
41
- a compile error, and if `userDb` is optional and was not configured the queue
42
- registers nothing: `queue.jobs.enabled` is `false` and `enqueue` throws, so the
43
- config needs no conditional. The
44
- database is looked up when a query runs, so the order of the plugins array
45
- does not matter.
22
+ Postgres and SQLite, which is what tome speaks.
46
23
 
47
- ## Enqueue
24
+ ## A job
48
25
 
49
- From anywhere on the server — import the queue:
26
+ ```ts
27
+ // src/jobs/send-welcome.ts — the job "send-welcome"
28
+ import { defineJob } from "@sigil-dev/plugin-jobs";
29
+ import { schema, t } from "@sigil-dev/tome";
30
+
31
+ export default defineJob({
32
+ payload: schema({ email: t.text().email() }),
33
+ attempts: 3,
34
+ backoff: "1m",
35
+ async handler({ email }, { id, attempt }) {
36
+ await sendEmail(email);
37
+ },
38
+ });
39
+ ```
50
40
 
51
41
  ```ts
52
- import { queue } from "$lib/jobs";
42
+ // anywhere on the server
43
+ import sendWelcome from "$jobs/send-welcome";
53
44
 
54
- await queue.jobs.enqueue("send-welcome", { email: "ada@example.com" });
55
- await queue.jobs.enqueue("send-welcome", { mail: "x" }); // type error: wrong payload
56
- await queue.jobs.enqueue("send-welcom", {}); // type error: no such handler
57
- await queue.jobs.enqueue("rankings.snapshot"); // no payload needed
45
+ await sendWelcome.enqueue({ email: "ada@example.com" });
46
+ await sendWelcome.in("1h 30m").enqueue({ email });
47
+ await sendWelcome.at(tomorrowAt9).enqueue({ email });
48
+ await sendWelcome.enqueue({ mail: "x" }); // type error, and refused at runtime
58
49
  ```
59
50
 
60
- In a route it is also `locals.jobs`. With plugin-sql's `perRequestTx`, a job
61
- enqueued during a request is part of the request's transaction: an order
62
- that rolls back takes its confirmation email with it.
63
-
64
- Options: `{ runAt: Date, maxAttempts: number }` (default 5).
51
+ - **The name is the file**: `src/jobs/mail/digest.ts` is "mail/digest". Files starting with `_` (helpers the jobs share) and tests are not jobs. The folder is compiled to a list (`.grimoire/jobs.ts`) when the app builds, like routes; nothing scans it while serving.
52
+ - **`payload`** is any [Standard Schema](https://standardschema.dev): tome's `schema()` and tables, zod, valibot, arktype. The handler's argument is typed from it, and it is checked twice: when you enqueue, so a bad payload fails where it was made, and when the job runs, so a row an older deploy queued that the job no longer accepts fails for good, with the reason, instead of running on `undefined`. Without `payload`, annotate the handler's argument, or take none.
53
+ - **One payload object**, not positional arguments: queued jobs outlive deploys, and a named field survives a change a position does not. Pass ids, not rows.
54
+ - **`attempts`** (default 5), **`backoff`** (default 30s, doubling, at most an hour) and **`every`** take durations: `"30s"`, `"5m"`, `"1h 30m"`, `"1d"`, or milliseconds. A typo is a compile error.
55
+ - `src/jobs/` is server-only: a browser import of a job fails the build.
65
56
 
66
- ## Handlers
57
+ `locals.jobs.enqueue(name, payload)` queues by name, for when the name is data (an admin screen, a webhook). Jobs that are not files — a package's own, or a test's — go in `jobs({ jobs: { name: job } })`.
67
58
 
68
- ```ts
69
- async (payload, { id, name, attempt }) => { ... }
70
- ```
59
+ ## Context
71
60
 
72
- `id` is stable across retries, so it can key idempotency. `attempt` is 1 on
73
- the first run. A handler that throws is retried after a backoff (30s by
74
- default); after `maxAttempts` it stays in the table as failed, for
75
- `sigil jobs retry`.
61
+ `handler(payload, { id, name, attempt })`. `id` is stable across retries, so it can key idempotency. `attempt` is 1 on the first run. A handler that throws is retried after its backoff; after `attempts` it stays in the table as failed, for `sigil jobs retry`.
76
62
 
77
- ## Schedules
63
+ ## Recurring jobs
78
64
 
79
65
  ```ts
80
- schedules: { "rankings.snapshot": { every: "5m", payload: { region: "jp" } } }
66
+ export default defineJob({ every: "5m", async handler() { await snapshotRankings(); } });
81
67
  ```
82
68
 
83
- `every` is milliseconds or `"30s"`, `"5m"`, `"1h"`, `"1d"`. Each schedule is
84
- one row that is rescheduled when it completes, so:
69
+ Each is one row that is rescheduled when it completes, so:
85
70
 
86
- - **It runs once however many workers there are.** Every worker makes sure
87
- the row exists at boot; the primary key admits one, and one worker claims
88
- it at a time.
89
- - **A restart keeps its clock**, and a deploy does not fire every schedule
90
- at once: the first run is one interval after the row is created.
91
- - **A schedule that keeps failing moves on** to its next run once its
92
- retries are used up, with the error kept in `last_error`.
93
- - **A schedule removed from the config is removed from the table.**
71
+ - **It runs once however many workers there are.** Every worker makes sure the row exists at boot; the primary key admits one, and one worker claims it at a time.
72
+ - **A restart keeps its clock**, and a deploy does not fire every schedule at once: the first run is one interval after the row is created.
73
+ - **One that keeps failing moves on** to its next run once its attempts are used up, with the error kept in `lastError`.
74
+ - **One whose file is gone is removed from the table.**
94
75
 
95
76
  ## Running it
96
77
 
97
- `inlineWorker: true` runs the worker inside the server. That is right for
98
- one process; with `sigil start --scale full=3` every worker also works jobs,
99
- which is safe (claims are race-free) but may be more than you want.
78
+ `jobs({ inlineWorker: true })` runs the worker inside the server. That is right for one process; with `sigil start --scale full=3` every worker also works jobs, which is safe (claims are race-free) but may be more than you want.
100
79
 
101
80
  The alternative is a separate process:
102
81
 
@@ -105,54 +84,21 @@ sigil jobs work # run the worker until stopped (drains on SIGTERM)
105
84
  sigil jobs list # most recent jobs [--limit 20]
106
85
  sigil jobs stats # counts by state
107
86
  sigil jobs retry # requeue every exhausted job, or one: retry <id>
108
- sigil jobs migrate # create the table (also done at boot)
109
- ```
110
-
111
- The CLI opens the database from `sigil.config.ts` itself, so it works
112
- without a running server.
113
-
114
- ## Other stores
115
-
116
- Instead of `database`, pass `sql` — any object with `query(sql, params)`
117
- filling `?` placeholders — for a queue that owns its connection:
118
-
119
- ```ts
120
- import { bunSqlStore } from "@sigil-dev/plugin-jobs";
121
- jobs({ sql: bunSqlStore(process.env.JOBS_URL!), handlers });
122
87
  ```
123
88
 
124
- A store can declare what its dialect can do; everything defaults to false,
125
- so a store that declares nothing takes the path that is correct everywhere.
126
-
127
- | capability | true on | what it changes |
128
- | --- | --- | --- |
129
- | `skipLocked` | postgres, MySQL | appends `FOR UPDATE SKIP LOCKED`, so a worker does not block behind rows another holds. Throughput only. |
130
- | `spaceSeparatedTimestamp` | MySQL | writes `2026-01-01 00:00:00` rather than ISO 8601, which MySQL rejects on a `TIMESTAMP` column. |
131
-
132
- ### Drizzle
133
-
134
- ```ts
135
- import { drizzleStore, bunSqlRunner } from "@sigil-dev/plugin-jobs";
136
- sql: drizzleStore(bunSqlRunner(client), "mysql"),
137
- ```
138
-
139
- This takes the `Bun.SQL` connection, not your Drizzle `db`: Drizzle's
140
- `execute` ignores a parameter array, and its bun-sql driver emits `$1` on
141
- every dialect.
89
+ The CLI opens the database from `sigil.config.ts` itself, so it works without a running server.
142
90
 
143
91
  ## Guarantees, precisely
144
92
 
145
93
  **Delivery is at-least-once.** A worker can die between your side effect and the row delete, so a job may run twice. **Handlers must be idempotent.** The alternative — holding a transaction open across every handler, including every HTTP call inside one — is a worse trade than a duplicate email.
146
94
 
147
- **Order is not guaranteed.** A job that fails and retries runs after jobs enqueued later. Two jobs that write the same thing should carry what they need to be applied in any order.
95
+ **Order is not guaranteed.** A job that fails and retries runs after jobs enqueued later.
148
96
 
149
- **Claiming is race-free without `SKIP LOCKED`.** The claim is an `UPDATE` guarded on the row being unclaimed or stale, tagged with a random token, followed by a `SELECT` of that token — so it works on MariaDB, which has no `UPDATE ... RETURNING`. Two workers racing means the loser gets an empty set, not a duplicate job.
97
+ **Claiming is race-free.** One `UPDATE` picks the due rows in a subquery and claims them with a random token, guarded on each row being unclaimed or stale, so two workers racing means the loser gets an empty batch, not a duplicate job. On postgres the subquery locks `FOR UPDATE SKIP LOCKED`, so a second worker takes the next rows instead of waiting.
150
98
 
151
99
  **A stale claim is reclaimed** after 60s by default, so a hard-killed worker does not strand its jobs.
152
100
 
153
- ## Dialects
154
-
155
- Tested against real SQLite, postgres and MariaDB.
101
+ **Enqueues are transactional.** Every query goes through tome, so with plugin-sql's `perRequestTx` a job enqueued during a request that fails is rolled back with it.
156
102
 
157
103
  ## License
158
104
 
package/define.ts ADDED
@@ -0,0 +1,223 @@
1
+ import { type Duration, ms } from "./duration.ts";
2
+ import type { EnqueueOptions, JobContext } from "./store.ts";
3
+
4
+ /**
5
+ * The part of Standard Schema (standardschema.dev) a payload needs: tome's
6
+ * `schema()` and tables, zod, valibot and arktype all have it.
7
+ */
8
+ export interface StandardSchema<Output = unknown> {
9
+ readonly "~standard": {
10
+ readonly version: 1;
11
+ readonly vendor: string;
12
+ readonly validate: (
13
+ value: unknown,
14
+ ) => StandardResult<Output> | Promise<StandardResult<Output>>;
15
+ readonly types?:
16
+ | { readonly input: unknown; readonly output: Output }
17
+ | undefined;
18
+ };
19
+ }
20
+
21
+ type StandardResult<Output> =
22
+ | { readonly value: Output; readonly issues?: undefined }
23
+ | {
24
+ readonly issues: ReadonlyArray<{
25
+ readonly message: string;
26
+ readonly path?: ReadonlyArray<
27
+ PropertyKey | { readonly key: PropertyKey }
28
+ >;
29
+ }>;
30
+ };
31
+
32
+ type OutputOf<S> = S extends {
33
+ readonly "~standard": { readonly types?: infer T };
34
+ }
35
+ ? NonNullable<T> extends { readonly output: infer O }
36
+ ? O
37
+ : unknown
38
+ : unknown;
39
+
40
+ interface Settings {
41
+ /** Runs before giving up, the first included. @default 5 */
42
+ attempts?: number;
43
+ /**
44
+ * How long to wait before the next try. Default: 30s, doubling each time,
45
+ * at most an hour.
46
+ */
47
+ backoff?: Duration;
48
+ /** Run on this interval, as one row however many workers there are. */
49
+ every?: Duration;
50
+ }
51
+
52
+ /** What a job knows and does, as `defineJob` returns it. */
53
+ export interface Job<P = undefined> {
54
+ /** The job's name: its file in src/jobs, or its key in `jobs({ jobs })`. */
55
+ readonly name: string;
56
+ readonly attempts: number;
57
+ readonly every: number | undefined;
58
+ readonly payload: StandardSchema<P> | undefined;
59
+ /** Milliseconds to wait after the try numbered `attempt` (1 for the first) failed. */
60
+ backoffAfter(attempt: number): number;
61
+ readonly handler: (payload: P, ctx: JobContext) => unknown;
62
+ /**
63
+ * Queue it to run now. The payload is checked against `payload` first,
64
+ * so a bad one fails here, where it was made, rather than in the worker.
65
+ */
66
+ enqueue(...args: EnqueueArgs<P>): Promise<string>;
67
+ /** Run it after a while: `job.in("1h 30m").enqueue(payload)`. */
68
+ in(duration: Duration): Scheduled<P>;
69
+ /** Run it at a time: `job.at(date).enqueue(payload)`. */
70
+ at(date: Date): Scheduled<P>;
71
+ }
72
+
73
+ /** A job with its timing set; `enqueue` queues it. */
74
+ export interface Scheduled<P> {
75
+ enqueue(...args: EnqueueArgs<P>): Promise<string>;
76
+ }
77
+
78
+ type EnqueueArgs<P> = undefined extends P ? [payload?: P] : [payload: P];
79
+
80
+ /** Where `enqueue` sends work: set by the jobs plugin. */
81
+ export interface JobSink {
82
+ enqueue(
83
+ job: Job<any>,
84
+ payload: unknown,
85
+ options: EnqueueOptions,
86
+ ): Promise<string>;
87
+ /** Load the jobs in src/jobs, so a job imported on its own learns its name. */
88
+ discover(): Promise<void>;
89
+ }
90
+
91
+ let sink: JobSink | undefined;
92
+ export function setJobSink(next: JobSink | undefined): void {
93
+ sink = next;
94
+ }
95
+
96
+ const names = new WeakMap<Job<any>, string>();
97
+
98
+ /** Give a job its name. One name per job: a second, different one is a mistake. */
99
+ export function nameJob(job: Job<any>, name: string): void {
100
+ const had = names.get(job);
101
+ if (had !== undefined && had !== name) {
102
+ throw new Error(
103
+ `plugin-jobs: one job is registered as both "${had}" and "${name}". Give each name its own defineJob().`,
104
+ );
105
+ }
106
+ names.set(job, name);
107
+ }
108
+
109
+ /**
110
+ * A job: what it takes, how it retries, and what it does.
111
+ *
112
+ * ```ts
113
+ * // src/jobs/send-welcome.ts
114
+ * export default defineJob({
115
+ * payload: schema({ email: t.text().email() }),
116
+ * attempts: 3,
117
+ * backoff: "1m",
118
+ * async handler({ email }, { attempt }) { ... },
119
+ * });
120
+ *
121
+ * // anywhere on the server
122
+ * await sendWelcome.in("1h").enqueue({ email });
123
+ * ```
124
+ *
125
+ * The payload is one object, not positional arguments: queued jobs outlive
126
+ * deploys, and a named field survives a change that a position does not.
127
+ */
128
+ export function defineJob<S extends StandardSchema<any>>(
129
+ options: Settings & {
130
+ payload: S;
131
+ handler: (payload: OutputOf<S>, ctx: JobContext) => unknown;
132
+ },
133
+ ): Job<OutputOf<S>>;
134
+ export function defineJob<P = undefined>(
135
+ options: Settings & {
136
+ payload?: undefined;
137
+ handler: (payload: P, ctx: JobContext) => unknown;
138
+ },
139
+ ): Job<P>;
140
+ export function defineJob(
141
+ options: Settings & {
142
+ payload?: StandardSchema<any>;
143
+ handler: (payload: any, ctx: JobContext) => unknown;
144
+ },
145
+ ): Job<any> {
146
+ const attempts = options.attempts ?? 5;
147
+ if (!Number.isInteger(attempts) || attempts < 1) {
148
+ throw new Error(
149
+ `plugin-jobs: attempts must be a whole number of at least 1, not ${attempts}`,
150
+ );
151
+ }
152
+ const fixed = options.backoff === undefined ? undefined : ms(options.backoff);
153
+ const every = options.every === undefined ? undefined : ms(options.every);
154
+ if (every !== undefined && every <= 0) {
155
+ throw new Error("plugin-jobs: every must be longer than zero");
156
+ }
157
+ const job: Job<any> = {
158
+ get name() {
159
+ const name = names.get(job);
160
+ if (name === undefined) {
161
+ throw new Error(
162
+ "plugin-jobs: this job has no name yet. Put it in src/jobs, or pass it to jobs({ jobs: { name: job } }).",
163
+ );
164
+ }
165
+ return name;
166
+ },
167
+ attempts,
168
+ every,
169
+ payload: options.payload,
170
+ backoffAfter: (attempt) =>
171
+ fixed ?? Math.min(30_000 * 2 ** Math.max(0, attempt - 1), 3_600_000),
172
+ handler: options.handler,
173
+ enqueue: (payload?: unknown) => send(job, payload, {}),
174
+ in: (duration) => {
175
+ const wait = ms(duration);
176
+ return {
177
+ enqueue: (payload?: unknown) => send(job, payload, { in: wait }),
178
+ };
179
+ },
180
+ at: (date) => {
181
+ if (!(date instanceof Date) || Number.isNaN(date.getTime())) {
182
+ throw new Error("plugin-jobs: at() takes a valid Date");
183
+ }
184
+ return {
185
+ enqueue: (payload?: unknown) => send(job, payload, { at: date }),
186
+ };
187
+ },
188
+ };
189
+ return job;
190
+ }
191
+
192
+ async function send(
193
+ job: Job<any>,
194
+ payload: unknown,
195
+ opts: EnqueueOptions,
196
+ ): Promise<string> {
197
+ if (!sink) {
198
+ throw new Error(
199
+ "plugin-jobs: no queue to enqueue into. Register jobs() in sigil.config.ts.",
200
+ );
201
+ }
202
+ if (!names.has(job)) await sink.discover();
203
+ return sink.enqueue(job, payload, opts);
204
+ }
205
+
206
+ /** Check a payload against a job's schema: the value to store, or the problems. */
207
+ export async function checkPayload(
208
+ job: Job<any>,
209
+ payload: unknown,
210
+ ): Promise<{ ok: true; value: unknown } | { ok: false; message: string }> {
211
+ if (!job.payload) return { ok: true, value: payload };
212
+ const result = await job.payload["~standard"].validate(payload ?? {});
213
+ if (!result.issues) return { ok: true, value: result.value };
214
+ const message = result.issues
215
+ .map((i) => {
216
+ const path = (i.path ?? [])
217
+ .map((p) => String(typeof p === "object" ? p.key : p))
218
+ .join(".");
219
+ return path ? `${path}: ${i.message}` : i.message;
220
+ })
221
+ .join("; ");
222
+ return { ok: false, message };
223
+ }
package/duration.ts ADDED
@@ -0,0 +1,7 @@
1
+ import { type Duration, ms as parse } from "@sigil-dev/grimoire";
2
+
3
+ export type { Duration };
4
+
5
+ /** A duration in milliseconds; see grimoire's `ms`. */
6
+ export const ms = (duration: Duration): number =>
7
+ parse(duration, "plugin-jobs");