@stonyx/cron 0.2.1-beta.121 → 0.2.1-beta.122
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +87 -3
- package/dist/main.js +22 -6
- package/dist/service.d.ts +104 -7
- package/dist/service.js +521 -46
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -47,6 +47,71 @@ When a job is executed, its next trigger time is updated, and it is re-inserted
|
|
|
47
47
|
|
|
48
48
|
> `MinHeap` is also exported as a public subpath (`@stonyx/cron/min-heap`) and can be imported directly for advanced usage.
|
|
49
49
|
|
|
50
|
+
## CronService
|
|
51
|
+
|
|
52
|
+
The default export above is `Cron`: a fire-and-forget interval registry. `@stonyx/cron/service` is a separate, heavier class for jobs that need CRUD, persistence, a run log and error backoff. It is not a drop-in replacement and the two do not share a scheduler.
|
|
53
|
+
|
|
54
|
+
The two classes agree on the guarantee — the same job is never run concurrently with itself, and different jobs may overlap — but not on the mechanism or on what you can observe. `Cron` invokes callbacks fire-and-forget and reports a skipped run only as an ungated `log.warn` line, one per stuck run, never as a value. `CronService` **awaits** `onJobDue`, its return value shapes `status`/`error`/`summary`, and a refused run comes back to the caller as a value that no log setting can suppress — it is not logged. The two therefore agree on one more thing than the contrast suggests: neither can have a *refused* run made silent by `config.cron.log` — `Cron`'s skip warning is ungated, and `CronService`'s refusal is a return value rather than a log line. A job *failure* is the opposite case, and there the two classes genuinely diverge; see [Configuration](#configuration).
|
|
55
|
+
|
|
56
|
+
```js
|
|
57
|
+
import CronService from '@stonyx/cron/service';
|
|
58
|
+
|
|
59
|
+
const service = new CronService();
|
|
60
|
+
service.onJobDue = async (job) => ({ status: 'ok', summary: 'done' });
|
|
61
|
+
|
|
62
|
+
await service.start();
|
|
63
|
+
const job = await service.add({ name: 'Nightly', schedule: { kind: 'every', everyMs: 86_400_000 }, payload: { kind: 'agentTurn', message: 'go' } });
|
|
64
|
+
|
|
65
|
+
const result = await service.run(job.id, 'force');
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
### The `run()` contract
|
|
69
|
+
|
|
70
|
+
`run(id, mode)` resolves with an `ExecuteResult`. It **never** invokes the callback twice for one job, and it will refuse rather than queue:
|
|
71
|
+
|
|
72
|
+
| `status` | `reason` | Meaning |
|
|
73
|
+
| :---------- | :------------------ | :----------------------------------------------------------------------------- |
|
|
74
|
+
| `'ok'` | — | The callback resolved. `summary` and `durationMs` are set. |
|
|
75
|
+
| `'error'` | — | The callback threw or rejected. `error` carries the message; backoff is applied. |
|
|
76
|
+
| `'skipped'` | `'not due'` | `mode` was `'due'` and the job's next run time has not arrived. Use `'force'` to run anyway. |
|
|
77
|
+
| `'skipped'` | `'already running'` | A previous invocation of **this** job has not settled. The call is refused, not queued, and nothing is logged. |
|
|
78
|
+
| `'skipped'` | `'removed'` | The job was removed between the lookup and the claim. The callback did not fire. |
|
|
79
|
+
|
|
80
|
+
`run()` throws (rather than returning a result) when `id` is not a registered job: `Error: Job not found: <id>`.
|
|
81
|
+
|
|
82
|
+
**Concurrency.** A job is bounded to one in-flight invocation on every path — manual `run()` and the timer both claim it first. **Different** jobs are not bounded: the callback is deliberately invoked outside the internal lock, so N concurrent `run()` calls on N distinct jobs produce N concurrent callbacks. The scheduler itself never generates that fan-out (its timer path invokes a due batch sequentially); only a caller can. If you drive `run()` from a request handler, bound it on your side. Taking the callback out of the lock is what stops a callback that never settles from blocking `add`/`update`/`remove`; restoring the bound by putting it back would restore that deadlock.
|
|
83
|
+
|
|
84
|
+
**The residual, stated plainly.** Taking the callback out of the lock fixes
|
|
85
|
+
`add`/`update`/`remove`; it does **not** bound the callback. A callback that
|
|
86
|
+
never settles still holds its job's claim forever, and because the timer's
|
|
87
|
+
re-entrancy latch is only released when the batch settles, it also **stops the
|
|
88
|
+
timer loop for every other job** — silently, with `status()` still reporting
|
|
89
|
+
`started: true`. There is no execution timeout by design, so bounding your own
|
|
90
|
+
callback is your responsibility, exactly as it is for `Cron`. Only a real
|
|
91
|
+
process restart recovers it: `start()` releases a claim whose owner is provably
|
|
92
|
+
gone, but an object still held in this process is deliberately left alone.
|
|
93
|
+
Tracked as #35.
|
|
94
|
+
|
|
95
|
+
### Breaking changes in this line
|
|
96
|
+
|
|
97
|
+
Five consumer-visible changes landed with the phase split (#34). Only the first
|
|
98
|
+
two are visible in the emitted `dist/service.d.ts`; the rest are runtime
|
|
99
|
+
behaviour and a type-checker will not find them for you. The measured
|
|
100
|
+
`dist/service.d.ts` delta against the previous release is exactly four things:
|
|
101
|
+
the class gained `#private;`, `reason` narrowed, five type declarations gained
|
|
102
|
+
`export` (`JobDueResult`, `ExecuteResult`, `ServiceStatus`, `ListOptions` and
|
|
103
|
+
`OnJobDueCallback`), and `SkipReason` was added as a new exported type — it did
|
|
104
|
+
not exist in the previous release, so it gained nothing. `HeapEntry` remains
|
|
105
|
+
unexported. `dist/main.d.ts` is unchanged.
|
|
106
|
+
|
|
107
|
+
1. **`ExecuteResult.reason` narrowed** from `string` to `'not due' | 'already running' | 'removed'`, and gained the `'removed'` member. Comparing it against a literal outside the union, or `switch`ing on one, is now a compile error (`TS2367` / `TS2678`). Assigning it into `string | undefined` and spreading it are unaffected. The type is exported as `SkipReason`.
|
|
108
|
+
2. **`CronService` is nominally typed.** It carries ECMAScript hard-private members, so the declarations emit `#private;` and a structurally hand-built test double no longer assigns to `CronService` (`TS2741: Property '#private' is missing`). The break is one-directional: `class X extends CronService` still compiles, and assigning a real `CronService` to your own hand-written interface still compiles. **Migration:** declare your own interface and depend on that instead of a `CronService`-typed mock.
|
|
109
|
+
3. **`claimJob`, `settleJob` and `executeClaimed` are not published.** Not a change against the previous release — these members did not exist there at all. Listed because they are new internals that look like API and are deliberately unreachable: a claim taken without its matching settle strands the job permanently in-process. Guarded by `test/unit/publish-surface-test.ts`.
|
|
110
|
+
4. **`run()` no longer serializes across jobs** — a runtime change, not a d.ts one. See the concurrency note above.
|
|
111
|
+
5. **`start()` now throws on a row it cannot use.** A `Job` whose `state` is missing or frozen — `structuredClone` + `Object.freeze` is an ordinary defensive rehydration — makes `start()` reject where the previous release resolved and carried on. Measured: previous release resolves and leaves the stale claim in place; this line throws `TypeError: Cannot assign to read only property 'runningAtMs'`. The timer is still armed for the rows loaded before the throw, so this surfaces at your `await` instead of as an unhandled rejection from inside a timer callback later. **Migration:** if your store hands back frozen rows, thaw `state` before passing them, or catch at the `start()` call site.
|
|
112
|
+
|
|
113
|
+
`SkipReason`, `ExecuteResult`, `JobDueResult`, `ServiceStatus`, `ListOptions` and `OnJobDueCallback` are all exported from `@stonyx/cron/service`, so an exhaustive handler over `reason` is expressible.
|
|
114
|
+
|
|
50
115
|
## Configuration
|
|
51
116
|
|
|
52
117
|
Optionally, informational logging and debugging can be controlled through `config.cron`:
|
|
@@ -59,9 +124,28 @@ config.cron = {
|
|
|
59
124
|
config.debug = true; // optional: debug logs for job registration and execution
|
|
60
125
|
```
|
|
61
126
|
|
|
62
|
-
`config.cron.log` gates
|
|
63
|
-
|
|
64
|
-
|
|
127
|
+
`config.cron.log` gates the `log.cron` channel only. **The two classes route job
|
|
128
|
+
failures differently, so what `log: false` costs you depends on which one you are
|
|
129
|
+
running.** Measured on this build, with a callback that throws:
|
|
130
|
+
|
|
131
|
+
| | Failure channel | Records with `log: false` |
|
|
132
|
+
| :--- | :--- | :--- |
|
|
133
|
+
| `Cron` (default export) | ungated `log.error` | emitted — unchanged |
|
|
134
|
+
| `CronService` | **gated** `log.cron` | **none** |
|
|
135
|
+
|
|
136
|
+
- **`Cron`** reports a callback's throw or rejection through `log.error`, and the
|
|
137
|
+
stuck-job warning described above through `log.warn`. Neither is gated, so
|
|
138
|
+
setting `log` to `false` cannot make a dropped or failed execution silent.
|
|
139
|
+
- **`CronService`** reports a job whose `onJobDue` callback throws through the
|
|
140
|
+
**gated** channel, so `log: false` yields **zero** log records for that
|
|
141
|
+
failure. It stays observable as `ExecuteResult.error` and as a run-log row —
|
|
142
|
+
but a job driven off the timer rather than `run()` has no caller to read that
|
|
143
|
+
return value, so the failure is visible only in the run log. Only the timer
|
|
144
|
+
path's *unexpected* internal throw (a fault in the scheduler itself, not in
|
|
145
|
+
your callback) is reported on the ungated `log.error` channel.
|
|
146
|
+
|
|
147
|
+
If you set `log: false` in production and rely on `CronService`, read failures
|
|
148
|
+
from the run log or from `ExecuteResult`, not from the log file.
|
|
65
149
|
|
|
66
150
|
## License
|
|
67
151
|
|
package/dist/main.js
CHANGED
|
@@ -29,12 +29,28 @@ const MAX_LOGGED_ERROR_LENGTH = 512;
|
|
|
29
29
|
* become the literal two characters so the content survives for a reader, and
|
|
30
30
|
* the length cap keeps one pathological value from swamping the file.
|
|
31
31
|
*
|
|
32
|
-
*
|
|
33
|
-
*
|
|
34
|
-
*
|
|
35
|
-
*
|
|
36
|
-
*
|
|
37
|
-
*
|
|
32
|
+
* DIVERGED from the `forLog` in `src/service.ts`, deliberately — this copy is
|
|
33
|
+
* NOT byte-identical to it and must not be folded into it by assuming it is.
|
|
34
|
+
* (An earlier version of this docblock claimed byte-identity; #34's `fac09cb`
|
|
35
|
+
* falsified that, and the two bodies now measure unequal.) `service.ts`'s is
|
|
36
|
+
* TOTAL: it wraps the coercion in `String(value)` and a `try`, returning
|
|
37
|
+
* `'<unrenderable value>'` rather than throwing. This one is not.
|
|
38
|
+
*
|
|
39
|
+
* The divergence is correct on the merits, and the reason is the call site, not
|
|
40
|
+
* the helper. Both of this copy's callers are the two `forLog(...)` calls in
|
|
41
|
+
* `describeError` directly below (`:82`, `:85`), and both are INSIDE that
|
|
42
|
+
* function's own `try`, so a `TypeError` from `value.replace` on a
|
|
43
|
+
* non-string degrades to `'<thrown value could not be rendered>'` and the log
|
|
44
|
+
* record is still produced. `service.ts`'s callers are not so contained: `:700`
|
|
45
|
+
* sits in a bare `catch` with nothing above it, and `:579`'s enclosing `catch`
|
|
46
|
+
* has nothing left to report to — so a throw there destroyed the failure
|
|
47
|
+
* record outright (measured: 0 records for a job that failed). That is why
|
|
48
|
+
* totality was a live defect there and is not one here.
|
|
49
|
+
*
|
|
50
|
+
* Duplicated rather than shared because the two landed on separate branches.
|
|
51
|
+
* The fold is still wanted, but whoever performs it MUST adopt `service.ts`'s
|
|
52
|
+
* total version as the survivor: folding to this one would silently revert
|
|
53
|
+
* `fac09cb`. See #66, which carries the consolidation.
|
|
38
54
|
*/
|
|
39
55
|
function forLog(value, maxLength) {
|
|
40
56
|
const flattened = value.replace(/\r\n|[\r\n\u2028\u2029]/g, '\\n');
|
package/dist/service.d.ts
CHANGED
|
@@ -4,29 +4,37 @@ import RunLog from './run-log.js';
|
|
|
4
4
|
interface HeapEntry extends HeapItem {
|
|
5
5
|
key: string;
|
|
6
6
|
}
|
|
7
|
-
|
|
7
|
+
/**
|
|
8
|
+
* Why a `run()` did not invoke the callback. Exported so a consumer can write a
|
|
9
|
+
* total handler over it: the union is closed and narrowed (it was `string`
|
|
10
|
+
* before #34), so an exhaustive `switch` is now both possible and expected.
|
|
11
|
+
*/
|
|
12
|
+
export type SkipReason = 'not due' | 'already running' | 'removed';
|
|
13
|
+
export interface JobDueResult {
|
|
8
14
|
status?: string;
|
|
9
15
|
error?: string;
|
|
10
16
|
summary?: string;
|
|
11
17
|
}
|
|
12
|
-
interface ExecuteResult {
|
|
18
|
+
export interface ExecuteResult {
|
|
13
19
|
status: string;
|
|
14
20
|
error?: string;
|
|
15
21
|
summary?: string;
|
|
16
22
|
durationMs?: number;
|
|
17
23
|
deleted?: boolean;
|
|
18
|
-
|
|
24
|
+
/** Only set when `status` is `'skipped'`. */
|
|
25
|
+
reason?: SkipReason;
|
|
19
26
|
}
|
|
20
|
-
interface ServiceStatus {
|
|
27
|
+
export interface ServiceStatus {
|
|
21
28
|
started: boolean;
|
|
22
29
|
jobCount: number;
|
|
23
30
|
nextWakeAtMs: number | undefined;
|
|
24
31
|
}
|
|
25
|
-
interface ListOptions {
|
|
32
|
+
export interface ListOptions {
|
|
26
33
|
includeDisabled?: boolean;
|
|
27
34
|
}
|
|
28
|
-
type OnJobDueCallback = (job: Job) => Promise<JobDueResult | void> | JobDueResult | void;
|
|
35
|
+
export type OnJobDueCallback = (job: Job) => Promise<JobDueResult | void> | JobDueResult | void;
|
|
29
36
|
export default class CronService {
|
|
37
|
+
#private;
|
|
30
38
|
jobs: Map<string, Job>;
|
|
31
39
|
heap: MinHeap<HeapEntry>;
|
|
32
40
|
timer: ReturnType<typeof setTimeout> | null;
|
|
@@ -36,7 +44,46 @@ export default class CronService {
|
|
|
36
44
|
onJobDue: OnJobDueCallback | null;
|
|
37
45
|
constructor();
|
|
38
46
|
/**
|
|
39
|
-
* Start the service. Loads jobs from store (if any), arms timer.
|
|
47
|
+
* Start the service. Loads jobs from store (if any), arms timer. A no-op if
|
|
48
|
+
* already started.
|
|
49
|
+
*
|
|
50
|
+
* `initialJobs` crosses a serialization boundary — it is whatever the
|
|
51
|
+
* consumer's store handed back — so `Job[]` is a compile-time claim about
|
|
52
|
+
* runtime data. Three behaviours follow from that and are worth knowing
|
|
53
|
+
* before you call this, because all three are deliberate and two of them
|
|
54
|
+
* differ from a plain "load and arm":
|
|
55
|
+
*
|
|
56
|
+
* 1. WRITES TO `row.state`. A STALE claim (`state.runningAtMs` set by a
|
|
57
|
+
* process that is gone) is released, because nothing else ever will —
|
|
58
|
+
* there is no lease on the field (#35) — and left in place it is a job
|
|
59
|
+
* that is dead forever while `status()` reports it healthy. A LIVE claim,
|
|
60
|
+
* held by an invocation still running in this process, is left alone:
|
|
61
|
+
* releasing it would let the timer start a second concurrent invocation of
|
|
62
|
+
* a job that is already running.
|
|
63
|
+
*
|
|
64
|
+
* 2. THROWS on a row this class cannot use, rather than accepting it. A row
|
|
65
|
+
* whose `state` is missing, or frozen (`structuredClone` + `Object.freeze`
|
|
66
|
+
* is an ordinary defensive rehydration), throws out of `start()` where the
|
|
67
|
+
* caller's own `await` can catch it. The alternative is a `TypeError` from
|
|
68
|
+
* inside a bare timer callback later — an unhandled rejection, and
|
|
69
|
+
* process-fatal under Node's default.
|
|
70
|
+
*
|
|
71
|
+
* 3. ARMS THE TIMER EVEN IF IT THROWS. The rows loaded before the throw are
|
|
72
|
+
* registered and scheduled. Without this, a throw leaves `started: true`
|
|
73
|
+
* (so a retry is a no-op) with jobs in the heap and no timer: nothing ever
|
|
74
|
+
* fires and `status()` still reports healthy.
|
|
75
|
+
*
|
|
76
|
+
* Which of the three you can observe depends on the CONTENT of the rows, not
|
|
77
|
+
* on whether they were deserialized — 1 fires only on a row that already
|
|
78
|
+
* carries a claim, and 2/3 only on a row this class cannot use. Hand it
|
|
79
|
+
* well-formed deserialized rows with no claim set and none of the three is
|
|
80
|
+
* observable. Hand it a deserialized row that DOES carry one and 1 and 2 are
|
|
81
|
+
* exactly what you get: measured, a stale `state.runningAtMs` of 1 comes back
|
|
82
|
+
* `undefined`, and a `structuredClone` + `Object.freeze` row makes `start()`
|
|
83
|
+
* throw `TypeError: Cannot assign to read only property 'runningAtMs'` with
|
|
84
|
+
* the timer still armed behind it. Hand it live `Job` objects this service is
|
|
85
|
+
* currently executing and only 1 is in play, by design — and on those it
|
|
86
|
+
* deliberately does nothing.
|
|
40
87
|
*/
|
|
41
88
|
start(initialJobs?: Job[]): Promise<void>;
|
|
42
89
|
/**
|
|
@@ -69,6 +116,37 @@ export default class CronService {
|
|
|
69
116
|
remove(id: string): Promise<void>;
|
|
70
117
|
/**
|
|
71
118
|
* Manually trigger a job.
|
|
119
|
+
*
|
|
120
|
+
* Returns `{ status: 'skipped', reason }` without invoking the callback when
|
|
121
|
+
* the job is not due (`mode: 'due'`), is already in flight
|
|
122
|
+
* (`'already running'`), or was removed before the claim landed
|
|
123
|
+
* (`'removed'`). Before the phase split, a forced run against an in-flight
|
|
124
|
+
* job launched a second concurrent invocation.
|
|
125
|
+
*
|
|
126
|
+
* THROWS (rather than returning a skip) when `id` is not a registered job:
|
|
127
|
+
* `Error("Job not found: <id>")`. A job that disappears between this lookup
|
|
128
|
+
* and the claim is the `'removed'` skip above, not a throw — the two differ
|
|
129
|
+
* only by the timing of a race, and the second is a legitimate outcome
|
|
130
|
+
* whereas the first is a caller error.
|
|
131
|
+
*
|
|
132
|
+
* CONCURRENCY: the same job is bounded to one in-flight invocation on every
|
|
133
|
+
* path, and the timer path invokes due jobs one at a time. `run()` fan-out
|
|
134
|
+
* across DIFFERENT jobs is deliberately unbounded — N concurrent `run()`
|
|
135
|
+
* calls produce N concurrent consumer callbacks. Before the phase split
|
|
136
|
+
* these serialized behind the module-global lock; that serialization was the
|
|
137
|
+
* bug rather than the feature (one hung callback wedged every other caller),
|
|
138
|
+
* so it is not restored here. The fan-out is caller-driven and the scheduler
|
|
139
|
+
* never produces it on its own. A per-invoke bound belongs above this layer;
|
|
140
|
+
* it is tracked on stonyx-cron#35 alongside the execution timeout.
|
|
141
|
+
*
|
|
142
|
+
* NOT FIXED HERE: the phase split fixes the LOCK wedge, not the TIMER. A
|
|
143
|
+
* callback that never settles still stops `onTimer`'s sequential loop
|
|
144
|
+
* forever — `running` stays true, every later tick early-returns and re-arms,
|
|
145
|
+
* and the hung job's batch siblings stay claimed and off-heap having never
|
|
146
|
+
* been invoked. CRUD still resolves and `status()` still reports
|
|
147
|
+
* `started: true`, so that failure is now silent where it used to be loud.
|
|
148
|
+
* Bounding the callback and releasing batch siblings is stonyx-cron#35. Do
|
|
149
|
+
* not read this method's doc as "the hang is fixed".
|
|
72
150
|
*/
|
|
73
151
|
run(id: string, mode?: 'due' | 'force'): Promise<ExecuteResult>;
|
|
74
152
|
/**
|
|
@@ -78,6 +156,25 @@ export default class CronService {
|
|
|
78
156
|
armTimer(): void;
|
|
79
157
|
onTimer(): Promise<void>;
|
|
80
158
|
findDueJobs(nowMs: number): Job[];
|
|
159
|
+
/**
|
|
160
|
+
* Execute a job in three phases:
|
|
161
|
+
*
|
|
162
|
+
* 1. claim (locked) — take ownership of the job, detach it from the heap
|
|
163
|
+
* 2. invoke (UNLOCKED) — await the consumer callback
|
|
164
|
+
* 3. settle (locked) — apply the result, log it, re-insert into the heap
|
|
165
|
+
*
|
|
166
|
+
* The critical section deliberately excludes phase 2. `onJobDue` is
|
|
167
|
+
* arbitrary, unbounded consumer code; awaiting it under the module-global
|
|
168
|
+
* lock is what wedged every subsequent `locked()` call (add/update/remove)
|
|
169
|
+
* when a callback never settled.
|
|
170
|
+
*
|
|
171
|
+
* `onTimer` performs the batch claim (`findDueJobs` + `markRunning`) for all
|
|
172
|
+
* due jobs under a single lock and then enters at phase 2 via
|
|
173
|
+
* `#executeClaimed`. That entry point is `#private` rather than a parameter
|
|
174
|
+
* on this method: as a published `alreadyClaimed` flag it would be a
|
|
175
|
+
* supported way to skip phase 1 entirely, defeating the claim guard and
|
|
176
|
+
* allowing concurrent `onJobDue` invocations for the same job.
|
|
177
|
+
*/
|
|
81
178
|
executeJob(job: Job): Promise<ExecuteResult>;
|
|
82
179
|
removeFromHeap(id: string): void;
|
|
83
180
|
log(message: string): void;
|
package/dist/service.js
CHANGED
|
@@ -2,7 +2,21 @@
|
|
|
2
2
|
* CronService - the main API for advanced job scheduling.
|
|
3
3
|
*
|
|
4
4
|
* Manages jobs in memory with a min-heap for efficient next-job lookup.
|
|
5
|
-
*
|
|
5
|
+
*
|
|
6
|
+
* Async locking, but never around the consumer callback. Execution is split
|
|
7
|
+
* into three phases: claim (locked), invoke (UNLOCKED), settle (locked). Only
|
|
8
|
+
* phases 1 and 3 are serialized; the critical section deliberately excludes
|
|
9
|
+
* phase 2, so a consumer callback that never settles cannot wedge the lock
|
|
10
|
+
* chain and block add/update/remove. See `run()` and `#executeClaimed`.
|
|
11
|
+
*
|
|
12
|
+
* CONSUMER NOTE — this class is NOMINALLY typed. It carries ECMAScript hard-
|
|
13
|
+
* private members, so `dist/service.d.ts` emits `#private;` on the class and a
|
|
14
|
+
* structurally-built test double will not assign to `CronService`
|
|
15
|
+
* (`TS2741: Property '#private' is missing`). The break is one-directional and
|
|
16
|
+
* has a zero-cost workaround: `extends CronService` still compiles, and
|
|
17
|
+
* assigning a real `CronService` to your own hand-written interface still
|
|
18
|
+
* compiles. Declare your own interface and depend on that rather than
|
|
19
|
+
* hand-building a `CronService`-typed mock. See README's `CronService` section.
|
|
6
20
|
*/
|
|
7
21
|
import config from 'stonyx/config';
|
|
8
22
|
import log from 'stonyx/log';
|
|
@@ -12,6 +26,121 @@ import { locked } from './locked.js';
|
|
|
12
26
|
import { normalizeJobInput, recoverFlatParams } from './normalize.js';
|
|
13
27
|
import RunLog from './run-log.js';
|
|
14
28
|
const MAX_TIMER_DELAY_MS = 60_000;
|
|
29
|
+
/** Longest error text that may reach a log line. Anything past this is truncated. */
|
|
30
|
+
const MAX_LOGGED_ERROR_LENGTH = 512;
|
|
31
|
+
/** Longest job name that may reach a log line. Anything past this is truncated. */
|
|
32
|
+
const MAX_LOGGED_NAME_LENGTH = 120;
|
|
33
|
+
/**
|
|
34
|
+
* Flatten a value for interpolation into a single log line.
|
|
35
|
+
*
|
|
36
|
+
* Chronicle writes `${timestamp} ${content}\n` to a newline-delimited file, so
|
|
37
|
+
* any `\r` or `\n` inside `content` ends the record early and everything after
|
|
38
|
+
* it is read back as a separate, attacker-shaped entry — including a forged
|
|
39
|
+
* `[timestamp] Cron — ...` prefix that is indistinguishable from a real one.
|
|
40
|
+
* Both values that reach these lines are untrusted: `job.name` is passed
|
|
41
|
+
* through `createJob` unvalidated and `normalize.ts` exists specifically to
|
|
42
|
+
* accept AI-shaped input, and an error message is arbitrary consumer-callback
|
|
43
|
+
* text. Newlines become the literal two characters so the content survives for
|
|
44
|
+
* a reader, and the length cap keeps one pathological value from swamping the
|
|
45
|
+
* file.
|
|
46
|
+
*
|
|
47
|
+
* TOTAL, for the same reason `describeError` below is total, and the parameter
|
|
48
|
+
* is coerced even though it is typed `string`. `job.name` is typed `string` but
|
|
49
|
+
* that is a compile-time claim about runtime data: `normalize.ts` only
|
|
50
|
+
* GENERATES a name when the field is falsy, so `add({ name: 12345 })` stores a
|
|
51
|
+
* number through the public API, and `start(initialJobs)` takes names verbatim
|
|
52
|
+
* from the consumer's store — the same untrusted boundary `start()` already
|
|
53
|
+
* hardens `state` against.
|
|
54
|
+
*
|
|
55
|
+
* Fixed HERE rather than by coercing in `normalize`, deliberately. Coercing at
|
|
56
|
+
* `normalize` closes the `add()` path only; the rehydration path bypasses both
|
|
57
|
+
* `normalize` and `createJob` entirely, and that is the path this class already
|
|
58
|
+
* treats as hostile. And this helper is on the ERROR path — a helper that
|
|
59
|
+
* throws while building an error report destroys the report it exists to
|
|
60
|
+
* produce. Measured pre-fix: `run()` rejected with `TypeError: value.replace is
|
|
61
|
+
* not a function` instead of returning an `ExecuteResult`, and on the timer
|
|
62
|
+
* path the failure record was swallowed entirely (0 records for a job that
|
|
63
|
+
* failed) because the throw happened inside the reporter's own `try`.
|
|
64
|
+
*
|
|
65
|
+
* `String(value)` alone is NOT enough: a value whose `toString` or
|
|
66
|
+
* `Symbol.toPrimitive` throws raises out of the coercion itself, so the `try`
|
|
67
|
+
* is load-bearing and not belt-and-braces. Kept typed `string` rather than
|
|
68
|
+
* widened to `unknown` so call sites still get compile-time pressure; the
|
|
69
|
+
* runtime coercion is the defence, not the signature.
|
|
70
|
+
*/
|
|
71
|
+
function forLog(value, maxLength) {
|
|
72
|
+
let flattened;
|
|
73
|
+
try {
|
|
74
|
+
flattened = String(value).replace(/\r\n|[\r\n\u2028\u2029]/g, '\\n');
|
|
75
|
+
}
|
|
76
|
+
catch {
|
|
77
|
+
return '<unrenderable value>';
|
|
78
|
+
}
|
|
79
|
+
return flattened.length > maxLength ? `${flattened.slice(0, maxLength)}...` : flattened;
|
|
80
|
+
}
|
|
81
|
+
/**
|
|
82
|
+
* Describe a thrown value without ever throwing.
|
|
83
|
+
*
|
|
84
|
+
* `String(err)` is not total: a null-prototype object, or any object whose
|
|
85
|
+
* `toString`/`Symbol.toPrimitive` throws, raises "Cannot convert object to
|
|
86
|
+
* primitive value". Consumer callbacks throw arbitrary values, so the error
|
|
87
|
+
* handler must not become a second failure source of its own.
|
|
88
|
+
*
|
|
89
|
+
* The `instanceof Error` branch needs the same guard as the other one. `Error`
|
|
90
|
+
* is subclassable and `message` is a plain writable property, so a consumer can
|
|
91
|
+
* hand back an `Error` whose `message` is a getter that throws, or one that is
|
|
92
|
+
* an object whose `toString` throws — `Error.prototype.message` is typed
|
|
93
|
+
* `string`, so TypeScript sees nothing wrong and the coercion is deferred to
|
|
94
|
+
* the caller's template literal, outside every guard here. That made `run()`
|
|
95
|
+
* reject instead of returning an `ExecuteResult`: a contract violation in the
|
|
96
|
+
* function written to prevent exactly that. Reading and coercing `message`
|
|
97
|
+
* inside the `try` is what closes it.
|
|
98
|
+
*
|
|
99
|
+
* `Cron` in `main.ts` carries its own `describeError` and the two deliberately
|
|
100
|
+
* differ: it renders for a log line only, so it prefers `err.stack`; this one
|
|
101
|
+
* is also returned to the caller as `ExecuteResult.error` and persisted in a
|
|
102
|
+
* per-job run log, where a stack would be an unbounded blob in every stored
|
|
103
|
+
* failure. Recorded in `docs/architecture.md` under Code Patterns &
|
|
104
|
+
* Conventions -> Private Members -> "Two `describeError` helpers, deliberately
|
|
105
|
+
* not shared (#34 / #36)" — do not merge them into a shared helper without
|
|
106
|
+
* reading that first. NOT the `### Error Handling` section further down, which
|
|
107
|
+
* is about the legacy `Cron` and does not carry this decision.
|
|
108
|
+
*/
|
|
109
|
+
function describeError(err) {
|
|
110
|
+
try {
|
|
111
|
+
return err instanceof Error ? String(err.message) : String(err);
|
|
112
|
+
}
|
|
113
|
+
catch {
|
|
114
|
+
return 'unknown error';
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
/**
|
|
118
|
+
* Job objects whose claim is held by an invocation still running IN THIS
|
|
119
|
+
* PROCESS. Added by phase 1, removed by phase 3 (or by the one hand-release in
|
|
120
|
+
* `#executeClaimed`), so membership is exactly "a settle is still coming".
|
|
121
|
+
*
|
|
122
|
+
* This exists so `start()` can tell a STALE claim from a LIVE one. Both look
|
|
123
|
+
* identical in `job.state.runningAtMs` — a number — but they need opposite
|
|
124
|
+
* treatment: a stale claim must be released (it is #34's permanently-dead job,
|
|
125
|
+
* and nothing reaps it) while a live one must be left alone (releasing it lets
|
|
126
|
+
* the timer launch a second concurrent invocation of a job that is already
|
|
127
|
+
* running, breaking the one invariant this class advertises).
|
|
128
|
+
*
|
|
129
|
+
* Keyed on the Job OBJECT, not the id, and module-level rather than per
|
|
130
|
+
* instance, for the two reasons that make those the only workable choices:
|
|
131
|
+
*
|
|
132
|
+
* - Object identity is what makes it correct across `CronService` instances.
|
|
133
|
+
* A second service handed live rows sees the same objects, so it inherits
|
|
134
|
+
* the answer rather than guessing; a per-instance set would report "not in
|
|
135
|
+
* flight" for a claim a sibling instance is holding. `locked()`'s chain is
|
|
136
|
+
* already module-global for the same reason.
|
|
137
|
+
* - Identity is also what makes it correct after a real crash. A restart
|
|
138
|
+
* deserializes its rows, so those are new objects and are never members —
|
|
139
|
+
* the #34 release still fires, which is the whole point of it.
|
|
140
|
+
*
|
|
141
|
+
* `WeakSet`, so a job that is removed and dropped mid-flight is not retained.
|
|
142
|
+
*/
|
|
143
|
+
const inFlight = new WeakSet();
|
|
15
144
|
export default class CronService {
|
|
16
145
|
jobs;
|
|
17
146
|
heap;
|
|
@@ -32,21 +161,122 @@ export default class CronService {
|
|
|
32
161
|
}
|
|
33
162
|
// -- Lifecycle -------------------------------------------------------
|
|
34
163
|
/**
|
|
35
|
-
* Start the service. Loads jobs from store (if any), arms timer.
|
|
164
|
+
* Start the service. Loads jobs from store (if any), arms timer. A no-op if
|
|
165
|
+
* already started.
|
|
166
|
+
*
|
|
167
|
+
* `initialJobs` crosses a serialization boundary — it is whatever the
|
|
168
|
+
* consumer's store handed back — so `Job[]` is a compile-time claim about
|
|
169
|
+
* runtime data. Three behaviours follow from that and are worth knowing
|
|
170
|
+
* before you call this, because all three are deliberate and two of them
|
|
171
|
+
* differ from a plain "load and arm":
|
|
172
|
+
*
|
|
173
|
+
* 1. WRITES TO `row.state`. A STALE claim (`state.runningAtMs` set by a
|
|
174
|
+
* process that is gone) is released, because nothing else ever will —
|
|
175
|
+
* there is no lease on the field (#35) — and left in place it is a job
|
|
176
|
+
* that is dead forever while `status()` reports it healthy. A LIVE claim,
|
|
177
|
+
* held by an invocation still running in this process, is left alone:
|
|
178
|
+
* releasing it would let the timer start a second concurrent invocation of
|
|
179
|
+
* a job that is already running.
|
|
180
|
+
*
|
|
181
|
+
* 2. THROWS on a row this class cannot use, rather than accepting it. A row
|
|
182
|
+
* whose `state` is missing, or frozen (`structuredClone` + `Object.freeze`
|
|
183
|
+
* is an ordinary defensive rehydration), throws out of `start()` where the
|
|
184
|
+
* caller's own `await` can catch it. The alternative is a `TypeError` from
|
|
185
|
+
* inside a bare timer callback later — an unhandled rejection, and
|
|
186
|
+
* process-fatal under Node's default.
|
|
187
|
+
*
|
|
188
|
+
* 3. ARMS THE TIMER EVEN IF IT THROWS. The rows loaded before the throw are
|
|
189
|
+
* registered and scheduled. Without this, a throw leaves `started: true`
|
|
190
|
+
* (so a retry is a no-op) with jobs in the heap and no timer: nothing ever
|
|
191
|
+
* fires and `status()` still reports healthy.
|
|
192
|
+
*
|
|
193
|
+
* Which of the three you can observe depends on the CONTENT of the rows, not
|
|
194
|
+
* on whether they were deserialized — 1 fires only on a row that already
|
|
195
|
+
* carries a claim, and 2/3 only on a row this class cannot use. Hand it
|
|
196
|
+
* well-formed deserialized rows with no claim set and none of the three is
|
|
197
|
+
* observable. Hand it a deserialized row that DOES carry one and 1 and 2 are
|
|
198
|
+
* exactly what you get: measured, a stale `state.runningAtMs` of 1 comes back
|
|
199
|
+
* `undefined`, and a `structuredClone` + `Object.freeze` row makes `start()`
|
|
200
|
+
* throw `TypeError: Cannot assign to read only property 'runningAtMs'` with
|
|
201
|
+
* the timer still armed behind it. Hand it live `Job` objects this service is
|
|
202
|
+
* currently executing and only 1 is in play, by design — and on those it
|
|
203
|
+
* deliberately does nothing.
|
|
36
204
|
*/
|
|
37
205
|
async start(initialJobs) {
|
|
38
206
|
if (this.started)
|
|
39
207
|
return;
|
|
40
208
|
this.started = true;
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
209
|
+
// `finally`, not a trailing statement. Reconciled with the same guard #53
|
|
210
|
+
// puts around `register`'s `runOnInit` invocation: a scheduler that is
|
|
211
|
+
// marked started but never armed is the terminal state both fixes exist to
|
|
212
|
+
// remove, reached here through the other entry point. `initialJobs` crosses
|
|
213
|
+
// a serialization boundary — it is whatever the consumer's store handed
|
|
214
|
+
// back — so `Job[]` is a compile-time claim about runtime data, and a row
|
|
215
|
+
// missing `state` throws mid-loop. Without this, `start()` leaves
|
|
216
|
+
// `started: true` (so it is now a no-op), the rows registered before the
|
|
217
|
+
// throw sitting in the heap, and NO timer: measured, `status()` then
|
|
218
|
+
// reports `{ started: true, jobCount: 1, nextWakeAtMs: <real> }` while
|
|
219
|
+
// nothing will ever fire. Silent and healthy-looking, again.
|
|
220
|
+
try {
|
|
221
|
+
if (initialJobs) {
|
|
222
|
+
for (const job of initialJobs) {
|
|
223
|
+
// A STALE `runningAtMs` records a claim taken by a process that is
|
|
224
|
+
// gone. Nothing will ever settle it, and nothing reaps it — there is
|
|
225
|
+
// no lease on the field (tracked on #35). Left in place it is a
|
|
226
|
+
// permanently dead job that still reports healthy: `isDue` returns
|
|
227
|
+
// false forever because of the flag, `run()` answers
|
|
228
|
+
// `'already running'` forever, `update()` never touches
|
|
229
|
+
// `state.runningAtMs`, and `status()` counts it like any other. The
|
|
230
|
+
// consumer's only recovery would be remove() + add(), losing the job
|
|
231
|
+
// id and its run history.
|
|
232
|
+
//
|
|
233
|
+
// Same hazard, same treatment as the hand-release on the `'removed'`
|
|
234
|
+
// path in `#executeClaimed`: a claim with no reachable settle must be
|
|
235
|
+
// released. Assigned directly rather than via `applyResult` for the same
|
|
236
|
+
// reason — this releases the claim and nothing else. The job did not
|
|
237
|
+
// run, so it gets no run-log row, no `lastStatus`, and no recomputed
|
|
238
|
+
// `nextRunAtMs`; it is rescheduled from the store's own value below.
|
|
239
|
+
//
|
|
240
|
+
// But NOT every `runningAtMs` here is stale, and the field cannot
|
|
241
|
+
// tell you which — it is a number either way. `start()` early-returns
|
|
242
|
+
// when `started`, so reaching this with a LIVE claim needs `stop()`
|
|
243
|
+
// then `start(sameObjects)` (an in-process restart against a store
|
|
244
|
+
// that hands back references rather than fresh rows) or a second
|
|
245
|
+
// `CronService` handed live rows. Measured on the unconditional
|
|
246
|
+
// version: the release cleared a live claim, the timer then found the
|
|
247
|
+
// job due, and one job got TWO concurrent in-flight callbacks. That
|
|
248
|
+
// is the single invariant this class advertises and that #34/#35
|
|
249
|
+
// exist to protect, so the release is guarded on `inFlight` —
|
|
250
|
+
// authoritative object identity, not a heuristic on the timestamp.
|
|
251
|
+
// See `inFlight`'s docblock for why identity is also what keeps the
|
|
252
|
+
// stale case working after a real crash.
|
|
253
|
+
//
|
|
254
|
+
// The guard is deliberately NOT a `job.state.runningAtMs` read.
|
|
255
|
+
// Guarding on THAT makes `start()` accept a row whose `state` is
|
|
256
|
+
// frozen — `structuredClone` + `Object.freeze` is an ordinary
|
|
257
|
+
// defensive rehydration — and that row is not usable by this class at
|
|
258
|
+
// all: `markRunning` writes the same field on every execution.
|
|
259
|
+
// Measured, that guard moves the failure from a throw out of
|
|
260
|
+
// `start()`, which the consumer's own `await` can catch, to a
|
|
261
|
+
// TypeError raised inside `onTimer`'s batch claim — a bare timer
|
|
262
|
+
// callback, so it surfaces as an unhandled rejection and is
|
|
263
|
+
// process-fatal under Node's default. `inFlight` does not have that
|
|
264
|
+
// problem: a deserialized row is never a member, so the write still
|
|
265
|
+
// happens and the frozen row still fails loudly at the boundary. Both
|
|
266
|
+
// properties are tested; do not collapse the two guards into one.
|
|
267
|
+
if (!inFlight.has(job)) {
|
|
268
|
+
job.state.runningAtMs = undefined;
|
|
269
|
+
}
|
|
270
|
+
this.jobs.set(job.id, job);
|
|
271
|
+
if (job.enabled && job.state.nextRunAtMs) {
|
|
272
|
+
this.heap.push({ key: job.id, nextTrigger: job.state.nextRunAtMs });
|
|
273
|
+
}
|
|
46
274
|
}
|
|
47
275
|
}
|
|
48
276
|
}
|
|
49
|
-
|
|
277
|
+
finally {
|
|
278
|
+
this.armTimer();
|
|
279
|
+
}
|
|
50
280
|
}
|
|
51
281
|
/**
|
|
52
282
|
* Stop the service. Clears timer.
|
|
@@ -136,6 +366,37 @@ export default class CronService {
|
|
|
136
366
|
}
|
|
137
367
|
/**
|
|
138
368
|
* Manually trigger a job.
|
|
369
|
+
*
|
|
370
|
+
* Returns `{ status: 'skipped', reason }` without invoking the callback when
|
|
371
|
+
* the job is not due (`mode: 'due'`), is already in flight
|
|
372
|
+
* (`'already running'`), or was removed before the claim landed
|
|
373
|
+
* (`'removed'`). Before the phase split, a forced run against an in-flight
|
|
374
|
+
* job launched a second concurrent invocation.
|
|
375
|
+
*
|
|
376
|
+
* THROWS (rather than returning a skip) when `id` is not a registered job:
|
|
377
|
+
* `Error("Job not found: <id>")`. A job that disappears between this lookup
|
|
378
|
+
* and the claim is the `'removed'` skip above, not a throw — the two differ
|
|
379
|
+
* only by the timing of a race, and the second is a legitimate outcome
|
|
380
|
+
* whereas the first is a caller error.
|
|
381
|
+
*
|
|
382
|
+
* CONCURRENCY: the same job is bounded to one in-flight invocation on every
|
|
383
|
+
* path, and the timer path invokes due jobs one at a time. `run()` fan-out
|
|
384
|
+
* across DIFFERENT jobs is deliberately unbounded — N concurrent `run()`
|
|
385
|
+
* calls produce N concurrent consumer callbacks. Before the phase split
|
|
386
|
+
* these serialized behind the module-global lock; that serialization was the
|
|
387
|
+
* bug rather than the feature (one hung callback wedged every other caller),
|
|
388
|
+
* so it is not restored here. The fan-out is caller-driven and the scheduler
|
|
389
|
+
* never produces it on its own. A per-invoke bound belongs above this layer;
|
|
390
|
+
* it is tracked on stonyx-cron#35 alongside the execution timeout.
|
|
391
|
+
*
|
|
392
|
+
* NOT FIXED HERE: the phase split fixes the LOCK wedge, not the TIMER. A
|
|
393
|
+
* callback that never settles still stops `onTimer`'s sequential loop
|
|
394
|
+
* forever — `running` stays true, every later tick early-returns and re-arms,
|
|
395
|
+
* and the hung job's batch siblings stay claimed and off-heap having never
|
|
396
|
+
* been invoked. CRUD still resolves and `status()` still reports
|
|
397
|
+
* `started: true`, so that failure is now silent where it used to be loud.
|
|
398
|
+
* Bounding the callback and releasing batch siblings is stonyx-cron#35. Do
|
|
399
|
+
* not read this method's doc as "the hang is fixed".
|
|
139
400
|
*/
|
|
140
401
|
async run(id, mode = 'force') {
|
|
141
402
|
const job = this.jobs.get(id);
|
|
@@ -144,6 +405,10 @@ export default class CronService {
|
|
|
144
405
|
if (mode === 'due' && !isDue(job, Date.now())) {
|
|
145
406
|
return { status: 'skipped', reason: 'not due' };
|
|
146
407
|
}
|
|
408
|
+
// Deliberately NOT wrapped in locked(): executeJob takes the lock itself,
|
|
409
|
+
// for its claim and settle phases only. Wrapping here would re-create the
|
|
410
|
+
// wedge through a second door, because the consumer callback would once
|
|
411
|
+
// again be awaited while a lock is held.
|
|
147
412
|
return this.executeJob(job);
|
|
148
413
|
}
|
|
149
414
|
/**
|
|
@@ -172,16 +437,78 @@ export default class CronService {
|
|
|
172
437
|
}
|
|
173
438
|
this.running = true;
|
|
174
439
|
try {
|
|
175
|
-
|
|
440
|
+
// -- Phase 1: claim (locked), batched --
|
|
441
|
+
// Collecting due jobs pops them off the heap, and marking them running
|
|
442
|
+
// makes them un-claimable by anyone else. Both must happen under the
|
|
443
|
+
// same lock turn, or a concurrent run() could claim a job this batch has
|
|
444
|
+
// already detached.
|
|
445
|
+
//
|
|
446
|
+
// This is the SECOND claim implementation — `#claimJob` is the other, and
|
|
447
|
+
// the two reach the same state by different routes. `#claimJob` guards
|
|
448
|
+
// with an explicit `job.state.runningAtMs` check; this path has no such
|
|
449
|
+
// check and relies entirely on `isDue`'s `!job.state.runningAtMs` clause
|
|
450
|
+
// (`job.ts`) to keep `findDueJobs` from re-claiming a job that `run()`
|
|
451
|
+
// already holds. THAT CLAUSE IS LOAD-BEARING HERE, not an optimisation:
|
|
452
|
+
// drop it and the timer path silently double-invokes a job that `run()`
|
|
453
|
+
// is mid-flight on, while `run()` keeps refusing correctly and looks
|
|
454
|
+
// healthy. The one-in-flight-invocation-per-job invariant this class
|
|
455
|
+
// advertises holds by two independent guards in two files; a change to
|
|
456
|
+
// either has to be checked against the other.
|
|
457
|
+
const dueJobs = await locked(() => {
|
|
176
458
|
const nowMs = Date.now();
|
|
177
|
-
const
|
|
178
|
-
for (const job of
|
|
459
|
+
const due = this.findDueJobs(nowMs);
|
|
460
|
+
for (const job of due) {
|
|
179
461
|
markRunning(job);
|
|
462
|
+
inFlight.add(job);
|
|
180
463
|
}
|
|
181
|
-
|
|
182
|
-
await this.executeJob(job);
|
|
183
|
-
}
|
|
464
|
+
return due;
|
|
184
465
|
});
|
|
466
|
+
// Phases 2 and 3 run OUTSIDE the claim lock. The consumer callback is
|
|
467
|
+
// awaited here holding no lock at all, so a callback that never settles
|
|
468
|
+
// cannot poison the lock chain and wedge add/update/remove.
|
|
469
|
+
for (const job of dueJobs) {
|
|
470
|
+
try {
|
|
471
|
+
await this.#executeClaimed(job);
|
|
472
|
+
}
|
|
473
|
+
catch (err) {
|
|
474
|
+
// One job's unexpected throw must not abort the batch. Every job in
|
|
475
|
+
// `dueJobs` is already claimed — marked running and detached from
|
|
476
|
+
// the heap — and only its own settle releases it, so aborting here
|
|
477
|
+
// would strand every sibling permanently un-due.
|
|
478
|
+
//
|
|
479
|
+
// Reported on an UNGATED channel. `this.log()` returns early when
|
|
480
|
+
// `config.cron.log` is false, which is a supported production
|
|
481
|
+
// setting, and a failure here permanently unschedules a job while
|
|
482
|
+
// `status()` keeps reporting the service healthy. Silent-and-healthy
|
|
483
|
+
// is exactly the failure class this split exists to remove.
|
|
484
|
+
//
|
|
485
|
+
// This is the outermost handler on the timer path, so it is the one
|
|
486
|
+
// that must not be able to throw: `log` is a shared singleton whose
|
|
487
|
+
// transports can reach the filesystem, so its own failure is
|
|
488
|
+
// swallowed rather than allowed to take the batch down.
|
|
489
|
+
//
|
|
490
|
+
// BOTH halves of that failure have to be caught, and they are caught
|
|
491
|
+
// by different constructs. `log.error` is a chronicle convenience
|
|
492
|
+
// method that returns `logAction(...)` -> `async log(...)`, so its
|
|
493
|
+
// console write, colour lookup, `mkdirSync` and `appendFile` all
|
|
494
|
+
// surface as REJECTIONS, never as synchronous throws. A bare call
|
|
495
|
+
// here escapes this `catch` entirely and terminates the process under
|
|
496
|
+
// Node's default `--unhandled-rejections=throw` — the handler written
|
|
497
|
+
// so it "must not be able to throw" would be the one taking the
|
|
498
|
+
// daemon down. The `try` covers the synchronous half (evaluating the
|
|
499
|
+
// template literal); `Promise.resolve(...).catch()` covers the async
|
|
500
|
+
// half. Deliberately not awaited: the batch must not block on a log
|
|
501
|
+
// transport, and `void` marks the floated promise as intentional.
|
|
502
|
+
try {
|
|
503
|
+
void Promise.resolve(log.error(`Cron — Job "${forLog(job.name, MAX_LOGGED_NAME_LENGTH)}" (${job.id}) execution failed unexpectedly: ${forLog(describeError(err), MAX_LOGGED_ERROR_LENGTH)}`)).catch(() => {
|
|
504
|
+
// Nothing left to report to.
|
|
505
|
+
});
|
|
506
|
+
}
|
|
507
|
+
catch {
|
|
508
|
+
// Nothing left to report to.
|
|
509
|
+
}
|
|
510
|
+
}
|
|
511
|
+
}
|
|
185
512
|
}
|
|
186
513
|
finally {
|
|
187
514
|
this.running = false;
|
|
@@ -202,50 +529,198 @@ export default class CronService {
|
|
|
202
529
|
}
|
|
203
530
|
return due;
|
|
204
531
|
}
|
|
532
|
+
/**
|
|
533
|
+
* Execute a job in three phases:
|
|
534
|
+
*
|
|
535
|
+
* 1. claim (locked) — take ownership of the job, detach it from the heap
|
|
536
|
+
* 2. invoke (UNLOCKED) — await the consumer callback
|
|
537
|
+
* 3. settle (locked) — apply the result, log it, re-insert into the heap
|
|
538
|
+
*
|
|
539
|
+
* The critical section deliberately excludes phase 2. `onJobDue` is
|
|
540
|
+
* arbitrary, unbounded consumer code; awaiting it under the module-global
|
|
541
|
+
* lock is what wedged every subsequent `locked()` call (add/update/remove)
|
|
542
|
+
* when a callback never settled.
|
|
543
|
+
*
|
|
544
|
+
* `onTimer` performs the batch claim (`findDueJobs` + `markRunning`) for all
|
|
545
|
+
* due jobs under a single lock and then enters at phase 2 via
|
|
546
|
+
* `#executeClaimed`. That entry point is `#private` rather than a parameter
|
|
547
|
+
* on this method: as a published `alreadyClaimed` flag it would be a
|
|
548
|
+
* supported way to skip phase 1 entirely, defeating the claim guard and
|
|
549
|
+
* allowing concurrent `onJobDue` invocations for the same job.
|
|
550
|
+
*/
|
|
205
551
|
async executeJob(job) {
|
|
552
|
+
// -- Phase 1: claim (locked) --
|
|
553
|
+
const refusal = await locked(() => this.#claimJob(job));
|
|
554
|
+
if (refusal)
|
|
555
|
+
return { status: 'skipped', reason: refusal };
|
|
556
|
+
return this.#executeClaimed(job);
|
|
557
|
+
}
|
|
558
|
+
/**
|
|
559
|
+
* Phases 2 and 3 for a job that has already been claimed — either by
|
|
560
|
+
* `executeJob` above or by `onTimer`'s batch claim.
|
|
561
|
+
*
|
|
562
|
+
* Private: reaching this without a claim would run the consumer callback for
|
|
563
|
+
* a job nobody owns, and would leave nothing to release the claim.
|
|
564
|
+
*/
|
|
565
|
+
async #executeClaimed(job) {
|
|
566
|
+
// Membership re-check. The claim and the invoke are no longer in the same
|
|
567
|
+
// critical section, and sibling callbacks run unlocked, so a `remove()` can
|
|
568
|
+
// now land in between AND RESOLVE — it used to deadlock. A resolved
|
|
569
|
+
// `remove()` must keep meaning "this callback will not fire"; the identity
|
|
570
|
+
// guard in `#settleJob` only cleans up afterwards, by which point the side
|
|
571
|
+
// effect has already happened. Identity, not id, so a removed-then-replaced
|
|
572
|
+
// key is caught too. Deliberately synchronous with the `onJobDue` call
|
|
573
|
+
// below — nothing can interleave between this check and the invocation.
|
|
574
|
+
//
|
|
575
|
+
// This is the one early return after a claim, so it is the one that has to
|
|
576
|
+
// release the claim by hand. Skipping settle is right — re-inserting or
|
|
577
|
+
// run-logging a removed job is the resurrection `#settleJob` refuses, and
|
|
578
|
+
// the heap entry is already gone. But the claim must still come off,
|
|
579
|
+
// because the detached object is NOT unreachable: it is the object `add()`
|
|
580
|
+
// returned and `get()`/`list()` hand out, and `start(initialJobs)`
|
|
581
|
+
// re-registers those objects verbatim, `state` included. A leftover
|
|
582
|
+
// `runningAtMs` rehydrates a permanently dead job — `isDue` false forever,
|
|
583
|
+
// `run()` refused forever, `status()` reporting it healthy.
|
|
584
|
+
//
|
|
585
|
+
// Assigned directly rather than via `applyResult`: this releases the claim
|
|
586
|
+
// and nothing else. No run-log row, no heap entry, no `lastStatus`, no
|
|
587
|
+
// recomputed `nextRunAtMs` — the job did not run.
|
|
588
|
+
if (this.jobs.get(job.id) !== job) {
|
|
589
|
+
job.state.runningAtMs = undefined;
|
|
590
|
+
inFlight.delete(job);
|
|
591
|
+
return { status: 'skipped', reason: 'removed' };
|
|
592
|
+
}
|
|
206
593
|
const startMs = Date.now();
|
|
207
594
|
let status = 'ok';
|
|
208
595
|
let error;
|
|
209
596
|
let summary;
|
|
597
|
+
let settled;
|
|
598
|
+
// The claim marked the job running and detached it from the heap. Phase 3
|
|
599
|
+
// is the ONLY thing that undoes either, so it must survive every non-local
|
|
600
|
+
// exit from phase 2 — including a throw from the catch handler itself
|
|
601
|
+
// (`this.log` is public and overridable and reaches a transport). A claim
|
|
602
|
+
// with no matching settle is not a degraded state, it is a permanently
|
|
603
|
+
// dead job.
|
|
210
604
|
try {
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
if (
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
605
|
+
// -- Phase 2: invoke (NOT locked) --
|
|
606
|
+
try {
|
|
607
|
+
if (this.onJobDue) {
|
|
608
|
+
const result = await this.onJobDue(job);
|
|
609
|
+
if (result) {
|
|
610
|
+
status = result.status || 'ok';
|
|
611
|
+
error = result.error;
|
|
612
|
+
summary = result.summary;
|
|
613
|
+
}
|
|
217
614
|
}
|
|
218
615
|
}
|
|
616
|
+
catch (err) {
|
|
617
|
+
status = 'error';
|
|
618
|
+
error = describeError(err);
|
|
619
|
+
this.log(`Job "${forLog(job.name, MAX_LOGGED_NAME_LENGTH)}" (${job.id}) failed: ${forLog(error, MAX_LOGGED_ERROR_LENGTH)}`);
|
|
620
|
+
}
|
|
219
621
|
}
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
this.log(`Job "${job.name}" (${job.id}) failed: ${error}`);
|
|
622
|
+
finally {
|
|
623
|
+
// -- Phase 3: settle (locked) --
|
|
624
|
+
settled = await locked(() => this.#settleJob(job, status, error, summary, startMs, Date.now() - startMs));
|
|
224
625
|
}
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
626
|
+
return settled;
|
|
627
|
+
}
|
|
628
|
+
/**
|
|
629
|
+
* Phase 1 — claim. Must be called while holding the lock (`locked()`, whose
|
|
630
|
+
* chain is module-global and therefore shared across CronService instances).
|
|
631
|
+
*
|
|
632
|
+
* Returns `null` on a successful claim, or the reason the claim was refused.
|
|
633
|
+
* `'already running'` is what makes a second `run()` report a skip instead of
|
|
634
|
+
* launching a concurrent invocation. `'removed'` covers the job being deleted
|
|
635
|
+
* between `run()`'s unlocked lookup and this lock turn — claiming then would
|
|
636
|
+
* `markRunning` an orphan and, worse, `removeFromHeap` an id that may now
|
|
637
|
+
* belong to a replacement.
|
|
638
|
+
*
|
|
639
|
+
* Detaching from the heap here — rather than relying on phase 3 to push a
|
|
640
|
+
* fresh entry — is what stops manual runs permanently duplicating entries.
|
|
641
|
+
*
|
|
642
|
+
* `#private`: published, this would be a supported call performing
|
|
643
|
+
* `markRunning` + `removeFromHeap` with no guaranteed settle and no lease on
|
|
644
|
+
* `runningAtMs`, so a single such call would strand the job forever. The
|
|
645
|
+
* lock-held precondition cannot be expressed in the type system, so the
|
|
646
|
+
* method must not be reachable from outside the class body.
|
|
647
|
+
*/
|
|
648
|
+
#claimJob(job) {
|
|
649
|
+
if (this.jobs.get(job.id) !== job)
|
|
650
|
+
return 'removed';
|
|
651
|
+
if (job.state.runningAtMs)
|
|
652
|
+
return 'already running';
|
|
653
|
+
markRunning(job);
|
|
654
|
+
inFlight.add(job);
|
|
655
|
+
this.removeFromHeap(job.id);
|
|
656
|
+
return null;
|
|
657
|
+
}
|
|
658
|
+
/**
|
|
659
|
+
* Phase 3 — settle. Must be called while holding the lock.
|
|
660
|
+
*
|
|
661
|
+
* `#private` for the same reason as `#claimJob`: unlocked it would run
|
|
662
|
+
* `applyResult`, a `runLog.record`, a full `removeFromHeap` rebuild, a
|
|
663
|
+
* `heap.push` and an `armTimer` with no mutual exclusion — exactly the
|
|
664
|
+
* corruption `locked()` exists to prevent.
|
|
665
|
+
*/
|
|
666
|
+
#settleJob(job, status, error, summary, startMs, durationMs) {
|
|
667
|
+
try {
|
|
668
|
+
const validStatus = (status === 'ok' || status === 'error' || status === 'skipped') ? status : 'error';
|
|
669
|
+
applyResult(job, validStatus, error, durationMs);
|
|
670
|
+
// The callback ran unlocked, so this job may have been removed — or
|
|
671
|
+
// removed and re-registered under the same id, the shape
|
|
672
|
+
// `start(initialJobs)` uses — while it was in flight. Identity, not id.
|
|
673
|
+
//
|
|
674
|
+
// Deliberately touch NOTHING here. The claim already detached this job's
|
|
675
|
+
// heap entry and nothing re-added it, so there is nothing to clean up;
|
|
676
|
+
// any entry now filed under this id belongs to the replacement, and
|
|
677
|
+
// removing it by id would silently unschedule a live job. Do not
|
|
678
|
+
// resurrect a removed job's heap entry or run log either.
|
|
679
|
+
if (this.jobs.get(job.id) !== job) {
|
|
680
|
+
return { status, error, summary, durationMs };
|
|
681
|
+
}
|
|
682
|
+
// Log the run
|
|
683
|
+
this.runLog.record({
|
|
684
|
+
jobId: job.id,
|
|
685
|
+
status,
|
|
686
|
+
error,
|
|
687
|
+
summary,
|
|
688
|
+
runAtMs: startMs,
|
|
689
|
+
durationMs,
|
|
690
|
+
nextRunAtMs: job.state.nextRunAtMs,
|
|
691
|
+
});
|
|
692
|
+
// Handle one-shot auto-delete. The callback ran unlocked and may have
|
|
693
|
+
// pushed a heap entry for this job via add()/update(), so drop it — the
|
|
694
|
+
// job is about to stop existing.
|
|
695
|
+
if (job.deleteAfterRun && status === 'ok' && !job.enabled) {
|
|
696
|
+
this.jobs.delete(job.id);
|
|
697
|
+
this.removeFromHeap(job.id);
|
|
698
|
+
this.runLog.removeJob(job.id);
|
|
699
|
+
return { status, summary, deleted: true };
|
|
700
|
+
}
|
|
701
|
+
// Re-insert into the heap if still active. Same reason as above: drop any
|
|
702
|
+
// entry the unlocked callback added for this job first, to preserve
|
|
703
|
+
// one-entry-per-key.
|
|
704
|
+
this.removeFromHeap(job.id);
|
|
705
|
+
if (job.enabled && job.state.nextRunAtMs) {
|
|
706
|
+
this.heap.push({ key: job.id, nextTrigger: job.state.nextRunAtMs });
|
|
707
|
+
}
|
|
708
|
+
return { status, error, summary, durationMs };
|
|
243
709
|
}
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
710
|
+
finally {
|
|
711
|
+
// The invocation is over, so this job is no longer in flight in this
|
|
712
|
+
// process — whatever `applyResult` did or did not manage to write. In the
|
|
713
|
+
// `finally` so it covers a throw out of `applyResult` or `runLog.record`
|
|
714
|
+
// too: past this point no settle is coming, which is precisely the state
|
|
715
|
+
// `start()`'s release is for.
|
|
716
|
+
inFlight.delete(job);
|
|
717
|
+
// One re-arm covering every exit, rather than one per branch. The claim
|
|
718
|
+
// detached this job from the heap, so a timer that fired during the
|
|
719
|
+
// unlocked invoke would have found nothing to arm — and `run()` has no
|
|
720
|
+
// `finally { armTimer() }` of its own the way `onTimer` does. Without
|
|
721
|
+
// this, a manual run() can leave the scheduler with no pending wake.
|
|
722
|
+
this.armTimer();
|
|
247
723
|
}
|
|
248
|
-
return { status, error, summary, durationMs };
|
|
249
724
|
}
|
|
250
725
|
// -- Helpers ---------------------------------------------------------
|
|
251
726
|
removeFromHeap(id) {
|