@task-master/cadence 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +206 -110
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,179 +1,275 @@
|
|
|
1
1
|
# @task-master/cadence
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
MongoDB-backed orchestration for recurring data acquisition, vendor exports, and backfills.
|
|
4
4
|
|
|
5
|
-
Cadence
|
|
5
|
+
Cadence tracks what data is due, admits work within capacity and traffic limits, and records the outcome. When a process crashes or an external request has an unknown result, it reconciles the run using evidence from your queue and vendor. Vendor-reported coverage gaps become repair work.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
You provide the queue, vendor client, file storage, and processes that call the library.
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
> **Early release:** the public API may change during 0.x. Pin an exact version when integrating.
|
|
10
|
+
|
|
11
|
+
## Installation
|
|
10
12
|
|
|
11
13
|
```sh
|
|
12
|
-
npm install @task-master/cadence mongodb
|
|
14
|
+
npm install --save-exact @task-master/cadence@0.1.0 "mongodb@^6.10.0"
|
|
13
15
|
```
|
|
14
16
|
|
|
15
|
-
|
|
17
|
+
Requirements:
|
|
16
18
|
|
|
17
|
-
|
|
19
|
+
- **Node.js 20 or later.**
|
|
20
|
+
- **MongoDB configured as a replica set.** A single-node replica set works for development; Cadence uses transactions.
|
|
21
|
+
- **MongoDB Node.js driver 6.10 or later within major version 6.** The driver is a peer dependency because collection handles and sessions cross the package boundary.
|
|
18
22
|
|
|
19
|
-
|
|
23
|
+
The package includes ESM, CommonJS, and TypeScript declarations:
|
|
20
24
|
|
|
21
25
|
```js
|
|
22
26
|
import { connect, plan, admit } from '@task-master/cadence';
|
|
23
|
-
const { connect, plan, admit } = require('@task-master/cadence');
|
|
24
27
|
```
|
|
25
28
|
|
|
26
|
-
## Shape of it
|
|
27
|
-
|
|
28
29
|
```js
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
const c = await connect(process.env.MONGO_URI, 'cadence');
|
|
32
|
-
await ensureIndexes(c); // once, before any traffic
|
|
33
|
-
|
|
34
|
-
const { proposals, skipped } = await plan(c, 'vendor-a', new Date());
|
|
35
|
-
for (const p of proposals) await admit(c, p, new Date());
|
|
30
|
+
const { connect, plan, admit } = require('@task-master/cadence');
|
|
36
31
|
```
|
|
37
32
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
33
|
+
## Core concepts
|
|
34
|
+
|
|
35
|
+
| Concept | Meaning |
|
|
36
|
+
| --- | --- |
|
|
37
|
+
| Integration | A vendor endpoint, its capacity, capabilities, and traffic policy. |
|
|
38
|
+
| Unit | A tenant, practice, feed, or other item acquired on a recurring schedule. |
|
|
39
|
+
| Request | Work to fulfill: a scheduled acquisition, an ad hoc backfill, or a repair. |
|
|
40
|
+
| Run | One attempt to fulfill a request. Retries belong to the same request. |
|
|
41
|
+
| Watermark | The data boundary reached by scheduled acquisition. |
|
|
42
|
+
| Repair | A recorded coverage gap that still needs acquisition. |
|
|
43
|
+
|
|
44
|
+
```text
|
|
45
|
+
plan -> admit -> publishOutbox -> submitKickoff -> pollRun
|
|
46
|
+
|
|
|
47
|
+
downloadNext
|
|
48
|
+
|
|
|
49
|
+
outcomeFor -> finalize
|
|
50
|
+
|
|
51
|
+
reconcile checks abandoned claims, uncertain requests, and stalled runs.
|
|
52
|
+
```
|
|
50
53
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
54
|
+
## Getting started
|
|
55
|
+
|
|
56
|
+
### 1. Connect and define the work
|
|
57
|
+
|
|
58
|
+
This TypeScript example creates one integration and one daily unit in a new database. Set `MONGO_URI` to your replica set connection string, then run this setup once.
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
import {
|
|
62
|
+
connect,
|
|
63
|
+
defaultIntegrationPolicy,
|
|
64
|
+
defaultTrafficPolicy,
|
|
65
|
+
type Integration,
|
|
66
|
+
type Unit,
|
|
67
|
+
} from '@task-master/cadence';
|
|
68
|
+
|
|
69
|
+
const uri = process.env.MONGO_URI;
|
|
70
|
+
if (!uri) throw new Error('Set MONGO_URI to a MongoDB replica set URI');
|
|
71
|
+
|
|
72
|
+
const c = await connect(uri, 'cadence');
|
|
73
|
+
|
|
74
|
+
const integration: Integration = {
|
|
75
|
+
_id: 'vendor-a',
|
|
76
|
+
quotaKey: 'vendor-a',
|
|
77
|
+
capacity: { budget: 4, reserved: 0, active: 0 },
|
|
78
|
+
trafficPolicy: defaultTrafficPolicy({ timezone: 'UTC' }),
|
|
79
|
+
vendorCooldownUntil: null,
|
|
80
|
+
profile: { until: false, typeFilter: false, patient: false },
|
|
81
|
+
// Use this classification only after verifying that repeating an acquisition is safe.
|
|
82
|
+
sideEffectClass: 'idempotent_acquisition',
|
|
83
|
+
sideEffectClassVerifiedBy: 'your-vendor-contract-review',
|
|
84
|
+
policy: defaultIntegrationPolicy({
|
|
85
|
+
defaultSince: new Date('2026-01-01T00:00:00.000Z'),
|
|
86
|
+
}),
|
|
87
|
+
};
|
|
88
|
+
|
|
89
|
+
const now = new Date();
|
|
90
|
+
const unit: Unit = {
|
|
91
|
+
_id: 'tenant-1',
|
|
92
|
+
integrationId: integration._id,
|
|
93
|
+
adminState: 'active',
|
|
94
|
+
executionState: 'idle',
|
|
95
|
+
availability: {
|
|
96
|
+
dispatchFrom: null,
|
|
97
|
+
dispatchUntil: null,
|
|
98
|
+
dataFrom: null,
|
|
99
|
+
dataUntil: null,
|
|
100
|
+
allowFinalCatchUp: false,
|
|
101
|
+
suspensions: [],
|
|
102
|
+
},
|
|
103
|
+
watermark: {
|
|
104
|
+
scheduledThrough: null, basis: null, seeded: false, updatedAt: null,
|
|
105
|
+
},
|
|
106
|
+
counters: {
|
|
107
|
+
scheduled: { consecutiveFailures: 0, emptyStreak: 0, totalRuns: 0 },
|
|
108
|
+
adhoc: { consecutiveFailures: 0, emptyStreak: 0, totalRuns: 0 },
|
|
109
|
+
repair: { consecutiveFailures: 0, emptyStreak: 0, totalRuns: 0 },
|
|
110
|
+
},
|
|
111
|
+
cadenceMs: 24 * 60 * 60_000,
|
|
112
|
+
scheduledCadenceAt: now,
|
|
113
|
+
scheduledNextEligibleAt: now,
|
|
114
|
+
activeRunId: null,
|
|
115
|
+
lastScheduledRunId: null,
|
|
116
|
+
lastAdhocRunId: null,
|
|
117
|
+
lastRepairRunId: null,
|
|
118
|
+
attention: [],
|
|
119
|
+
};
|
|
120
|
+
|
|
121
|
+
try {
|
|
122
|
+
await c.integrations.insertOne(integration);
|
|
123
|
+
await c.units.insertOne(unit);
|
|
124
|
+
} finally {
|
|
125
|
+
await c.client.close();
|
|
55
126
|
}
|
|
56
127
|
```
|
|
57
128
|
|
|
58
|
-
|
|
129
|
+
Set capability flags to `true` only after verifying the vendor supports them. In particular, `profile.until` means the vendor honors an upper time bound; without it, Cadence cannot enforce `maxCatchUpWindowMs` against that vendor.
|
|
59
130
|
|
|
60
|
-
If you
|
|
131
|
+
`connect()` creates the required indexes. If you construct a `Cadence` handle with your own `MongoClient`, call `ensureIndexes(c)` before starting any work. The unique indexes are required for deduplication.
|
|
61
132
|
|
|
62
|
-
|
|
63
|
-
import { isEnqueueRejected, isEnqueueUncertain } from '@task-master/cadence';
|
|
64
|
-
```
|
|
133
|
+
### 2. Plan and admit work
|
|
65
134
|
|
|
66
|
-
|
|
135
|
+
In your scheduler process, open a connection with `connect()` and run a tick like this. The following snippets use that open handle as `c`.
|
|
67
136
|
|
|
68
|
-
|
|
137
|
+
```ts
|
|
138
|
+
import { admit, plan } from '@task-master/cadence';
|
|
69
139
|
|
|
70
|
-
|
|
140
|
+
const { proposals, skipped } = await plan(c, 'vendor-a', new Date());
|
|
71
141
|
|
|
72
|
-
|
|
73
|
-
- **Polling is not progress.** A job polled a thousand times with nothing new to report has not progressed, or the no-progress deadline would never fire. The *first* report is news; identical repeats are not.
|
|
74
|
-
- **A capped catch-up window is not a gap.** The marker stops at the capped boundary and the next scheduled window resumes exactly there.
|
|
75
|
-
- **Attention is a list.** Three unrelated problems can be true about one unit at once.
|
|
142
|
+
for (const diagnostic of skipped) console.log('Skipped:', diagnostic);
|
|
76
143
|
|
|
77
|
-
|
|
144
|
+
for (const proposal of proposals) {
|
|
145
|
+
const result = await admit(c, proposal, new Date());
|
|
146
|
+
if (!result.admitted) {
|
|
147
|
+
console.log('Admission refused:', result.reason);
|
|
148
|
+
if (result.reason === 'no_capacity') break;
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
```
|
|
78
152
|
|
|
79
|
-
|
|
153
|
+
`plan()` reads state without reserving capacity. `admit()` checks the proposal again inside a transaction and records admitted work in the outbox. Repeat the tick at an interval appropriate to your application.
|
|
80
154
|
|
|
81
|
-
|
|
155
|
+
### 3. Implement your adapters
|
|
82
156
|
|
|
83
|
-
|
|
84
|
-
|---|---|---|
|
|
85
|
-
| Scheduler | one per integration is simplest | seconds to minutes |
|
|
86
|
-
| Publisher | many; they compete safely | continuous |
|
|
87
|
-
| Worker | many; they compete safely | continuous |
|
|
88
|
-
| Reconciler | **exactly one** | a minute or more |
|
|
157
|
+
Import the interfaces from the package root:
|
|
89
158
|
|
|
90
|
-
|
|
159
|
+
```ts
|
|
160
|
+
import type { Queue, Vendor, PollingVendor, FileSink } from '@task-master/cadence';
|
|
161
|
+
```
|
|
91
162
|
|
|
92
|
-
|
|
163
|
+
| Adapter | Required method | Your implementation |
|
|
164
|
+
| --- | --- | --- |
|
|
165
|
+
| `Queue` | `enqueue(payload, { idempotencyKey })` | Enqueue work and return `{ taskId }`. Pass the stable idempotency key to your queue. |
|
|
166
|
+
| `Vendor` | `kickoff(run)` | Start the vendor job and return `{ jobRef, contentLocation }`. Use `run._id` as the vendor request key where supported. |
|
|
167
|
+
| `PollingVendor` | `poll(run)` | Map vendor responses to `accepted`, `complete`, `throttled`, `expired`, or `failed`. |
|
|
168
|
+
| `FileSink` | `fetch(entry, run)` | Download and persist one manifest entry, then return `{ bytes, count }`. |
|
|
93
169
|
|
|
94
|
-
|
|
170
|
+
The queue adapter must distinguish a definite refusal from an unknown result:
|
|
95
171
|
|
|
96
|
-
|
|
97
|
-
|
|
172
|
+
| Enqueue result | Adapter behavior |
|
|
173
|
+
| --- | --- |
|
|
174
|
+
| Definitely accepted | Return `{ taskId }`. |
|
|
175
|
+
| Definitely refused; no task created | Throw `EnqueueRejected`. |
|
|
176
|
+
| May have been accepted, such as a timeout after sending | Throw `EnqueueUncertain`, or another error. Cadence treats all other thrown errors as uncertain. |
|
|
98
177
|
|
|
99
|
-
|
|
100
|
-
for (const s of skipped) count(s.reason);
|
|
178
|
+
Both error classes are exported from `@task-master/cadence`. Classify responses according to your queue's actual contract. Reporting an unknown result as a rejection can cause duplicate work. If you inspect these errors yourself, use the exported `isEnqueueRejected` and `isEnqueueUncertain` guards; they work across separate package copies.
|
|
101
179
|
|
|
102
|
-
|
|
103
|
-
const result = await admit(c, p, new Date());
|
|
104
|
-
if (!result.admitted && result.reason === 'no_capacity') break;
|
|
105
|
-
}
|
|
106
|
-
```
|
|
180
|
+
### 4. Publish and execute runs
|
|
107
181
|
|
|
108
|
-
|
|
182
|
+
Call `publishOutbox()` repeatedly from a publisher process with your `Queue` adapter:
|
|
183
|
+
|
|
184
|
+
```ts
|
|
185
|
+
import { DEFAULTS, publishOutbox } from '@task-master/cadence';
|
|
109
186
|
|
|
110
|
-
```js
|
|
111
187
|
const result = await publishOutbox(c, queue, new Date(), {
|
|
112
|
-
instanceId,
|
|
188
|
+
instanceId: 'publisher-1',
|
|
113
189
|
claimTtlMs: DEFAULTS.outboxClaimTtlMs,
|
|
114
190
|
});
|
|
115
|
-
|
|
116
|
-
if (!result.published && result.reason === 'nothing_pending') await sleep(1000);
|
|
117
|
-
// 'claim_lost' means a peer took it; 'uncertain' is already parked for the reconciler.
|
|
118
191
|
```
|
|
119
192
|
|
|
120
|
-
|
|
193
|
+
Each call attempts to publish one pending entry. `queue` is your adapter. Back off when the result is `{ published: false, reason: 'nothing_pending' }`.
|
|
121
194
|
|
|
122
|
-
|
|
123
|
-
const worker = { workerId, claimTtlMs: DEFAULTS.submissionClaimTtlMs };
|
|
124
|
-
const exec = { workerId, entryClaimTtlMs: 300_000 };
|
|
125
|
-
|
|
126
|
-
const submit = await submitKickoff(c, vendor, integration, runId, new Date(), worker);
|
|
127
|
-
if (submit.status !== 'submitted' && submit.status !== 'already_confirmed') return;
|
|
128
|
-
|
|
129
|
-
for (;;) {
|
|
130
|
-
const step = await pollRun(c, poller, integration, runId, new Date(), exec);
|
|
131
|
-
if (step.status === 'terminal') return finalize(c, runId, step.outcome, new Date());
|
|
132
|
-
if (step.status === 'manifest_ready') break;
|
|
133
|
-
if (step.status === 'not_pollable') return;
|
|
134
|
-
await sleep(15_000);
|
|
135
|
-
}
|
|
195
|
+
Each queued payload contains `{ runId, unitKey, kind }`. When your queue delivers a task, load the run from `c.runs` using `runId`, then load its integration from `c.integrations` using the run's `integrationId`. Your worker calls the following functions in order, using your `Vendor`, `PollingVendor`, and `FileSink` adapters:
|
|
136
196
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
197
|
+
| Call | How to handle the result |
|
|
198
|
+
| --- | --- |
|
|
199
|
+
| `submitKickoff(c, vendor, integration, runId, now, workerOptions)` | Continue on `submitted` or `already_confirmed`. Reschedule `deferred` work; other statuses require yielding ownership or reconciliation. |
|
|
200
|
+
| `pollRun(c, poller, integration, runId, now, executeOptions)` | Poll with a delay while `working`; honor `deferred` results. On `terminal`, pass its `outcome` to `finalize()`. On `manifest_ready`, start downloads. Stop on `not_pollable`. |
|
|
201
|
+
| `downloadNext(c, sink, integration, runId, now, executeOptions)` | Claims and downloads one entry per call. Repeat, delaying on `deferred`, until `nothing_claimable`. |
|
|
202
|
+
| `outcomeFor(c, runId)` | Returns an outcome when the manifest is finished, or `null` while entries remain outstanding. Revisit unfinished runs. |
|
|
203
|
+
| `finalize(c, runId, outcome, now)` | Applies the outcome, updates request and unit state, and releases capacity. |
|
|
204
|
+
|
|
205
|
+
Use a fresh `new Date()` for each call's `now` argument. The option shapes are:
|
|
142
206
|
|
|
143
|
-
|
|
144
|
-
const
|
|
145
|
-
|
|
207
|
+
```ts
|
|
208
|
+
const workerOptions = { workerId: 'worker-1', claimTtlMs: 600_000 };
|
|
209
|
+
const executeOptions = { workerId: 'worker-1', entryClaimTtlMs: 300_000 };
|
|
146
210
|
```
|
|
147
211
|
|
|
148
|
-
|
|
212
|
+
Your application owns queue acknowledgment, redelivery, delays, and shutdown. A deferred step needs another execution attempt. `nothing_claimable` can also mean another worker holds the remaining entries, so always check `outcomeFor()` before finalizing.
|
|
213
|
+
|
|
214
|
+
### 5. Reconcile interrupted work
|
|
215
|
+
|
|
216
|
+
Run one reconciler periodically, for example once a minute:
|
|
217
|
+
|
|
218
|
+
```ts
|
|
219
|
+
import { DEFAULTS, reconcile } from '@task-master/cadence';
|
|
149
220
|
|
|
150
|
-
```js
|
|
151
221
|
const report = await reconcile(c, new Date(), {
|
|
152
222
|
noProgressDeadlineMs: DEFAULTS.noProgressDeadlineMs,
|
|
153
223
|
publicationResolutionGraceMs: 15 * 60_000,
|
|
154
224
|
}, { vendor, queue });
|
|
225
|
+
|
|
226
|
+
if (report.submissionsEscalated || report.publicationsEscalated || report.runsBlocked) {
|
|
227
|
+
console.error('Cadence needs operator attention:', report);
|
|
228
|
+
}
|
|
155
229
|
```
|
|
156
230
|
|
|
157
|
-
|
|
231
|
+
Recovery is more effective when your adapters support lookups:
|
|
158
232
|
|
|
159
|
-
|
|
233
|
+
- `vendor.findByClientRequestId(runId)` returns `{ jobRef, contentLocation }` or `null`, allowing a submission interrupted by a crash to be recovered.
|
|
234
|
+
- The queue passed to `reconcile()` may also implement `findByIdempotencyKey(key)`, returning `{ taskId }` or `null` to resolve uncertain publication.
|
|
160
235
|
|
|
161
|
-
|
|
236
|
+
These lookups are optional. When the outcome cannot be established, Cadence may escalate or block work for an operator to resolve.
|
|
162
237
|
|
|
163
|
-
|
|
238
|
+
## Running in production
|
|
164
239
|
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
240
|
+
The four roles can share a process or run separately:
|
|
241
|
+
|
|
242
|
+
| Role | Concurrency |
|
|
243
|
+
| --- | --- |
|
|
244
|
+
| Scheduler | One per integration is simplest. Admission rechecks proposals and unique indexes guard against duplicates. |
|
|
245
|
+
| Publisher | Multiple instances can compete for claimed outbox entries. |
|
|
246
|
+
| Worker | Multiple instances can claim submission work and individual manifest entries. |
|
|
247
|
+
| Reconciler | Run exactly one, with no overlapping sweeps. |
|
|
248
|
+
|
|
249
|
+
Keep MongoDB connections open for each process and close them with `await c.client.close()` during shutdown.
|
|
250
|
+
|
|
251
|
+
Inspect operational state using the collections on `c`:
|
|
252
|
+
|
|
253
|
+
```ts
|
|
254
|
+
const needsAttention = await c.units.find({
|
|
255
|
+
attention: { $not: { $size: 0 } },
|
|
256
|
+
}).toArray();
|
|
257
|
+
const blockedRuns = await c.runs.find({ state: 'blocked' }).toArray();
|
|
258
|
+
const openRepairs = await c.repairs.find({ state: 'open' }).toArray();
|
|
259
|
+
const abandonedRepairs = await c.repairs.find({ state: 'accepted' }).toArray();
|
|
170
260
|
```
|
|
171
261
|
|
|
172
|
-
|
|
262
|
+
Monitor escalations, blocked runs, open attention episodes, and accepted repairs. An accepted repair is a gap the engine has stopped trying to fill.
|
|
263
|
+
|
|
264
|
+
Use `patchAvailability`, `addSuspension`, `removeSuspension`, and `setTrafficPolicy` to change scheduling policy; these helpers refresh eligibility hints. After changing a unit's `adminState` directly, call `refreshEligibility`. Leave `executionState` to the engine.
|
|
173
265
|
|
|
174
|
-
|
|
266
|
+
## Outcome and retry behavior
|
|
175
267
|
|
|
176
|
-
|
|
268
|
+
- A failed download is a failed attempt. Report `partial` only when the vendor reports a coverage gap.
|
|
269
|
+
- Repeating an unchanged polling response does not count as progress. Pass through the vendor's progress note so stalled work can be detected.
|
|
270
|
+
- A catch-up window capped by policy leaves later data for the next scheduled run. The cap alone does not create a repair.
|
|
271
|
+
- A blocked run is nonterminal and retains its reservation while evidence is outstanding.
|
|
272
|
+
- Inspect both `plan().skipped` and stored unit state when diagnosing missing work; units excluded by the planner's initial query do not appear in `skipped`.
|
|
177
273
|
|
|
178
274
|
## License
|
|
179
275
|
|
package/package.json
CHANGED