@task-master/cadence 0.0.0-stage → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Task Master
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,276 @@
1
- # Temporary Holding Version
2
-
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
1
+ # @task-master/cadence
2
+
3
+ MongoDB-backed orchestration for recurring data acquisition, vendor exports, and backfills.
4
+
5
+ Cadence tracks what data is due, admits work within capacity and traffic limits, and records the outcome. When a process crashes or an external request has an unknown result, it reconciles the run using evidence from your queue and vendor. Vendor-reported coverage gaps become repair work.
6
+
7
+ You provide the queue, vendor client, file storage, and processes that call the library.
8
+
9
+ > **Early release:** the public API may change during 0.x. Pin an exact version when integrating.
10
+
11
+ ## Installation
12
+
13
+ ```sh
14
+ npm install --save-exact @task-master/cadence@0.1.0 "mongodb@^6.10.0"
15
+ ```
16
+
17
+ Requirements:
18
+
19
+ - **Node.js 20 or later.**
20
+ - **MongoDB configured as a replica set.** A single-node replica set works for development; Cadence uses transactions.
21
+ - **MongoDB Node.js driver 6.10 or later within major version 6.** The driver is a peer dependency because collection handles and sessions cross the package boundary.
22
+
23
+ The package includes ESM, CommonJS, and TypeScript declarations:
24
+
25
+ ```js
26
+ import { connect, plan, admit } from '@task-master/cadence';
27
+ ```
28
+
29
+ ```js
30
+ const { connect, plan, admit } = require('@task-master/cadence');
31
+ ```
32
+
33
+ ## Core concepts
34
+
35
+ | Concept | Meaning |
36
+ | --- | --- |
37
+ | Integration | A vendor endpoint, its capacity, capabilities, and traffic policy. |
38
+ | Unit | A tenant, practice, feed, or other item acquired on a recurring schedule. |
39
+ | Request | Work to fulfill: a scheduled acquisition, an ad hoc backfill, or a repair. |
40
+ | Run | One attempt to fulfill a request. Retries belong to the same request. |
41
+ | Watermark | The data boundary reached by scheduled acquisition. |
42
+ | Repair | A recorded coverage gap that still needs acquisition. |
43
+
44
+ ```text
45
+ plan -> admit -> publishOutbox -> submitKickoff -> pollRun
46
+ |
47
+ downloadNext
48
+ |
49
+ outcomeFor -> finalize
50
+
51
+ reconcile checks abandoned claims, uncertain requests, and stalled runs.
52
+ ```
53
+
54
+ ## Getting started
55
+
56
+ ### 1. Connect and define the work
57
+
58
+ This TypeScript example creates one integration and one daily unit in a new database. Set `MONGO_URI` to your replica set connection string, then run this setup once.
59
+
60
+ ```ts
61
+ import {
62
+ connect,
63
+ defaultIntegrationPolicy,
64
+ defaultTrafficPolicy,
65
+ type Integration,
66
+ type Unit,
67
+ } from '@task-master/cadence';
68
+
69
+ const uri = process.env.MONGO_URI;
70
+ if (!uri) throw new Error('Set MONGO_URI to a MongoDB replica set URI');
71
+
72
+ const c = await connect(uri, 'cadence');
73
+
74
+ const integration: Integration = {
75
+ _id: 'vendor-a',
76
+ quotaKey: 'vendor-a',
77
+ capacity: { budget: 4, reserved: 0, active: 0 },
78
+ trafficPolicy: defaultTrafficPolicy({ timezone: 'UTC' }),
79
+ vendorCooldownUntil: null,
80
+ profile: { until: false, typeFilter: false, patient: false },
81
+ // Use this classification only after verifying that repeating an acquisition is safe.
82
+ sideEffectClass: 'idempotent_acquisition',
83
+ sideEffectClassVerifiedBy: 'your-vendor-contract-review',
84
+ policy: defaultIntegrationPolicy({
85
+ defaultSince: new Date('2026-01-01T00:00:00.000Z'),
86
+ }),
87
+ };
88
+
89
+ const now = new Date();
90
+ const unit: Unit = {
91
+ _id: 'tenant-1',
92
+ integrationId: integration._id,
93
+ adminState: 'active',
94
+ executionState: 'idle',
95
+ availability: {
96
+ dispatchFrom: null,
97
+ dispatchUntil: null,
98
+ dataFrom: null,
99
+ dataUntil: null,
100
+ allowFinalCatchUp: false,
101
+ suspensions: [],
102
+ },
103
+ watermark: {
104
+ scheduledThrough: null, basis: null, seeded: false, updatedAt: null,
105
+ },
106
+ counters: {
107
+ scheduled: { consecutiveFailures: 0, emptyStreak: 0, totalRuns: 0 },
108
+ adhoc: { consecutiveFailures: 0, emptyStreak: 0, totalRuns: 0 },
109
+ repair: { consecutiveFailures: 0, emptyStreak: 0, totalRuns: 0 },
110
+ },
111
+ cadenceMs: 24 * 60 * 60_000,
112
+ scheduledCadenceAt: now,
113
+ scheduledNextEligibleAt: now,
114
+ activeRunId: null,
115
+ lastScheduledRunId: null,
116
+ lastAdhocRunId: null,
117
+ lastRepairRunId: null,
118
+ attention: [],
119
+ };
120
+
121
+ try {
122
+ await c.integrations.insertOne(integration);
123
+ await c.units.insertOne(unit);
124
+ } finally {
125
+ await c.client.close();
126
+ }
127
+ ```
128
+
129
+ Set capability flags to `true` only after verifying the vendor supports them. In particular, `profile.until` means the vendor honors an upper time bound; without it, Cadence cannot enforce `maxCatchUpWindowMs` against that vendor.
130
+
131
+ `connect()` creates the required indexes. If you construct a `Cadence` handle with your own `MongoClient`, call `ensureIndexes(c)` before starting any work. The unique indexes are required for deduplication.
132
+
133
+ ### 2. Plan and admit work
134
+
135
+ In your scheduler process, open a connection with `connect()` and run a tick like this. The following snippets use that open handle as `c`.
136
+
137
+ ```ts
138
+ import { admit, plan } from '@task-master/cadence';
139
+
140
+ const { proposals, skipped } = await plan(c, 'vendor-a', new Date());
141
+
142
+ for (const diagnostic of skipped) console.log('Skipped:', diagnostic);
143
+
144
+ for (const proposal of proposals) {
145
+ const result = await admit(c, proposal, new Date());
146
+ if (!result.admitted) {
147
+ console.log('Admission refused:', result.reason);
148
+ if (result.reason === 'no_capacity') break;
149
+ }
150
+ }
151
+ ```
152
+
153
+ `plan()` reads state without reserving capacity. `admit()` checks the proposal again inside a transaction and records admitted work in the outbox. Repeat the tick at an interval appropriate to your application.
154
+
155
+ ### 3. Implement your adapters
156
+
157
+ Import the interfaces from the package root:
158
+
159
+ ```ts
160
+ import type { Queue, Vendor, PollingVendor, FileSink } from '@task-master/cadence';
161
+ ```
162
+
163
+ | Adapter | Required method | Your implementation |
164
+ | --- | --- | --- |
165
+ | `Queue` | `enqueue(payload, { idempotencyKey })` | Enqueue work and return `{ taskId }`. Pass the stable idempotency key to your queue. |
166
+ | `Vendor` | `kickoff(run)` | Start the vendor job and return `{ jobRef, contentLocation }`. Use `run._id` as the vendor request key where supported. |
167
+ | `PollingVendor` | `poll(run)` | Map vendor responses to `accepted`, `complete`, `throttled`, `expired`, or `failed`. |
168
+ | `FileSink` | `fetch(entry, run)` | Download and persist one manifest entry, then return `{ bytes, count }`. |
169
+
170
+ The queue adapter must distinguish a definite refusal from an unknown result:
171
+
172
+ | Enqueue result | Adapter behavior |
173
+ | --- | --- |
174
+ | Definitely accepted | Return `{ taskId }`. |
175
+ | Definitely refused; no task created | Throw `EnqueueRejected`. |
176
+ | May have been accepted, such as a timeout after sending | Throw `EnqueueUncertain`, or another error. Cadence treats all other thrown errors as uncertain. |
177
+
178
+ Both error classes are exported from `@task-master/cadence`. Classify responses according to your queue's actual contract. Reporting an unknown result as a rejection can cause duplicate work. If you inspect these errors yourself, use the exported `isEnqueueRejected` and `isEnqueueUncertain` guards; they work across separate package copies.
179
+
180
+ ### 4. Publish and execute runs
181
+
182
+ Call `publishOutbox()` repeatedly from a publisher process with your `Queue` adapter:
183
+
184
+ ```ts
185
+ import { DEFAULTS, publishOutbox } from '@task-master/cadence';
186
+
187
+ const result = await publishOutbox(c, queue, new Date(), {
188
+ instanceId: 'publisher-1',
189
+ claimTtlMs: DEFAULTS.outboxClaimTtlMs,
190
+ });
191
+ ```
192
+
193
+ Each call attempts to publish one pending entry. `queue` is your adapter. Back off when the result is `{ published: false, reason: 'nothing_pending' }`.
194
+
195
+ Each queued payload contains `{ runId, unitKey, kind }`. When your queue delivers a task, load the run from `c.runs` using `runId`, then load its integration from `c.integrations` using the run's `integrationId`. Your worker calls the following functions in order, using your `Vendor`, `PollingVendor`, and `FileSink` adapters:
196
+
197
+ | Call | How to handle the result |
198
+ | --- | --- |
199
+ | `submitKickoff(c, vendor, integration, runId, now, workerOptions)` | Continue on `submitted` or `already_confirmed`. Reschedule `deferred` work; other statuses require yielding ownership or reconciliation. |
200
+ | `pollRun(c, poller, integration, runId, now, executeOptions)` | Poll with a delay while `working`; honor `deferred` results. On `terminal`, pass its `outcome` to `finalize()`. On `manifest_ready`, start downloads. Stop on `not_pollable`. |
201
+ | `downloadNext(c, sink, integration, runId, now, executeOptions)` | Claims and downloads one entry per call. Repeat, delaying on `deferred`, until `nothing_claimable`. |
202
+ | `outcomeFor(c, runId)` | Returns an outcome when the manifest is finished, or `null` while entries remain outstanding. Revisit unfinished runs. |
203
+ | `finalize(c, runId, outcome, now)` | Applies the outcome, updates request and unit state, and releases capacity. |
204
+
205
+ Use a fresh `new Date()` for each call's `now` argument. The option shapes are:
206
+
207
+ ```ts
208
+ const workerOptions = { workerId: 'worker-1', claimTtlMs: 600_000 };
209
+ const executeOptions = { workerId: 'worker-1', entryClaimTtlMs: 300_000 };
210
+ ```
211
+
212
+ Your application owns queue acknowledgment, redelivery, delays, and shutdown. A deferred step needs another execution attempt. `nothing_claimable` can also mean another worker holds the remaining entries, so always check `outcomeFor()` before finalizing.
213
+
214
+ ### 5. Reconcile interrupted work
215
+
216
+ Run one reconciler periodically, for example once a minute:
217
+
218
+ ```ts
219
+ import { DEFAULTS, reconcile } from '@task-master/cadence';
220
+
221
+ const report = await reconcile(c, new Date(), {
222
+ noProgressDeadlineMs: DEFAULTS.noProgressDeadlineMs,
223
+ publicationResolutionGraceMs: 15 * 60_000,
224
+ }, { vendor, queue });
225
+
226
+ if (report.submissionsEscalated || report.publicationsEscalated || report.runsBlocked) {
227
+ console.error('Cadence needs operator attention:', report);
228
+ }
229
+ ```
230
+
231
+ Recovery is more effective when your adapters support lookups:
232
+
233
+ - `vendor.findByClientRequestId(runId)` returns `{ jobRef, contentLocation }` or `null`, allowing a submission interrupted by a crash to be recovered.
234
+ - The queue passed to `reconcile()` may also implement `findByIdempotencyKey(key)`, returning `{ taskId }` or `null` to resolve uncertain publication.
235
+
236
+ These lookups are optional. When the outcome cannot be established, Cadence may escalate or block work for an operator to resolve.
237
+
238
+ ## Running in production
239
+
240
+ The four roles can share a process or run separately:
241
+
242
+ | Role | Concurrency |
243
+ | --- | --- |
244
+ | Scheduler | One per integration is simplest. Admission rechecks proposals and unique indexes guard against duplicates. |
245
+ | Publisher | Multiple instances can compete for claimed outbox entries. |
246
+ | Worker | Multiple instances can claim submission work and individual manifest entries. |
247
+ | Reconciler | Run exactly one, with no overlapping sweeps. |
248
+
249
+ Keep MongoDB connections open for each process and close them with `await c.client.close()` during shutdown.
250
+
251
+ Inspect operational state using the collections on `c`:
252
+
253
+ ```ts
254
+ const needsAttention = await c.units.find({
255
+ attention: { $not: { $size: 0 } },
256
+ }).toArray();
257
+ const blockedRuns = await c.runs.find({ state: 'blocked' }).toArray();
258
+ const openRepairs = await c.repairs.find({ state: 'open' }).toArray();
259
+ const abandonedRepairs = await c.repairs.find({ state: 'accepted' }).toArray();
260
+ ```
261
+
262
+ Monitor escalations, blocked runs, open attention episodes, and accepted repairs. An accepted repair is a gap the engine has stopped trying to fill.
263
+
264
+ Use `patchAvailability`, `addSuspension`, `removeSuspension`, and `setTrafficPolicy` to change scheduling policy; these helpers refresh eligibility hints. After changing a unit's `adminState` directly, call `refreshEligibility`. Leave `executionState` to the engine.
265
+
266
+ ## Outcome and retry behavior
267
+
268
+ - A failed download is a failed attempt. Report `partial` only when the vendor reports a coverage gap.
269
+ - Repeating an unchanged polling response does not count as progress. Pass through the vendor's progress note so stalled work can be detected.
270
+ - A catch-up window capped by policy leaves later data for the next scheduled run. The cap alone does not create a repair.
271
+ - A blocked run is nonterminal and retains its reservation while evidence is outstanding.
272
+ - Inspect both `plan().skipped` and stored unit state when diagnosing missing work; units excluded by the planner's initial query do not appear in `skipped`.
273
+
274
+ ## License
275
+
276
+ MIT