@skedulo/integration-sdk 0.11.3 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +31 -0
- package/README.md +6 -2
- package/dist/actions/sync/run-flow.action.js +1 -1
- package/dist/actions/sync/run-flow.action.js.map +1 -1
- package/dist/actions/sync/run-job.action.d.ts +6 -3
- package/dist/actions/sync/run-job.action.d.ts.map +1 -1
- package/dist/actions/sync/run-job.action.js +105 -14
- package/dist/actions/sync/run-job.action.js.map +1 -1
- package/dist/browser.d.ts +4 -3
- package/dist/browser.d.ts.map +1 -1
- package/dist/browser.js +7 -1
- package/dist/browser.js.map +1 -1
- package/dist/clients/dataflow.client.d.ts +5 -1
- package/dist/clients/dataflow.client.d.ts.map +1 -1
- package/dist/clients/dataflow.client.js +11 -1
- package/dist/clients/dataflow.client.js.map +1 -1
- package/dist/constants.d.ts +8 -0
- package/dist/constants.d.ts.map +1 -1
- package/dist/constants.js +15 -1
- package/dist/constants.js.map +1 -1
- package/dist/index.d.ts +4 -3
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +9 -2
- package/dist/index.js.map +1 -1
- package/dist/interfaces/adapter.interface.d.ts +27 -0
- package/dist/interfaces/adapter.interface.d.ts.map +1 -1
- package/dist/interfaces/data-flow-spec.interface.d.ts +28 -1
- package/dist/interfaces/data-flow-spec.interface.d.ts.map +1 -1
- package/dist/interfaces/sync-flow-strategy.interface.d.ts +2 -0
- package/dist/interfaces/sync-flow-strategy.interface.d.ts.map +1 -1
- package/dist/interfaces/sync-result.interface.d.ts +5 -1
- package/dist/interfaces/sync-result.interface.d.ts.map +1 -1
- package/dist/services/dataflow.service.d.ts +5 -1
- package/dist/services/dataflow.service.d.ts.map +1 -1
- package/dist/services/dataflow.service.js +3 -0
- package/dist/services/dataflow.service.js.map +1 -1
- package/dist/services/id-mapping-sync.strategy.d.ts.map +1 -1
- package/dist/services/id-mapping-sync.strategy.js +2 -1
- package/dist/services/id-mapping-sync.strategy.js.map +1 -1
- package/dist/services/run-log-builder.d.ts +23 -1
- package/dist/services/run-log-builder.d.ts.map +1 -1
- package/dist/services/run-log-builder.js +84 -2
- package/dist/services/run-log-builder.js.map +1 -1
- package/dist/services/sync-runner.d.ts +3 -0
- package/dist/services/sync-runner.d.ts.map +1 -1
- package/dist/services/sync-runner.js +434 -60
- package/dist/services/sync-runner.js.map +1 -1
- package/dist/services/sync.service.d.ts +13 -2
- package/dist/services/sync.service.d.ts.map +1 -1
- package/dist/services/sync.service.js +133 -51
- package/dist/services/sync.service.js.map +1 -1
- package/dist/types.d.ts +3 -3
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js.map +1 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,36 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.12.0] - 2026-10-12
|
|
4
|
+
|
|
5
|
+
**A sync too large for one run now continues across runs instead of starting over (COR-1283).**
|
|
6
|
+
|
|
7
|
+
In 0.11.x a whole sync had to finish in one function call (900 s). A large initial load or a mass update was cut off, saved nothing and started over on every run, so it never finished.
|
|
8
|
+
|
|
9
|
+
For adapter authors:
|
|
10
|
+
|
|
11
|
+
- New optional `Adapter.fetchResumable(ctx, flow, since, filter, resumeToken)`. It yields `FetchPage` objects (`{ records, next }`). `next` resumes right after the page and `null` means the load is done. Keep `next` under about 2000 chars. A source implements `fetch` or `fetchResumable`.
|
|
12
|
+
- Throw `ResumeTokenError` for a saved token you can read but can't resume from. After 3 such runs over at least 1 hour, the saved token is dropped and the load reads from the start again (it keeps its start time). Any other error keeps it. To start over by hand, reset the flow's cursor (`PATCH /data-flows/:id/cursor` with `{"lastRunAt": null}`). The saved token comes back from storage an admin can change, so check its shape.
|
|
13
|
+
- `DataFlow.createdAt` (ISO string, sent by integration-service) is now typed. `sync.run-job` uses it when a load has no valid `startedAt` and the flow has no `lastRunAt`.
|
|
14
|
+
- `SyncService.run` takes optional resume args (`ResumeArgs`). Without them it reads every page and saves nothing. `DataFlowService.patchCursor` saves only the keys it's given (`lastRunAt`, `syncProgress`).
|
|
15
|
+
- New exports from the root and `/browser`: `FetchPage`, `SyncProgress`, `CheckpointStop`, `DEFAULT_RUN_BUDGET_MS`, `MAX_SYNC_PROGRESS_CHARS`, `MAX_CHECKPOINT_HOLDS`, `MAX_FAILED_RESUMES`, `FAILED_RESUME_MIN_MS`. From the root only: `ResumeTokenError`, `ResumeArgs`, `MAX_RECORD_ERRORS`.
|
|
16
|
+
|
|
17
|
+
What changes for users:
|
|
18
|
+
|
|
19
|
+
- A run on a `fetchResumable` source stops after 12 minutes. In `sync.run-job` all flows in the job share one budget: at most 12 minutes and at most the job interval minus 1 minute, but never less than 1 minute. It saves its progress after each page and the next run continues from there. Scheduled runs and a plain `run-flow` (no `since`, no `fullResync`) save and resume progress. `lastRunAt` moves only when the load finishes, to the time the load started. Deletes sync only on the run that finishes the load. A stopped run sets `BatchResult.stoppedEarly`, and its run log says the sync continues next run. Editing the flow's objects, filter, mapping or match key starts the load over, unless a run already going during the edit finishes the load first: then the load doesn't start over, and records synced under the old settings stay as they are. Only a match-key edit, or a run that was still running during the edit, logs `sync_checkpoint_ignored`: for the other edits integration-service clears the checkpoint itself. A token too large to save (`MAX_SYNC_PROGRESS_CHARS`) is skipped, and the run saves the next page that fits.
|
|
20
|
+
- On a run that saves progress, a page with a failure that can't be recorded (no source id, a source id over 255 characters, or a failed count with no errors), or that looks like an outage (no successes, every failure transient), isn't saved. The run stops and tries the page again next run. After 3 tries over at least 1 hour it records what it can and moves on. This also applies to the last page of a load that has saved progress. A load with no saved progress (for example, one that fits in one page) holds with no try limit, as in 0.11.x.
|
|
21
|
+
- If saving a page's failed records fails, the run stops (`checkpoint_failed`) and the next run tries again from the last saved page. A failed progress save doesn't stop the run (a `sync_progress_save_failed` warn log line); the next run may re-read those pages. A page held later in that run starts its own count of 3 tries.
|
|
22
|
+
- A run that saves progress records every failed record on every page and keeps going. It saves a page's progress only after that page's failures are written. There's no cap of 100 failed-record rows per run on these runs, so a bad mapping on a large load writes one failed-record row for each failing record (about 2 million rows for a 2 million record load). A backfill or full resync keeps the 0.11.x cap of 100 rows per run, and a full resync that couldn't record every failure doesn't move `lastRunAt`.
|
|
23
|
+
- **Behaviour change:** in `sync.run-job`, later flows wait while an earlier flow is still loading. A flow is still loading if it has a saved checkpoint after its run, had one when the run started, or has never finished a load (`lastRunAt` is null). When a flow stops early for any reason (`time_budget`, `not_drained`, `checkpoint_held`, `checkpoint_failed`), or holds its last page (`holdCursor`) or throws while still loading, the job's later flows are skipped that tick with `waiting for flow N to finish its load` (an INFO run log on each). A paused flow with a saved checkpoint makes them wait too. If the flow can't be read back after the run, they wait too. A flow that finished its run doesn't hold up later flows, and nor does a flow that has finished a load before and throws with no checkpoint. The wait lasts up to 24 hours or 3 job intervals, whichever is longer, from the load's `startedAt` (3 days for a daily job). If there's no checkpoint, or its `startedAt` doesn't parse or is in the future, the wait counts from the flow's `lastRunAt`, else its `createdAt`. With none, later flows don't wait. After that, later flows run so a slow or stuck load can't hold up the job for good: the job logs `sync_job_wait_skipped` and writes a WARNING run log on the loading flow and on each later flow that runs, saying they may sync against a partly loaded parent. A child flow that runs against a partly loaded parent can skip records for good: a lookup to a parent not loaded yet skips the child record, writes it without the parent link, or records a failure, depending on the lookup's `onMissing`, and the child's cursor moves past it. [COR-1468](https://linear.app/skedulo/issue/COR-1468) tracks a fix. A child flow in another job, a realtime event or a manual `run-flow` can also run against a partly loaded parent. The wait includes flows that don't depend on the loading flow; put those in a separate job. This also covers flows on sources without `fetchResumable`: a new flow that throws on every run (for example, a bad login) holds up later flows for 24 hours or 3 job intervals, whichever is longer, from when it was created, then they run with a WARNING run log every tick until it finishes a load.
|
|
24
|
+
- For a first sync, or after a bulk import or mass update (for example 100k Salesforce records), sync the affected flows by hand with a plain `sync.run-flow` (no `since`, no `fullResync`): one flow at a time, parents before children, and re-run each until `run-flow` succeeds with no `stoppedEarly` and no `holdCursor` (a finished run's log has no “continues next run” or “retries” note). A scheduled job can carry on a large load by itself, but a child flow in the same job may then run against a partly loaded parent ([COR-1468](https://linear.app/skedulo/issue/COR-1468)).
|
|
25
|
+
- **Behaviour change:** a `run-flow` `since` backfill never moves `lastRunAt` (0.11.x moved it, which could skip records). A full resync saves no progress, so one too big for one run never finishes: reset the flow's cursor instead.
|
|
26
|
+
- A run that throws after saving pages says in its run log how many pages and failed records it saved.
|
|
27
|
+
- To spot a stuck load: `sync_load_slow` (a warn log line, every run once a load is over 24 hours old) and the metrics `sync.checkpoint.held` and `sync.run.stopped_early`. `sync.checkpoint.failed` counts two things: a failed progress save (the run keeps going) and a failed write of a page's failed records (the run stops with `checkpoint_failed`).
|
|
28
|
+
|
|
29
|
+
Deploy and rollback:
|
|
30
|
+
|
|
31
|
+
- Deploy integration-service with the `data_flow.sync_progress` column and the progress-only `PATCH /data-flows/:id/cursor` first.
|
|
32
|
+
- Roll back functions on SDK 0.12.0 before integration-service. An older integration-service rejects `syncProgress` with a 400, so no resumable run can save progress: each run reads as far as its time budget allows and the next one starts again. A load that finishes in one run still moves `lastRunAt`, since the finishing write only sends `syncProgress` when a checkpoint was saved.
|
|
33
|
+
|
|
3
34
|
## [0.11.3] - 2026-10-06
|
|
4
35
|
|
|
5
36
|
Bump `@skedulo/field-mapping-engine` to `^1.2.1` so FME transform errors log their real JSONata message instead of `[object Object]` (COR-1363).
|
package/README.md
CHANGED
|
@@ -192,7 +192,7 @@ Throw a typed subclass instead of `new ActionError(CODE, ...)`. The dispatcher m
|
|
|
192
192
|
There are two ways a data flow runs:
|
|
193
193
|
|
|
194
194
|
1. **Realtime** (per-flow triggered action) — the flow's `triggerConfig` carries `sync_config: { strategy: 'realtime' }`. Activation provisions one PSS triggered action; record changes dispatch `event.on-skedulo` (Skedulo → vendor) or `event.on-vendor` (vendor → Skedulo) immediately.
|
|
195
|
-
2. **Batch** (integration-level scheduled jobs) — the flow has **no** `sync_config` (`triggerConfig` is `null`). Operators group batch flows into ordered **scheduled jobs** at the integration level. Each job provisions one PSS scheduled webhook that fires `sync.run-job`, which runs the job's `flow_ids` sequentially (continue-on-error).
|
|
195
|
+
2. **Batch** (integration-level scheduled jobs) — the flow has **no** `sync_config` (`triggerConfig` is `null`). Operators group batch flows into ordered **scheduled jobs** at the integration level. Each job provisions one PSS scheduled webhook that fires `sync.run-job`, which runs the job's `flow_ids` sequentially (continue-on-error; later flows wait while an earlier one is still loading).
|
|
196
196
|
|
|
197
197
|
```jsonc
|
|
198
198
|
// Realtime flow — triggerConfig
|
|
@@ -237,7 +237,11 @@ sked-int-<integrationType>-<sourceObject>-to-<targetObject>-flow<id>
|
|
|
237
237
|
sked-int-<integrationType>-job-<slug(name)>
|
|
238
238
|
```
|
|
239
239
|
|
|
240
|
-
The webhook posts `sync.run-job` with the `sked-scheduled-job-name` header carrying the `webhook_name`. The SDK looks up the job in config and runs each flow in order; one flow failing does not stop the others.
|
|
240
|
+
The webhook posts `sync.run-job` with the `sked-scheduled-job-name` header carrying the `webhook_name`. The SDK looks up the job in config and runs each flow in order; one flow failing does not stop the others. When a flow stops part way through its load, holds its last page or throws while still loading (it has a checkpoint, had one when the run started, or has never finished a load), or is paused with a checkpoint, the later flows in the job wait until it finishes, for up to 24 hours or 3 job intervals, whichever is longer, from the load's start (if the start isn't saved, from its last finished load, else from when the flow was created; with none of these, they don't wait). After that they run with a WARNING run log (see [Multi-run syncs](./docs/SYNC-PIPELINE.md#multi-run-syncs-fetchresumable)).
|
|
241
|
+
|
|
242
|
+
A load too big for one run (a large initial sync or a mass update) can span several runs when the source adapter implements `fetchResumable`: on every run but a backfill the SDK saves a checkpoint after each page and the next run carries on. A `run-flow` backfill or `fullResync` saves no checkpoint. This needs integration-service with the `sync_progress` column — deploy it before SDK 0.12.0. See [docs/SYNC-PIPELINE.md](./docs/SYNC-PIPELINE.md#multi-run-syncs-fetchresumable).
|
|
243
|
+
|
|
244
|
+
**Bulk loads:** For a first sync, or after a bulk import or mass update (for example 100k Salesforce records), sync the affected flows by hand with a plain `sync.run-flow` (no `since`, no `fullResync`). Run one flow at a time, parents before children, and re-run each until `run-flow` succeeds with no `stoppedEarly` and no `holdCursor` (a finished run's log has no “continues next run” or “retries” note). Scheduled jobs can carry on a large load by themselves, but a child flow in the same job may then run against a partly loaded parent ([COR-1468](https://linear.app/skedulo/issue/COR-1468)).
|
|
241
245
|
|
|
242
246
|
Artifact names act as idempotency keys and are capped (the SDK throws rather than truncating silently): **255 chars** for per-flow realtime artifacts (system-derived) and **200 chars** for scheduled-job webhooks (derived from the user-supplied job name, so a long display name is rejected up-front). The callback URL baked into each artifact is `{{ SKEDULO_API_URL }}/function/<type>/<type>` — the platform substitutes the tenant API host when the artifact fires.
|
|
243
247
|
|
|
@@ -84,7 +84,7 @@ function makeRunFlowAction(deps) {
|
|
|
84
84
|
// Admin-only action, so a deactivated flow can still be run on demand for
|
|
85
85
|
// testing or a backfill — only unattended runs are gated on status.
|
|
86
86
|
allowInactive: true,
|
|
87
|
-
options: { lookupStrategy },
|
|
87
|
+
options: { lookupStrategy, deadlineAt: Date.now() + constants_1.DEFAULT_RUN_BUDGET_MS },
|
|
88
88
|
}, ctx);
|
|
89
89
|
// errors (raw records) + errorCount go to the run logs, not the HTTP response.
|
|
90
90
|
const { errors: _errors, errorCount: _errorCount } = stats, counts = __rest(stats, ["errors", "errorCount"]);
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"run-flow.action.js","sourceRoot":"","sources":["../../../src/actions/sync/run-flow.action.ts"],"names":[],"mappings":";;;;;;;;;;;;;AAkCA,8CAuDC;AAzFD,mCAAoC;AACpC,+
|
|
1
|
+
{"version":3,"file":"run-flow.action.js","sourceRoot":"","sources":["../../../src/actions/sync/run-flow.action.ts"],"names":[],"mappings":";;;;;;;;;;;;;AAkCA,8CAuDC;AAzFD,mCAAoC;AACpC,+CAAwE;AACxE,4DAAwD;AACxD,wDAAgD;AAChD,8CAAyD;AACzD,4CAAmE;AACnE,0DAAsD;AAItD,SAAS,gBAAgB,CAAC,GAAY,EAAE,KAAa;IACnD,MAAM,GAAG,GAAG,CAAC,GAAG,aAAH,GAAG,cAAH,GAAG,GAAI,EAAE,CAA4B,CAAC;IACnD,MAAM,CAAC,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC;IACrB,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,IAAK,CAAY,IAAI,CAAC,EAAE,CAAC;QAC/C,MAAM,IAAI,0BAAiB,CAAC,GAAG,KAAK,6BAA6B,CAAC,CAAC;IACrE,CAAC;IACD,OAAO,CAAW,CAAC;AACrB,CAAC;AAED,SAAS,cAAc,CAAC,KAAc;IACpC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,MAAM,IAAI,0BAAiB,CAAC,uCAAuC,CAAC,CAAC;IACvE,CAAC;IACD,MAAM,IAAI,GAAG,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,IAAI,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC,EAAE,CAAC;QACjC,MAAM,IAAI,0BAAiB,CAAC,oCAAoC,KAAK,GAAG,CAAC,CAAC;IAC5E,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;GAGG;AACH,SAAgB,iBAAiB,CAAC,IAAc;IAC9C,OAAO,IAAA,4BAAY,EAA8B;QAC/C,IAAI,EAAE,sBAAO,CAAC,IAAI,CAAC,OAAO;QAC1B,YAAY,EAAE,IAAI;QAClB,UAAU,CAAC,GAAG;YACZ,MAAM,GAAG,GAAG,CAAC,GAAG,aAAH,GAAG,cAAH,GAAG,GAAI,EAAE,CAA4B,CAAC;YACnD,MAAM,KAAK,GAAiB,EAAE,MAAM,EAAE,gBAAgB,CAAC,GAAG,EAAE,QAAQ,CAAC,EAAE,CAAC;YACxE,iEAAiE;YACjE,IAAI,GAAG,CAAC,UAAU,KAAK,IAAI,EAAE,CAAC;gBAC5B,KAAK,CAAC,UAAU,GAAG,IAAI,CAAC;YAC1B,CAAC;iBAAM,IAAI,GAAG,CAAC,KAAK,KAAK,SAAS,EAAE,CAAC;gBACnC,KAAK,CAAC,KAAK,GAAG,cAAc,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;YAC1C,CAAC;YACD,OAAO,KAAK,CAAC;QACf,CAAC;QACD,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE,GAAG;;YACtB,MAAM,eAAe,GAAG,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC;YACrD,MAAM,cAAc,GAAG,IAAA,+BAAqB,EAAC,IAAI,CAAC,OAAO,CAAC,CAAC;YAE3D,MAAM,KAAK,GAAG,IAAA,mBAAU,GAAE,CAAC;YAC3B,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,eAAe,CAAC,WAAW,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;YACjE,IAAI,CAAC,IAAI,EAAE,CAAC;gBACV,MAAM,IAAI,sBAAa,CAAC,gBAAgB,EAAE,QAAQ,KAAK,CAAC,MAAM,YAAY,CAAC,CAAC;YAC9E,CAAC;YAED,2FAA2F;YAC3F,MAAM,aAAa,GAAG,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC;YAC5D,yFAAyF;YACzF,GAAG,CAAC,MAAM,CAAC,IAAI,CACb,qBAAqB,KAAK,CAAC,MAAM,UAAU,KAAK,EAAE;gBAChD,CAAC,aAAa,KAAK,IAAI;oBACrB,CAAC,CAAC,cAAc;oBAChB,CAAC,CAAC,aAAa;wBACb,CAAC,CAAC,mBAAmB,aAAa,CAAC,WAAW,EAAE,EAAE;wBAClD,CAAC,CAAC,EAAE,CAAC,CACZ,CAAC;YACF,MAAM,KAAK,GAAG,MAAM,IAAI,wBAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,GAAG,CACnD;gBACE,MAAM,EAAE,KAAK,CAAC,MAAM;gBACpB,KAAK,EAAE,IAAI;gBACX,aAAa;gBACb,YAAY,EAAE,MAAA,IAAI,CAAC,YAAY,mCAAI,eAAe;gBAClD,YAAY,EAAE,MAAA,IAAI,CAAC,YAAY,mCAAI,0BAAc;gBACjD,0EAA0E;gBAC1E,oEAAoE;gBACpE,aAAa,EAAE,IAAI;gBACnB,OAAO,EAAE,EAAE,cAAc,EAAE,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,iCAAqB,EAAE;aAC5E,EACD,GAAG,CACJ,CAAC;YACF,+EAA+E;YAC/E,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE,WAAW,KAAgB,KAAK,EAAhB,MAAM,UAAK,KAAK,EAA/D,wBAAuD,CAAQ,CAAC;YACtE,uBAAS,KAAK,EAAE,MAAM,EAAE,KAAK,CAAC,MAAM,IAAK,MAAM,EAAG;QACpD,CAAC;KACF,CAAC,CAAC;AACL,CAAC"}
|
|
@@ -3,9 +3,12 @@ import type { RunJobInput, RunJobOutput, SyncDeps } from './types';
|
|
|
3
3
|
/**
|
|
4
4
|
* Run a scheduled job: load it from config by `webhook_name`, then run its
|
|
5
5
|
* `flow_ids` sequentially in order. Continue-on-error — one flow failing does
|
|
6
|
-
* not stop the rest
|
|
7
|
-
*
|
|
8
|
-
*
|
|
6
|
+
* not stop the rest. A flow that stops early, holds its last page or throws while still loading
|
|
7
|
+
* (a checkpoint after or before the run, or no `lastRunAt` yet), or is paused with a checkpoint, makes
|
|
8
|
+
* the job skip its later flows this tick. After 24 hours or 3 job intervals, whichever is longer,
|
|
9
|
+
* from the load's `startedAt` (else the flow's `lastRunAt`, else its `createdAt`; a future time
|
|
10
|
+
* doesn't count; with none, at once) later flows run with a WARNING run log.
|
|
11
|
+
* Per-flow direction is decided by the flow's own `sourceSystem`.
|
|
9
12
|
*/
|
|
10
13
|
export declare function makeRunJobAction(deps: SyncDeps): Action<RunJobInput, RunJobOutput>;
|
|
11
14
|
//# sourceMappingURL=run-job.action.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"run-job.action.d.ts","sourceRoot":"","sources":["../../../src/actions/sync/run-job.action.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"run-job.action.d.ts","sourceRoot":"","sources":["../../../src/actions/sync/run-job.action.ts"],"names":[],"mappings":"AAmBA,OAAO,KAAK,EAAE,MAAM,EAAa,MAAM,gBAAgB,CAAC;AAExD,OAAO,KAAK,EAAiB,WAAW,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,SAAS,CAAC;AA0BlF;;;;;;;;;GASG;AACH,wBAAgB,gBAAgB,CAAC,IAAI,EAAE,QAAQ,GAAG,MAAM,CAAC,WAAW,EAAE,YAAY,CAAC,CA2IlF"}
|
|
@@ -14,18 +14,43 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
14
14
|
exports.makeRunJobAction = makeRunJobAction;
|
|
15
15
|
const crypto_1 = require("crypto");
|
|
16
16
|
const constants_1 = require("../../constants");
|
|
17
|
+
const integration_log_interface_1 = require("../../interfaces/integration-log.interface");
|
|
18
|
+
const run_log_builder_1 = require("../../services/run-log-builder");
|
|
17
19
|
const sync_runner_1 = require("../../services/sync-runner");
|
|
20
|
+
const artifact_builder_1 = require("../../utils/artifact-builder");
|
|
18
21
|
const action_names_1 = require("../utils/action-names");
|
|
19
22
|
const helpers_1 = require("../utils/helpers");
|
|
20
23
|
const errors_1 = require("../utils/errors");
|
|
21
24
|
const define_action_1 = require("../utils/define-action");
|
|
22
25
|
const shared_1 = require("../scheduled-job/shared");
|
|
26
|
+
function jobRunBudgetMs(job, logger) {
|
|
27
|
+
let intervalMs;
|
|
28
|
+
try {
|
|
29
|
+
intervalMs = (0, artifact_builder_1.jobIntervalToMinutes)(job) * 60 * 1000;
|
|
30
|
+
}
|
|
31
|
+
catch (err) {
|
|
32
|
+
logger.warn(`[run-job] ${job.webhook_name} interval can't be read (${err instanceof Error ? err.message : String(err)}) — using the default budget of ${constants_1.DEFAULT_RUN_BUDGET_MS / 60000} min`);
|
|
33
|
+
return constants_1.DEFAULT_RUN_BUDGET_MS;
|
|
34
|
+
}
|
|
35
|
+
return Math.max(constants_1.MIN_RUN_BUDGET_MS, Math.min(constants_1.DEFAULT_RUN_BUDGET_MS, intervalMs - constants_1.RUN_BUDGET_INTERVAL_MARGIN_MS));
|
|
36
|
+
}
|
|
37
|
+
function jobWaitLimitMs(job) {
|
|
38
|
+
try {
|
|
39
|
+
return Math.max(constants_1.SLOW_LOAD_MS, 3 * (0, artifact_builder_1.jobIntervalToMinutes)(job) * 60000);
|
|
40
|
+
}
|
|
41
|
+
catch (_a) {
|
|
42
|
+
return constants_1.SLOW_LOAD_MS;
|
|
43
|
+
}
|
|
44
|
+
}
|
|
23
45
|
/**
|
|
24
46
|
* Run a scheduled job: load it from config by `webhook_name`, then run its
|
|
25
47
|
* `flow_ids` sequentially in order. Continue-on-error — one flow failing does
|
|
26
|
-
* not stop the rest
|
|
27
|
-
*
|
|
28
|
-
*
|
|
48
|
+
* not stop the rest. A flow that stops early, holds its last page or throws while still loading
|
|
49
|
+
* (a checkpoint after or before the run, or no `lastRunAt` yet), or is paused with a checkpoint, makes
|
|
50
|
+
* the job skip its later flows this tick. After 24 hours or 3 job intervals, whichever is longer,
|
|
51
|
+
* from the load's `startedAt` (else the flow's `lastRunAt`, else its `createdAt`; a future time
|
|
52
|
+
* doesn't count; with none, at once) later flows run with a WARNING run log.
|
|
53
|
+
* Per-flow direction is decided by the flow's own `sourceSystem`.
|
|
29
54
|
*/
|
|
30
55
|
function makeRunJobAction(deps) {
|
|
31
56
|
return (0, define_action_1.defineAction)({
|
|
@@ -38,7 +63,7 @@ function makeRunJobAction(deps) {
|
|
|
38
63
|
return { webhookName: (0, shared_1.parseWebhookName)(raw) };
|
|
39
64
|
},
|
|
40
65
|
async handler(input, ctx) {
|
|
41
|
-
var _a, _b;
|
|
66
|
+
var _a, _b, _c, _d, _e, _f, _g;
|
|
42
67
|
const integrationType = deps.adapter.integrationType;
|
|
43
68
|
const lookupStrategy = (0, helpers_1.resolveLookupStrategy)(deps.adapter);
|
|
44
69
|
const config = await (0, shared_1.readConfig)(ctx, integrationType);
|
|
@@ -56,10 +81,25 @@ function makeRunJobAction(deps) {
|
|
|
56
81
|
ctx.jobRunId = (0, crypto_1.randomUUID)();
|
|
57
82
|
const runner = new sync_runner_1.SyncRunner(deps.registry);
|
|
58
83
|
const results = [];
|
|
84
|
+
// All flows in a job run in one function call, so they share one budget.
|
|
85
|
+
const deadlineAt = Date.now() + jobRunBudgetMs(job, ctx.logger);
|
|
86
|
+
const waitLimitMs = jobWaitLimitMs(job);
|
|
87
|
+
let waitingFor;
|
|
88
|
+
let partlyLoaded;
|
|
59
89
|
for (const flowId of job.flow_ids) {
|
|
90
|
+
if (waitingFor !== undefined) {
|
|
91
|
+
const reason = `waiting for flow ${waitingFor} to finish its load`;
|
|
92
|
+
ctx.logger.info(`[run-job] ${job.webhook_name} flow=${flowId} skipped — ${reason}`);
|
|
93
|
+
await logJobNote(ctx, flowId, integration_log_interface_1.LogLevel.INFO, `Flow ${flowId} skipped, ${reason} — nothing synced`);
|
|
94
|
+
results.push({ flowId, skipped: true, reason });
|
|
95
|
+
continue;
|
|
96
|
+
}
|
|
60
97
|
const startedAt = Date.now();
|
|
98
|
+
let unfinished = false;
|
|
99
|
+
let stopped = false;
|
|
100
|
+
let before = null;
|
|
61
101
|
try {
|
|
62
|
-
const flow = await ctx.dataFlowService.getFlowById(flowId);
|
|
102
|
+
const flow = (before = await ctx.dataFlowService.getFlowById(flowId));
|
|
63
103
|
if (!flow) {
|
|
64
104
|
ctx.logger.warn(`[run-job] ${job.webhook_name} flow=${flowId} not found — skipping`);
|
|
65
105
|
results.push({ flowId, skipped: true, reason: 'flow not found' });
|
|
@@ -70,28 +110,79 @@ function makeRunJobAction(deps) {
|
|
|
70
110
|
since: null,
|
|
71
111
|
sourceSystem: (_a = flow.sourceSystem) !== null && _a !== void 0 ? _a : integrationType,
|
|
72
112
|
targetSystem: (_b = flow.targetSystem) !== null && _b !== void 0 ? _b : constants_1.SKEDULO_SYSTEM,
|
|
73
|
-
options: { lookupStrategy },
|
|
113
|
+
options: { lookupStrategy, deadlineAt },
|
|
74
114
|
}, ctx);
|
|
75
115
|
if (stats.skippedReason === 'flow_inactive') {
|
|
76
116
|
results.push({ flowId, skipped: true, reason: 'flow inactive' });
|
|
77
|
-
|
|
117
|
+
unfinished = !!flow.syncProgress;
|
|
118
|
+
}
|
|
119
|
+
else {
|
|
120
|
+
if (partlyLoaded)
|
|
121
|
+
await logJobNote(ctx, flowId, integration_log_interface_1.LogLevel.WARNING, partlyLoaded);
|
|
122
|
+
stopped = stats.stoppedEarly !== undefined;
|
|
123
|
+
unfinished = stopped || stats.holdCursor !== undefined;
|
|
124
|
+
const durationMs = Date.now() - startedAt;
|
|
125
|
+
// errors (raw records) + errorCount go to the run logs, never stdout or the HTTP response.
|
|
126
|
+
// Strip BEFORE logging so a record's PII can't leak via JSON.stringify(stats).
|
|
127
|
+
const { errors: _errors, errorCount: _errorCount } = stats, counts = __rest(stats, ["errors", "errorCount"]);
|
|
128
|
+
ctx.logger.info(`[run-job] ${job.webhook_name} flow=${flowId} ${flow.sourceObject}→${flow.targetObject} ` +
|
|
129
|
+
`${JSON.stringify(counts)} took=${durationMs}ms`);
|
|
130
|
+
results.push(Object.assign({ flowId, sourceObject: flow.sourceObject, targetObject: flow.targetObject, durationMs }, counts));
|
|
78
131
|
}
|
|
79
|
-
const durationMs = Date.now() - startedAt;
|
|
80
|
-
// errors (raw records) + errorCount go to the run logs, never stdout or the HTTP response.
|
|
81
|
-
// Strip BEFORE logging so a record's PII can't leak via JSON.stringify(stats).
|
|
82
|
-
const { errors: _errors, errorCount: _errorCount } = stats, counts = __rest(stats, ["errors", "errorCount"]);
|
|
83
|
-
ctx.logger.info(`[run-job] ${job.webhook_name} flow=${flowId} ${flow.sourceObject}→${flow.targetObject} ` +
|
|
84
|
-
`${JSON.stringify(counts)} took=${durationMs}ms`);
|
|
85
|
-
results.push(Object.assign({ flowId, sourceObject: flow.sourceObject, targetObject: flow.targetObject, durationMs }, counts));
|
|
86
132
|
}
|
|
87
133
|
catch (err) {
|
|
88
134
|
const msg = err instanceof Error ? err.message : String(err);
|
|
89
135
|
ctx.logger.error(`[run-job] ${job.webhook_name} flow=${flowId} failed: ${msg}`);
|
|
90
136
|
results.push({ flowId, error: msg });
|
|
137
|
+
unfinished = true;
|
|
138
|
+
}
|
|
139
|
+
if (!unfinished)
|
|
140
|
+
continue;
|
|
141
|
+
// Read it again: the run may have saved or dropped the checkpoint. If the read fails, wait.
|
|
142
|
+
const after = await ctx.dataFlowService.getFlowById(flowId).catch(() => undefined);
|
|
143
|
+
const loading = after === undefined ||
|
|
144
|
+
(after !== null &&
|
|
145
|
+
(stopped || !!(after.syncProgress || (before === null || before === void 0 ? void 0 : before.syncProgress) || !after.lastRunAt)));
|
|
146
|
+
if (!loading)
|
|
147
|
+
continue;
|
|
148
|
+
// No valid start time (a future one isn't valid, as in SyncRunner): count from the last finished load,
|
|
149
|
+
// else from when the flow was created, else don't wait.
|
|
150
|
+
const loadStart = [
|
|
151
|
+
(_c = after === null || after === void 0 ? void 0 : after.syncProgress) === null || _c === void 0 ? void 0 : _c.startedAt,
|
|
152
|
+
(_d = before === null || before === void 0 ? void 0 : before.syncProgress) === null || _d === void 0 ? void 0 : _d.startedAt,
|
|
153
|
+
(_f = (_e = (after !== null && after !== void 0 ? after : before)) === null || _e === void 0 ? void 0 : _e.lastRunAt) === null || _f === void 0 ? void 0 : _f.toJSON(),
|
|
154
|
+
(_g = (after !== null && after !== void 0 ? after : before)) === null || _g === void 0 ? void 0 : _g.createdAt,
|
|
155
|
+
]
|
|
156
|
+
.map((t) => Date.parse(t !== null && t !== void 0 ? t : ''))
|
|
157
|
+
.find((t) => t <= Date.now());
|
|
158
|
+
if (loadStart === undefined || startedAt - loadStart > waitLimitMs) {
|
|
159
|
+
ctx.logger.warn(`sync_job_wait_skipped: flowId=${flowId} load started over ${waitLimitMs / 3600000}h ago or has no start time, ` +
|
|
160
|
+
`later flows run against a partly loaded parent`);
|
|
161
|
+
partlyLoaded =
|
|
162
|
+
`Flow ${flowId}'s load started over ${waitLimitMs / 3600000} hours ago or has no start time, and hasn't finished, so later flows in this job run now. ` +
|
|
163
|
+
`They may sync against a partly loaded flow ${flowId} and skip records that point to it; see "Multi-run syncs" in the SDK's SYNC-PIPELINE docs`;
|
|
164
|
+
await logJobNote(ctx, flowId, integration_log_interface_1.LogLevel.WARNING, partlyLoaded);
|
|
165
|
+
}
|
|
166
|
+
else {
|
|
167
|
+
waitingFor = flowId;
|
|
91
168
|
}
|
|
92
169
|
}
|
|
93
170
|
return { webhookName: job.webhook_name, results };
|
|
94
171
|
},
|
|
95
172
|
});
|
|
96
173
|
}
|
|
174
|
+
async function logJobNote(ctx, flowId, level, message) {
|
|
175
|
+
const entries = (0, run_log_builder_1.buildRunLogEntries)({
|
|
176
|
+
integrationType: ctx.integrationType,
|
|
177
|
+
flowId,
|
|
178
|
+
executionId: (0, crypto_1.randomUUID)(),
|
|
179
|
+
source: integration_log_interface_1.LogSource.SCHEDULED_JOB,
|
|
180
|
+
durationMs: 0,
|
|
181
|
+
summary: { succeeded: 0, filtered: 0, skipped: 0, failed: 0, dlqWrites: 0 },
|
|
182
|
+
summaryLevel: level,
|
|
183
|
+
summaryMessage: message,
|
|
184
|
+
jobRunId: ctx.jobRunId,
|
|
185
|
+
});
|
|
186
|
+
await (0, run_log_builder_1.flushRunLog)(ctx, entries, flowId);
|
|
187
|
+
}
|
|
97
188
|
//# sourceMappingURL=run-job.action.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"run-job.action.js","sourceRoot":"","sources":["../../../src/actions/sync/run-job.action.ts"],"names":[],"mappings":";;;;;;;;;;;;;
|
|
1
|
+
{"version":3,"file":"run-job.action.js","sourceRoot":"","sources":["../../../src/actions/sync/run-job.action.ts"],"names":[],"mappings":";;;;;;;;;;;;;AAyDA,4CA2IC;AApMD,mCAAoC;AACpC,+CAMyB;AAIzB,0FAAiF;AACjF,oEAAiF;AACjF,4DAAwD;AACxD,mEAAoE;AACpE,wDAAgD;AAChD,8CAAyD;AACzD,4CAAgD;AAChD,0DAAsD;AAEtD,oDAAqF;AAGrF,SAAS,cAAc,CAAC,GAAiB,EAAE,MAAoC;IAC7E,IAAI,UAAkB,CAAC;IACvB,IAAI,CAAC;QACH,UAAU,GAAG,IAAA,uCAAoB,EAAC,GAAG,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC;IACrD,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,CAAC,IAAI,CACT,aAAa,GAAG,CAAC,YAAY,4BAA4B,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,mCAAmC,iCAAqB,GAAG,KAAM,MAAM,CACjL,CAAC;QACF,OAAO,iCAAqB,CAAC;IAC/B,CAAC;IACD,OAAO,IAAI,CAAC,GAAG,CACb,6BAAiB,EACjB,IAAI,CAAC,GAAG,CAAC,iCAAqB,EAAE,UAAU,GAAG,yCAA6B,CAAC,CAC5E,CAAC;AACJ,CAAC;AAED,SAAS,cAAc,CAAC,GAAiB;IACvC,IAAI,CAAC;QACH,OAAO,IAAI,CAAC,GAAG,CAAC,wBAAY,EAAE,CAAC,GAAG,IAAA,uCAAoB,EAAC,GAAG,CAAC,GAAG,KAAM,CAAC,CAAC;IACxE,CAAC;IAAC,WAAM,CAAC;QACP,OAAO,wBAAY,CAAC;IACtB,CAAC;AACH,CAAC;AAED;;;;;;;;;GASG;AACH,SAAgB,gBAAgB,CAAC,IAAc;IAC7C,OAAO,IAAA,4BAAY,EAA4B;QAC7C,IAAI,EAAE,sBAAO,CAAC,IAAI,CAAC,MAAM;QACzB,UAAU,CAAC,GAAG,EAAE,IAA2B;YACzC,MAAM,QAAQ,GAAG,IAAI,aAAJ,IAAI,uBAAJ,IAAI,CAAE,gBAAgB,CAAC;YACxC,IAAI,OAAO,QAAQ,KAAK,QAAQ,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBACxD,OAAO,EAAE,WAAW,EAAE,QAAQ,EAAE,CAAC;YACnC,CAAC;YACD,OAAO,EAAE,WAAW,EAAE,IAAA,yBAAgB,EAAC,GAAG,CAAC,EAAE,CAAC;QAChD,CAAC;QACD,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE,GAAG;;YACtB,MAAM,eAAe,GAAG,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC;YACrD,MAAM,cAAc,GAAG,IAAA,+BAAqB,EAAC,IAAI,CAAC,OAAO,CAAC,CAAC;YAE3D,MAAM,MAAM,GAAG,MAAM,IAAA,mBAAU,EAAC,GAAG,EAAE,eAAe,CAAC,CAAC;YACtD,MAAM,GAAG,GAAG,MAAM,CAAC,cAAc,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,YAAY,KAAK,KAAK,CAAC,WAAW,CAAC,CAAC;YACpF,IAAI,CAAC,GAAG,EAAE,CAAC;gBACT,MAAM,IAAI,sBAAa,CAAC,eAAe,EAAE,kBAAkB,KAAK,CAAC,WAAW,aAAa,CAAC,CAAC;YAC7F,CAAC;YAED,IAAI,CAAC,IAAA,qBAAY,EAAC,GAAG,CAAC,EAAE,CAAC;gBACvB,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,aAAa,GAAG,CAAC,YAAY,iCAAiC,CAAC,CAAC;gBAChF,OAAO,EAAE,WAAW,EAAE,GAAG,CAAC,YAAY,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;YACvE,CAAC;YAED,GAAG,CAAC,MAAM,CAAC,IAAI,CACb,aAAa,GAAG,CAAC,YAAY,YAAY,GAAG,CAAC,QAAQ,CAAC,MAAM,qBAAqB;gBAC/E,GAAG,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,CAC1B,CAAC;YAEF,4EAA4E;YAC5E,GAAG,CAAC,QAAQ,GAAG,IAAA,mBAAU,GAAE,CAAC;YAE5B,MAAM,MAAM,GAAG,IAAI,wBAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;YAC7C,MAAM,OAAO,GAAoB,EAAE,CAAC;YACpC,yEAAyE;YACzE,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,cAAc,CAAC,GAAG,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;YAChE,MAAM,WAAW,GAAG,cAAc,CAAC,GAAG,CAAC,CAAC;YAExC,IAAI,UAA8B,CAAC;YACnC,IAAI,YAAgC,CAAC;YACrC,KAAK,MAAM,MAAM,IAAI,GAAG,CAAC,QAAQ,EAAE,CAAC;gBAClC,IAAI,UAAU,KAAK,SAAS,EAAE,CAAC;oBAC7B,MAAM,MAAM,GAAG,oBAAoB,UAAU,qBAAqB,CAAC;oBACnE,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,aAAa,GAAG,CAAC,YAAY,SAAS,MAAM,cAAc,MAAM,EAAE,CAAC,CAAC;oBACpF,MAAM,UAAU,CACd,GAAG,EACH,MAAM,EACN,oCAAQ,CAAC,IAAI,EACb,QAAQ,MAAM,aAAa,MAAM,mBAAmB,CACrD,CAAC;oBACF,OAAO,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC,CAAC;oBAChD,SAAS;gBACX,CAAC;gBACD,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;gBAC7B,IAAI,UAAU,GAAG,KAAK,CAAC;gBACvB,IAAI,OAAO,GAAG,KAAK,CAAC;gBACpB,IAAI,MAAM,GAAoB,IAAI,CAAC;gBACnC,IAAI,CAAC;oBACH,MAAM,IAAI,GAAG,CAAC,MAAM,GAAG,MAAM,GAAG,CAAC,eAAe,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC,CAAC;oBACtE,IAAI,CAAC,IAAI,EAAE,CAAC;wBACV,GAAG,CAAC,MAAM,CAAC,IAAI,CAAC,aAAa,GAAG,CAAC,YAAY,SAAS,MAAM,uBAAuB,CAAC,CAAC;wBACrF,OAAO,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,gBAAgB,EAAE,CAAC,CAAC;wBAClE,SAAS;oBACX,CAAC;oBACD,MAAM,KAAK,GAAG,MAAM,MAAM,CAAC,GAAG,CAC5B;wBACE,MAAM;wBACN,KAAK,EAAE,IAAI;wBACX,YAAY,EAAE,MAAA,IAAI,CAAC,YAAY,mCAAI,eAAe;wBAClD,YAAY,EAAE,MAAA,IAAI,CAAC,YAAY,mCAAI,0BAAc;wBACjD,OAAO,EAAE,EAAE,cAAc,EAAE,UAAU,EAAE;qBACxC,EACD,GAAG,CACJ,CAAC;oBACF,IAAI,KAAK,CAAC,aAAa,KAAK,eAAe,EAAE,CAAC;wBAC5C,OAAO,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,eAAe,EAAE,CAAC,CAAC;wBACjE,UAAU,GAAG,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC;oBACnC,CAAC;yBAAM,CAAC;wBACN,IAAI,YAAY;4BAAE,MAAM,UAAU,CAAC,GAAG,EAAE,MAAM,EAAE,oCAAQ,CAAC,OAAO,EAAE,YAAY,CAAC,CAAC;wBAChF,OAAO,GAAG,KAAK,CAAC,YAAY,KAAK,SAAS,CAAC;wBAC3C,UAAU,GAAG,OAAO,IAAI,KAAK,CAAC,UAAU,KAAK,SAAS,CAAC;wBACvD,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;wBAC1C,2FAA2F;wBAC3F,+EAA+E;wBAC/E,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE,WAAW,KAAgB,KAAK,EAAhB,MAAM,UAAK,KAAK,EAA/D,wBAAuD,CAAQ,CAAC;wBACtE,GAAG,CAAC,MAAM,CAAC,IAAI,CACb,aAAa,GAAG,CAAC,YAAY,SAAS,MAAM,IAAI,IAAI,CAAC,YAAY,IAAI,IAAI,CAAC,YAAY,GAAG;4BACvF,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,SAAS,UAAU,IAAI,CACnD,CAAC;wBACF,OAAO,CAAC,IAAI,iBACV,MAAM,EACN,YAAY,EAAE,IAAI,CAAC,YAAY,EAC/B,YAAY,EAAE,IAAI,CAAC,YAAY,EAC/B,UAAU,IACP,MAAM,EACT,CAAC;oBACL,CAAC;gBACH,CAAC;gBAAC,OAAO,GAAG,EAAE,CAAC;oBACb,MAAM,GAAG,GAAG,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;oBAC7D,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,aAAa,GAAG,CAAC,YAAY,SAAS,MAAM,YAAY,GAAG,EAAE,CAAC,CAAC;oBAChF,OAAO,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,EAAE,CAAC,CAAC;oBACrC,UAAU,GAAG,IAAI,CAAC;gBACpB,CAAC;gBACD,IAAI,CAAC,UAAU;oBAAE,SAAS;gBAC1B,4FAA4F;gBAC5F,MAAM,KAAK,GAAG,MAAM,GAAG,CAAC,eAAe,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;gBACnF,MAAM,OAAO,GACX,KAAK,KAAK,SAAS;oBACnB,CAAC,KAAK,KAAK,IAAI;wBACb,CAAC,OAAO,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,YAAY,KAAI,MAAM,aAAN,MAAM,uBAAN,MAAM,CAAE,YAAY,CAAA,IAAI,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC;gBACrF,IAAI,CAAC,OAAO;oBAAE,SAAS;gBACvB,uGAAuG;gBACvG,wDAAwD;gBACxD,MAAM,SAAS,GAAG;oBAChB,MAAA,KAAK,aAAL,KAAK,uBAAL,KAAK,CAAE,YAAY,0CAAE,SAAS;oBAC9B,MAAA,MAAM,aAAN,MAAM,uBAAN,MAAM,CAAE,YAAY,0CAAE,SAAS;oBAC/B,MAAA,MAAA,CAAC,KAAK,aAAL,KAAK,cAAL,KAAK,GAAI,MAAM,CAAC,0CAAE,SAAS,0CAAE,MAAM,EAAE;oBACtC,MAAA,CAAC,KAAK,aAAL,KAAK,cAAL,KAAK,GAAI,MAAM,CAAC,0CAAE,SAAS;iBAC7B;qBACE,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,aAAD,CAAC,cAAD,CAAC,GAAI,EAAE,CAAC,CAAC;qBAC/B,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;gBAChC,IAAI,SAAS,KAAK,SAAS,IAAI,SAAS,GAAG,SAAS,GAAG,WAAW,EAAE,CAAC;oBACnE,GAAG,CAAC,MAAM,CAAC,IAAI,CACb,iCAAiC,MAAM,sBAAsB,WAAW,GAAG,OAAS,8BAA8B;wBAChH,gDAAgD,CACnD,CAAC;oBACF,YAAY;wBACV,QAAQ,MAAM,wBAAwB,WAAW,GAAG,OAAS,4FAA4F;4BACzJ,8CAA8C,MAAM,2FAA2F,CAAC;oBAClJ,MAAM,UAAU,CAAC,GAAG,EAAE,MAAM,EAAE,oCAAQ,CAAC,OAAO,EAAE,YAAY,CAAC,CAAC;gBAChE,CAAC;qBAAM,CAAC;oBACN,UAAU,GAAG,MAAM,CAAC;gBACtB,CAAC;YACH,CAAC;YAED,OAAO,EAAE,WAAW,EAAE,GAAG,CAAC,YAAY,EAAE,OAAO,EAAE,CAAC;QACpD,CAAC;KACF,CAAC,CAAC;AACL,CAAC;AAED,KAAK,UAAU,UAAU,CACvB,GAAuB,EACvB,MAAc,EACd,KAAe,EACf,OAAe;IAEf,MAAM,OAAO,GAAG,IAAA,oCAAkB,EAAC;QACjC,eAAe,EAAE,GAAG,CAAC,eAAe;QACpC,MAAM;QACN,WAAW,EAAE,IAAA,mBAAU,GAAE;QACzB,MAAM,EAAE,qCAAS,CAAC,aAAa;QAC/B,UAAU,EAAE,CAAC;QACb,OAAO,EAAE,EAAE,SAAS,EAAE,CAAC,EAAE,QAAQ,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE;QAC3E,YAAY,EAAE,KAAK;QACnB,cAAc,EAAE,OAAO;QACvB,QAAQ,EAAE,GAAG,CAAC,QAAQ;KACvB,CAAC,CAAC;IACH,MAAM,IAAA,6BAAW,EAAC,GAAG,EAAE,OAAO,EAAE,MAAM,CAAC,CAAC;AAC1C,CAAC"}
|
package/dist/browser.d.ts
CHANGED
|
@@ -2,18 +2,19 @@ export type { LookupRule } from './interfaces/lookup-rule.interface';
|
|
|
2
2
|
export type { FieldMapping, MappingConfigDto, MappingConfigMapping, TranslationTable, ValidationAction, } from './interfaces/field-mapping.interface';
|
|
3
3
|
export type { FilterOperator, FilterOperatorOption, FilterRule, FilterSpec, } from './interfaces/filter.interface';
|
|
4
4
|
export { FILTER_OPERATORS } from './interfaces/filter.interface';
|
|
5
|
-
export type { DataFlow } from './interfaces/data-flow-spec.interface';
|
|
5
|
+
export type { DataFlow, SyncProgress } from './interfaces/data-flow-spec.interface';
|
|
6
6
|
export type { ScheduledJob, IntegrationConfig } from './interfaces/scheduled-job.interface';
|
|
7
7
|
export { JobIntervalUnit } from './interfaces/scheduled-job.interface';
|
|
8
8
|
export type { MappedEvent } from './interfaces/mapped-event.interface';
|
|
9
|
-
export type { SyncResult, BatchResult, PreviewResult } from './interfaces/sync-result.interface';
|
|
9
|
+
export type { SyncResult, BatchResult, CheckpointStop, PreviewResult, } from './interfaces/sync-result.interface';
|
|
10
10
|
export type { SampleFetcher } from './interfaces/sample-fetcher.interface';
|
|
11
|
-
export type { Adapter, EnrichmentPair, UpsertOutcome, ObjectInfo, FieldInfo, SampleRecord, DataFlowTemplate, SampleAPI, } from './interfaces/adapter.interface';
|
|
11
|
+
export type { Adapter, EnrichmentPair, FetchPage, UpsertOutcome, ObjectInfo, FieldInfo, SampleRecord, DataFlowTemplate, SampleAPI, } from './interfaces/adapter.interface';
|
|
12
12
|
export type { IntegrationCapabilities } from './interfaces/integration-capabilities.interface';
|
|
13
13
|
export { SyncStage } from './interfaces/sync-pipeline.interface';
|
|
14
14
|
export type { SyncPipelineState, SyncPipelineError } from './interfaces/sync-pipeline.interface';
|
|
15
15
|
export { LogLevel, LogKind, LogSource } from './interfaces/integration-log.interface';
|
|
16
16
|
export type { IntegrationLogEntry } from './interfaces/integration-log.interface';
|
|
17
|
+
export { DEFAULT_RUN_BUDGET_MS, MAX_SYNC_PROGRESS_CHARS, MAX_CHECKPOINT_HOLDS, MAX_FAILED_RESUMES, FAILED_RESUME_MIN_MS, } from './constants';
|
|
17
18
|
export { ACTION_HEADER, ACTIONS, SCHEDULED_JOB_NAME_HEADER } from './actions/utils/action-names';
|
|
18
19
|
export type { ActionName } from './actions/utils/action-names';
|
|
19
20
|
export type { DataFlowPreviewInput, DataFlowPreviewOutput } from './actions/data-flow/types';
|
package/dist/browser.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"browser.d.ts","sourceRoot":"","sources":["../src/browser.ts"],"names":[],"mappings":"AAIA,YAAY,EAAE,UAAU,EAAE,MAAM,oCAAoC,CAAC;AACrE,YAAY,EACV,YAAY,EACZ,gBAAgB,EAChB,oBAAoB,EACpB,gBAAgB,EAChB,gBAAgB,GACjB,MAAM,sCAAsC,CAAC;AAC9C,YAAY,EACV,cAAc,EACd,oBAAoB,EACpB,UAAU,EACV,UAAU,GACX,MAAM,+BAA+B,CAAC;AACvC,OAAO,EAAE,gBAAgB,EAAE,MAAM,+BAA+B,CAAC;AACjE,YAAY,EAAE,QAAQ,EAAE,MAAM,uCAAuC,CAAC;
|
|
1
|
+
{"version":3,"file":"browser.d.ts","sourceRoot":"","sources":["../src/browser.ts"],"names":[],"mappings":"AAIA,YAAY,EAAE,UAAU,EAAE,MAAM,oCAAoC,CAAC;AACrE,YAAY,EACV,YAAY,EACZ,gBAAgB,EAChB,oBAAoB,EACpB,gBAAgB,EAChB,gBAAgB,GACjB,MAAM,sCAAsC,CAAC;AAC9C,YAAY,EACV,cAAc,EACd,oBAAoB,EACpB,UAAU,EACV,UAAU,GACX,MAAM,+BAA+B,CAAC;AACvC,OAAO,EAAE,gBAAgB,EAAE,MAAM,+BAA+B,CAAC;AACjE,YAAY,EAAE,QAAQ,EAAE,YAAY,EAAE,MAAM,uCAAuC,CAAC;AACpF,YAAY,EAAE,YAAY,EAAE,iBAAiB,EAAE,MAAM,sCAAsC,CAAC;AAC5F,OAAO,EAAE,eAAe,EAAE,MAAM,sCAAsC,CAAC;AACvE,YAAY,EAAE,WAAW,EAAE,MAAM,qCAAqC,CAAC;AACvE,YAAY,EACV,UAAU,EACV,WAAW,EACX,cAAc,EACd,aAAa,GACd,MAAM,oCAAoC,CAAC;AAC5C,YAAY,EAAE,aAAa,EAAE,MAAM,uCAAuC,CAAC;AAC3E,YAAY,EACV,OAAO,EACP,cAAc,EACd,SAAS,EACT,aAAa,EACb,UAAU,EACV,SAAS,EACT,YAAY,EACZ,gBAAgB,EAChB,SAAS,GACV,MAAM,gCAAgC,CAAC;AACxC,YAAY,EAAE,uBAAuB,EAAE,MAAM,iDAAiD,CAAC;AAC/F,OAAO,EAAE,SAAS,EAAE,MAAM,sCAAsC,CAAC;AACjE,YAAY,EAAE,iBAAiB,EAAE,iBAAiB,EAAE,MAAM,sCAAsC,CAAC;AACjG,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,wCAAwC,CAAC;AACtF,YAAY,EAAE,mBAAmB,EAAE,MAAM,wCAAwC,CAAC;AAClF,OAAO,EACL,qBAAqB,EACrB,uBAAuB,EACvB,oBAAoB,EACpB,kBAAkB,EAClB,oBAAoB,GACrB,MAAM,aAAa,CAAC;AAIrB,OAAO,EAAE,aAAa,EAAE,OAAO,EAAE,yBAAyB,EAAE,MAAM,8BAA8B,CAAC;AACjG,YAAY,EAAE,UAAU,EAAE,MAAM,8BAA8B,CAAC;AAC/D,YAAY,EAAE,oBAAoB,EAAE,qBAAqB,EAAE,MAAM,2BAA2B,CAAC"}
|
package/dist/browser.js
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// No Node.js built-ins, no PSS, no FME.
|
|
4
4
|
// Use this in browser/UI bundles: import { FILTER_OPERATORS } from '@skedulo/integration-sdk/browser'
|
|
5
5
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
6
|
-
exports.SCHEDULED_JOB_NAME_HEADER = exports.ACTIONS = exports.ACTION_HEADER = exports.LogSource = exports.LogKind = exports.LogLevel = exports.SyncStage = exports.JobIntervalUnit = exports.FILTER_OPERATORS = void 0;
|
|
6
|
+
exports.SCHEDULED_JOB_NAME_HEADER = exports.ACTIONS = exports.ACTION_HEADER = exports.FAILED_RESUME_MIN_MS = exports.MAX_FAILED_RESUMES = exports.MAX_CHECKPOINT_HOLDS = exports.MAX_SYNC_PROGRESS_CHARS = exports.DEFAULT_RUN_BUDGET_MS = exports.LogSource = exports.LogKind = exports.LogLevel = exports.SyncStage = exports.JobIntervalUnit = exports.FILTER_OPERATORS = void 0;
|
|
7
7
|
var filter_interface_1 = require("./interfaces/filter.interface");
|
|
8
8
|
Object.defineProperty(exports, "FILTER_OPERATORS", { enumerable: true, get: function () { return filter_interface_1.FILTER_OPERATORS; } });
|
|
9
9
|
var scheduled_job_interface_1 = require("./interfaces/scheduled-job.interface");
|
|
@@ -14,6 +14,12 @@ var integration_log_interface_1 = require("./interfaces/integration-log.interfac
|
|
|
14
14
|
Object.defineProperty(exports, "LogLevel", { enumerable: true, get: function () { return integration_log_interface_1.LogLevel; } });
|
|
15
15
|
Object.defineProperty(exports, "LogKind", { enumerable: true, get: function () { return integration_log_interface_1.LogKind; } });
|
|
16
16
|
Object.defineProperty(exports, "LogSource", { enumerable: true, get: function () { return integration_log_interface_1.LogSource; } });
|
|
17
|
+
var constants_1 = require("./constants");
|
|
18
|
+
Object.defineProperty(exports, "DEFAULT_RUN_BUDGET_MS", { enumerable: true, get: function () { return constants_1.DEFAULT_RUN_BUDGET_MS; } });
|
|
19
|
+
Object.defineProperty(exports, "MAX_SYNC_PROGRESS_CHARS", { enumerable: true, get: function () { return constants_1.MAX_SYNC_PROGRESS_CHARS; } });
|
|
20
|
+
Object.defineProperty(exports, "MAX_CHECKPOINT_HOLDS", { enumerable: true, get: function () { return constants_1.MAX_CHECKPOINT_HOLDS; } });
|
|
21
|
+
Object.defineProperty(exports, "MAX_FAILED_RESUMES", { enumerable: true, get: function () { return constants_1.MAX_FAILED_RESUMES; } });
|
|
22
|
+
Object.defineProperty(exports, "FAILED_RESUME_MIN_MS", { enumerable: true, get: function () { return constants_1.FAILED_RESUME_MIN_MS; } });
|
|
17
23
|
// Dispatcher wire constants — pure strings, safe in the browser. Import these
|
|
18
24
|
// in the UI so action names / the action header are never hardcoded.
|
|
19
25
|
var action_names_1 = require("./actions/utils/action-names");
|
package/dist/browser.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"browser.js","sourceRoot":"","sources":["../src/browser.ts"],"names":[],"mappings":";AAAA,oEAAoE;AACpE,wCAAwC;AACxC,sGAAsG;;;AAgBtG,kEAAiE;AAAxD,oHAAA,gBAAgB,OAAA;AAGzB,gFAAuE;AAA9D,0HAAA,eAAe,OAAA;
|
|
1
|
+
{"version":3,"file":"browser.js","sourceRoot":"","sources":["../src/browser.ts"],"names":[],"mappings":";AAAA,oEAAoE;AACpE,wCAAwC;AACxC,sGAAsG;;;AAgBtG,kEAAiE;AAAxD,oHAAA,gBAAgB,OAAA;AAGzB,gFAAuE;AAA9D,0HAAA,eAAe,OAAA;AAqBxB,gFAAiE;AAAxD,oHAAA,SAAS,OAAA;AAElB,oFAAsF;AAA7E,qHAAA,QAAQ,OAAA;AAAE,oHAAA,OAAO,OAAA;AAAE,sHAAA,SAAS,OAAA;AAErC,yCAMqB;AALnB,kHAAA,qBAAqB,OAAA;AACrB,oHAAA,uBAAuB,OAAA;AACvB,iHAAA,oBAAoB,OAAA;AACpB,+GAAA,kBAAkB,OAAA;AAClB,iHAAA,oBAAoB,OAAA;AAGtB,8EAA8E;AAC9E,qEAAqE;AACrE,6DAAiG;AAAxF,6GAAA,aAAa,OAAA;AAAE,uGAAA,OAAO,OAAA;AAAE,yHAAA,yBAAyB,OAAA"}
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { BaseClient } from '@skedulo/pulse-solution-services';
|
|
2
2
|
import type { BaseConfig, ExecutionOptions } from '@skedulo/pulse-solution-services';
|
|
3
|
-
import type { DataFlow } from '../interfaces/data-flow-spec.interface';
|
|
3
|
+
import type { DataFlow, SyncProgress } from '../interfaces/data-flow-spec.interface';
|
|
4
4
|
import type { MappingConfigDto } from '../interfaces/field-mapping.interface';
|
|
5
5
|
import type { FilterSpec } from '../interfaces/filter.interface';
|
|
6
6
|
export interface DataFlowListParams {
|
|
@@ -48,5 +48,9 @@ export declare class DataFlowClient extends BaseClient {
|
|
|
48
48
|
activate(id: number): Promise<void>;
|
|
49
49
|
deactivate(id: number): Promise<void>;
|
|
50
50
|
updateCursor(flowId: number, cursor: Date | null): Promise<void>;
|
|
51
|
+
patchCursor(flowId: number, patch: {
|
|
52
|
+
lastRunAt?: Date | null;
|
|
53
|
+
syncProgress?: SyncProgress | null;
|
|
54
|
+
}): Promise<void>;
|
|
51
55
|
}
|
|
52
56
|
//# sourceMappingURL=dataflow.client.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"dataflow.client.d.ts","sourceRoot":"","sources":["../../src/clients/dataflow.client.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,kCAAkC,CAAC;AAC9D,OAAO,KAAK,EAAE,UAAU,EAAE,gBAAgB,EAAE,MAAM,kCAAkC,CAAC;AACrF,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,wCAAwC,CAAC;
|
|
1
|
+
{"version":3,"file":"dataflow.client.d.ts","sourceRoot":"","sources":["../../src/clients/dataflow.client.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,kCAAkC,CAAC;AAC9D,OAAO,KAAK,EAAE,UAAU,EAAE,gBAAgB,EAAE,MAAM,kCAAkC,CAAC;AACrF,OAAO,KAAK,EAAE,QAAQ,EAAE,YAAY,EAAE,MAAM,wCAAwC,CAAC;AACrF,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,uCAAuC,CAAC;AAC9E,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,gCAAgC,CAAC;AAIjE,MAAM,WAAW,kBAAkB;IACjC,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB;AAED;;;GAGG;AACH,MAAM,WAAW,mBAAmB;IAClC,eAAe,EAAE,MAAM,CAAC;IACxB,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,YAAY,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,YAAY,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,YAAY,EAAE,MAAM,CAAC;IACrB,YAAY,EAAE,MAAM,CAAC;IACrB,aAAa,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,IAAI,CAAC;IAC/C,aAAa,CAAC,EAAE,gBAAgB,GAAG,IAAI,CAAC;IACxC,MAAM,CAAC,EAAE,UAAU,GAAG,IAAI,CAAC;IAC3B,aAAa,EAAE,MAAM,CAAC;IACtB,cAAc,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;CAChC;AAED;;;;;GAKG;AACH,MAAM,MAAM,mBAAmB,GAAG,OAAO,CACvC,IAAI,CAAC,mBAAmB,EAAE,cAAc,GAAG,cAAc,CAAC,CAC3D,CAAC;AAEF,qBAAa,cAAe,SAAQ,UAAU;IAC5C,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAS;gBAEnB,MAAM,EAAE,UAAU,EAAE,gBAAgB,GAAE,OAAO,CAAC,gBAAgB,CAAM;IAM1E,IAAI,CAAC,MAAM,EAAE,kBAAkB,GAAG,OAAO,CAAC,KAAK,CAAC;QAAE,EAAE,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IAWhE,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAO7C,MAAM,CAAC,KAAK,EAAE,mBAAmB,GAAG,OAAO,CAAC,QAAQ,CAAC;IASrD,MAAM,CAAC,EAAE,EAAE,MAAM,EAAE,KAAK,EAAE,mBAAmB,GAAG,OAAO,CAAC,QAAQ,CAAC;IASjE,MAAM,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAOjC,QAAQ,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAOnC,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAOrC,YAAY,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,IAAI,GAAG,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC;IAKhE,WAAW,CACf,MAAM,EAAE,MAAM,EACd,KAAK,EAAE;QAAE,SAAS,CAAC,EAAE,IAAI,GAAG,IAAI,CAAC;QAAC,YAAY,CAAC,EAAE,YAAY,GAAG,IAAI,CAAA;KAAE,GACrE,OAAO,CAAC,IAAI,CAAC;CAiBjB"}
|
|
@@ -58,10 +58,20 @@ class DataFlowClient extends pulse_solution_services_1.BaseClient {
|
|
|
58
58
|
});
|
|
59
59
|
}
|
|
60
60
|
async updateCursor(flowId, cursor) {
|
|
61
|
+
await this.patchCursor(flowId, { lastRunAt: cursor });
|
|
62
|
+
}
|
|
63
|
+
// Sends only the keys given; IS treats an empty body as a full reset.
|
|
64
|
+
async patchCursor(flowId, patch) {
|
|
65
|
+
var _a, _b;
|
|
66
|
+
if (patch.lastRunAt === undefined && patch.syncProgress === undefined) {
|
|
67
|
+
throw new Error('patchCursor needs lastRunAt or syncProgress; use updateCursor(id, null) to reset');
|
|
68
|
+
}
|
|
61
69
|
await this._performRequest({
|
|
62
70
|
endpoint: `${this._base}/${encodeURIComponent(String(flowId))}/cursor`,
|
|
63
71
|
method: 'PATCH',
|
|
64
|
-
body: { lastRunAt
|
|
72
|
+
body: Object.assign(Object.assign({}, (patch.lastRunAt !== undefined
|
|
73
|
+
? { lastRunAt: (_b = (_a = patch.lastRunAt) === null || _a === void 0 ? void 0 : _a.toISOString()) !== null && _b !== void 0 ? _b : null }
|
|
74
|
+
: {})), (patch.syncProgress !== undefined ? { syncProgress: patch.syncProgress } : {})),
|
|
65
75
|
});
|
|
66
76
|
}
|
|
67
77
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"dataflow.client.js","sourceRoot":"","sources":["../../src/clients/dataflow.client.ts"],"names":[],"mappings":";;;AAAA,8EAA8D;AAK9D,8CAAqD;AACrD,qDAAsE;AAuCtE,MAAa,cAAe,SAAQ,oCAAU;IAG5C,YAAY,MAAkB,EAAE,mBAA8C,EAAE;QAC9E,KAAK,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAAC;QAChC,MAAM,SAAS,GAAG,IAAA,mDAAkC,GAAE,CAAC;QACvD,IAAI,CAAC,KAAK,GAAG,SAAS,CAAC,CAAC,CAAC,GAAG,SAAS,aAAa,CAAC,CAAC,CAAC,wBAAwB,CAAC;IAChF,CAAC;IAED,KAAK,CAAC,IAAI,CAAC,MAA0B;QACnC,MAAM,WAAW,GAAG,MAAM,CAAC,WAAW,CACpC,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,SAAS,CAAC,CAChC,CAAC;QAE5B,OAAO,IAAI,CAAC,eAAe,CAAwB;YACjD,QAAQ,EAAE,IAAI,CAAC,KAAK;YACpB,WAAW;SACZ,CAAC,CAAC;IACL,CAAC;IAED,KAAK,CAAC,OAAO,CAAC,EAAU;QACtB,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAkB;YACvD,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,EAAE;SACpD,CAAC,CAAC;QACH,OAAO,IAAI,CAAC,CAAC,CAAC,IAAA,2BAAiB,EAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IAC/C,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,KAA0B;QACrC,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAW;YAChD,QAAQ,EAAE,IAAI,CAAC,KAAK;YACpB,MAAM,EAAE,MAAM;YACd,IAAI,EAAE,KAAK;SACZ,CAAC,CAAC;QACH,OAAO,IAAA,2BAAiB,EAAC,IAAI,CAAC,CAAC;IACjC,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,EAAU,EAAE,KAA0B;QACjD,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAW;YAChD,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,EAAE;YACnD,MAAM,EAAE,OAAO;YACf,IAAI,EAAE,KAAK;SACZ,CAAC,CAAC;QACH,OAAO,IAAA,2BAAiB,EAAC,IAAI,CAAC,CAAC;IACjC,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,EAAU;QACrB,MAAM,IAAI,CAAC,eAAe,CAAO;YAC/B,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,EAAE;YACnD,MAAM,EAAE,QAAQ;SACjB,CAAC,CAAC;IACL,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,EAAU;QACvB,MAAM,IAAI,CAAC,eAAe,CAAO;YAC/B,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,WAAW;YAC5D,MAAM,EAAE,MAAM;SACf,CAAC,CAAC;IACL,CAAC;IAED,KAAK,CAAC,UAAU,CAAC,EAAU;QACzB,MAAM,IAAI,CAAC,eAAe,CAAO;YAC/B,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,aAAa;YAC9D,MAAM,EAAE,MAAM;SACf,CAAC,CAAC;IACL,CAAC;IAED,KAAK,CAAC,YAAY,CAAC,MAAc,EAAE,MAAmB;QACpD,MAAM,IAAI,CAAC,eAAe,CAAO;YAC/B,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,SAAS;YACtE,MAAM,EAAE,OAAO;YACf,IAAI,EAAE,EAAE,SAAS,EAAE,
|
|
1
|
+
{"version":3,"file":"dataflow.client.js","sourceRoot":"","sources":["../../src/clients/dataflow.client.ts"],"names":[],"mappings":";;;AAAA,8EAA8D;AAK9D,8CAAqD;AACrD,qDAAsE;AAuCtE,MAAa,cAAe,SAAQ,oCAAU;IAG5C,YAAY,MAAkB,EAAE,mBAA8C,EAAE;QAC9E,KAAK,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAAC;QAChC,MAAM,SAAS,GAAG,IAAA,mDAAkC,GAAE,CAAC;QACvD,IAAI,CAAC,KAAK,GAAG,SAAS,CAAC,CAAC,CAAC,GAAG,SAAS,aAAa,CAAC,CAAC,CAAC,wBAAwB,CAAC;IAChF,CAAC;IAED,KAAK,CAAC,IAAI,CAAC,MAA0B;QACnC,MAAM,WAAW,GAAG,MAAM,CAAC,WAAW,CACpC,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,SAAS,CAAC,CAChC,CAAC;QAE5B,OAAO,IAAI,CAAC,eAAe,CAAwB;YACjD,QAAQ,EAAE,IAAI,CAAC,KAAK;YACpB,WAAW;SACZ,CAAC,CAAC;IACL,CAAC;IAED,KAAK,CAAC,OAAO,CAAC,EAAU;QACtB,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAkB;YACvD,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,EAAE;SACpD,CAAC,CAAC;QACH,OAAO,IAAI,CAAC,CAAC,CAAC,IAAA,2BAAiB,EAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IAC/C,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,KAA0B;QACrC,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAW;YAChD,QAAQ,EAAE,IAAI,CAAC,KAAK;YACpB,MAAM,EAAE,MAAM;YACd,IAAI,EAAE,KAAK;SACZ,CAAC,CAAC;QACH,OAAO,IAAA,2BAAiB,EAAC,IAAI,CAAC,CAAC;IACjC,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,EAAU,EAAE,KAA0B;QACjD,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAW;YAChD,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,EAAE;YACnD,MAAM,EAAE,OAAO;YACf,IAAI,EAAE,KAAK;SACZ,CAAC,CAAC;QACH,OAAO,IAAA,2BAAiB,EAAC,IAAI,CAAC,CAAC;IACjC,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,EAAU;QACrB,MAAM,IAAI,CAAC,eAAe,CAAO;YAC/B,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,EAAE;YACnD,MAAM,EAAE,QAAQ;SACjB,CAAC,CAAC;IACL,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,EAAU;QACvB,MAAM,IAAI,CAAC,eAAe,CAAO;YAC/B,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,WAAW;YAC5D,MAAM,EAAE,MAAM;SACf,CAAC,CAAC;IACL,CAAC;IAED,KAAK,CAAC,UAAU,CAAC,EAAU;QACzB,MAAM,IAAI,CAAC,eAAe,CAAO;YAC/B,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,EAAE,CAAC,aAAa;YAC9D,MAAM,EAAE,MAAM;SACf,CAAC,CAAC;IACL,CAAC;IAED,KAAK,CAAC,YAAY,CAAC,MAAc,EAAE,MAAmB;QACpD,MAAM,IAAI,CAAC,WAAW,CAAC,MAAM,EAAE,EAAE,SAAS,EAAE,MAAM,EAAE,CAAC,CAAC;IACxD,CAAC;IAED,sEAAsE;IACtE,KAAK,CAAC,WAAW,CACf,MAAc,EACd,KAAsE;;QAEtE,IAAI,KAAK,CAAC,SAAS,KAAK,SAAS,IAAI,KAAK,CAAC,YAAY,KAAK,SAAS,EAAE,CAAC;YACtE,MAAM,IAAI,KAAK,CACb,kFAAkF,CACnF,CAAC;QACJ,CAAC;QACD,MAAM,IAAI,CAAC,eAAe,CAAO;YAC/B,QAAQ,EAAE,GAAG,IAAI,CAAC,KAAK,IAAI,kBAAkB,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,SAAS;YACtE,MAAM,EAAE,OAAO;YACf,IAAI,kCACC,CAAC,KAAK,CAAC,SAAS,KAAK,SAAS;gBAC/B,CAAC,CAAC,EAAE,SAAS,EAAE,MAAA,MAAA,KAAK,CAAC,SAAS,0CAAE,WAAW,EAAE,mCAAI,IAAI,EAAE;gBACvD,CAAC,CAAC,EAAE,CAAC,GACJ,CAAC,KAAK,CAAC,YAAY,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,KAAK,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,CAClF;SACF,CAAC,CAAC;IACL,CAAC;CACF;AA3FD,wCA2FC"}
|
package/dist/constants.d.ts
CHANGED
|
@@ -27,6 +27,14 @@ export declare const LOOKUP_CHUNK_SIZE = 100;
|
|
|
27
27
|
* 200 leaves headroom under the 500 hard limit.
|
|
28
28
|
*/
|
|
29
29
|
export declare const MAX_MUTATION_BATCH_SIZE = 200;
|
|
30
|
+
export declare const DEFAULT_RUN_BUDGET_MS: number;
|
|
31
|
+
export declare const RUN_BUDGET_INTERVAL_MARGIN_MS: number;
|
|
32
|
+
export declare const MIN_RUN_BUDGET_MS: number;
|
|
33
|
+
export declare const SLOW_LOAD_MS: number;
|
|
34
|
+
export declare const MAX_SYNC_PROGRESS_CHARS = 4096;
|
|
35
|
+
export declare const MAX_CHECKPOINT_HOLDS = 3;
|
|
36
|
+
export declare const MAX_FAILED_RESUMES = 3;
|
|
37
|
+
export declare const FAILED_RESUME_MIN_MS: number;
|
|
30
38
|
/** Persisted lifecycle status of a data flow (IS `status` column / list filter). */
|
|
31
39
|
export declare const DataFlowStatus: {
|
|
32
40
|
readonly ACTIVE: "ACTIVE";
|
package/dist/constants.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"constants.d.ts","sourceRoot":"","sources":["../src/constants.ts"],"names":[],"mappings":"AAAA,eAAO,MAAM,QAAQ;;;;CAIX,CAAC;AAEX,eAAO,MAAM,aAAa;;gCAEL,MAAM;;;wBAGd,MAAM;0CACY,MAAM;;2BAErB,MAAM;;;;;;;CAOZ,CAAC;AAEX,eAAO,MAAM,iBAAiB,MAAM,CAAC;AAErC;;;;;GAKG;AACH,eAAO,MAAM,uBAAuB,MAAM,CAAC;AAE3C,oFAAoF;AACpF,eAAO,MAAM,cAAc;;;CAGjB,CAAC;AACX,MAAM,MAAM,cAAc,GAAG,CAAC,OAAO,cAAc,CAAC,CAAC,MAAM,OAAO,cAAc,CAAC,CAAC;AAElF,wEAAwE;AACxE,eAAO,MAAM,cAAc,YAAY,CAAC;AAExC;;;;GAIG;AACH,eAAO,MAAM,iBAAiB,GAAI,iBAAiB,MAAM,KAAG,MAC1B,CAAC;AAEnC,eAAO,MAAM,WAAW;;;CAGd,CAAC;AACX,MAAM,MAAM,WAAW,GAAG,CAAC,OAAO,WAAW,CAAC,CAAC,MAAM,OAAO,WAAW,CAAC,CAAC;AAEzE,eAAO,MAAM,QAAQ;;;CAGX,CAAC;AACX,MAAM,MAAM,QAAQ,GAAG,CAAC,OAAO,QAAQ,CAAC,CAAC,MAAM,OAAO,QAAQ,CAAC,CAAC;AAEhE,eAAO,MAAM,cAAc;IACzB,kFAAkF;;IAElF,sFAAsF;;CAE9E,CAAC;AACX,MAAM,MAAM,cAAc,GAAG,CAAC,OAAO,cAAc,CAAC,CAAC,MAAM,OAAO,cAAc,CAAC,CAAC"}
|
|
1
|
+
{"version":3,"file":"constants.d.ts","sourceRoot":"","sources":["../src/constants.ts"],"names":[],"mappings":"AAAA,eAAO,MAAM,QAAQ;;;;CAIX,CAAC;AAEX,eAAO,MAAM,aAAa;;gCAEL,MAAM;;;wBAGd,MAAM;0CACY,MAAM;;2BAErB,MAAM;;;;;;;CAOZ,CAAC;AAEX,eAAO,MAAM,iBAAiB,MAAM,CAAC;AAErC;;;;;GAKG;AACH,eAAO,MAAM,uBAAuB,MAAM,CAAC;AAG3C,eAAO,MAAM,qBAAqB,QAAiB,CAAC;AAGpD,eAAO,MAAM,6BAA6B,QAAY,CAAC;AAEvD,eAAO,MAAM,iBAAiB,QAAY,CAAC;AAE3C,eAAO,MAAM,YAAY,QAAmB,CAAC;AAG7C,eAAO,MAAM,uBAAuB,OAAO,CAAC;AAI5C,eAAO,MAAM,oBAAoB,IAAI,CAAC;AAGtC,eAAO,MAAM,kBAAkB,IAAI,CAAC;AACpC,eAAO,MAAM,oBAAoB,QAAiB,CAAC;AAEnD,oFAAoF;AACpF,eAAO,MAAM,cAAc;;;CAGjB,CAAC;AACX,MAAM,MAAM,cAAc,GAAG,CAAC,OAAO,cAAc,CAAC,CAAC,MAAM,OAAO,cAAc,CAAC,CAAC;AAElF,wEAAwE;AACxE,eAAO,MAAM,cAAc,YAAY,CAAC;AAExC;;;;GAIG;AACH,eAAO,MAAM,iBAAiB,GAAI,iBAAiB,MAAM,KAAG,MAC1B,CAAC;AAEnC,eAAO,MAAM,WAAW;;;CAGd,CAAC;AACX,MAAM,MAAM,WAAW,GAAG,CAAC,OAAO,WAAW,CAAC,CAAC,MAAM,OAAO,WAAW,CAAC,CAAC;AAEzE,eAAO,MAAM,QAAQ;;;CAGX,CAAC;AACX,MAAM,MAAM,QAAQ,GAAG,CAAC,OAAO,QAAQ,CAAC,CAAC,MAAM,OAAO,QAAQ,CAAC,CAAC;AAEhE,eAAO,MAAM,cAAc;IACzB,kFAAkF;;IAElF,sFAAsF;;CAE9E,CAAC;AACX,MAAM,MAAM,cAAc,GAAG,CAAC,OAAO,cAAc,CAAC,CAAC,MAAM,OAAO,cAAc,CAAC,CAAC"}
|
package/dist/constants.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
-
exports.LookupStrategy = exports.SyncMode = exports.AdapterRole = exports.integrationSource = exports.SKEDULO_SYSTEM = exports.DataFlowStatus = exports.MAX_MUTATION_BATCH_SIZE = exports.LOOKUP_CHUNK_SIZE = exports.OperationName = exports.Strategy = void 0;
|
|
3
|
+
exports.LookupStrategy = exports.SyncMode = exports.AdapterRole = exports.integrationSource = exports.SKEDULO_SYSTEM = exports.DataFlowStatus = exports.FAILED_RESUME_MIN_MS = exports.MAX_FAILED_RESUMES = exports.MAX_CHECKPOINT_HOLDS = exports.MAX_SYNC_PROGRESS_CHARS = exports.SLOW_LOAD_MS = exports.MIN_RUN_BUDGET_MS = exports.RUN_BUDGET_INTERVAL_MARGIN_MS = exports.DEFAULT_RUN_BUDGET_MS = exports.MAX_MUTATION_BATCH_SIZE = exports.LOOKUP_CHUNK_SIZE = exports.OperationName = exports.Strategy = void 0;
|
|
4
4
|
exports.Strategy = {
|
|
5
5
|
PARAGON_WORKFLOW: 'PARAGON_WORKFLOW',
|
|
6
6
|
SCHEDULED_PULL: 'SCHEDULED_PULL',
|
|
@@ -30,6 +30,20 @@ exports.LOOKUP_CHUNK_SIZE = 100;
|
|
|
30
30
|
* 200 leaves headroom under the 500 hard limit.
|
|
31
31
|
*/
|
|
32
32
|
exports.MAX_MUTATION_BATCH_SIZE = 200;
|
|
33
|
+
// Max time a run spends on resumable fetches. Stays under the 900s function timeout.
|
|
34
|
+
exports.DEFAULT_RUN_BUDGET_MS = 12 * 60 * 1000;
|
|
35
|
+
// Stop this long before the job's next run so runs don't overlap.
|
|
36
|
+
exports.RUN_BUDGET_INTERVAL_MARGIN_MS = 60 * 1000;
|
|
37
|
+
exports.MIN_RUN_BUDGET_MS = 60 * 1000;
|
|
38
|
+
exports.SLOW_LOAD_MS = 24 * 60 * 60000;
|
|
39
|
+
// Must equal integration-service's SYNC_PROGRESS_MAX_JSON_LENGTH (update-cursor.dto.ts); it rejects a larger syncProgress (as JSON) with a 400.
|
|
40
|
+
exports.MAX_SYNC_PROGRESS_CHARS = 4096;
|
|
41
|
+
// Runs in a row a resumable load holds since its saved `token`, over at least FAILED_RESUME_MIN_MS, before it
|
|
42
|
+
// records what it can and moves on. Counted per `token`, not per page, so a later held page can move on sooner.
|
|
43
|
+
exports.MAX_CHECKPOINT_HOLDS = 3;
|
|
44
|
+
// A resumed load is started over only after this many failed runs that span at least FAILED_RESUME_MIN_MS.
|
|
45
|
+
exports.MAX_FAILED_RESUMES = 3;
|
|
46
|
+
exports.FAILED_RESUME_MIN_MS = 60 * 60 * 1000;
|
|
33
47
|
/** Persisted lifecycle status of a data flow (IS `status` column / list filter). */
|
|
34
48
|
exports.DataFlowStatus = {
|
|
35
49
|
ACTIVE: 'ACTIVE',
|
package/dist/constants.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"constants.js","sourceRoot":"","sources":["../src/constants.ts"],"names":[],"mappings":";;;AAAa,QAAA,QAAQ,GAAG;IACtB,gBAAgB,EAAE,kBAAkB;IACpC,cAAc,EAAE,gBAAgB;IAChC,kBAAkB,EAAE,oBAAoB;CAChC,CAAC;AAEE,QAAA,aAAa,GAAG;IAC3B,eAAe,EAAE,gBAAgB;IACjC,YAAY,EAAE,CAAC,EAAU,EAAE,EAAE,CAAC,gBAAgB,EAAE,EAAE;IAClD,qBAAqB,EAAE,6BAA6B;IACpD,cAAc,EAAE,uBAAuB;IACvC,IAAI,EAAE,CAAC,EAAU,EAAE,EAAE,CAAC,QAAQ,EAAE,EAAE;IAClC,cAAc,EAAE,CAAC,UAAkB,EAAE,EAAE,CAAC,iBAAiB,UAAU,EAAE;IACrE,UAAU,EAAE,WAAW;IACvB,OAAO,EAAE,CAAC,EAAU,EAAE,EAAE,CAAC,WAAW,EAAE,EAAE;IACxC,eAAe,EAAE,kBAAkB;IACnC,aAAa,EAAE,gBAAgB;IAC/B,cAAc,EAAE,iBAAiB;IACjC,qBAAqB,EAAE,uBAAuB;IAC9C,cAAc,EAAE,iBAAiB;IACjC,mBAAmB,EAAE,qBAAqB;CAClC,CAAC;AAEE,QAAA,iBAAiB,GAAG,GAAG,CAAC;AAErC;;;;;GAKG;AACU,QAAA,uBAAuB,GAAG,GAAG,CAAC;AAE3C,oFAAoF;AACvE,QAAA,cAAc,GAAG;IAC5B,MAAM,EAAE,QAAQ;IAChB,QAAQ,EAAE,UAAU;CACZ,CAAC;AAGX,wEAAwE;AAC3D,QAAA,cAAc,GAAG,SAAS,CAAC;AAExC;;;;GAIG;AACI,MAAM,iBAAiB,GAAG,CAAC,eAAuB,EAAU,EAAE,CACnE,eAAe,eAAe,EAAE,CAAC;AADtB,QAAA,iBAAiB,qBACK;AAEtB,QAAA,WAAW,GAAG;IACzB,MAAM,EAAE,QAAQ;IAChB,MAAM,EAAE,QAAQ;CACR,CAAC;AAGE,QAAA,QAAQ,GAAG;IACtB,KAAK,EAAE,OAAO;IACd,KAAK,EAAE,OAAO;CACN,CAAC;AAGE,QAAA,cAAc,GAAG;IAC5B,kFAAkF;IAClF,UAAU,EAAE,YAAY;IACxB,sFAAsF;IACtF,SAAS,EAAE,WAAW;CACd,CAAC"}
|
|
1
|
+
{"version":3,"file":"constants.js","sourceRoot":"","sources":["../src/constants.ts"],"names":[],"mappings":";;;AAAa,QAAA,QAAQ,GAAG;IACtB,gBAAgB,EAAE,kBAAkB;IACpC,cAAc,EAAE,gBAAgB;IAChC,kBAAkB,EAAE,oBAAoB;CAChC,CAAC;AAEE,QAAA,aAAa,GAAG;IAC3B,eAAe,EAAE,gBAAgB;IACjC,YAAY,EAAE,CAAC,EAAU,EAAE,EAAE,CAAC,gBAAgB,EAAE,EAAE;IAClD,qBAAqB,EAAE,6BAA6B;IACpD,cAAc,EAAE,uBAAuB;IACvC,IAAI,EAAE,CAAC,EAAU,EAAE,EAAE,CAAC,QAAQ,EAAE,EAAE;IAClC,cAAc,EAAE,CAAC,UAAkB,EAAE,EAAE,CAAC,iBAAiB,UAAU,EAAE;IACrE,UAAU,EAAE,WAAW;IACvB,OAAO,EAAE,CAAC,EAAU,EAAE,EAAE,CAAC,WAAW,EAAE,EAAE;IACxC,eAAe,EAAE,kBAAkB;IACnC,aAAa,EAAE,gBAAgB;IAC/B,cAAc,EAAE,iBAAiB;IACjC,qBAAqB,EAAE,uBAAuB;IAC9C,cAAc,EAAE,iBAAiB;IACjC,mBAAmB,EAAE,qBAAqB;CAClC,CAAC;AAEE,QAAA,iBAAiB,GAAG,GAAG,CAAC;AAErC;;;;;GAKG;AACU,QAAA,uBAAuB,GAAG,GAAG,CAAC;AAE3C,qFAAqF;AACxE,QAAA,qBAAqB,GAAG,EAAE,GAAG,EAAE,GAAG,IAAI,CAAC;AAEpD,kEAAkE;AACrD,QAAA,6BAA6B,GAAG,EAAE,GAAG,IAAI,CAAC;AAE1C,QAAA,iBAAiB,GAAG,EAAE,GAAG,IAAI,CAAC;AAE9B,QAAA,YAAY,GAAG,EAAE,GAAG,EAAE,GAAG,KAAM,CAAC;AAE7C,gJAAgJ;AACnI,QAAA,uBAAuB,GAAG,IAAI,CAAC;AAE5C,8GAA8G;AAC9G,gHAAgH;AACnG,QAAA,oBAAoB,GAAG,CAAC,CAAC;AAEtC,2GAA2G;AAC9F,QAAA,kBAAkB,GAAG,CAAC,CAAC;AACvB,QAAA,oBAAoB,GAAG,EAAE,GAAG,EAAE,GAAG,IAAI,CAAC;AAEnD,oFAAoF;AACvE,QAAA,cAAc,GAAG;IAC5B,MAAM,EAAE,QAAQ;IAChB,QAAQ,EAAE,UAAU;CACZ,CAAC;AAGX,wEAAwE;AAC3D,QAAA,cAAc,GAAG,SAAS,CAAC;AAExC;;;;GAIG;AACI,MAAM,iBAAiB,GAAG,CAAC,eAAuB,EAAU,EAAE,CACnE,eAAe,eAAe,EAAE,CAAC;AADtB,QAAA,iBAAiB,qBACK;AAEtB,QAAA,WAAW,GAAG;IACzB,MAAM,EAAE,QAAQ;IAChB,MAAM,EAAE,QAAQ;CACR,CAAC;AAGE,QAAA,QAAQ,GAAG;IACtB,KAAK,EAAE,OAAO;IACd,KAAK,EAAE,OAAO;CACN,CAAC;AAGE,QAAA,cAAc,GAAG;IAC5B,kFAAkF;IAClF,UAAU,EAAE,YAAY;IACxB,sFAAsF;IACtF,SAAS,EAAE,WAAW;CACd,CAAC"}
|
package/dist/index.d.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
export * from './types';
|
|
2
|
-
export { Strategy, OperationName, LOOKUP_CHUNK_SIZE, AdapterRole, SyncMode, LookupStrategy, DataFlowStatus, SKEDULO_SYSTEM, } from './constants';
|
|
2
|
+
export { Strategy, OperationName, LOOKUP_CHUNK_SIZE, AdapterRole, SyncMode, LookupStrategy, DataFlowStatus, SKEDULO_SYSTEM, DEFAULT_RUN_BUDGET_MS, MAX_SYNC_PROGRESS_CHARS, MAX_CHECKPOINT_HOLDS, MAX_FAILED_RESUMES, FAILED_RESUME_MIN_MS, } from './constants';
|
|
3
3
|
export { IntegrationContext, createIntegrationContext, createContextFactory } from './context';
|
|
4
4
|
export type { ContextFactoryOptions } from './context';
|
|
5
5
|
export { ProxyClient } from './clients/proxy.client';
|
|
@@ -10,13 +10,14 @@ export { IntegrationConfigClient } from './clients/integration-config.client';
|
|
|
10
10
|
export { IntegrationLogClient } from './clients/integration-log.client';
|
|
11
11
|
export { LogLevel, LogKind, LogSource } from './interfaces/integration-log.interface';
|
|
12
12
|
export type { IntegrationLogEntry } from './interfaces/integration-log.interface';
|
|
13
|
-
export { buildRunLogEntries } from './services/run-log-builder';
|
|
13
|
+
export { buildRunLogEntries, MAX_RECORD_ERRORS } from './services/run-log-builder';
|
|
14
14
|
export type { BuildRunLogArgs } from './services/run-log-builder';
|
|
15
15
|
export { DataFlowService } from './services/dataflow.service';
|
|
16
16
|
export type { ScheduledJob, IntegrationConfig } from './interfaces/scheduled-job.interface';
|
|
17
17
|
export { asIntegrationConfig, JobIntervalUnit } from './interfaces/scheduled-job.interface';
|
|
18
18
|
export { PreviewService } from './services/preview.service';
|
|
19
|
-
export { SyncService } from './services/sync.service';
|
|
19
|
+
export { SyncService, ResumeTokenError } from './services/sync.service';
|
|
20
|
+
export type { ResumeArgs } from './services/sync.service';
|
|
20
21
|
export { SyncRunner, AdapterNotRegisteredError, AdapterRoleError } from './services/sync-runner';
|
|
21
22
|
export { SkeduloAdapter, skeduloAdapter } from './adapters/skedulo.adapter';
|
|
22
23
|
export { BaseAdapter } from './adapters/base.adapter';
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,SAAS,CAAC;AACxB,OAAO,EACL,QAAQ,EACR,aAAa,EACb,iBAAiB,EACjB,WAAW,EACX,QAAQ,EACR,cAAc,EACd,cAAc,EACd,cAAc,
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,SAAS,CAAC;AACxB,OAAO,EACL,QAAQ,EACR,aAAa,EACb,iBAAiB,EACjB,WAAW,EACX,QAAQ,EACR,cAAc,EACd,cAAc,EACd,cAAc,EACd,qBAAqB,EACrB,uBAAuB,EACvB,oBAAoB,EACpB,kBAAkB,EAClB,oBAAoB,GACrB,MAAM,aAAa,CAAC;AACrB,OAAO,EAAE,kBAAkB,EAAE,wBAAwB,EAAE,oBAAoB,EAAE,MAAM,WAAW,CAAC;AAC/F,YAAY,EAAE,qBAAqB,EAAE,MAAM,WAAW,CAAC;AACvD,OAAO,EAAE,WAAW,EAAE,MAAM,wBAAwB,CAAC;AACrD,YAAY,EAAE,UAAU,EAAE,mBAAmB,EAAE,WAAW,EAAE,MAAM,wBAAwB,CAAC;AAC3F,OAAO,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAC3D,YAAY,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AACpE,OAAO,EAAE,uBAAuB,EAAE,MAAM,qCAAqC,CAAC;AAC9E,OAAO,EAAE,oBAAoB,EAAE,MAAM,kCAAkC,CAAC;AACxE,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,wCAAwC,CAAC;AACtF,YAAY,EAAE,mBAAmB,EAAE,MAAM,wCAAwC,CAAC;AAClF,OAAO,EAAE,kBAAkB,EAAE,iBAAiB,EAAE,MAAM,4BAA4B,CAAC;AACnF,YAAY,EAAE,eAAe,EAAE,MAAM,4BAA4B,CAAC;AAClE,OAAO,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAC;AAC9D,YAAY,EAAE,YAAY,EAAE,iBAAiB,EAAE,MAAM,sCAAsC,CAAC;AAC5F,OAAO,EAAE,mBAAmB,EAAE,eAAe,EAAE,MAAM,sCAAsC,CAAC;AAC5F,OAAO,EAAE,cAAc,EAAE,MAAM,4BAA4B,CAAC;AAC5D,OAAO,EAAE,WAAW,EAAE,gBAAgB,EAAE,MAAM,yBAAyB,CAAC;AACxE,YAAY,EAAE,UAAU,EAAE,MAAM,yBAAyB,CAAC;AAC1D,OAAO,EAAE,UAAU,EAAE,yBAAyB,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AACjG,OAAO,EAAE,cAAc,EAAE,cAAc,EAAE,MAAM,4BAA4B,CAAC;AAC5E,OAAO,EAAE,WAAW,EAAE,MAAM,yBAAyB,CAAC;AACtD,YAAY,EAAE,UAAU,EAAE,gBAAgB,EAAE,MAAM,yBAAyB,CAAC;AAC5E,OAAO,EAAE,gBAAgB,EAAE,MAAM,+BAA+B,CAAC;AACjE,YAAY,EAAE,YAAY,EAAE,MAAM,+BAA+B,CAAC;AAClE,YAAY,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,2CAA2C,CAAC;AAClG,OAAO,EACL,cAAc,EACd,cAAc,EACd,sBAAsB,EACtB,kBAAkB,EAClB,iBAAiB,GAClB,MAAM,kCAAkC,CAAC;AAC1C,YAAY,EAAE,aAAa,EAAE,MAAM,kCAAkC,CAAC;AACtE,OAAO,EAAE,kBAAkB,EAAE,MAAM,4BAA4B,CAAC;AAChE,OAAO,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAC;AAC9D,OAAO,EAAE,kBAAkB,EAAE,MAAM,iCAAiC,CAAC;AACrE,YAAY,EAAE,wBAAwB,EAAE,YAAY,EAAE,MAAM,iCAAiC,CAAC;AAC9F,YAAY,EACV,oBAAoB,EACpB,gBAAgB,EAChB,gBAAgB,EAChB,gBAAgB,GACjB,MAAM,sCAAsC,CAAC;AAC9C,YAAY,EACV,mBAAmB,EACnB,gBAAgB,EAChB,iBAAiB,EACjB,SAAS,GACV,MAAM,gCAAgC,CAAC;AACxC,OAAO,EAAE,SAAS,EAAE,MAAM,sCAAsC,CAAC;AACjE,YAAY,EAAE,iBAAiB,EAAE,iBAAiB,EAAE,MAAM,sCAAsC,CAAC;AAGjG,OAAO,EAAE,aAAa,EAAE,OAAO,EAAE,yBAAyB,EAAE,MAAM,8BAA8B,CAAC;AACjG,YAAY,EAAE,UAAU,EAAE,MAAM,8BAA8B,CAAC;AAG/D,OAAO,EACL,WAAW,EACX,aAAa,EACb,cAAc,EACd,iBAAiB,EACjB,aAAa,EACb,mBAAmB,EACnB,uBAAuB,EACvB,sBAAsB,EACtB,sBAAsB,EACtB,iBAAiB,EACjB,gBAAgB,GACjB,MAAM,wBAAwB,CAAC;AAChC,YAAY,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAC;AAC5D,OAAO,EAAE,YAAY,EAAE,MAAM,+BAA+B,CAAC;AAC7D,YAAY,EAAE,gBAAgB,EAAE,MAAM,+BAA+B,CAAC;AACtE,OAAO,EACL,OAAO,EACP,YAAY,IAAI,sBAAsB,EACtC,SAAS,EACT,WAAW,EACX,WAAW,GACZ,MAAM,4BAA4B,CAAC;AACpC,YAAY,EAAE,aAAa,EAAE,UAAU,EAAE,cAAc,EAAE,MAAM,4BAA4B,CAAC;AAC5F,OAAO,EAAE,WAAW,EAAE,MAAM,yBAAyB,CAAC;AACtD,YAAY,EAAE,OAAO,EAAE,MAAM,yBAAyB,CAAC;AAGvD,OAAO,EAAE,uBAAuB,EAAE,eAAe,EAAE,MAAM,4BAA4B,CAAC;AACtF,YAAY,EAAE,qBAAqB,EAAE,MAAM,4BAA4B,CAAC;AAGxE,OAAO,EACL,WAAW,EACX,UAAU,EACV,aAAa,EACb,aAAa,EACb,cAAc,EACd,YAAY,EACZ,iBAAiB,GAClB,MAAM,iBAAiB,CAAC;AACzB,YAAY,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC"}
|