@celigo/api-specs 0.2.189 → 0.2.191
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/connection.yml +16 -1
- package/dist/export.yml +60 -52
- package/dist/flow.yml +24 -4
- package/dist/job.yml +7 -2
- package/dist/tool.yml +48 -53
- package/package.json +1 -1
- package/schemas.json +2 -2
package/dist/connection.yml
CHANGED
|
@@ -1879,7 +1879,12 @@ paths:
|
|
|
1879
1879
|
searchValue: SEED-01
|
|
1880
1880
|
responses:
|
|
1881
1881
|
'200':
|
|
1882
|
-
description:
|
|
1882
|
+
description: |-
|
|
1883
|
+
The records the source returned, with a per-record link when the adaptor provides one.
|
|
1884
|
+
When paging stopped on a repeated page (observed 2026-10-23), the body was instead
|
|
1885
|
+
`text/plain` holding two JSON objects in a row: the records read, then
|
|
1886
|
+
`{"errors": [...]}` carrying the `page_response_error`. A client that parses that body as
|
|
1887
|
+
one JSON document fails on it; split it into its objects.
|
|
1883
1888
|
content:
|
|
1884
1889
|
application/json:
|
|
1885
1890
|
schema:
|
|
@@ -1912,6 +1917,16 @@ paths:
|
|
|
1912
1917
|
externalid: SEED-01
|
|
1913
1918
|
dataURIs:
|
|
1914
1919
|
- https://1234567.app.netsuite.com/app/common/custom/custrecordentry.nl?rectype=923&id=1
|
|
1920
|
+
text/plain:
|
|
1921
|
+
schema:
|
|
1922
|
+
type: string
|
|
1923
|
+
description: |-
|
|
1924
|
+
Observed only when paging stopped on a repeated page (2026-10-23): two JSON objects
|
|
1925
|
+
in a row, the records read and then the errors.
|
|
1926
|
+
examples:
|
|
1927
|
+
paging_stopped:
|
|
1928
|
+
summary: Paging stopped because a next page repeated the previous one
|
|
1929
|
+
value: '{"data":[{"id":"o-a"},{"id":"o-b"}],"dataURIs":[null,null]}{"errors":[{"code":"page_response_error","message":"Identical response received from next page request. No more page requests will be made.","source":"resource","resolved":false}]}'
|
|
1915
1930
|
'400':
|
|
1916
1931
|
description: The top-level `export` object is missing.
|
|
1917
1932
|
content:
|
package/dist/export.yml
CHANGED
|
@@ -695,7 +695,10 @@ paths:
|
|
|
695
695
|
(after its transform, output filter and preSavePage hook) or an
|
|
696
696
|
`errors` array if the export encountered application-level failures
|
|
697
697
|
(connection timeout, file not found, etc.). Always inspect for errors
|
|
698
|
-
even on 200.
|
|
698
|
+
even on 200. When paging stopped on a repeated page (observed 2026-10-23), the body
|
|
699
|
+
was instead `text/plain` holding two JSON objects in a row: the records read (`data`,
|
|
700
|
+
`dataURIs`), then `{"errors": [...]}` carrying the `page_response_error`. A client that
|
|
701
|
+
parses that body as one JSON document fails on it; split it into its objects.
|
|
699
702
|
content:
|
|
700
703
|
application/json:
|
|
701
704
|
schema:
|
|
@@ -715,6 +718,16 @@ paths:
|
|
|
715
718
|
service: file-adaptor
|
|
716
719
|
resolved: false
|
|
717
720
|
occurredAt: 1777645522601
|
|
721
|
+
text/plain:
|
|
722
|
+
schema:
|
|
723
|
+
type: string
|
|
724
|
+
description: |-
|
|
725
|
+
Observed only when paging stopped on a repeated page (2026-10-23): two JSON objects
|
|
726
|
+
in a row, the records read and then the errors.
|
|
727
|
+
examples:
|
|
728
|
+
paging_stopped:
|
|
729
|
+
summary: Paging stopped because a next page repeated the previous one
|
|
730
|
+
value: '{"data":[{"id":"o-a"},{"id":"o-b"}],"dataURIs":[null,null]}{"errors":[{"code":"page_response_error","message":"Identical response received from next page request. No more page requests will be made.","source":"resource","resolved":false}]}'
|
|
718
731
|
'401':
|
|
719
732
|
$ref: '#/components/responses/401-unauthorized'
|
|
720
733
|
'404':
|
|
@@ -995,10 +1008,12 @@ components:
|
|
|
995
1008
|
Configures incremental exports that retrieve only records created or modified since the
|
|
996
1009
|
flow's delta checkpoint; omit it for other export types. Adaptors that compare a record field
|
|
997
1010
|
name it in `dateField`, while HTTP exports and SQL queries carry {{lastExportDateTime}} in the
|
|
998
|
-
request and save without a delta object. Each run
|
|
999
|
-
|
|
1000
|
-
|
|
1001
|
-
|
|
1011
|
+
request and save without a delta object. Each run reads the window from {{lastExportDateTime}}
|
|
1012
|
+
to {{currentExportDateTime}}: the flow's checkpoint and the time the run started, both less
|
|
1013
|
+
`lagOffset` when it is set. Until a run has exported records the flow has no checkpoint, and
|
|
1014
|
+
{{lastExportDateTime}} is `startDate` itself, unshifted. A first run is not a full export: with
|
|
1015
|
+
no `startDate`, its {{lastExportDateTime}} is the run's own timestamp, so it picks up no
|
|
1016
|
+
earlier changes. The checkpoint moves to a run's start time only when the run
|
|
1002
1017
|
exports records, so a run that exports nothing leaves it where it was, and after a failed run
|
|
1003
1018
|
the next run reuses the last checkpoint. On success, when `POST /v1/flows/{_id}/run` sends
|
|
1004
1019
|
`export.endDate`, the flow's checkpoint is left at that value, so an end date in the past
|
|
@@ -1025,7 +1040,7 @@ components:
|
|
|
1025
1040
|
default ``startDate`` to **today at midnight UTC** in ISO-8601 format
|
|
1026
1041
|
(e.g. ``"2026-05-14T00:00:00.000Z"``). This means the first run picks
|
|
1027
1042
|
up records changed today; subsequent runs use the platform-managed
|
|
1028
|
-
``{{lastExportDateTime}}`` (the
|
|
1043
|
+
``{{lastExportDateTime}}`` (the start of the last run that exported records, less ``lagOffset``).
|
|
1029
1044
|
|
|
1030
1045
|
- First run: filters records changed since ``startDate``.
|
|
1031
1046
|
- Subsequent runs: uses ``{{lastExportDateTime}}`` as the lower bound;
|
|
@@ -1041,10 +1056,13 @@ components:
|
|
|
1041
1056
|
2. The system automatically maintains the checkpoint
|
|
1042
1057
|
- No need to store or manage timestamps in your own code
|
|
1043
1058
|
- A first run without ``startDate`` fetches nothing older than the run itself
|
|
1044
|
-
- A run that exports records moves the checkpoint to its
|
|
1045
|
-
|
|
1046
|
-
|
|
1047
|
-
|
|
1059
|
+
- A run that exports records moves the checkpoint to its start time. The next run
|
|
1060
|
+
reads from the checkpoint less ``lagOffset``, which is this run's
|
|
1061
|
+
{{currentExportDateTime}} while ``lagOffset`` is unchanged
|
|
1062
|
+
- Use {{lastExportDateTime}} as the lower bound. {{currentExportDateTime}} as the
|
|
1063
|
+
lower bound asks only for records changed after this run's start less ``lagOffset``
|
|
1064
|
+
(nothing when it is 0), so the changes from {{lastExportDateTime}} up to that bound are
|
|
1065
|
+
never exported
|
|
1048
1066
|
|
|
1049
1067
|
3. Error handling and recovery:
|
|
1050
1068
|
- If an export fails, the next run uses the last successful timestamp
|
|
@@ -1055,7 +1073,7 @@ components:
|
|
|
1055
1073
|
- A run given its own window (export.startDate / export.endDate on
|
|
1056
1074
|
POST /v1/flows/{_id}/run) that exports records leaves the checkpoint at endDate,
|
|
1057
1075
|
even a past one, so the next run re-reads everything after it
|
|
1058
|
-
-
|
|
1076
|
+
- ``lagOffset`` delays every window by its value, for a source that shows changes late
|
|
1059
1077
|
- |-
|
|
1060
1078
|
Omit this object for export types other than "delta". An HTTP export with type
|
|
1061
1079
|
"delta" saves without it, since the request carries {{lastExportDateTime}}, but
|
|
@@ -1225,46 +1243,36 @@ components:
|
|
|
1225
1243
|
lagOffset:
|
|
1226
1244
|
type: integer
|
|
1227
1245
|
description: |-
|
|
1228
|
-
|
|
1229
|
-
|
|
1230
|
-
|
|
1231
|
-
|
|
1232
|
-
|
|
1246
|
+
Milliseconds taken off a run's start time to give `{{currentExportDateTime}}`, the window's
|
|
1247
|
+
upper bound, and off the flow's checkpoint, the start time of the last run that exported
|
|
1248
|
+
records, to give `{{lastExportDateTime}}`, its lower bound. Until a run has exported records
|
|
1249
|
+
there is no checkpoint, and the lower bound is `startDate` itself, unshifted, so a
|
|
1250
|
+
`startDate` within the last `lagOffset` gives the run an empty window. While the value stays the same, windows stay back to back and nothing is read twice:
|
|
1251
|
+
each record just reaches a run `lagOffset` later, after it has had that long to become
|
|
1252
|
+
visible. Changing it moves the next window's start by the change: raising it re-reads the
|
|
1253
|
+
added span once, and lowering it skips the removed span. Set it when records
|
|
1254
|
+
changed shortly before a run are sometimes missing because the source shows them with a
|
|
1255
|
+
delay (indexing, replication), and keep it no larger than that delay, since every record
|
|
1256
|
+
waits that much longer. Never set a negative value: once the flow has a checkpoint it moves
|
|
1257
|
+
both ends into the future, so a record changed in the `lagOffset` after each run falls in no
|
|
1258
|
+
window and is never exported (before a checkpoint only `{{currentExportDateTime}}` moves).
|
|
1233
1259
|
x-celigo-ai-guidance:
|
|
1234
|
-
- Specifies a time buffer (in milliseconds) to account for system data propagation delays.
|
|
1235
|
-
- |-
|
|
1236
|
-
## FIELD BEHAVIOR
|
|
1237
|
-
|
|
1238
|
-
This field addresses synchronization issues caused by replication or indexing delays:
|
|
1239
|
-
|
|
1240
|
-
- OPTIONAL: Only needed for systems with known data visibility delays
|
|
1241
|
-
- Value is SUBTRACTED from the last successful run timestamp
|
|
1242
|
-
- Creates an overlapping window to catch records that were being processed
|
|
1243
|
-
during the previous export
|
|
1244
|
-
- Measured in milliseconds (1000ms = 1 second)
|
|
1245
1260
|
- |-
|
|
1246
|
-
|
|
1247
|
-
|
|
1248
|
-
|
|
1249
|
-
|
|
1250
|
-
|
|
1251
|
-
|
|
1252
|
-
|
|
1253
|
-
|
|
1254
|
-
|
|
1255
|
-
- 15000 (15 seconds): Typical for systems with short indexing delays
|
|
1256
|
-
- 60000 (1 minute): Common for systems with moderate replication lag
|
|
1257
|
-
- 300000 (5 minutes): For systems with significant processing delays
|
|
1261
|
+
Verified live (2026-10-23, an HTTP delta export run twice with `lagOffset` 3600000 and
|
|
1262
|
+
then twice with -3600000): each run's `{{currentExportDateTime}}` was its start minus
|
|
1263
|
+
`lagOffset`; the next run's `{{lastExportDateTime}}` was the previous run's start minus
|
|
1264
|
+
`lagOffset`, which equals the previous `{{currentExportDateTime}}` while the value is
|
|
1265
|
+
unchanged; the first successful run's `{{lastExportDateTime}}` was `startDate` unshifted,
|
|
1266
|
+
after two failed runs that hadn't moved it. With -3600000 both ends were an hour after
|
|
1267
|
+
the run, and the switch from 3600000 moved the next window's start two hours past the
|
|
1268
|
+
previous `{{currentExportDateTime}}`, skipping that span.
|
|
1258
1269
|
- |-
|
|
1259
1270
|
## DIAGNOSIS
|
|
1260
1271
|
|
|
1261
|
-
|
|
1262
|
-
|
|
1263
|
-
|
|
1264
|
-
|
|
1265
|
-
|
|
1266
|
-
IMPORTANT: Setting this value too high decreases efficiency by processing
|
|
1267
|
-
redundant records. Set only as high as needed to avoid missed records.
|
|
1272
|
+
Set it when records created or modified near the run time are occasionally missing from
|
|
1273
|
+
a delta export because the source makes them visible late. Start from the observed delay
|
|
1274
|
+
(15000 to 300000 is typical) and raise it only as far as the misses stop: each record
|
|
1275
|
+
arrives `lagOffset` later, and each raise re-reads the added span once.
|
|
1268
1276
|
examples:
|
|
1269
1277
|
- 15000
|
|
1270
1278
|
- 30000
|
|
@@ -1275,10 +1283,10 @@ components:
|
|
|
1275
1283
|
- 'null'
|
|
1276
1284
|
format: date-time
|
|
1277
1285
|
description: |-
|
|
1278
|
-
Lower-bound cutoff for the first run. Omitting it doesn't export history: the first run
|
|
1286
|
+
Lower-bound cutoff for the first run, and for every run until one exports records. Omitting it doesn't export history: the first run
|
|
1279
1287
|
then starts at its own timestamp, an empty window. Set it to the earliest change the first
|
|
1280
|
-
run should pick up. Later runs start from the flow's checkpoint, the
|
|
1281
|
-
that exported records.
|
|
1288
|
+
run should pick up. Later runs start from the flow's checkpoint, the start of the last run
|
|
1289
|
+
that exported records (less `lagOffset`).
|
|
1282
1290
|
x-celigo-ai-guidance:
|
|
1283
1291
|
- |-
|
|
1284
1292
|
Initial cutoff timestamp for the first export run, in ISO-8601 UTC format (e.g.
|
|
@@ -1297,8 +1305,8 @@ components:
|
|
|
1297
1305
|
- |-
|
|
1298
1306
|
## SUBSEQUENT RUNS
|
|
1299
1307
|
After the first run that exports records, the platform keeps the
|
|
1300
|
-
flow's checkpoint at the
|
|
1301
|
-
and exposes it as ``{{lastExportDateTime}}``; runs that export
|
|
1308
|
+
flow's checkpoint at the start of the last run that exported records
|
|
1309
|
+
and exposes it, less ``lagOffset``, as ``{{lastExportDateTime}}``; runs that export
|
|
1302
1310
|
nothing leave it unchanged. ``startDate`` becomes the floor used
|
|
1303
1311
|
only when the platform timestamp is absent (e.g. after a reset).
|
|
1304
1312
|
- |-
|
|
@@ -6701,7 +6709,7 @@ components:
|
|
|
6701
6709
|
|
|
6702
6710
|
When this export is the first lookup of a builder-mode API with `pagination.enabled` set, it is that API's volume driver: saving the export (even unchanged) invalidates the API's issued cursors, so a caller mid-walk gets an empty final page, and removing `paging` is accepted on save but makes every call to the API fail with `400 api_pagination_source_not_paged` until paging is restored or the API's flag is turned off.
|
|
6703
6711
|
|
|
6704
|
-
When a next page's response is identical to the previous one, paging stops: the export records an open error with code `page_response_error` ("Identical response received from next page request. No more page requests will be made.") and
|
|
6712
|
+
When a next page's response is identical to the previous one, paging stops: the export records an open error with code `page_response_error` ("Identical response received from next page request. No more page requests will be made.") and later pages are never exported. A request that doesn't change between pages produces this, as observed with `method: page` and the pagination variable missing from the relative URI and body; reference the variable so each request asks for the next page. In that probe (an HTTP export, 2026-10-23) the run completed with the pages already read when the export had no `type`, and failed when it had `type: "delta"`: the steps after the export received nothing and the checkpoint didn't move. Other `type` values weren't tried. An invoke of that export, or a virtual export of the same document, answered 200 with the records read, then `{"errors": [...]}` carrying the `page_response_error`, as two JSON objects in one `text/plain` body.
|
|
6705
6713
|
x-celigo-ai-guidance:
|
|
6706
6714
|
- Configuration object for navigating through multi-page API responses.
|
|
6707
6715
|
- |-
|
package/dist/flow.yml
CHANGED
|
@@ -904,7 +904,22 @@ paths:
|
|
|
904
904
|
'401':
|
|
905
905
|
$ref: '#/components/responses/401-unauthorized'
|
|
906
906
|
'404':
|
|
907
|
-
|
|
907
|
+
description: |-
|
|
908
|
+
Flow not found ("Flow not found"). A flow that exists but has page generators and no step
|
|
909
|
+
after them (no page processor or router) answers the same 404, enabled or disabled
|
|
910
|
+
(observed 2026-10-24), so the answer doesn't tell it from an unknown id: give the flow a
|
|
911
|
+
step after the export before running it. `POST /v1/flows/{_id}/test/run` refuses that flow
|
|
912
|
+
with a 404 of its own wording.
|
|
913
|
+
content:
|
|
914
|
+
application/json:
|
|
915
|
+
schema:
|
|
916
|
+
$ref: '#/components/schemas/Error'
|
|
917
|
+
examples:
|
|
918
|
+
not_found:
|
|
919
|
+
summary: Unknown flow id, or a flow with nothing after its page generators
|
|
920
|
+
value:
|
|
921
|
+
errors:
|
|
922
|
+
- message: Flow not found
|
|
908
923
|
'409':
|
|
909
924
|
description: |-
|
|
910
925
|
Another instance of this flow is already queued or running. Wait for it to
|
|
@@ -5115,7 +5130,7 @@ components:
|
|
|
5115
5130
|
description: Overrides the fixed start timestamp used for the delta window on the abstract flow's export, in place of the last successful run time.
|
|
5116
5131
|
lagOffset:
|
|
5117
5132
|
type: integer
|
|
5118
|
-
description: Overrides the
|
|
5133
|
+
description: Overrides the abstract flow's export `delta.lagOffset` -- milliseconds taken off both ends of a delta window once the flow has a checkpoint, so records reach a run that much later. While the value stays the same, windows don't overlap; raising it re-reads the added span once, and lowering it skips the removed span. Never negative (once the flow has a checkpoint, that moves the window into the future and skips records).
|
|
5119
5134
|
endDateField:
|
|
5120
5135
|
type: string
|
|
5121
5136
|
description: Overrides the record timestamp field used as the upper bound of the delta window on the abstract flow's export.
|
|
@@ -7492,14 +7507,19 @@ components:
|
|
|
7492
7507
|
doneExporting:
|
|
7493
7508
|
type: boolean
|
|
7494
7509
|
readOnly: true
|
|
7495
|
-
description:
|
|
7510
|
+
description: |-
|
|
7511
|
+
When true, all export pages have been generated. Observed exception: parent flow jobs
|
|
7512
|
+
read false after their runs completed (runs of an HTTP export into an HTTP import, read
|
|
7513
|
+
again minutes after they ended, 2026-10-23); `status` says whether a run finished.
|
|
7496
7514
|
numPagesProcessed:
|
|
7497
7515
|
type: integer
|
|
7498
7516
|
readOnly: true
|
|
7499
7517
|
description: |-
|
|
7500
7518
|
Number of pages processed by downstream steps: imports and lookup steps. A source export's
|
|
7501
7519
|
child job reads 0, a lookup step's and an import's count the pages they processed, and a
|
|
7502
|
-
parent flow job sums its children.
|
|
7520
|
+
parent flow job sums its children. Observed exception: on those same runs the parent read 0
|
|
7521
|
+
while its import job read 1; when a parent reads 0, read the step jobs
|
|
7522
|
+
(`GET /v1/jobs/{_id}/family`).
|
|
7503
7523
|
examples:
|
|
7504
7524
|
- 10
|
|
7505
7525
|
oIndex:
|
package/dist/job.yml
CHANGED
|
@@ -1599,14 +1599,19 @@ components:
|
|
|
1599
1599
|
doneExporting:
|
|
1600
1600
|
type: boolean
|
|
1601
1601
|
readOnly: true
|
|
1602
|
-
description:
|
|
1602
|
+
description: |-
|
|
1603
|
+
When true, all export pages have been generated. Observed exception: parent flow jobs
|
|
1604
|
+
read false after their runs completed (runs of an HTTP export into an HTTP import, read
|
|
1605
|
+
again minutes after they ended, 2026-10-23); `status` says whether a run finished.
|
|
1603
1606
|
numPagesProcessed:
|
|
1604
1607
|
type: integer
|
|
1605
1608
|
readOnly: true
|
|
1606
1609
|
description: |-
|
|
1607
1610
|
Number of pages processed by downstream steps: imports and lookup steps. A source export's
|
|
1608
1611
|
child job reads 0, a lookup step's and an import's count the pages they processed, and a
|
|
1609
|
-
parent flow job sums its children.
|
|
1612
|
+
parent flow job sums its children. Observed exception: on those same runs the parent read 0
|
|
1613
|
+
while its import job read 1; when a parent reads 0, read the step jobs
|
|
1614
|
+
(`GET /v1/jobs/{_id}/family`).
|
|
1610
1615
|
examples:
|
|
1611
1616
|
- 10
|
|
1612
1617
|
oIndex:
|
package/dist/tool.yml
CHANGED
|
@@ -26745,10 +26745,12 @@ components:
|
|
|
26745
26745
|
Configures incremental exports that retrieve only records created or modified since the
|
|
26746
26746
|
flow's delta checkpoint; omit it for other export types. Adaptors that compare a record field
|
|
26747
26747
|
name it in `dateField`, while HTTP exports and SQL queries carry {{lastExportDateTime}} in the
|
|
26748
|
-
request and save without a delta object. Each run
|
|
26749
|
-
|
|
26750
|
-
|
|
26751
|
-
|
|
26748
|
+
request and save without a delta object. Each run reads the window from {{lastExportDateTime}}
|
|
26749
|
+
to {{currentExportDateTime}}: the flow's checkpoint and the time the run started, both less
|
|
26750
|
+
`lagOffset` when it is set. Until a run has exported records the flow has no checkpoint, and
|
|
26751
|
+
{{lastExportDateTime}} is `startDate` itself, unshifted. A first run is not a full export: with
|
|
26752
|
+
no `startDate`, its {{lastExportDateTime}} is the run's own timestamp, so it picks up no
|
|
26753
|
+
earlier changes. The checkpoint moves to a run's start time only when the run
|
|
26752
26754
|
exports records, so a run that exports nothing leaves it where it was, and after a failed run
|
|
26753
26755
|
the next run reuses the last checkpoint. On success, when `POST /v1/flows/{_id}/run` sends
|
|
26754
26756
|
`export.endDate`, the flow's checkpoint is left at that value, so an end date in the past
|
|
@@ -26775,7 +26777,7 @@ components:
|
|
|
26775
26777
|
default ``startDate`` to **today at midnight UTC** in ISO-8601 format
|
|
26776
26778
|
(e.g. ``"2026-05-14T00:00:00.000Z"``). This means the first run picks
|
|
26777
26779
|
up records changed today; subsequent runs use the platform-managed
|
|
26778
|
-
``{{lastExportDateTime}}`` (the
|
|
26780
|
+
``{{lastExportDateTime}}`` (the start of the last run that exported records, less ``lagOffset``).
|
|
26779
26781
|
|
|
26780
26782
|
- First run: filters records changed since ``startDate``.
|
|
26781
26783
|
- Subsequent runs: uses ``{{lastExportDateTime}}`` as the lower bound;
|
|
@@ -26791,10 +26793,13 @@ components:
|
|
|
26791
26793
|
2. The system automatically maintains the checkpoint
|
|
26792
26794
|
- No need to store or manage timestamps in your own code
|
|
26793
26795
|
- A first run without ``startDate`` fetches nothing older than the run itself
|
|
26794
|
-
- A run that exports records moves the checkpoint to its
|
|
26795
|
-
|
|
26796
|
-
|
|
26797
|
-
|
|
26796
|
+
- A run that exports records moves the checkpoint to its start time. The next run
|
|
26797
|
+
reads from the checkpoint less ``lagOffset``, which is this run's
|
|
26798
|
+
{{currentExportDateTime}} while ``lagOffset`` is unchanged
|
|
26799
|
+
- Use {{lastExportDateTime}} as the lower bound. {{currentExportDateTime}} as the
|
|
26800
|
+
lower bound asks only for records changed after this run's start less ``lagOffset``
|
|
26801
|
+
(nothing when it is 0), so the changes from {{lastExportDateTime}} up to that bound are
|
|
26802
|
+
never exported
|
|
26798
26803
|
|
|
26799
26804
|
3. Error handling and recovery:
|
|
26800
26805
|
- If an export fails, the next run uses the last successful timestamp
|
|
@@ -26805,7 +26810,7 @@ components:
|
|
|
26805
26810
|
- A run given its own window (export.startDate / export.endDate on
|
|
26806
26811
|
POST /v1/flows/{_id}/run) that exports records leaves the checkpoint at endDate,
|
|
26807
26812
|
even a past one, so the next run re-reads everything after it
|
|
26808
|
-
-
|
|
26813
|
+
- ``lagOffset`` delays every window by its value, for a source that shows changes late
|
|
26809
26814
|
- |-
|
|
26810
26815
|
Omit this object for export types other than "delta". An HTTP export with type
|
|
26811
26816
|
"delta" saves without it, since the request carries {{lastExportDateTime}}, but
|
|
@@ -26975,46 +26980,36 @@ components:
|
|
|
26975
26980
|
lagOffset:
|
|
26976
26981
|
type: integer
|
|
26977
26982
|
description: |-
|
|
26978
|
-
|
|
26979
|
-
|
|
26980
|
-
|
|
26981
|
-
|
|
26982
|
-
|
|
26983
|
-
|
|
26984
|
-
|
|
26985
|
-
|
|
26986
|
-
|
|
26987
|
-
|
|
26988
|
-
|
|
26989
|
-
|
|
26990
|
-
|
|
26991
|
-
|
|
26992
|
-
|
|
26993
|
-
|
|
26994
|
-
-
|
|
26995
|
-
|
|
26996
|
-
|
|
26997
|
-
|
|
26998
|
-
|
|
26999
|
-
|
|
27000
|
-
|
|
27001
|
-
```
|
|
27002
|
-
- |-
|
|
27003
|
-
## COMMON VALUES
|
|
27004
|
-
|
|
27005
|
-
- 15000 (15 seconds): Typical for systems with short indexing delays
|
|
27006
|
-
- 60000 (1 minute): Common for systems with moderate replication lag
|
|
27007
|
-
- 300000 (5 minutes): For systems with significant processing delays
|
|
26983
|
+
Milliseconds taken off a run's start time to give `{{currentExportDateTime}}`, the window's
|
|
26984
|
+
upper bound, and off the flow's checkpoint, the start time of the last run that exported
|
|
26985
|
+
records, to give `{{lastExportDateTime}}`, its lower bound. Until a run has exported records
|
|
26986
|
+
there is no checkpoint, and the lower bound is `startDate` itself, unshifted, so a
|
|
26987
|
+
`startDate` within the last `lagOffset` gives the run an empty window. While the value stays the same, windows stay back to back and nothing is read twice:
|
|
26988
|
+
each record just reaches a run `lagOffset` later, after it has had that long to become
|
|
26989
|
+
visible. Changing it moves the next window's start by the change: raising it re-reads the
|
|
26990
|
+
added span once, and lowering it skips the removed span. Set it when records
|
|
26991
|
+
changed shortly before a run are sometimes missing because the source shows them with a
|
|
26992
|
+
delay (indexing, replication), and keep it no larger than that delay, since every record
|
|
26993
|
+
waits that much longer. Never set a negative value: once the flow has a checkpoint it moves
|
|
26994
|
+
both ends into the future, so a record changed in the `lagOffset` after each run falls in no
|
|
26995
|
+
window and is never exported (before a checkpoint only `{{currentExportDateTime}}` moves).
|
|
26996
|
+
x-celigo-ai-guidance:
|
|
26997
|
+
- |-
|
|
26998
|
+
Verified live (2026-10-23, an HTTP delta export run twice with `lagOffset` 3600000 and
|
|
26999
|
+
then twice with -3600000): each run's `{{currentExportDateTime}}` was its start minus
|
|
27000
|
+
`lagOffset`; the next run's `{{lastExportDateTime}}` was the previous run's start minus
|
|
27001
|
+
`lagOffset`, which equals the previous `{{currentExportDateTime}}` while the value is
|
|
27002
|
+
unchanged; the first successful run's `{{lastExportDateTime}}` was `startDate` unshifted,
|
|
27003
|
+
after two failed runs that hadn't moved it. With -3600000 both ends were an hour after
|
|
27004
|
+
the run, and the switch from 3600000 moved the next window's start two hours past the
|
|
27005
|
+
previous `{{currentExportDateTime}}`, skipping that span.
|
|
27008
27006
|
- |-
|
|
27009
27007
|
## DIAGNOSIS
|
|
27010
27008
|
|
|
27011
|
-
|
|
27012
|
-
|
|
27013
|
-
|
|
27014
|
-
|
|
27015
|
-
|
|
27016
|
-
IMPORTANT: Setting this value too high decreases efficiency by processing
|
|
27017
|
-
redundant records. Set only as high as needed to avoid missed records.
|
|
27009
|
+
Set it when records created or modified near the run time are occasionally missing from
|
|
27010
|
+
a delta export because the source makes them visible late. Start from the observed delay
|
|
27011
|
+
(15000 to 300000 is typical) and raise it only as far as the misses stop: each record
|
|
27012
|
+
arrives `lagOffset` later, and each raise re-reads the added span once.
|
|
27018
27013
|
examples:
|
|
27019
27014
|
- 15000
|
|
27020
27015
|
- 30000
|
|
@@ -27025,10 +27020,10 @@ components:
|
|
|
27025
27020
|
- 'null'
|
|
27026
27021
|
format: date-time
|
|
27027
27022
|
description: |-
|
|
27028
|
-
Lower-bound cutoff for the first run. Omitting it doesn't export history: the first run
|
|
27023
|
+
Lower-bound cutoff for the first run, and for every run until one exports records. Omitting it doesn't export history: the first run
|
|
27029
27024
|
then starts at its own timestamp, an empty window. Set it to the earliest change the first
|
|
27030
|
-
run should pick up. Later runs start from the flow's checkpoint, the
|
|
27031
|
-
that exported records.
|
|
27025
|
+
run should pick up. Later runs start from the flow's checkpoint, the start of the last run
|
|
27026
|
+
that exported records (less `lagOffset`).
|
|
27032
27027
|
x-celigo-ai-guidance:
|
|
27033
27028
|
- |-
|
|
27034
27029
|
Initial cutoff timestamp for the first export run, in ISO-8601 UTC format (e.g.
|
|
@@ -27047,8 +27042,8 @@ components:
|
|
|
27047
27042
|
- |-
|
|
27048
27043
|
## SUBSEQUENT RUNS
|
|
27049
27044
|
After the first run that exports records, the platform keeps the
|
|
27050
|
-
flow's checkpoint at the
|
|
27051
|
-
and exposes it as ``{{lastExportDateTime}}``; runs that export
|
|
27045
|
+
flow's checkpoint at the start of the last run that exported records
|
|
27046
|
+
and exposes it, less ``lagOffset``, as ``{{lastExportDateTime}}``; runs that export
|
|
27052
27047
|
nothing leave it unchanged. ``startDate`` becomes the floor used
|
|
27053
27048
|
only when the platform timestamp is absent (e.g. after a reset).
|
|
27054
27049
|
- |-
|
|
@@ -31956,7 +31951,7 @@ components:
|
|
|
31956
31951
|
|
|
31957
31952
|
When this export is the first lookup of a builder-mode API with `pagination.enabled` set, it is that API's volume driver: saving the export (even unchanged) invalidates the API's issued cursors, so a caller mid-walk gets an empty final page, and removing `paging` is accepted on save but makes every call to the API fail with `400 api_pagination_source_not_paged` until paging is restored or the API's flag is turned off.
|
|
31958
31953
|
|
|
31959
|
-
When a next page's response is identical to the previous one, paging stops: the export records an open error with code `page_response_error` ("Identical response received from next page request. No more page requests will be made.") and
|
|
31954
|
+
When a next page's response is identical to the previous one, paging stops: the export records an open error with code `page_response_error` ("Identical response received from next page request. No more page requests will be made.") and later pages are never exported. A request that doesn't change between pages produces this, as observed with `method: page` and the pagination variable missing from the relative URI and body; reference the variable so each request asks for the next page. In that probe (an HTTP export, 2026-10-23) the run completed with the pages already read when the export had no `type`, and failed when it had `type: "delta"`: the steps after the export received nothing and the checkpoint didn't move. Other `type` values weren't tried. An invoke of that export, or a virtual export of the same document, answered 200 with the records read, then `{"errors": [...]}` carrying the `page_response_error`, as two JSON objects in one `text/plain` body.
|
|
31960
31955
|
x-celigo-ai-guidance:
|
|
31961
31956
|
- Configuration object for navigating through multi-page API responses.
|
|
31962
31957
|
- |-
|
package/package.json
CHANGED