@celigo/api-specs 0.2.197 → 0.2.198
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/export.yml +145 -32
- package/dist/tool.yml +133 -30
- package/package.json +1 -1
- package/schemas.json +2 -1
package/dist/export.yml
CHANGED
|
@@ -5926,12 +5926,22 @@ components:
|
|
|
5926
5926
|
description: |-
|
|
5927
5927
|
Change-data-capture configuration for real-time listener exports (the "Listen for real-time
|
|
5928
5928
|
data" source step). Present only when `type` is `stream`. Set on `MongodbExport` (MongoDB change
|
|
5929
|
-
streams) and `RDBMSExport` (SQL Server CDC
|
|
5930
|
-
streams change events continuously instead of fetching records on a schedule.
|
|
5929
|
+
streams) and `RDBMSExport` (SQL Server CDC, PostgreSQL logical replication, and MySQL binary-log
|
|
5930
|
+
CDC); the listener streams change events continuously instead of fetching records on a schedule.
|
|
5931
5931
|
|
|
5932
5932
|
On save the API normalizes the `properties` grid into the typed fields: a `snapshot.mode`
|
|
5933
5933
|
entry populates `snapshotMode` and a `capture.mode` entry populates `captureMode`. Every
|
|
5934
5934
|
stream export must resolve a snapshot mode from one of those two places.
|
|
5935
|
+
|
|
5936
|
+
The prerequisites live on the database, not on the export: a listener saves (201) on a
|
|
5937
|
+
connection whose database host is unreachable, so a successful save does not confirm them.
|
|
5938
|
+
The platform's listener documentation requires: for PostgreSQL, logical replication
|
|
5939
|
+
(`wal_level=logical`) and a replication-capable user; for SQL Server, CDC enabled on the
|
|
5940
|
+
database and tables; for MongoDB change streams, a replica set; for MySQL, binary logging in
|
|
5941
|
+
row format (`log_bin=ON`, `binlog_format=ROW`, `binlog_row_image=FULL`) and a connection user
|
|
5942
|
+
holding `REPLICATION SLAVE`, `REPLICATION CLIENT` and `SELECT` on the captured tables
|
|
5943
|
+
(`RELOAD` is not needed on Amazon RDS or Aurora, where the listener takes no global read
|
|
5944
|
+
lock).
|
|
5935
5945
|
x-celigo-ai-guidance:
|
|
5936
5946
|
- |-
|
|
5937
5947
|
A stream export that resolves no snapshot mode — neither `snapshotMode` nor a
|
|
@@ -5950,7 +5960,13 @@ components:
|
|
|
5950
5960
|
description: |-
|
|
5951
5961
|
How MongoDB change-stream events are materialized into records. Use
|
|
5952
5962
|
`change_streams_update_full_with_pre_image` when downstream steps need the document's prior
|
|
5953
|
-
state. MongoDB listeners only — `RDBMSExport` (SQL Server CDC, PostgreSQL) does not
|
|
5963
|
+
state. MongoDB listeners only — `RDBMSExport` (SQL Server CDC, PostgreSQL, MySQL) does not
|
|
5964
|
+
set this.
|
|
5965
|
+
x-celigo-ai-guidance:
|
|
5966
|
+
- |-
|
|
5967
|
+
Never author this on an `RDBMSExport`: the API stores it on a MySQL stream export without
|
|
5968
|
+
rejecting it (unlike `slotName`, it is not gated per adaptor), so nothing at save time
|
|
5969
|
+
catches the mistake and the dead setting rides on the document.
|
|
5954
5970
|
examples:
|
|
5955
5971
|
- change_streams_update_full
|
|
5956
5972
|
- change_streams_update_full_with_pre_image
|
|
@@ -5968,13 +5984,17 @@ components:
|
|
|
5968
5984
|
initial_only: Takes the initial snapshot and then stops, without streaming ongoing changes.
|
|
5969
5985
|
default: no_data
|
|
5970
5986
|
description: |-
|
|
5971
|
-
Whether the listener seeds existing rows before streaming changes.
|
|
5972
|
-
existing data on the first run
|
|
5987
|
+
Whether the listener seeds existing rows before streaming changes. `no_data` streams only
|
|
5988
|
+
new changes; `initial` backfills existing data on the first run where the listener form
|
|
5989
|
+
offers it (MongoDB, SQL Server). The PostgreSQL and MySQL forms offer `no_data` and
|
|
5990
|
+
`when_needed` only; the MySQL listener documentation describes `when_needed` as a one-time
|
|
5991
|
+
read of the captured tables when no stream position is stored or the stored position is no
|
|
5992
|
+
longer on the server.
|
|
5973
5993
|
x-celigo-ai-guidance:
|
|
5974
5994
|
- |-
|
|
5975
|
-
The PostgreSQL listener
|
|
5976
|
-
values on
|
|
5977
|
-
|
|
5995
|
+
The PostgreSQL and MySQL listener forms offer only `no_data` and `when_needed`. The API
|
|
5996
|
+
stores other values on those exports without rejecting them, but they are outside the
|
|
5997
|
+
supported matrix — stay within the form's subset.
|
|
5978
5998
|
examples:
|
|
5979
5999
|
- initial
|
|
5980
6000
|
- no_data
|
|
@@ -5989,8 +6009,9 @@ components:
|
|
|
5989
6009
|
underscore).
|
|
5990
6010
|
x-celigo-ai-guidance:
|
|
5991
6011
|
- |-
|
|
5992
|
-
PostgreSQL connections only: on any other connection the save is
|
|
5993
|
-
`not_allowed` ("CDC slotName can only be used with postgresql adaptor type").
|
|
6012
|
+
PostgreSQL connections only: on any other connection — SQL Server, MySQL — the save is
|
|
6013
|
+
rejected 422 `not_allowed` ("CDC slotName can only be used with postgresql adaptor type").
|
|
6014
|
+
MySQL CDC reads the binary log directly; it has no replication slot.
|
|
5994
6015
|
- |-
|
|
5995
6016
|
The identifier pattern is enforced server-side — an invalid name (uppercase, dashes) is
|
|
5996
6017
|
rejected 422 `invalid_field_value` at save time.
|
|
@@ -6010,17 +6031,27 @@ components:
|
|
|
6010
6031
|
sent, so nothing catches the ambiguity at save time — the exclusivity is the listener
|
|
6011
6032
|
form's rule (publication wins as the sole table filter).
|
|
6012
6033
|
- |-
|
|
6013
|
-
PostgreSQL connections only: rejected 422 `not_allowed` on other connections
|
|
6014
|
-
|
|
6034
|
+
PostgreSQL connections only: rejected 422 `not_allowed` on other connections (SQL Server,
|
|
6035
|
+
MySQL — "CDC publicationName can only be used with postgresql adaptor type"), and the
|
|
6036
|
+
identifier pattern is enforced server-side like `slotName`. A MySQL listener scopes
|
|
6037
|
+
tables only through `rdbms.tables` and the include / exclude `properties`.
|
|
6038
|
+
- |-
|
|
6039
|
+
On a PostgreSQL connection one of this field or `rdbms.tables` must be present: a stream
|
|
6040
|
+
export carrying neither is rejected 422 `missing_required_field` on both paths
|
|
6041
|
+
("publicationName is required when tables is not provided." / "tables is required when
|
|
6042
|
+
publicationName is not provided."). SQL Server and MySQL listeners save without either.
|
|
6015
6043
|
examples:
|
|
6016
6044
|
- celigo_pub
|
|
6017
6045
|
- orders_publication
|
|
6018
6046
|
cursor:
|
|
6019
6047
|
type: string
|
|
6020
6048
|
description: |-
|
|
6021
|
-
Server-managed stream position marker (where the listener resumes reading change events
|
|
6049
|
+
Server-managed stream position marker (where the listener resumes reading change events;
|
|
6050
|
+
the MySQL listener documentation describes it as the binlog file and position last saved).
|
|
6022
6051
|
Maintained by the platform as events are consumed; the UI exposes a reset workflow for it
|
|
6023
|
-
under Cursor Management.
|
|
6052
|
+
under Cursor Management, which appears once a cursor exists. The form's reset is a full save that omits `cursor` and carries the
|
|
6053
|
+
top-level request flag `pendingCursorReset: true`; the API accepts the flag and does not
|
|
6054
|
+
store it on the document.
|
|
6024
6055
|
x-celigo-ai-guidance:
|
|
6025
6056
|
- Never author or modify this field — the platform owns the stream position.
|
|
6026
6057
|
paths:
|
|
@@ -6029,17 +6060,33 @@ components:
|
|
|
6029
6060
|
type: string
|
|
6030
6061
|
description: |-
|
|
6031
6062
|
Dot-notation paths within each raw change event to retain in the emitted record. MongoDB
|
|
6032
|
-
listeners default to `["payload"]` (the whole change document); SQL Server and
|
|
6033
|
-
listeners commonly select `payload.after`, `payload.op`, and related fields to expose
|
|
6034
|
-
changed row and operation type.
|
|
6063
|
+
listeners default to `["payload"]` (the whole change document); SQL Server, PostgreSQL and
|
|
6064
|
+
MySQL listeners commonly select `payload.after`, `payload.op`, and related fields to expose
|
|
6065
|
+
the changed row and operation type. The MySQL listener form offers `payload.after`,
|
|
6066
|
+
`payload.before`, `schema`, `payload.source`, `payload.op`, `payload.transaction`,
|
|
6067
|
+
`payload.ts_ms`, `payload.ts_us` and `payload.ts_ns`, preselecting `payload.after`; it has
|
|
6068
|
+
no `updateDescription`-derived options and no `payload.id`. The MySQL listener
|
|
6069
|
+
documentation describes `after` as the row after the change (empty on a delete), `before`
|
|
6070
|
+
as the row before an update or delete (needs `binlog_row_image=FULL`), `op` as `c` / `u` /
|
|
6071
|
+
`d` / `r` (insert / update / delete / snapshot read) and `source` as the database, table,
|
|
6072
|
+
binlog file and position.
|
|
6035
6073
|
examples:
|
|
6036
6074
|
- - payload
|
|
6037
6075
|
- - payload.after
|
|
6038
6076
|
- payload.op
|
|
6077
|
+
- - payload.after
|
|
6078
|
+
- payload.before
|
|
6079
|
+
- payload.op
|
|
6039
6080
|
intervalTime:
|
|
6040
6081
|
type: integer
|
|
6041
6082
|
default: 300
|
|
6042
|
-
|
|
6083
|
+
minimum: 30
|
|
6084
|
+
x-celigo-ui-override: The minimum mirrors the listener forms, which label this field Max wait time and reject a value under 30 ("Max wait time must be at least 30 seconds."); the API itself stores a lower value.
|
|
6085
|
+
description: |-
|
|
6086
|
+
Interval, in seconds, that bounds how long the listener holds change events before sending
|
|
6087
|
+
a page downstream. The listener forms label it Max wait time, and the platform's listener
|
|
6088
|
+
documentation describes it as the longest a page waits when fewer than `pageSize` events
|
|
6089
|
+
have arrived — a full page is sent as soon as it fills.
|
|
6043
6090
|
examples:
|
|
6044
6091
|
- 300
|
|
6045
6092
|
- 30
|
|
@@ -6048,11 +6095,25 @@ components:
|
|
|
6048
6095
|
description: |-
|
|
6049
6096
|
Connector-engine settings passed through to the underlying CDC connector as name/value
|
|
6050
6097
|
pairs. Mirrors the form's additional-properties grid: the required keys (`capture.mode`
|
|
6051
|
-
and `snapshot.mode` for MongoDB, `snapshot.mode` for SQL Server and
|
|
6098
|
+
and `snapshot.mode` for MongoDB, `snapshot.mode` for SQL Server, PostgreSQL and MySQL) are
|
|
6052
6099
|
serialized first, followed by optional keys such as `field.exclude.list`,
|
|
6053
6100
|
`column.include.list`, or `snapshot.collection.filter.overrides.<db>.<collection>`.
|
|
6054
6101
|
On save the API also lifts `snapshot.mode` / `capture.mode` entries into the typed
|
|
6055
6102
|
`snapshotMode` / `captureMode` fields.
|
|
6103
|
+
|
|
6104
|
+
The MySQL listener form offers six keys: `snapshot.mode` (preselected `no_data`) and five
|
|
6105
|
+
optional comma-separated regular-expression lists — `table.exclude.list` (matched on
|
|
6106
|
+
`database.table`), `database.include.list`, `database.exclude.list`,
|
|
6107
|
+
`column.include.list` and `column.exclude.list` (matched on `database.table.column`).
|
|
6108
|
+
MySQL has no schema layer, so there are no `schema.include.list` / `schema.exclude.list`
|
|
6109
|
+
keys, and the MongoDB keys (`capture.mode`, `collection.exclude.list`,
|
|
6110
|
+
`field.exclude.list`) do not apply. The MySQL listener documentation describes a blank
|
|
6111
|
+
watch list as every table the connection's user can read, trimmed by `table.exclude.list`.
|
|
6112
|
+
x-celigo-ai-guidance:
|
|
6113
|
+
- |-
|
|
6114
|
+
The API persists any key sent here without validating it against the connector, so a
|
|
6115
|
+
MongoDB or PostgreSQL key on a MySQL listener saves cleanly and does nothing — author only
|
|
6116
|
+
the keys the listener's form offers.
|
|
6056
6117
|
items:
|
|
6057
6118
|
type: object
|
|
6058
6119
|
required:
|
|
@@ -11977,8 +12038,8 @@ components:
|
|
|
11977
12038
|
for other connection types. For query-type exports (standard, delta, once) it is REQUIRED
|
|
11978
12039
|
when the _connectionId field references an RDBMS database connection and holds the SQL
|
|
11979
12040
|
query. For real-time stream listeners (`type: "stream"`) it holds the CDC watch list
|
|
11980
|
-
(`tables`) instead — and a
|
|
11981
|
-
(`cdc.publicationName`) may omit this object entirely.
|
|
12041
|
+
(`tables`) instead — and a SQL Server or MySQL listener with no watch list, or a PostgreSQL
|
|
12042
|
+
listener scoped by a publication (`cdc.publicationName`), may omit this object entirely.
|
|
11982
12043
|
|
|
11983
12044
|
**Rdbms export capabilities**
|
|
11984
12045
|
- Execute custom SQL SELECT statements
|
|
@@ -12032,8 +12093,8 @@ components:
|
|
|
12032
12093
|
```
|
|
12033
12094
|
|
|
12034
12095
|
**Real-time CDC listener exports (type: "stream")**
|
|
12035
|
-
SQL Server and
|
|
12036
|
-
changes continuously (the `cdc` object holds the listener settings). A stream export has NO
|
|
12096
|
+
SQL Server, PostgreSQL and MySQL connections support change-data-capture listeners that stream
|
|
12097
|
+
row changes continuously (the `cdc` object holds the listener settings). A stream export has NO
|
|
12037
12098
|
`query` — it watches `tables`:
|
|
12038
12099
|
```json
|
|
12039
12100
|
{
|
|
@@ -12042,9 +12103,26 @@ components:
|
|
|
12042
12103
|
"cdc": { "properties": [{ "name": "snapshot.mode", "value": "no_data" }] }
|
|
12043
12104
|
}
|
|
12044
12105
|
```
|
|
12106
|
+
A MySQL listener names its tables as `database.table` (MySQL has no schema layer):
|
|
12107
|
+
```json
|
|
12108
|
+
{
|
|
12109
|
+
"type": "stream",
|
|
12110
|
+
"rdbms": { "tables": "sales.orders,sales.order_lines" },
|
|
12111
|
+
"cdc": {
|
|
12112
|
+
"properties": [{ "name": "snapshot.mode", "value": "no_data" }],
|
|
12113
|
+
"paths": ["payload.after", "payload.op"]
|
|
12114
|
+
}
|
|
12115
|
+
}
|
|
12116
|
+
```
|
|
12045
12117
|
A PostgreSQL listener can instead scope tables through a publication
|
|
12046
12118
|
(`cdc.publicationName`), in which case `tables` stays unset and the `rdbms` object may be
|
|
12047
|
-
omitted entirely.
|
|
12119
|
+
omitted entirely; on PostgreSQL one of the two must be present. SQL Server and MySQL
|
|
12120
|
+
listeners save with no `tables` and no `rdbms` object at all (a blank `tables` string is
|
|
12121
|
+
dropped on save); the MySQL listener documentation describes that as every table the
|
|
12122
|
+
connection's user can read, trimmed by the `table.exclude.list` property.
|
|
12123
|
+
`POST /v1/exports/preview` rejects a stream export both with `rdbms.tables` sent and with
|
|
12124
|
+
no `rdbms` object at all, with the same 422 `missing_required_field` ("rdbms subschema not
|
|
12125
|
+
defined") — listeners have no preview.
|
|
12048
12126
|
properties:
|
|
12049
12127
|
tables:
|
|
12050
12128
|
type: string
|
|
@@ -12052,16 +12130,26 @@ components:
|
|
|
12052
12130
|
Comma-separated list of tables the CDC listener watches for changes. Used only by real-time
|
|
12053
12131
|
listener exports (`type` is `stream`); standard SQL exports use `query` instead. SQL Server
|
|
12054
12132
|
listeners use fully-qualified three-part names (`database.schema.table`); PostgreSQL
|
|
12055
|
-
listeners use two-part names (`schema.table`)
|
|
12133
|
+
listeners use two-part names (`schema.table`); MySQL listeners use two-part
|
|
12134
|
+
`database.table` names, since MySQL has no schema layer between the database and the
|
|
12135
|
+
table. A PostgreSQL listener must carry this field or `cdc.publicationName`; SQL Server
|
|
12136
|
+
and MySQL listeners may leave it out (for MySQL the listener documentation describes that
|
|
12137
|
+
as every table the connection's user can read, trimmed by `table.exclude.list`).
|
|
12056
12138
|
x-celigo-ai-guidance:
|
|
12057
12139
|
- |-
|
|
12058
12140
|
For a PostgreSQL listener, author EITHER this field OR `cdc.publicationName`, never both —
|
|
12059
12141
|
when a publication is set it is the sole table filter and the listener form clears this
|
|
12060
12142
|
field (the API stores both if sent, so nothing catches the conflict at save time).
|
|
12143
|
+
- |-
|
|
12144
|
+
The API does not validate the names against the engine — a three-part name on a MySQL
|
|
12145
|
+
listener saves cleanly — so nothing at save time catches the wrong form. Use the engine's
|
|
12146
|
+
own: `database.table` on MySQL, `schema.table` on PostgreSQL, `database.schema.table` on
|
|
12147
|
+
SQL Server.
|
|
12061
12148
|
examples:
|
|
12062
12149
|
- UKGODS.dbo.UKG_MetadataStaging
|
|
12063
12150
|
- dbo.WorkOrderT,dbo.OrderLineT
|
|
12064
12151
|
- public.orders,public.customers
|
|
12152
|
+
- sales.orders,sales.order_lines
|
|
12065
12153
|
query:
|
|
12066
12154
|
type: string
|
|
12067
12155
|
x-celigo-agent:
|
|
@@ -16575,8 +16663,8 @@ components:
|
|
|
16575
16663
|
- **Listen for inbound events from NetSuite or Salesforce**
|
|
16576
16664
|
-> `distributed`
|
|
16577
16665
|
- **Stream database changes in real time as they happen**
|
|
16578
|
-
(change data capture from MongoDB, SQL Server,
|
|
16579
|
-
|
|
16666
|
+
(change data capture from MongoDB, SQL Server, PostgreSQL
|
|
16667
|
+
or MySQL — "when a row changes", "listen to my table")
|
|
16580
16668
|
-> `stream`, configured via the `cdc` object
|
|
16581
16669
|
- **Subscribe to an Event Streams topic** ("listen to topic X",
|
|
16582
16670
|
"consume messages published to …") -> `stream` with
|
|
@@ -16600,8 +16688,8 @@ components:
|
|
|
16600
16688
|
delta config is a separate concern that scopes how you fill in
|
|
16601
16689
|
the chosen config, not whether to use it.
|
|
16602
16690
|
- |-
|
|
16603
|
-
`stream` is accepted only on MongoDB and RDBMS exports (SQL Server
|
|
16604
|
-
or
|
|
16691
|
+
`stream` is accepted only on MongoDB and RDBMS exports (SQL Server,
|
|
16692
|
+
PostgreSQL or MySQL connections) and on Event Streams listeners
|
|
16605
16693
|
(`adaptorType: TopicExport`); every other adaptor rejects the save
|
|
16606
16694
|
with `unsupported_export_type`. A MongoDB or RDBMS stream export
|
|
16607
16695
|
must carry `cdc.snapshotMode` (and `cdc.captureMode` on MongoDB) —
|
|
@@ -16626,7 +16714,7 @@ components:
|
|
|
16626
16714
|
simple: Allows manual file uploads through the data loader UI.
|
|
16627
16715
|
blob: Retrieves raw files without parsing them into structured data records.
|
|
16628
16716
|
distributed: Listens for real-time events from NetSuite or Salesforce.
|
|
16629
|
-
stream: Streams events in real time — database change data capture (MongoDB change streams, SQL Server CDC, PostgreSQL logical replication) configured through the `cdc` object, or an Event Streams topic subscription (`TopicExport`) configured through `topic`.
|
|
16717
|
+
stream: Streams events in real time — database change data capture (MongoDB change streams, SQL Server CDC, PostgreSQL logical replication, MySQL binary log) configured through the `cdc` object, or an Event Streams topic subscription (`TopicExport`) configured through `topic`.
|
|
16630
16718
|
all: Explicitly retrieves all available records — equivalent to omitting `type`.
|
|
16631
16719
|
examples:
|
|
16632
16720
|
- webhook
|
|
@@ -16753,6 +16841,21 @@ components:
|
|
|
16753
16841
|
$ref: '#/components/schemas/Distributed'
|
|
16754
16842
|
cdc:
|
|
16755
16843
|
$ref: '#/components/schemas/Cdc'
|
|
16844
|
+
pendingCursorReset:
|
|
16845
|
+
type: boolean
|
|
16846
|
+
writeOnly: true
|
|
16847
|
+
description: |-
|
|
16848
|
+
When true on a save of a stream listener, requests a reset of the listener's stored stream
|
|
16849
|
+
position — the Cursor Management reset the listener form sends, with `cdc.cursor` omitted
|
|
16850
|
+
from the same document; the listener documentation describes the save as clearing the
|
|
16851
|
+
stored cursor and restarting the listener. Accepted on the request and not stored on the
|
|
16852
|
+
export.
|
|
16853
|
+
x-celigo-ai-guidance:
|
|
16854
|
+
- |-
|
|
16855
|
+
Never author this flag — like `cdc.cursor`, the stream position is the platform's, and
|
|
16856
|
+
the reset is the listener form's Cursor Management action. The API accepts it silently
|
|
16857
|
+
(200) and the readback carries no trace that it was sent, so a save that carried it by
|
|
16858
|
+
mistake cannot be seen afterwards on the export.
|
|
16756
16859
|
topic:
|
|
16757
16860
|
$ref: '#/components/schemas/Topic'
|
|
16758
16861
|
filesystem:
|
|
@@ -17547,8 +17650,18 @@ components:
|
|
|
17547
17650
|
required:
|
|
17548
17651
|
- adaptorType
|
|
17549
17652
|
then:
|
|
17550
|
-
|
|
17551
|
-
|
|
17653
|
+
if:
|
|
17654
|
+
properties:
|
|
17655
|
+
type:
|
|
17656
|
+
const: stream
|
|
17657
|
+
required:
|
|
17658
|
+
- type
|
|
17659
|
+
then:
|
|
17660
|
+
required:
|
|
17661
|
+
- cdc
|
|
17662
|
+
else:
|
|
17663
|
+
required:
|
|
17664
|
+
- rdbms
|
|
17552
17665
|
else:
|
|
17553
17666
|
if:
|
|
17554
17667
|
properties:
|
package/dist/tool.yml
CHANGED
|
@@ -31247,12 +31247,22 @@ components:
|
|
|
31247
31247
|
description: |-
|
|
31248
31248
|
Change-data-capture configuration for real-time listener exports (the "Listen for real-time
|
|
31249
31249
|
data" source step). Present only when `type` is `stream`. Set on `MongodbExport` (MongoDB change
|
|
31250
|
-
streams) and `RDBMSExport` (SQL Server CDC
|
|
31251
|
-
streams change events continuously instead of fetching records on a schedule.
|
|
31250
|
+
streams) and `RDBMSExport` (SQL Server CDC, PostgreSQL logical replication, and MySQL binary-log
|
|
31251
|
+
CDC); the listener streams change events continuously instead of fetching records on a schedule.
|
|
31252
31252
|
|
|
31253
31253
|
On save the API normalizes the `properties` grid into the typed fields: a `snapshot.mode`
|
|
31254
31254
|
entry populates `snapshotMode` and a `capture.mode` entry populates `captureMode`. Every
|
|
31255
31255
|
stream export must resolve a snapshot mode from one of those two places.
|
|
31256
|
+
|
|
31257
|
+
The prerequisites live on the database, not on the export: a listener saves (201) on a
|
|
31258
|
+
connection whose database host is unreachable, so a successful save does not confirm them.
|
|
31259
|
+
The platform's listener documentation requires: for PostgreSQL, logical replication
|
|
31260
|
+
(`wal_level=logical`) and a replication-capable user; for SQL Server, CDC enabled on the
|
|
31261
|
+
database and tables; for MongoDB change streams, a replica set; for MySQL, binary logging in
|
|
31262
|
+
row format (`log_bin=ON`, `binlog_format=ROW`, `binlog_row_image=FULL`) and a connection user
|
|
31263
|
+
holding `REPLICATION SLAVE`, `REPLICATION CLIENT` and `SELECT` on the captured tables
|
|
31264
|
+
(`RELOAD` is not needed on Amazon RDS or Aurora, where the listener takes no global read
|
|
31265
|
+
lock).
|
|
31256
31266
|
x-celigo-ai-guidance:
|
|
31257
31267
|
- |-
|
|
31258
31268
|
A stream export that resolves no snapshot mode — neither `snapshotMode` nor a
|
|
@@ -31271,7 +31281,13 @@ components:
|
|
|
31271
31281
|
description: |-
|
|
31272
31282
|
How MongoDB change-stream events are materialized into records. Use
|
|
31273
31283
|
`change_streams_update_full_with_pre_image` when downstream steps need the document's prior
|
|
31274
|
-
state. MongoDB listeners only — `RDBMSExport` (SQL Server CDC, PostgreSQL) does not
|
|
31284
|
+
state. MongoDB listeners only — `RDBMSExport` (SQL Server CDC, PostgreSQL, MySQL) does not
|
|
31285
|
+
set this.
|
|
31286
|
+
x-celigo-ai-guidance:
|
|
31287
|
+
- |-
|
|
31288
|
+
Never author this on an `RDBMSExport`: the API stores it on a MySQL stream export without
|
|
31289
|
+
rejecting it (unlike `slotName`, it is not gated per adaptor), so nothing at save time
|
|
31290
|
+
catches the mistake and the dead setting rides on the document.
|
|
31275
31291
|
examples:
|
|
31276
31292
|
- change_streams_update_full
|
|
31277
31293
|
- change_streams_update_full_with_pre_image
|
|
@@ -31289,13 +31305,17 @@ components:
|
|
|
31289
31305
|
initial_only: Takes the initial snapshot and then stops, without streaming ongoing changes.
|
|
31290
31306
|
default: no_data
|
|
31291
31307
|
description: |-
|
|
31292
|
-
Whether the listener seeds existing rows before streaming changes.
|
|
31293
|
-
existing data on the first run
|
|
31308
|
+
Whether the listener seeds existing rows before streaming changes. `no_data` streams only
|
|
31309
|
+
new changes; `initial` backfills existing data on the first run where the listener form
|
|
31310
|
+
offers it (MongoDB, SQL Server). The PostgreSQL and MySQL forms offer `no_data` and
|
|
31311
|
+
`when_needed` only; the MySQL listener documentation describes `when_needed` as a one-time
|
|
31312
|
+
read of the captured tables when no stream position is stored or the stored position is no
|
|
31313
|
+
longer on the server.
|
|
31294
31314
|
x-celigo-ai-guidance:
|
|
31295
31315
|
- |-
|
|
31296
|
-
The PostgreSQL listener
|
|
31297
|
-
values on
|
|
31298
|
-
|
|
31316
|
+
The PostgreSQL and MySQL listener forms offer only `no_data` and `when_needed`. The API
|
|
31317
|
+
stores other values on those exports without rejecting them, but they are outside the
|
|
31318
|
+
supported matrix — stay within the form's subset.
|
|
31299
31319
|
examples:
|
|
31300
31320
|
- initial
|
|
31301
31321
|
- no_data
|
|
@@ -31310,8 +31330,9 @@ components:
|
|
|
31310
31330
|
underscore).
|
|
31311
31331
|
x-celigo-ai-guidance:
|
|
31312
31332
|
- |-
|
|
31313
|
-
PostgreSQL connections only: on any other connection the save is
|
|
31314
|
-
`not_allowed` ("CDC slotName can only be used with postgresql adaptor type").
|
|
31333
|
+
PostgreSQL connections only: on any other connection — SQL Server, MySQL — the save is
|
|
31334
|
+
rejected 422 `not_allowed` ("CDC slotName can only be used with postgresql adaptor type").
|
|
31335
|
+
MySQL CDC reads the binary log directly; it has no replication slot.
|
|
31315
31336
|
- |-
|
|
31316
31337
|
The identifier pattern is enforced server-side — an invalid name (uppercase, dashes) is
|
|
31317
31338
|
rejected 422 `invalid_field_value` at save time.
|
|
@@ -31331,17 +31352,27 @@ components:
|
|
|
31331
31352
|
sent, so nothing catches the ambiguity at save time — the exclusivity is the listener
|
|
31332
31353
|
form's rule (publication wins as the sole table filter).
|
|
31333
31354
|
- |-
|
|
31334
|
-
PostgreSQL connections only: rejected 422 `not_allowed` on other connections
|
|
31335
|
-
|
|
31355
|
+
PostgreSQL connections only: rejected 422 `not_allowed` on other connections (SQL Server,
|
|
31356
|
+
MySQL — "CDC publicationName can only be used with postgresql adaptor type"), and the
|
|
31357
|
+
identifier pattern is enforced server-side like `slotName`. A MySQL listener scopes
|
|
31358
|
+
tables only through `rdbms.tables` and the include / exclude `properties`.
|
|
31359
|
+
- |-
|
|
31360
|
+
On a PostgreSQL connection one of this field or `rdbms.tables` must be present: a stream
|
|
31361
|
+
export carrying neither is rejected 422 `missing_required_field` on both paths
|
|
31362
|
+
("publicationName is required when tables is not provided." / "tables is required when
|
|
31363
|
+
publicationName is not provided."). SQL Server and MySQL listeners save without either.
|
|
31336
31364
|
examples:
|
|
31337
31365
|
- celigo_pub
|
|
31338
31366
|
- orders_publication
|
|
31339
31367
|
cursor:
|
|
31340
31368
|
type: string
|
|
31341
31369
|
description: |-
|
|
31342
|
-
Server-managed stream position marker (where the listener resumes reading change events
|
|
31370
|
+
Server-managed stream position marker (where the listener resumes reading change events;
|
|
31371
|
+
the MySQL listener documentation describes it as the binlog file and position last saved).
|
|
31343
31372
|
Maintained by the platform as events are consumed; the UI exposes a reset workflow for it
|
|
31344
|
-
under Cursor Management.
|
|
31373
|
+
under Cursor Management, which appears once a cursor exists. The form's reset is a full save that omits `cursor` and carries the
|
|
31374
|
+
top-level request flag `pendingCursorReset: true`; the API accepts the flag and does not
|
|
31375
|
+
store it on the document.
|
|
31345
31376
|
x-celigo-ai-guidance:
|
|
31346
31377
|
- Never author or modify this field — the platform owns the stream position.
|
|
31347
31378
|
paths:
|
|
@@ -31350,17 +31381,33 @@ components:
|
|
|
31350
31381
|
type: string
|
|
31351
31382
|
description: |-
|
|
31352
31383
|
Dot-notation paths within each raw change event to retain in the emitted record. MongoDB
|
|
31353
|
-
listeners default to `["payload"]` (the whole change document); SQL Server and
|
|
31354
|
-
listeners commonly select `payload.after`, `payload.op`, and related fields to expose
|
|
31355
|
-
changed row and operation type.
|
|
31384
|
+
listeners default to `["payload"]` (the whole change document); SQL Server, PostgreSQL and
|
|
31385
|
+
MySQL listeners commonly select `payload.after`, `payload.op`, and related fields to expose
|
|
31386
|
+
the changed row and operation type. The MySQL listener form offers `payload.after`,
|
|
31387
|
+
`payload.before`, `schema`, `payload.source`, `payload.op`, `payload.transaction`,
|
|
31388
|
+
`payload.ts_ms`, `payload.ts_us` and `payload.ts_ns`, preselecting `payload.after`; it has
|
|
31389
|
+
no `updateDescription`-derived options and no `payload.id`. The MySQL listener
|
|
31390
|
+
documentation describes `after` as the row after the change (empty on a delete), `before`
|
|
31391
|
+
as the row before an update or delete (needs `binlog_row_image=FULL`), `op` as `c` / `u` /
|
|
31392
|
+
`d` / `r` (insert / update / delete / snapshot read) and `source` as the database, table,
|
|
31393
|
+
binlog file and position.
|
|
31356
31394
|
examples:
|
|
31357
31395
|
- - payload
|
|
31358
31396
|
- - payload.after
|
|
31359
31397
|
- payload.op
|
|
31398
|
+
- - payload.after
|
|
31399
|
+
- payload.before
|
|
31400
|
+
- payload.op
|
|
31360
31401
|
intervalTime:
|
|
31361
31402
|
type: integer
|
|
31362
31403
|
default: 300
|
|
31363
|
-
|
|
31404
|
+
minimum: 30
|
|
31405
|
+
x-celigo-ui-override: The minimum mirrors the listener forms, which label this field Max wait time and reject a value under 30 ("Max wait time must be at least 30 seconds."); the API itself stores a lower value.
|
|
31406
|
+
description: |-
|
|
31407
|
+
Interval, in seconds, that bounds how long the listener holds change events before sending
|
|
31408
|
+
a page downstream. The listener forms label it Max wait time, and the platform's listener
|
|
31409
|
+
documentation describes it as the longest a page waits when fewer than `pageSize` events
|
|
31410
|
+
have arrived — a full page is sent as soon as it fills.
|
|
31364
31411
|
examples:
|
|
31365
31412
|
- 300
|
|
31366
31413
|
- 30
|
|
@@ -31369,11 +31416,25 @@ components:
|
|
|
31369
31416
|
description: |-
|
|
31370
31417
|
Connector-engine settings passed through to the underlying CDC connector as name/value
|
|
31371
31418
|
pairs. Mirrors the form's additional-properties grid: the required keys (`capture.mode`
|
|
31372
|
-
and `snapshot.mode` for MongoDB, `snapshot.mode` for SQL Server and
|
|
31419
|
+
and `snapshot.mode` for MongoDB, `snapshot.mode` for SQL Server, PostgreSQL and MySQL) are
|
|
31373
31420
|
serialized first, followed by optional keys such as `field.exclude.list`,
|
|
31374
31421
|
`column.include.list`, or `snapshot.collection.filter.overrides.<db>.<collection>`.
|
|
31375
31422
|
On save the API also lifts `snapshot.mode` / `capture.mode` entries into the typed
|
|
31376
31423
|
`snapshotMode` / `captureMode` fields.
|
|
31424
|
+
|
|
31425
|
+
The MySQL listener form offers six keys: `snapshot.mode` (preselected `no_data`) and five
|
|
31426
|
+
optional comma-separated regular-expression lists — `table.exclude.list` (matched on
|
|
31427
|
+
`database.table`), `database.include.list`, `database.exclude.list`,
|
|
31428
|
+
`column.include.list` and `column.exclude.list` (matched on `database.table.column`).
|
|
31429
|
+
MySQL has no schema layer, so there are no `schema.include.list` / `schema.exclude.list`
|
|
31430
|
+
keys, and the MongoDB keys (`capture.mode`, `collection.exclude.list`,
|
|
31431
|
+
`field.exclude.list`) do not apply. The MySQL listener documentation describes a blank
|
|
31432
|
+
watch list as every table the connection's user can read, trimmed by `table.exclude.list`.
|
|
31433
|
+
x-celigo-ai-guidance:
|
|
31434
|
+
- |-
|
|
31435
|
+
The API persists any key sent here without validating it against the connector, so a
|
|
31436
|
+
MongoDB or PostgreSQL key on a MySQL listener saves cleanly and does nothing — author only
|
|
31437
|
+
the keys the listener's form offers.
|
|
31377
31438
|
items:
|
|
31378
31439
|
type: object
|
|
31379
31440
|
required:
|
|
@@ -37298,8 +37359,8 @@ components:
|
|
|
37298
37359
|
for other connection types. For query-type exports (standard, delta, once) it is REQUIRED
|
|
37299
37360
|
when the _connectionId field references an RDBMS database connection and holds the SQL
|
|
37300
37361
|
query. For real-time stream listeners (`type: "stream"`) it holds the CDC watch list
|
|
37301
|
-
(`tables`) instead — and a
|
|
37302
|
-
(`cdc.publicationName`) may omit this object entirely.
|
|
37362
|
+
(`tables`) instead — and a SQL Server or MySQL listener with no watch list, or a PostgreSQL
|
|
37363
|
+
listener scoped by a publication (`cdc.publicationName`), may omit this object entirely.
|
|
37303
37364
|
|
|
37304
37365
|
**Rdbms export capabilities**
|
|
37305
37366
|
- Execute custom SQL SELECT statements
|
|
@@ -37353,8 +37414,8 @@ components:
|
|
|
37353
37414
|
```
|
|
37354
37415
|
|
|
37355
37416
|
**Real-time CDC listener exports (type: "stream")**
|
|
37356
|
-
SQL Server and
|
|
37357
|
-
changes continuously (the `cdc` object holds the listener settings). A stream export has NO
|
|
37417
|
+
SQL Server, PostgreSQL and MySQL connections support change-data-capture listeners that stream
|
|
37418
|
+
row changes continuously (the `cdc` object holds the listener settings). A stream export has NO
|
|
37358
37419
|
`query` — it watches `tables`:
|
|
37359
37420
|
```json
|
|
37360
37421
|
{
|
|
@@ -37363,9 +37424,26 @@ components:
|
|
|
37363
37424
|
"cdc": { "properties": [{ "name": "snapshot.mode", "value": "no_data" }] }
|
|
37364
37425
|
}
|
|
37365
37426
|
```
|
|
37427
|
+
A MySQL listener names its tables as `database.table` (MySQL has no schema layer):
|
|
37428
|
+
```json
|
|
37429
|
+
{
|
|
37430
|
+
"type": "stream",
|
|
37431
|
+
"rdbms": { "tables": "sales.orders,sales.order_lines" },
|
|
37432
|
+
"cdc": {
|
|
37433
|
+
"properties": [{ "name": "snapshot.mode", "value": "no_data" }],
|
|
37434
|
+
"paths": ["payload.after", "payload.op"]
|
|
37435
|
+
}
|
|
37436
|
+
}
|
|
37437
|
+
```
|
|
37366
37438
|
A PostgreSQL listener can instead scope tables through a publication
|
|
37367
37439
|
(`cdc.publicationName`), in which case `tables` stays unset and the `rdbms` object may be
|
|
37368
|
-
omitted entirely.
|
|
37440
|
+
omitted entirely; on PostgreSQL one of the two must be present. SQL Server and MySQL
|
|
37441
|
+
listeners save with no `tables` and no `rdbms` object at all (a blank `tables` string is
|
|
37442
|
+
dropped on save); the MySQL listener documentation describes that as every table the
|
|
37443
|
+
connection's user can read, trimmed by the `table.exclude.list` property.
|
|
37444
|
+
`POST /v1/exports/preview` rejects a stream export both with `rdbms.tables` sent and with
|
|
37445
|
+
no `rdbms` object at all, with the same 422 `missing_required_field` ("rdbms subschema not
|
|
37446
|
+
defined") — listeners have no preview.
|
|
37369
37447
|
properties:
|
|
37370
37448
|
tables:
|
|
37371
37449
|
type: string
|
|
@@ -37373,16 +37451,26 @@ components:
|
|
|
37373
37451
|
Comma-separated list of tables the CDC listener watches for changes. Used only by real-time
|
|
37374
37452
|
listener exports (`type` is `stream`); standard SQL exports use `query` instead. SQL Server
|
|
37375
37453
|
listeners use fully-qualified three-part names (`database.schema.table`); PostgreSQL
|
|
37376
|
-
listeners use two-part names (`schema.table`)
|
|
37454
|
+
listeners use two-part names (`schema.table`); MySQL listeners use two-part
|
|
37455
|
+
`database.table` names, since MySQL has no schema layer between the database and the
|
|
37456
|
+
table. A PostgreSQL listener must carry this field or `cdc.publicationName`; SQL Server
|
|
37457
|
+
and MySQL listeners may leave it out (for MySQL the listener documentation describes that
|
|
37458
|
+
as every table the connection's user can read, trimmed by `table.exclude.list`).
|
|
37377
37459
|
x-celigo-ai-guidance:
|
|
37378
37460
|
- |-
|
|
37379
37461
|
For a PostgreSQL listener, author EITHER this field OR `cdc.publicationName`, never both —
|
|
37380
37462
|
when a publication is set it is the sole table filter and the listener form clears this
|
|
37381
37463
|
field (the API stores both if sent, so nothing catches the conflict at save time).
|
|
37464
|
+
- |-
|
|
37465
|
+
The API does not validate the names against the engine — a three-part name on a MySQL
|
|
37466
|
+
listener saves cleanly — so nothing at save time catches the wrong form. Use the engine's
|
|
37467
|
+
own: `database.table` on MySQL, `schema.table` on PostgreSQL, `database.schema.table` on
|
|
37468
|
+
SQL Server.
|
|
37382
37469
|
examples:
|
|
37383
37470
|
- UKGODS.dbo.UKG_MetadataStaging
|
|
37384
37471
|
- dbo.WorkOrderT,dbo.OrderLineT
|
|
37385
37472
|
- public.orders,public.customers
|
|
37473
|
+
- sales.orders,sales.order_lines
|
|
37386
37474
|
query:
|
|
37387
37475
|
type: string
|
|
37388
37476
|
x-celigo-agent:
|
|
@@ -38214,8 +38302,8 @@ components:
|
|
|
38214
38302
|
- **Listen for inbound events from NetSuite or Salesforce**
|
|
38215
38303
|
-> `distributed`
|
|
38216
38304
|
- **Stream database changes in real time as they happen**
|
|
38217
|
-
(change data capture from MongoDB, SQL Server,
|
|
38218
|
-
|
|
38305
|
+
(change data capture from MongoDB, SQL Server, PostgreSQL
|
|
38306
|
+
or MySQL — "when a row changes", "listen to my table")
|
|
38219
38307
|
-> `stream`, configured via the `cdc` object
|
|
38220
38308
|
- **Subscribe to an Event Streams topic** ("listen to topic X",
|
|
38221
38309
|
"consume messages published to …") -> `stream` with
|
|
@@ -38239,8 +38327,8 @@ components:
|
|
|
38239
38327
|
delta config is a separate concern that scopes how you fill in
|
|
38240
38328
|
the chosen config, not whether to use it.
|
|
38241
38329
|
- |-
|
|
38242
|
-
`stream` is accepted only on MongoDB and RDBMS exports (SQL Server
|
|
38243
|
-
or
|
|
38330
|
+
`stream` is accepted only on MongoDB and RDBMS exports (SQL Server,
|
|
38331
|
+
PostgreSQL or MySQL connections) and on Event Streams listeners
|
|
38244
38332
|
(`adaptorType: TopicExport`); every other adaptor rejects the save
|
|
38245
38333
|
with `unsupported_export_type`. A MongoDB or RDBMS stream export
|
|
38246
38334
|
must carry `cdc.snapshotMode` (and `cdc.captureMode` on MongoDB) —
|
|
@@ -38265,7 +38353,7 @@ components:
|
|
|
38265
38353
|
simple: Allows manual file uploads through the data loader UI.
|
|
38266
38354
|
blob: Retrieves raw files without parsing them into structured data records.
|
|
38267
38355
|
distributed: Listens for real-time events from NetSuite or Salesforce.
|
|
38268
|
-
stream: Streams events in real time — database change data capture (MongoDB change streams, SQL Server CDC, PostgreSQL logical replication) configured through the `cdc` object, or an Event Streams topic subscription (`TopicExport`) configured through `topic`.
|
|
38356
|
+
stream: Streams events in real time — database change data capture (MongoDB change streams, SQL Server CDC, PostgreSQL logical replication, MySQL binary log) configured through the `cdc` object, or an Event Streams topic subscription (`TopicExport`) configured through `topic`.
|
|
38269
38357
|
all: Explicitly retrieves all available records — equivalent to omitting `type`.
|
|
38270
38358
|
examples:
|
|
38271
38359
|
- webhook
|
|
@@ -38392,6 +38480,21 @@ components:
|
|
|
38392
38480
|
$ref: '#/components/schemas/Distributed'
|
|
38393
38481
|
cdc:
|
|
38394
38482
|
$ref: '#/components/schemas/Cdc'
|
|
38483
|
+
pendingCursorReset:
|
|
38484
|
+
type: boolean
|
|
38485
|
+
writeOnly: true
|
|
38486
|
+
description: |-
|
|
38487
|
+
When true on a save of a stream listener, requests a reset of the listener's stored stream
|
|
38488
|
+
position — the Cursor Management reset the listener form sends, with `cdc.cursor` omitted
|
|
38489
|
+
from the same document; the listener documentation describes the save as clearing the
|
|
38490
|
+
stored cursor and restarting the listener. Accepted on the request and not stored on the
|
|
38491
|
+
export.
|
|
38492
|
+
x-celigo-ai-guidance:
|
|
38493
|
+
- |-
|
|
38494
|
+
Never author this flag — like `cdc.cursor`, the stream position is the platform's, and
|
|
38495
|
+
the reset is the listener form's Cursor Management action. The API accepts it silently
|
|
38496
|
+
(200) and the readback carries no trace that it was sent, so a save that carried it by
|
|
38497
|
+
mistake cannot be seen afterwards on the export.
|
|
38395
38498
|
topic:
|
|
38396
38499
|
$ref: '#/components/schemas/Topic-2'
|
|
38397
38500
|
filesystem:
|
package/package.json
CHANGED
package/schemas.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"provenance": {
|
|
3
3
|
"repo": "celigo/integrator-api-specs",
|
|
4
|
-
"iasCommit": "
|
|
4
|
+
"iasCommit": "a2635897",
|
|
5
5
|
"generatedAt": "2026-10-27"
|
|
6
6
|
},
|
|
7
7
|
"roots": {
|
|
@@ -5761,6 +5761,7 @@
|
|
|
5761
5761
|
"pageSize": 352,
|
|
5762
5762
|
"parsers": 455,
|
|
5763
5763
|
"pathToMany": 0,
|
|
5764
|
+
"pendingCursorReset": 8,
|
|
5764
5765
|
"preSave": 104,
|
|
5765
5766
|
"rawData": 0,
|
|
5766
5767
|
"rdbms": 451,
|