@celigo/api-specs 0.2.123 → 0.2.124

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/dist/api.yml CHANGED
@@ -7857,14 +7857,15 @@ components:
7857
7857
  - awsIam
7858
7858
  default: accesskey
7859
7859
  x-enumDescriptions:
7860
- accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey`.
7861
- awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection.
7860
+ accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey` — the connection form's "Username & Password" option (its fields are Access key ID, Secret access key and the database Username).
7861
+ awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection — the form's "AWS IAM role" option.
7862
7862
  description: |-
7863
7863
  Selects the Redshift authentication mode; omitting it behaves as
7864
7864
  `accesskey`. With `awsIam`, the platform assumes the iClient's
7865
7865
  role and requires exactly one of `clusterIdentifier` (provisioned
7866
7866
  cluster) or `workgroupName` (Redshift Serverless) — supplying
7867
- both or neither fails with 422.
7867
+ both or neither fails with 422. A bulk-load import saves against
7868
+ either mode, cluster or Serverless workgroup alike.
7868
7869
  aws:
7869
7870
  type: object
7870
7871
  description: |-
@@ -6968,14 +6968,15 @@ components:
6968
6968
  - awsIam
6969
6969
  default: accesskey
6970
6970
  x-enumDescriptions:
6971
- accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey`.
6972
- awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection.
6971
+ accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey` — the connection form's "Username & Password" option (its fields are Access key ID, Secret access key and the database Username).
6972
+ awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection — the form's "AWS IAM role" option.
6973
6973
  description: |-
6974
6974
  Selects the Redshift authentication mode; omitting it behaves as
6975
6975
  `accesskey`. With `awsIam`, the platform assumes the iClient's
6976
6976
  role and requires exactly one of `clusterIdentifier` (provisioned
6977
6977
  cluster) or `workgroupName` (Redshift Serverless) — supplying
6978
- both or neither fails with 422.
6978
+ both or neither fails with 422. A bulk-load import saves against
6979
+ either mode, cluster or Serverless workgroup alike.
6979
6980
  aws:
6980
6981
  type: object
6981
6982
  description: |-
package/dist/enduser.yml CHANGED
@@ -5069,14 +5069,15 @@ components:
5069
5069
  - awsIam
5070
5070
  default: accesskey
5071
5071
  x-enumDescriptions:
5072
- accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey`.
5073
- awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection.
5072
+ accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey` — the connection form's "Username & Password" option (its fields are Access key ID, Secret access key and the database Username).
5073
+ awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection — the form's "AWS IAM role" option.
5074
5074
  description: |-
5075
5075
  Selects the Redshift authentication mode; omitting it behaves as
5076
5076
  `accesskey`. With `awsIam`, the platform assumes the iClient's
5077
5077
  role and requires exactly one of `clusterIdentifier` (provisioned
5078
5078
  cluster) or `workgroupName` (Redshift Serverless) — supplying
5079
- both or neither fails with 422.
5079
+ both or neither fails with 422. A bulk-load import saves against
5080
+ either mode, cluster or Serverless workgroup alike.
5080
5081
  aws:
5081
5082
  type: object
5082
5083
  description: |-
package/dist/import.yml CHANGED
@@ -5221,15 +5221,15 @@ components:
5221
5221
  per_page: Execute one SQL statement per page (batch) of records.
5222
5222
  first_page: Execute the SQL statement once for the first page only, typically for setup or cleanup scripts.
5223
5223
  bulk_load: Stage data as a file and load via database-native COPY/bulk mechanism for maximum throughput.
5224
- x-celigo-ui-override: Required by the RDBMS import form (rdbms.queryType is required:true). Form default is bulk_insert, overridden to bulk_load for Snowflake and BigQuery. Encoded to mirror the form so builders produce connectable configurations.
5224
+ x-celigo-ui-override: Required by the RDBMS import form (rdbms.queryType is required:true). The form defaults a new import to bulk_load wherever it offers that option — Snowflake always; Google BigQuery, Microsoft SQL Server (cloud connections only), Azure Synapse and Amazon Redshift when the account's enableSizeBasedAggregation feature flag is on for that database — and to bulk_insert otherwise. Encoded to mirror the form so builders produce connectable configurations.
5225
5225
  description: |-
5226
5226
  Execution strategy for the SQL operation, which determines the required companion field: per_record, per_page, and first_page require `query`; bulk_insert requires `bulkInsert`; bulk_load requires `bulkLoad`.
5227
- Prefer the highest-performance type the operation supports: bulk_load (currently Snowflake and NSAW — handles INSERT, upsert via `bulkLoad.primaryKeys`, and custom merge or ignore-existing logic via `bulkLoad.overrideMergeQuery`), then bulk_insert for pure INSERTs, then per_page, then per_record only when the logic cannot be expressed as a batch operation.
5227
+ Prefer the highest-performance type the operation supports: bulk_load (Snowflake, Google BigQuery, Microsoft SQL Server on a cloud connection, Azure Synapse and Amazon Redshift — handles INSERT, upsert via `bulkLoad.primaryKeys`, and custom merge or ignore-existing logic via `bulkLoad.overrideMergeQuery`), then bulk_insert for pure INSERTs, then per_page, then per_record only when the logic cannot be expressed as a batch operation.
5228
5228
  For UPDATE, UPSERT, or ignore-existing logic on databases without bulk_load support, use per_page or per_record with `query` — bulk_insert has no duplicate-checking logic.
5229
5229
  Set a single value; do not combine types or use the legacy INSERT and UPDATE values.
5230
5230
  per_page statements render against the whole batch (`batch_of_records`) rather than a single record — loop with `{{#each batch_of_records}}...{{{record.fieldName}}}...{{/each}}`.
5231
5231
 
5232
- `bulk_load` — Stages data as a file and loads via database-native COPY/bulk mechanism for maximum throughput; supports INSERT, upsert via bulkLoad.primaryKeys, and custom merge or ignore-existing logic via bulkLoad.overrideMergeQuery; currently supported for Snowflake and NSAW.
5232
+ `bulk_load` — Stages the records as files and loads them with the database's native COPY / bulk mechanism for maximum throughput; supports INSERT, upsert via bulkLoad.primaryKeys, and custom merge or ignore-existing logic via bulkLoad.overrideMergeQuery; available on Snowflake, Google BigQuery, Microsoft SQL Server (cloud connections), Azure Synapse and Amazon Redshift (NetSuite Analytics Warehouse has it through the JDBC import). The import form calls it "Use optimized bulk load (recommended for larger imports)" and, beside it, calls bulk_insert "Use batch insert SQL query".
5233
5233
  `bulk_insert` — Batch INSERT via multi-row VALUES clause for efficient bulk data loading; has no upsert or ignore-existing logic and is available for all RDBMS types.
5234
5234
  `per_page` — Executes one SQL statement per page (batch) of records; available for all RDBMS types.
5235
5235
  `per_record` — Executes one SQL statement per record, giving full control over individual record SQL; available for all RDBMS types.
@@ -5238,13 +5238,18 @@ components:
5238
5238
  - |-
5239
5239
  ## DECISION TREE (Follow in order):
5240
5240
 
5241
+ **STEP 0: Is this an update of an existing import?** Keep its query type unless the user asks to change it — an import set to bulk_insert stays bulk_insert when the request is about something else (the query type is additive and opt-in; nothing migrates). The steps below are for a NEW import or an explicit request to change the type.
5242
+
5241
5243
  **STEP 1: Does the database support bulk_load?**
5242
- - If Snowflake, NSAW, or another database with bulk_load support:
5244
+ - Snowflake, Google BigQuery, Microsoft SQL Server (a cloud connection — not an on-premise agent connection), Azure Synapse and Amazon Redshift do (NetSuite Analytics Warehouse does through the JDBC import):
5243
5245
  → **USE `["bulk_load"]`** with:
5244
- - `bulkLoad.tableName` (required)
5246
+ - `bulkLoad.tableName` (required — omitting it fails the save with 422 `missing_required_field` "rdbms imports require a tableName.")
5245
5247
  - `bulkLoad.primaryKeys` for upsert/merge (auto-generates MERGE)
5246
- - `bulkLoad.overrideMergeQuery: true` for custom SQL (ignore-existing, conditional updates, multi-table ops). Override SQL references `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}` for the staging table.
5248
+ - `bulkLoad.overrideMergeQuery: true` for custom SQL (ignore-existing, conditional updates, multi-table ops). The override SQL goes in `query` and references `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) for the staging table and `{{tableName}}` for the destination.
5247
5249
  - No `primaryKeys` for pure INSERT (auto-generates INSERT)
5250
+ - No `bulkInsert` beside it (the server drops a `bulkInsert` sent with bulk_load)
5251
+ - **Amazon Redshift needs no AWS setup for bulk_load**: Celigo stages the files in Celigo-managed encrypted storage and Redshift loads them with COPY — no S3 bucket, IAM role, cluster role attachment or extra connection field. The save is accepted on either `redshift.authType` (accesskey and awsIam) and against a provisioned cluster or a Serverless workgroup alike. Never steer a Redshift import to bulk_insert for lack of setup.
5252
+ - The import form shows the bulk_load option only when the account's `enableSizeBasedAggregation` feature flag is on for that database (BigQuery, SQL Server, Synapse, Redshift; Snowflake always shows it). The flag is the form's switch, not the API's — a Redshift bulk_load import saves on an account whose flag has no redshift entry (off, to the form) — so author it, and tell the user the option appears on their import form once the flag is on for the account.
5248
5253
 
5249
5254
  **STEP 2: Database does NOT support bulk_load**
5250
5255
  - **Pure INSERT**: → **USE `["bulk_insert"]`** (requires `bulkInsert` object)
@@ -5501,47 +5506,38 @@ components:
5501
5506
  overrideMergeQuery:
5502
5507
  type: boolean
5503
5508
  description: |-
5504
- When true, a custom merge query replaces the auto-generated MERGE statement, enabling ignore-existing logic, conditional updates, or multi-table operations.
5505
- The override SQL references `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}` for the staging table.
5509
+ When true, a custom merge query replaces the auto-generated MERGE statement, enabling ignore-existing logic, conditional updates, or multi-table operations ("Override merge/insert statement" on the import form).
5510
+ The override SQL lives in `query` and references `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) for the staging table and `{{tableName}}` for the destination table.
5511
+ The SQL itself in this field fails the save: 422 `invalid_field` "Invalid value '<the statement>' for field 'rdbms.bulkLoad.overrideMergeQuery'. Reason: CastError."
5506
5512
  x-celigo-ai-guidance:
5507
- - |-
5508
- A custom SQL query string that fully overrides the default merge operation
5509
- executed during bulk load processes in a relational database management system
5510
- (RDBMS).
5511
- This property empowers users to define precise,
5512
- tailored merge logic that governs how records are inserted, updated,
5513
- or deleted in the target database table when handling large-scale data operations.
5514
- By specifying this query, users can implement complex matching conditions,
5515
- custom conflict resolution strategies, additional filtering criteria,
5516
- or conditional logic that surpasses the system-generated default behavior,
5517
- ensuring the bulk load aligns perfectly with specific business rules,
5518
- data integrity requirements, or performance optimizations.
5519
- - |-
5520
- ## FIELD BEHAVIOR:
5521
- - When provided, this query completely replaces the standard merge statement used in bulk load operations.
5522
- - Supports detailed customization of merge logic, including custom join conditions, conditional updates, selective inserts, and optional deletes.
5523
- - If omitted, the system automatically generates and executes a default merge query based on the schema, keys, and data mappings.
5524
- - The query must be syntactically compatible with the target RDBMS and support any required parameterization for dynamic data injection.
5525
- - Executed within the transactional context of the bulk load to maintain atomicity, consistency, and rollback capabilities in case of failure.
5526
- - The system expects the query to handle all necessary merge scenarios to avoid partial or inconsistent data states.
5527
- - Overrides apply globally for the bulk load operation, affecting all records processed in that batch.
5528
- - |-
5529
- ## IMPLEMENTATION GUIDANCE:
5530
- - Validate the custom SQL syntax thoroughly before execution to prevent runtime errors and ensure compatibility with the target RDBMS.
5531
- - Ensure the query comprehensively addresses all required merge operations—insert, update, and optionally delete—to maintain data integrity.
5532
- - Support parameter placeholders or bind variables if the system injects dynamic values during execution, and document their usage clearly.
5533
- - Provide users with clear documentation, templates, or examples outlining the expected query structure, required clauses, and best practices.
5534
- - Implement robust safeguards against SQL injection and other security vulnerabilities when accepting and executing custom queries.
5535
- - Test the custom query extensively in a controlled staging environment to verify correctness, performance, and side effects before deploying to production.
5536
- - Consider transaction isolation levels and locking behavior to avoid deadlocks or contention during bulk load operations.
5537
- - Encourage users to include comprehensive error handling and logging within the query or surrounding execution context to facilitate troubleshooting.
5538
- - |-
5539
- ## EXAMPLES:
5540
- - A MERGE statement that updates existing records based on a composite primary key and inserts new records when no match is found.
5541
- - A merge query incorporating additional WHERE clauses to exclude certain
5513
+ - This is the switch, not the SQL. Set it to true and put the statement in `query` (one string); the import form shows the SQL box only while "Override merge/insert statement" is checked. With it false or absent the platform generates the INSERT / MERGE from `bulkLoad.tableName` and `bulkLoad.primaryKeys`, and `query` stays empty.
5514
+ - 'The statement runs after the files are loaded into the staging table: read from `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) and write to `{{tableName}}`; the form warns when the staging-table placeholder is missing. Use it for ignore-existing, conditional updates or multi-table work the generated MERGE cannot express — for example `INSERT INTO {{tableName}} (id, total) SELECT id, total FROM {{preMergeTemporaryTable}}`.'
5515
+ aggregation:
5516
+ type: object
5517
+ description: How the staged files are cut and handed to the database. The import form writes it for a new bulk-load import when the account's `enableSizeBasedAggregation` flag is on for that database (Snowflake included — its form offers bulk_load without the flag but writes `aggregation` only with it); the server fills `type` and `maxFileSizeInMB` when they are omitted, so `loadDataAsAvailable` is the only value to author.
5518
+ properties:
5519
+ type:
5520
+ type: string
5521
+ enum:
5522
+ - size
5523
+ x-enumDescriptions:
5524
+ size: Files are cut by size
5525
+ description: Filled by the server on save; the only mode is size-based cutting.
5526
+ maxFileSizeInMB:
5527
+ type: integer
5528
+ description: Size of each staged file in MB; the server fills 250 when the field is omitted and the import form never sets it — leave it out.
5529
+ examples:
5530
+ - 250
5531
+ loadDataAsAvailable:
5532
+ type: boolean
5533
+ default: false
5534
+ description: When true, each staged file is sent for loading as soon as it is written, so the dataset arrives as several smaller loads ("Load data as available" on the import form). When false, every file is held until all are ready and they are loaded together.
5535
+ x-celigo-ai-guidance:
5536
+ - 'Send a real boolean and default to false unless the user asks for files to load as they are ready: a non-boolean value (for example the string "yes") makes the server drop the whole `aggregation` block without an error.'
5537
+ - The block outlives a query-type change — after a PUT that switches the import to bulk_insert and omits `bulkLoad`, the read-back still carries `bulkLoad.aggregation`. That is server behavior, not a failed write; do not re-send the document to clear it.
5542
5538
  description: |-
5543
- Configuration for bulk loading: data is staged as a file and loaded via the database's native COPY/bulk mechanism for maximum throughput.
5544
- Required when queryType is `["bulk_load"]`; currently supported for Snowflake and NSAW.
5539
+ Configuration for bulk loading: the records are staged as files and loaded with the database's native COPY / bulk mechanism for maximum throughput.
5540
+ Required when queryType is `["bulk_load"]`; available on Snowflake, Google BigQuery, Microsoft SQL Server (cloud connections), Azure Synapse and Amazon Redshift.
5545
5541
  x-celigo-ai-guidance:
5546
5542
  - |-
5547
5543
  Specifies whether bulk loading is enabled for relational database management
@@ -5716,9 +5712,25 @@ components:
5716
5712
  then:
5717
5713
  required:
5718
5714
  - bulkLoad
5719
- properties:
5720
- query:
5721
- maxItems: 0
5715
+ if:
5716
+ required:
5717
+ - bulkLoad
5718
+ properties:
5719
+ bulkLoad:
5720
+ type: object
5721
+ required:
5722
+ - overrideMergeQuery
5723
+ properties:
5724
+ overrideMergeQuery:
5725
+ const: true
5726
+ then:
5727
+ properties:
5728
+ query:
5729
+ type: array
5730
+ else:
5731
+ properties:
5732
+ query:
5733
+ maxItems: 0
5722
5734
  S3:
5723
5735
  type: object
5724
5736
  description: |-
@@ -6894,14 +6894,15 @@ components:
6894
6894
  - awsIam
6895
6895
  default: accesskey
6896
6896
  x-enumDescriptions:
6897
- accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey`.
6898
- awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection.
6897
+ accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey` — the connection form's "Username & Password" option (its fields are Access key ID, Secret access key and the database Username).
6898
+ awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection — the form's "AWS IAM role" option.
6899
6899
  description: |-
6900
6900
  Selects the Redshift authentication mode; omitting it behaves as
6901
6901
  `accesskey`. With `awsIam`, the platform assumes the iClient's
6902
6902
  role and requires exactly one of `clusterIdentifier` (provisioned
6903
6903
  cluster) or `workgroupName` (Redshift Serverless) — supplying
6904
- both or neither fails with 422.
6904
+ both or neither fails with 422. A bulk-load import saves against
6905
+ either mode, cluster or Serverless workgroup alike.
6905
6906
  aws:
6906
6907
  type: object
6907
6908
  description: |-
package/dist/tool.yml CHANGED
@@ -10223,14 +10223,15 @@ components:
10223
10223
  - awsIam
10224
10224
  default: accesskey
10225
10225
  x-enumDescriptions:
10226
- accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey`.
10227
- awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection.
10226
+ accesskey: Authenticate with static AWS access keys in `aws.accessKeyId` / `aws.secretAccessKey` — the connection form's "Username & Password" option (its fields are Access key ID, Secret access key and the database Username).
10227
+ awsIam: Authenticate by assuming the AWS IAM role held in the iClient referenced by `rdbms._iClientId`; no static keys stored on the connection — the form's "AWS IAM role" option.
10228
10228
  description: |-
10229
10229
  Selects the Redshift authentication mode; omitting it behaves as
10230
10230
  `accesskey`. With `awsIam`, the platform assumes the iClient's
10231
10231
  role and requires exactly one of `clusterIdentifier` (provisioned
10232
10232
  cluster) or `workgroupName` (Redshift Serverless) — supplying
10233
- both or neither fails with 422.
10233
+ both or neither fails with 422. A bulk-load import saves against
10234
+ either mode, cluster or Serverless workgroup alike.
10234
10235
  aws:
10235
10236
  type: object
10236
10237
  description: |-
@@ -16532,15 +16533,15 @@ components:
16532
16533
  per_page: Execute one SQL statement per page (batch) of records.
16533
16534
  first_page: Execute the SQL statement once for the first page only, typically for setup or cleanup scripts.
16534
16535
  bulk_load: Stage data as a file and load via database-native COPY/bulk mechanism for maximum throughput.
16535
- x-celigo-ui-override: Required by the RDBMS import form (rdbms.queryType is required:true). Form default is bulk_insert, overridden to bulk_load for Snowflake and BigQuery. Encoded to mirror the form so builders produce connectable configurations.
16536
+ x-celigo-ui-override: Required by the RDBMS import form (rdbms.queryType is required:true). The form defaults a new import to bulk_load wherever it offers that option — Snowflake always; Google BigQuery, Microsoft SQL Server (cloud connections only), Azure Synapse and Amazon Redshift when the account's enableSizeBasedAggregation feature flag is on for that database — and to bulk_insert otherwise. Encoded to mirror the form so builders produce connectable configurations.
16536
16537
  description: |-
16537
16538
  Execution strategy for the SQL operation, which determines the required companion field: per_record, per_page, and first_page require `query`; bulk_insert requires `bulkInsert`; bulk_load requires `bulkLoad`.
16538
- Prefer the highest-performance type the operation supports: bulk_load (currently Snowflake and NSAW — handles INSERT, upsert via `bulkLoad.primaryKeys`, and custom merge or ignore-existing logic via `bulkLoad.overrideMergeQuery`), then bulk_insert for pure INSERTs, then per_page, then per_record only when the logic cannot be expressed as a batch operation.
16539
+ Prefer the highest-performance type the operation supports: bulk_load (Snowflake, Google BigQuery, Microsoft SQL Server on a cloud connection, Azure Synapse and Amazon Redshift — handles INSERT, upsert via `bulkLoad.primaryKeys`, and custom merge or ignore-existing logic via `bulkLoad.overrideMergeQuery`), then bulk_insert for pure INSERTs, then per_page, then per_record only when the logic cannot be expressed as a batch operation.
16539
16540
  For UPDATE, UPSERT, or ignore-existing logic on databases without bulk_load support, use per_page or per_record with `query` — bulk_insert has no duplicate-checking logic.
16540
16541
  Set a single value; do not combine types or use the legacy INSERT and UPDATE values.
16541
16542
  per_page statements render against the whole batch (`batch_of_records`) rather than a single record — loop with `{{#each batch_of_records}}...{{{record.fieldName}}}...{{/each}}`.
16542
16543
 
16543
- `bulk_load` — Stages data as a file and loads via database-native COPY/bulk mechanism for maximum throughput; supports INSERT, upsert via bulkLoad.primaryKeys, and custom merge or ignore-existing logic via bulkLoad.overrideMergeQuery; currently supported for Snowflake and NSAW.
16544
+ `bulk_load` — Stages the records as files and loads them with the database's native COPY / bulk mechanism for maximum throughput; supports INSERT, upsert via bulkLoad.primaryKeys, and custom merge or ignore-existing logic via bulkLoad.overrideMergeQuery; available on Snowflake, Google BigQuery, Microsoft SQL Server (cloud connections), Azure Synapse and Amazon Redshift (NetSuite Analytics Warehouse has it through the JDBC import). The import form calls it "Use optimized bulk load (recommended for larger imports)" and, beside it, calls bulk_insert "Use batch insert SQL query".
16544
16545
  `bulk_insert` — Batch INSERT via multi-row VALUES clause for efficient bulk data loading; has no upsert or ignore-existing logic and is available for all RDBMS types.
16545
16546
  `per_page` — Executes one SQL statement per page (batch) of records; available for all RDBMS types.
16546
16547
  `per_record` — Executes one SQL statement per record, giving full control over individual record SQL; available for all RDBMS types.
@@ -16549,13 +16550,18 @@ components:
16549
16550
  - |-
16550
16551
  ## DECISION TREE (Follow in order):
16551
16552
 
16553
+ **STEP 0: Is this an update of an existing import?** Keep its query type unless the user asks to change it — an import set to bulk_insert stays bulk_insert when the request is about something else (the query type is additive and opt-in; nothing migrates). The steps below are for a NEW import or an explicit request to change the type.
16554
+
16552
16555
  **STEP 1: Does the database support bulk_load?**
16553
- - If Snowflake, NSAW, or another database with bulk_load support:
16556
+ - Snowflake, Google BigQuery, Microsoft SQL Server (a cloud connection — not an on-premise agent connection), Azure Synapse and Amazon Redshift do (NetSuite Analytics Warehouse does through the JDBC import):
16554
16557
  → **USE `["bulk_load"]`** with:
16555
- - `bulkLoad.tableName` (required)
16558
+ - `bulkLoad.tableName` (required — omitting it fails the save with 422 `missing_required_field` "rdbms imports require a tableName.")
16556
16559
  - `bulkLoad.primaryKeys` for upsert/merge (auto-generates MERGE)
16557
- - `bulkLoad.overrideMergeQuery: true` for custom SQL (ignore-existing, conditional updates, multi-table ops). Override SQL references `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}` for the staging table.
16560
+ - `bulkLoad.overrideMergeQuery: true` for custom SQL (ignore-existing, conditional updates, multi-table ops). The override SQL goes in `query` and references `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) for the staging table and `{{tableName}}` for the destination.
16558
16561
  - No `primaryKeys` for pure INSERT (auto-generates INSERT)
16562
+ - No `bulkInsert` beside it (the server drops a `bulkInsert` sent with bulk_load)
16563
+ - **Amazon Redshift needs no AWS setup for bulk_load**: Celigo stages the files in Celigo-managed encrypted storage and Redshift loads them with COPY — no S3 bucket, IAM role, cluster role attachment or extra connection field. The save is accepted on either `redshift.authType` (accesskey and awsIam) and against a provisioned cluster or a Serverless workgroup alike. Never steer a Redshift import to bulk_insert for lack of setup.
16564
+ - The import form shows the bulk_load option only when the account's `enableSizeBasedAggregation` feature flag is on for that database (BigQuery, SQL Server, Synapse, Redshift; Snowflake always shows it). The flag is the form's switch, not the API's — a Redshift bulk_load import saves on an account whose flag has no redshift entry (off, to the form) — so author it, and tell the user the option appears on their import form once the flag is on for the account.
16559
16565
 
16560
16566
  **STEP 2: Database does NOT support bulk_load**
16561
16567
  - **Pure INSERT**: → **USE `["bulk_insert"]`** (requires `bulkInsert` object)
@@ -16812,47 +16818,38 @@ components:
16812
16818
  overrideMergeQuery:
16813
16819
  type: boolean
16814
16820
  description: |-
16815
- When true, a custom merge query replaces the auto-generated MERGE statement, enabling ignore-existing logic, conditional updates, or multi-table operations.
16816
- The override SQL references `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}` for the staging table.
16821
+ When true, a custom merge query replaces the auto-generated MERGE statement, enabling ignore-existing logic, conditional updates, or multi-table operations ("Override merge/insert statement" on the import form).
16822
+ The override SQL lives in `query` and references `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) for the staging table and `{{tableName}}` for the destination table.
16823
+ The SQL itself in this field fails the save: 422 `invalid_field` "Invalid value '<the statement>' for field 'rdbms.bulkLoad.overrideMergeQuery'. Reason: CastError."
16817
16824
  x-celigo-ai-guidance:
16818
- - |-
16819
- A custom SQL query string that fully overrides the default merge operation
16820
- executed during bulk load processes in a relational database management system
16821
- (RDBMS).
16822
- This property empowers users to define precise,
16823
- tailored merge logic that governs how records are inserted, updated,
16824
- or deleted in the target database table when handling large-scale data operations.
16825
- By specifying this query, users can implement complex matching conditions,
16826
- custom conflict resolution strategies, additional filtering criteria,
16827
- or conditional logic that surpasses the system-generated default behavior,
16828
- ensuring the bulk load aligns perfectly with specific business rules,
16829
- data integrity requirements, or performance optimizations.
16830
- - |-
16831
- ## FIELD BEHAVIOR:
16832
- - When provided, this query completely replaces the standard merge statement used in bulk load operations.
16833
- - Supports detailed customization of merge logic, including custom join conditions, conditional updates, selective inserts, and optional deletes.
16834
- - If omitted, the system automatically generates and executes a default merge query based on the schema, keys, and data mappings.
16835
- - The query must be syntactically compatible with the target RDBMS and support any required parameterization for dynamic data injection.
16836
- - Executed within the transactional context of the bulk load to maintain atomicity, consistency, and rollback capabilities in case of failure.
16837
- - The system expects the query to handle all necessary merge scenarios to avoid partial or inconsistent data states.
16838
- - Overrides apply globally for the bulk load operation, affecting all records processed in that batch.
16839
- - |-
16840
- ## IMPLEMENTATION GUIDANCE:
16841
- - Validate the custom SQL syntax thoroughly before execution to prevent runtime errors and ensure compatibility with the target RDBMS.
16842
- - Ensure the query comprehensively addresses all required merge operations—insert, update, and optionally delete—to maintain data integrity.
16843
- - Support parameter placeholders or bind variables if the system injects dynamic values during execution, and document their usage clearly.
16844
- - Provide users with clear documentation, templates, or examples outlining the expected query structure, required clauses, and best practices.
16845
- - Implement robust safeguards against SQL injection and other security vulnerabilities when accepting and executing custom queries.
16846
- - Test the custom query extensively in a controlled staging environment to verify correctness, performance, and side effects before deploying to production.
16847
- - Consider transaction isolation levels and locking behavior to avoid deadlocks or contention during bulk load operations.
16848
- - Encourage users to include comprehensive error handling and logging within the query or surrounding execution context to facilitate troubleshooting.
16849
- - |-
16850
- ## EXAMPLES:
16851
- - A MERGE statement that updates existing records based on a composite primary key and inserts new records when no match is found.
16852
- - A merge query incorporating additional WHERE clauses to exclude certain
16825
+ - This is the switch, not the SQL. Set it to true and put the statement in `query` (one string); the import form shows the SQL box only while "Override merge/insert statement" is checked. With it false or absent the platform generates the INSERT / MERGE from `bulkLoad.tableName` and `bulkLoad.primaryKeys`, and `query` stays empty.
16826
+ - 'The statement runs after the files are loaded into the staging table: read from `{{preMergeTemporaryTable}}` (also written `{{import.rdbms.bulkLoad.preMergeTemporaryTable}}`) and write to `{{tableName}}`; the form warns when the staging-table placeholder is missing. Use it for ignore-existing, conditional updates or multi-table work the generated MERGE cannot express — for example `INSERT INTO {{tableName}} (id, total) SELECT id, total FROM {{preMergeTemporaryTable}}`.'
16827
+ aggregation:
16828
+ type: object
16829
+ description: How the staged files are cut and handed to the database. The import form writes it for a new bulk-load import when the account's `enableSizeBasedAggregation` flag is on for that database (Snowflake included — its form offers bulk_load without the flag but writes `aggregation` only with it); the server fills `type` and `maxFileSizeInMB` when they are omitted, so `loadDataAsAvailable` is the only value to author.
16830
+ properties:
16831
+ type:
16832
+ type: string
16833
+ enum:
16834
+ - size
16835
+ x-enumDescriptions:
16836
+ size: Files are cut by size
16837
+ description: Filled by the server on save; the only mode is size-based cutting.
16838
+ maxFileSizeInMB:
16839
+ type: integer
16840
+ description: Size of each staged file in MB; the server fills 250 when the field is omitted and the import form never sets it — leave it out.
16841
+ examples:
16842
+ - 250
16843
+ loadDataAsAvailable:
16844
+ type: boolean
16845
+ default: false
16846
+ description: When true, each staged file is sent for loading as soon as it is written, so the dataset arrives as several smaller loads ("Load data as available" on the import form). When false, every file is held until all are ready and they are loaded together.
16847
+ x-celigo-ai-guidance:
16848
+ - 'Send a real boolean and default to false unless the user asks for files to load as they are ready: a non-boolean value (for example the string "yes") makes the server drop the whole `aggregation` block without an error.'
16849
+ - The block outlives a query-type change — after a PUT that switches the import to bulk_insert and omits `bulkLoad`, the read-back still carries `bulkLoad.aggregation`. That is server behavior, not a failed write; do not re-send the document to clear it.
16853
16850
  description: |-
16854
- Configuration for bulk loading: data is staged as a file and loaded via the database's native COPY/bulk mechanism for maximum throughput.
16855
- Required when queryType is `["bulk_load"]`; currently supported for Snowflake and NSAW.
16851
+ Configuration for bulk loading: the records are staged as files and loaded with the database's native COPY / bulk mechanism for maximum throughput.
16852
+ Required when queryType is `["bulk_load"]`; available on Snowflake, Google BigQuery, Microsoft SQL Server (cloud connections), Azure Synapse and Amazon Redshift.
16856
16853
  x-celigo-ai-guidance:
16857
16854
  - |-
16858
16855
  Specifies whether bulk loading is enabled for relational database management
@@ -17027,9 +17024,25 @@ components:
17027
17024
  then:
17028
17025
  required:
17029
17026
  - bulkLoad
17030
- properties:
17031
- query:
17032
- maxItems: 0
17027
+ if:
17028
+ required:
17029
+ - bulkLoad
17030
+ properties:
17031
+ bulkLoad:
17032
+ type: object
17033
+ required:
17034
+ - overrideMergeQuery
17035
+ properties:
17036
+ overrideMergeQuery:
17037
+ const: true
17038
+ then:
17039
+ properties:
17040
+ query:
17041
+ type: array
17042
+ else:
17043
+ properties:
17044
+ query:
17045
+ maxItems: 0
17033
17046
  S3-2:
17034
17047
  type: object
17035
17048
  description: |-
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@celigo/api-specs",
3
- "version": "0.2.123",
3
+ "version": "0.2.124",
4
4
  "private": false,
5
5
  "description": "Public OpenAPI bundles (dist/) and the celigo-cli lint schema pack (schemas.json).",
6
6
  "keywords": [