@thinkingai/ae-cli 6.0.51 → 6.0.53

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (127) hide show
  1. package/README.md +7 -1
  2. package/README.zh.md +7 -1
  3. package/dist/{auth-XZNXFRJS.js → auth-LHM7NQPR.js} +167 -7
  4. package/dist/{capability-RJZ4KRGP.js → capability-F2RIR5DS.js} +11 -6
  5. package/dist/{chunk-57RRFUEZ.js → chunk-5XRQ3CZR.js} +4 -4
  6. package/dist/{chunk-L2QUIZTS.js → chunk-647BRKZK.js} +3 -3
  7. package/dist/{chunk-EQ255HKO.js → chunk-7WZACOAI.js} +2 -2
  8. package/dist/{chunk-DX5CDO34.js → chunk-BW7JUDEI.js} +84 -19
  9. package/dist/{chunk-7XGFSUOM.js → chunk-DU72X2IO.js} +271 -76
  10. package/dist/{chunk-V2NUHPXF.js → chunk-GK7WDY7E.js} +1 -1
  11. package/dist/{chunk-7FTTULED.js → chunk-IG3DYEIR.js} +4 -1
  12. package/dist/{chunk-HBGADCKA.js → chunk-KJDOTPYU.js} +3 -3
  13. package/dist/{chunk-4NRCTUZJ.js → chunk-KS4DE3MX.js} +4 -4
  14. package/dist/{chunk-JOCDD4ON.js → chunk-N4BLPJC7.js} +1 -1
  15. package/dist/{chunk-WEPZ5M3Z.js → chunk-PGCIKZRE.js} +7 -4
  16. package/dist/{chunk-HFGZGYCG.js → chunk-QYXDJOLA.js} +252 -6
  17. package/dist/{chunk-DVMLWQD4.js → chunk-SO5LKIWP.js} +2 -2
  18. package/dist/{chunk-MAPZ7VB6.js → chunk-YVGXGTD6.js} +13 -0
  19. package/dist/{community-report-client-CU22W53K.js → community-report-client-6VJDDMJJ.js} +1 -1
  20. package/dist/{config-XO3LI3SA.js → config-XQWOPDHN.js} +25 -6
  21. package/dist/{context-IFJI3LIT.js → context-M3HDGEIR.js} +4 -4
  22. package/dist/{data-integration-HNGUKFIC.js → data-integration-XBTCGXQT.js} +6 -6
  23. package/dist/index.js +77 -40
  24. package/dist/{local-data-upload-client-GTRIT64F.js → local-data-upload-client-3P7NIVQ4.js} +1 -1
  25. package/dist/{memory-QGPSNCQS.js → memory-6APEB2JB.js} +4 -4
  26. package/dist/{metadata-JAQW7KGZ.js → metadata-V5ZDTKWZ.js} +7 -7
  27. package/dist/{model-RNPQXIBI.js → model-ZS7HVGKN.js} +4 -4
  28. package/dist/{personal-semantic-preference-QCZ2IZTK.js → personal-semantic-preference-2XANU3F6.js} +8 -6
  29. package/dist/{project-semantic-64JURXKF.js → project-semantic-LP3LY7QP.js} +7 -7
  30. package/dist/{sync-O45NXEQY.js → sync-EQLSTUS7.js} +7 -7
  31. package/dist/{te-agent-K5MSPNHJ.js → te-agent-Z3HCTKR7.js} +348 -55
  32. package/dist/{te-analysis-KQKCAI5H.js → te-analysis-3EVYQ3IO.js} +803 -57
  33. package/dist/{te-community-4OYU3BDA.js → te-community-Z7JACGF3.js} +3 -3
  34. package/dist/{te-dataops-NFIPB6FX.js → te-dataops-F7GQOIS6.js} +644 -276
  35. package/dist/{te-engage-HSVSRW22.js → te-engage-LFBU6WWH.js} +6 -6
  36. package/dist/{te-kb-3TS73NO6.js → te-kb-QJLJEX5K.js} +323 -181
  37. package/dist/{te-system-STMHLS5J.js → te-system-IXC43QHZ.js} +6 -6
  38. package/dist/{te-team-3I5AOX3N.js → te-team-BIZIKIHZ.js} +6 -6
  39. package/dist/{update-RZXLPCRH.js → update-W2SGQNKT.js} +4 -4
  40. package/package.json +11 -4
  41. package/skills/ae-agent/SKILL.md +9 -4
  42. package/skills/ae-agent/references/notification.md +40 -0
  43. package/skills/ae-analysis/SKILL.md +65 -13
  44. package/skills/ae-analysis/references/agent_review_submit_to_page.md +3 -1
  45. package/skills/ae-analysis/references/ai_models/event.md +6 -1
  46. package/skills/ae-analysis/references/ai_models/funnel.md +7 -1
  47. package/skills/ae-analysis/references/ai_models/heat_map.md +23 -0
  48. package/skills/ae-analysis/references/ai_models/interval.md +24 -0
  49. package/skills/ae-analysis/references/ai_models/path.md +20 -0
  50. package/skills/ae-analysis/references/ai_models/prop_analysis.md +2 -0
  51. package/skills/ae-analysis/references/ai_models/rank_list.md +30 -0
  52. package/skills/ae-analysis/references/ai_models/retention.md +45 -1
  53. package/skills/ae-analysis/references/ai_models/revenue.md +20 -2
  54. package/skills/ae-analysis/references/ai_models/tag.md +2 -0
  55. package/skills/ae-analysis/references/ai_models.md +72 -3
  56. package/skills/ae-analysis/references/analysis_data_retrieval.md +4 -0
  57. package/skills/ae-analysis/references/asset_batch_info_export.md +10 -0
  58. package/skills/ae-analysis/references/asset_batch_sql_export.md +13 -1
  59. package/skills/ae-analysis/references/asset_export.md +12 -1
  60. package/skills/ae-analysis/references/asset_list.md +1 -0
  61. package/skills/ae-analysis/references/asset_search.md +11 -5
  62. package/skills/ae-analysis/references/bi_panel_list.md +1 -1
  63. package/skills/ae-analysis/references/catalog_list.md +1 -1
  64. package/skills/ae-analysis/references/collaboration.md +48 -0
  65. package/skills/ae-analysis/references/command_index.md +16 -15
  66. package/skills/ae-analysis/references/dashboard_list.md +4 -4
  67. package/skills/ae-analysis/references/dashboard_report_data_export.md +2 -0
  68. package/skills/ae-analysis/references/event_export.md +3 -1
  69. package/skills/ae-analysis/references/governance_recommendation_auto_review.md +71 -0
  70. package/skills/ae-analysis/references/governance_recommendation_export.md +11 -0
  71. package/skills/ae-analysis/references/metadata_resolution.md +3 -3
  72. package/skills/ae-analysis/references/metric_export.md +3 -1
  73. package/skills/ae-analysis/references/metric_list.md +1 -1
  74. package/skills/ae-analysis/references/operation_record_export.md +10 -0
  75. package/skills/ae-analysis/references/personal_semantic_preference_add.md +2 -2
  76. package/skills/ae-analysis/references/personal_semantic_preference_get.md +6 -4
  77. package/skills/ae-analysis/references/personal_semantic_preference_list.md +5 -3
  78. package/skills/ae-analysis/references/project_semantic_knowledge_wiki.md +3 -3
  79. package/skills/ae-analysis/references/property_export.md +3 -1
  80. package/skills/ae-analysis/references/report_create.md +4 -0
  81. package/skills/ae-analysis/references/report_data_export.md +6 -0
  82. package/skills/ae-analysis/references/report_get.md +2 -0
  83. package/skills/ae-analysis/references/report_list.md +5 -5
  84. package/skills/ae-analysis/references/report_update.md +6 -0
  85. package/skills/ae-analysis/references/sql_table_columns.md +4 -4
  86. package/skills/ae-analysis/references/sql_table_list.md +5 -5
  87. package/skills/ae-analysis/references/user_cluster_models.md +8 -0
  88. package/skills/ae-analysis/references/user_tag_create.md +7 -1
  89. package/skills/ae-analysis/references/user_tag_get.md +1 -1
  90. package/skills/ae-analysis/references/user_tag_models.md +18 -2
  91. package/skills/ae-analysis/references/user_tag_refresh.md +1 -1
  92. package/skills/ae-analysis/references/user_tag_update.md +2 -2
  93. package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/default-compile-rules.md +1 -1
  94. package/skills/ae-capability/SKILL.md +63 -1
  95. package/skills/ae-capability/references/collaboration.md +48 -0
  96. package/skills/ae-community/SKILL.md +5 -1
  97. package/skills/ae-community/references/collaboration.md +48 -0
  98. package/skills/ae-data-integration/SKILL.md +4 -0
  99. package/skills/ae-data-integration/references/collaboration.md +48 -0
  100. package/skills/ae-data-integration-helper/references/cpp_server_sdk_faq.md +2 -2
  101. package/skills/ae-data-integration-helper/references/logbus2_parser_plugin.md +6 -6
  102. package/skills/ae-data-integration-helper/references/sdk_log_guide.md +4 -4
  103. package/skills/ae-dataops/SKILL.md +54 -74
  104. package/skills/ae-dataops/references/collaboration.md +48 -0
  105. package/skills/ae-dataops/references/dataops-backfill.md +23 -18
  106. package/skills/ae-dataops/references/dataops-flow-create.md +74 -23
  107. package/skills/ae-dataops/references/dataops-flow-monitor.md +45 -13
  108. package/skills/ae-dataops/references/dataops-integration.md +57 -30
  109. package/skills/ae-dataops/references/dataops-query.md +21 -4
  110. package/skills/ae-dataops/references/dataops-table.md +143 -11
  111. package/skills/ae-engage/SKILL.md +21 -6
  112. package/skills/ae-engage/references/add-channel.md +3 -3
  113. package/skills/ae-engage/references/channel-mgmt.md +3 -3
  114. package/skills/ae-engage/references/collaboration.md +48 -0
  115. package/skills/ae-engage/references/common-metric.md +2 -2
  116. package/skills/ae-engage/references/save-task.md +2 -2
  117. package/skills/ae-engage/references/scene-config-channel.md +4 -4
  118. package/skills/ae-generate-tracking-code/SKILL.md +1 -1
  119. package/skills/ae-generate-tracking-plan/SKILL.md +4 -0
  120. package/skills/ae-generate-tracking-plan/references/collaboration.md +48 -0
  121. package/skills/ae-kb/SKILL.md +9 -1
  122. package/skills/ae-kb/references/collaboration.md +48 -0
  123. package/skills/ae-kb/references/schema-import.md +30 -0
  124. package/skills/ae-kb-discovery/SKILL.md +7 -2
  125. package/skills/ae-kb-discovery/references/collaboration.md +48 -0
  126. package/skills/ae-metadata/SKILL.md +20 -12
  127. package/skills/ae-metadata/references/collaboration.md +48 -0
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: dataops-flow-create
3
3
  version: 1.0.0
4
- description: "Flow creation and configuration: create flows, add task nodes, configure dependencies, save task definitions, configure scheduling, test, and release to production. Trigger keywords: create flow, new workflow, configure schedule, add task node, release, cron, scheduled execution."
4
+ description: "Flow creation and configuration: create flows, manage custom flow parameters, add task nodes, configure dependencies, save task definitions, configure scheduling, test, and release to production. Trigger keywords: create flow, new workflow, configure schedule, add task node, release, cron, scheduled execution."
5
5
  metadata:
6
6
  requires:
7
7
  bins: ["ae-cli"]
@@ -13,22 +13,74 @@ metadata:
13
13
 
14
14
  Use the `dataops_flow` subcommand to manage flow lifecycle.
15
15
 
16
- **Flow Lifecycle: Create → Configure Nodes → Configure Schedule → Preview Release → Release to PROD → Manual PROD Execution / Online Schedule**
16
+ **Flow Lifecycle: Create → Configure Nodes → Configure Schedule → Preview Release → Release to PROD → Read Back PROD Configuration**
17
17
 
18
18
  ---
19
19
 
20
20
  ## Complete Flow Creation Process
21
21
 
22
- Follow these steps in order to create a production-ready flow from scratch.
22
+ Follow the applicable steps for the requested outcome. A DEV-only change stops before release; a request to publish or enable scheduling ends after PROD readback. Manually execute only when the user also requested a run. For scheduled integration, first obtain the intended sync solution from [the integration reference](dataops-integration.md), bind it as an integration task, then configure and release the flow.
23
23
 
24
24
  ### Step 1: Create Flow
25
25
 
26
26
  ```bash
27
27
  ae-cli dataops_flow +create_flow --spaceCode "${spaceCode}" \
28
28
  --flowName "Daily ETL Process" --remark "Process user data"
29
- # Returns flowCode, required for subsequent steps
29
+ # Default CLI JSON: data.result.flowCode, required for subsequent steps
30
30
  ```
31
31
 
32
+ ### Custom Flow Parameters
33
+
34
+ These commands manage one custom workflow definition parameter at a time. They do not set temporary execution inputs or replace the complete parameter list. All four require workflow edit permission (`dwWorkflowEdit`) in the containing space; reading PROD parameters has the same requirement.
35
+
36
+ ```bash
37
+ # Read the current DEV custom parameters before editing
38
+ ae-cli dataops_flow +get_flow_params --spaceCode "${spaceCode}" \
39
+ --flowCode ${flowCode}
40
+
41
+ # Create text; VARCHAR is the default
42
+ ae-cli dataops_flow +create_flow_param --spaceCode "${spaceCode}" \
43
+ --flowCode ${flowCode} --paramKey "run_date" \
44
+ --paramValue "2026-09-08" --remark "Date to process"
45
+
46
+ # Use an expression already validated with the platform's supported expression syntax
47
+ ae-cli dataops_flow +create_flow_param --spaceCode "${spaceCode}" \
48
+ --flowCode ${flowCode} --paramKey "derived_date" \
49
+ --paramDataType EXPRESSION --paramValue "${validatedExpression}"
50
+
51
+ # Change only the value; preserve type and remark
52
+ ae-cli dataops_flow +update_flow_param --spaceCode "${spaceCode}" \
53
+ --flowCode ${flowCode} --paramKey "run_date" --paramValue "2026-09-09"
54
+
55
+ # Rename: originParamKey is the old name and paramKey is the new name
56
+ ae-cli dataops_flow +update_flow_param --spaceCode "${spaceCode}" \
57
+ --flowCode ${flowCode} --originParamKey "run_date" --paramKey "processing_date"
58
+
59
+ # Explicitly clear the remark
60
+ ae-cli dataops_flow +update_flow_param --spaceCode "${spaceCode}" \
61
+ --flowCode ${flowCode} --paramKey "processing_date" --remark ""
62
+
63
+ # Inspect the target above, then preview deletion
64
+ ae-cli dataops_flow +delete_flow_param --spaceCode "${spaceCode}" \
65
+ --flowCode ${flowCode} --paramKey "processing_date" --dry-run
66
+
67
+ # Run only after explicit user confirmation
68
+ ae-cli dataops_flow +delete_flow_param --spaceCode "${spaceCode}" \
69
+ --flowCode ${flowCode} --paramKey "processing_date" --yes
70
+
71
+ # Read the published definition separately
72
+ ae-cli dataops_flow +get_flow_params --spaceCode "${spaceCode}" \
73
+ --flowCode ${flowCode} --env PROD
74
+ ```
75
+
76
+ - Query defaults to `DEV`; `PROD` is the only other environment. It returns an array containing `paramKey`, `paramValue`, `paramDataType`, `remark`, `paramFormat`, and expression reference metadata. Only custom flow parameters are returned.
77
+ - `flowCode` must be a positive safe integer. Names must match `[a-z][a-z0-9_]{0,39}`; the `ws_` prefix, `env`, and space built-in parameter names are reserved. Built-in parameters cannot be created, updated, or deleted with these commands.
78
+ - `paramValue` is required and non-empty on create; it remains literal, including whitespace. Single-quote literal expressions to prevent shell expansion of `${...}`. Types are `VARCHAR` (text, create default) and `EXPRESSION`. Expression syntax and supported references are validated by Gaia; arbitrary references between custom parameters are not supported.
79
+ - Update accepts `paramValue`, `paramDataType`, and `remark` independently. Omitted fields retain their values; REST/MCP `null` also retains them. `--remark ""` clears the remark. An empty `paramValue` is rejected. Remarks allow up to 200 characters. Supply at least one update field or an actual rename.
80
+ - Omit `originParamKey` for an in-place update. When renaming, it identifies the old name while `paramKey` identifies the new name. Duplicate names, rename conflicts, and missing targets return errors. Renaming does not replace old references in SQL; inspect affected tasks and update their SQL when needed.
81
+ - Writes configure DEV. Adding a parameter to an existing definition, changing value/type/name, and deleting a parameter may stop running DEV debug executions. Remark-only and no-op updates do not stop debug runs. Value/type/name/deletion changes require a later release to affect PROD; a remark update without renaming also updates an existing PROD parameter with the same name.
82
+ - Deletion is `high-risk-write`: inspect, preview, confirm, then use `--yes`. Task references fall back to a same-name space parameter if one exists; otherwise they remain code-parsed references whose source and value fields are null or omitted. Inspect affected SQL through `+get_flow_overview` task parameters. Read parameters again after each write to verify the saved state.
83
+
32
84
  ### Step 2: Create Task Nodes (can be called multiple times)
33
85
 
34
86
  ```bash
@@ -36,7 +88,7 @@ ae-cli dataops_flow +create_flow --spaceCode "${spaceCode}" \
36
88
  ae-cli dataops_flow +create_sql_task --spaceCode "${spaceCode}" \
37
89
  --flowCode ${flowCode} --taskName "Process User Data" \
38
90
  --sql "SELECT * FROM dwd_user"
39
- # Returns result.taskCode
91
+ # Default CLI JSON: data.result.taskCode
40
92
 
41
93
  # Create task with upstream dependency
42
94
  ae-cli dataops_flow +create_sql_task --spaceCode "${spaceCode}" \
@@ -145,7 +197,7 @@ ae-cli dataops_flow +save_schedule_config --spaceCode "${spaceCode}" \
145
197
 
146
198
  **CRON Format** (6 fields): `second minute hour day month weekday`
147
199
  - `0 0 */4 * * ?` — Every 4 hours
148
- - `0 30 8 * * 1-5` — Weekdays at 8:30
200
+ - `0 30 8 ? * MON-FRI` — Weekdays at 8:30 (Quartz)
149
201
 
150
202
  ### Step 8: Preview Release
151
203
 
@@ -155,27 +207,22 @@ ae-cli dataops_flow +preview_release_flow --spaceCode "${spaceCode}" \
155
207
  --flowCode ${flowCode}
156
208
  ```
157
209
 
210
+ Inspect `data.releaseStatus` in the default CLI JSON: `READY` permits the requested release; `NO_CHANGES` means skip submission and inspect PROD. For `CHECK_FAILED` or `FAILED`, report the message and failed change items and stop before publishing or executing. Check that the preview contains only the intended changes.
211
+
158
212
  ### Step 9: Release to Production and Verify
159
213
 
160
214
  ```bash
161
215
  ae-cli dataops_flow +release_flow --spaceCode "${spaceCode}" \
162
216
  --flowCode ${flowCode}
163
217
 
164
- # Manually trigger PROD execution when verification is needed
165
- ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
166
- --flowCode ${flowCode}
218
+ # Read the published definition without starting a run
219
+ ae-cli dataops_flow +get_flow_overview --spaceCode "${spaceCode}" \
220
+ --flowCode ${flowCode} --env PROD
221
+ ```
167
222
 
168
- # Check execution status
169
- ae-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
170
- --keyword "${flowCode}" --pageSize 20
223
+ Check `data.result.releaseStatus`, not outer `ok` or action `status`: `FAILED`, `CHECK_FAILED`, `FAIL`, `PART_SUCCESS`, and `TABLE_FAIL` are not successful publication. Stop and report their details. `TO_RELEASE`, `RELEASING`, or `SUBMITTED` means completion is unverified; retain `data.result.packageCode` and do not resubmit the release.
171
224
 
172
- # Inspect one instance and task logs when troubleshooting
173
- ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
174
- --flowCode ${flowCode} --flowInstanceId ${flowInstanceId}
175
- ae-cli dataops_operations +get_task_instance_detail --spaceCode "${spaceCode}" \
176
- --flowCode ${flowCode} --flowInstanceId ${flowInstanceId} \
177
- --taskInstanceId ${taskInstanceId} --includeLog true
178
- ```
225
+ Confirm the PROD overview has `data.success=true` and compare its visible `schedule`, `dag`, and `flowParams` with the requested changes, including CRON and task presence/type when applicable. Even a release response of `SUCCESS` needs this readback. The overview does not expose task SQL or the integration task's `syncId`; retain the create/update result and release preview as evidence, and state this readback limit instead of claiming those fields were independently verified in PROD. If publication is pending, make read-only checks within a finite deadline; on expiry or unavailable evidence, report the package/flow IDs and what remains unverified. A release-only request ends here. For an explicitly requested manual run, continue with [execution and completion checks](dataops-flow-monitor.md#workflow-c-manual-execution-and-stop).
179
226
 
180
227
  ---
181
228
 
@@ -195,7 +242,11 @@ ae-cli dataops_operations +get_task_instance_detail --spaceCode "${spaceCode}" \
195
242
  | `+delete_task` | Delete one DEV task node after confirmation | `--spaceCode` `--flowCode` `--taskCode` `--yes` |
196
243
  | `+add_task_relation` | Add DEV dependency | `--spaceCode` `--flowCode` `--preTaskCode` `--taskCode` |
197
244
  | `+save_schedule_config` | Save DEV schedule config | `--spaceCode` `--flowCode` `--enabled` `[--cron]` |
198
- | `+get_task_params` | View DEV task parameter list | `--spaceCode` `--flowCode` `--taskCode` |
245
+ | `+get_flow_params` | List custom workflow parameters | `--spaceCode` `--flowCode` `[--env DEV\|PROD]` |
246
+ | `+create_flow_param` | Create one DEV custom parameter | `--spaceCode` `--flowCode` `--paramKey` `--paramValue` `[--paramDataType VARCHAR\|EXPRESSION]` `[--remark]` |
247
+ | `+update_flow_param` | Update or rename one DEV custom parameter | `--spaceCode` `--flowCode` `--paramKey` `[--originParamKey]` `[--paramValue]` `[--paramDataType]` `[--remark]` |
248
+ | `+delete_flow_param` | Delete one DEV custom parameter after confirmation | `--spaceCode` `--flowCode` `--paramKey` `--yes` |
249
+ | `+get_flow_overview` | View DEV/PROD flow definitions and all task parameter references | `--spaceCode` (`--flowCode` or `--flowName`) `[--env DEV\|PROD]` |
199
250
  | `+execute_flow` | Manual PROD execution | `--spaceCode` `--flowCode` `[--baseDate]` |
200
251
  | `dataops_operations +get_flow_instance_detail` | View operations instance DAG and task statuses | `--spaceCode` `--flowCode` `--flowInstanceId` |
201
252
  | `dataops_operations +get_task_instance_detail` | View operations task detail and optional logs | `--spaceCode` `--flowCode` `--flowInstanceId` (`--taskInstanceId` or `--taskCode` or `--taskName`) `[--includeLog]` |
@@ -207,14 +258,14 @@ ae-cli dataops_operations +get_task_instance_detail --spaceCode "${spaceCode}" \
207
258
  - **Parameter Reference**: Reference workspace parameters in tasks using `${paramKey}` (e.g., `${ws_run_date}`)
208
259
  - **Execution**: `+execute_flow` requires `--spaceCode` and `--flowCode`; `--baseDate` is optional and maps to runtime parameter `bd`. It always runs PROD and returns `action/result/status`; `result` includes `flowCode`, `executeId`, `operationStatus`, `nextAction`, and optional `flowInstanceId`.
209
260
  - **Schedule config**: `+save_schedule_config` requires `--spaceCode`, `--flowCode`, and `--enabled`. `--cron` is required only when `--enabled true`; omit it when disabling scheduling. It returns `action/result/status`; `result` includes `enabled`, `flow`, `message`, and `cron` only when enabled.
210
- - **Release preview**: `+preview_release_flow` requires `--spaceCode` and `--flowCode`; it has no optional flags. It returns `flowCode`, `releaseStatus`, `message`, and `changes`. Each change may include `scheduleConfigChange` and `tasks`; task entries may include `changed`, `contentCompare`, and `targetTable`.
261
+ - **Release preview**: `+preview_release_flow` requires `--spaceCode` and `--flowCode`; it has no optional flags. It returns `flowCode`, `releaseStatus`, `message`, and `changes`. Each change may include `scheduleConfigChange` and `tasks`; `scheduleConfigChange.diff.old.parameters` and `.new.parameters` show the flow parameter comparison. Parameter-only changes use the existing flow configuration release. Task entries may include `changed`, `contentCompare`, and `targetTable`.
211
262
  - **Release**: `+release_flow` requires `--spaceCode` and `--flowCode`; it has no optional flags. It returns `action/result/status`; `result` includes `flowCode`, `releaseStatus`, `message`, optional `packageCode`, and optional `changes`. Each change may include `scheduleConfigChange` and `tasks`; task entries include `changed`.
212
263
  - **Task dependencies**: `--preTasks` is a JSON array of upstream task codes on SQL, integration, workflow instance check, and task instance check create/update commands. Omit `--preTasks` on update to preserve existing dependencies; pass `--preTasks '[]'` to clear them; pass a non-empty array to replace them.
213
264
  - **Retry policy**: All four task types accept `--failRetryTimes`, `--failRetryInterval`, and `--failRetryUnit`. Create defaults to `3`, `5`, and `MINUTE`. Update preserves every omitted retry field. `MINUTE` is the only supported unit.
214
265
  - **SQL task creation**: `+create_sql_task` requires `--spaceCode`, `--flowCode`, `--taskName`, and `--sql`; SQL hooks, dependencies, retry policy, and remark are optional. It returns `action/result/status`; `result` includes `flowCode`, `taskCode`, `taskName`, `taskType=TRINO_SQL`, and `sqlSaved=true`.
215
266
  - **SQL task update**: `+update_sql_task` requires `--spaceCode`, `--flowCode`, `--taskCode`, and `--sql`; omitted SQL hooks, dependencies, and retry fields keep existing values. It returns `action/result/status`; `result` includes `sqlSaved`, `flowCode`, `taskCode`, `taskType=TRINO_SQL`, and `task`.
216
267
  - **Task dependency**: `+add_task_relation` requires `--spaceCode`, `--flowCode`, `--preTaskCode`, and `--taskCode`. `preTaskCode` is upstream and `taskCode` is downstream. It returns `action/result/status`; `result` includes `status`, `flowCode`, `preTaskCode`, `taskCode`, and `message`.
217
- - **Task parameters**: `+get_task_params` requires `--spaceCode`, `--flowCode`, and `--taskCode`; it has no optional flags. It queries DEV and returns `data` as an array. Items include fields such as `paramKey`, `paramType`, `paramDataType`, `paramFrom`, and built-in flags like `isBd`.
268
+ - **Flow and task parameters**: `+get_flow_overview` requires `dwWorkflowEdit` and supports DEV (default) or PROD. It returns all custom flow definitions in `flowParams` and task references in `dag.tasks[].taskParams`, including source, definition value, type, description, and built-in flags. Expressions remain literal; task references retain `paramDataType=QUOTE`. Source is `FLOW` or `SPACE` for configured parameters. A code-parsed reference without a configured source remains visible with null or omitted `paramFrom` and `paramValue` fields; either form means no configured source/value. Empty parameter lists are arrays. Use the selected environment to compare unpublished DEV changes with PROD.
218
269
  - **Integration task creation**: `+create_integration_task` requires `--spaceCode`, `--flowCode`, `--taskName`, and `--syncId`; dependencies, retry policy, and remark are optional. It returns `action/result/status`; `result` includes `syncTaskSaved`, `flowCode`, `taskCode`, `taskName`, `taskType=OFFLINE_SYNC`, `syncId`, and `nextAction`.
219
270
  - **Workflow instance check tasks**: `checkItems` is required and replaces the complete check item list. Create defaults `relation/checkInterval/checkTime` to `AND/5/3`; update preserves omitted scalar values. This command creates `FLOW_CHECK`, not task-instance `TASK_CHECK`.
220
271
  - **Task instance check tasks**: `checkItems` is required and replaces the complete check item list. Each item identifies a target using `flowCode` and `taskCode`. Create defaults `relation/checkInterval/checkTime` to `AND/10/3`; update preserves omitted scalar values. This command creates `TASK_CHECK`.
@@ -228,7 +279,7 @@ Owning module: gaia-mcp workflow
228
279
 
229
280
  Current transport: DataOps CLI REST
230
281
 
231
- Covered workflow tools: `flow_create_workflow_instance_check_task`, `flow_update_workflow_instance_check_task`, `flow_create_task_instance_check_task`, `flow_update_task_instance_check_task`, and `flow_delete_task`
282
+ Covered workflow tools: `flow_create_workflow_instance_check_task`, `flow_update_workflow_instance_check_task`, `flow_create_task_instance_check_task`, `flow_update_task_instance_check_task`, `flow_delete_task`, `flow_get_flow_params`, `flow_create_flow_param`, `flow_update_flow_param`, and `flow_delete_flow_param`
232
283
 
233
284
  Gateway target: TBD after DataOps workflow Gateway schema review
234
285
 
@@ -18,6 +18,23 @@ Use `dataops_flow` to execute and control flows. Use `dataops_operations` for sp
18
18
  - **flowInstanceId** — Operations workflow instance ID returned by `dataops_operations +search_flow_instances`; use it for operations detail, task logs, and stop
19
19
  - **Instance list queries PROD environment by default** (unlike other tools which default to DEV)
20
20
 
21
+ In the default CLI JSON, take `executeId` from `data.result.executeId` and an already available `flowInstanceId` from `data.result.flowInstanceId`. Otherwise select `data.instances[]` from operations search and keep its `flowCode` and `flowInstanceId` together. Instance detail supplies `data.taskInstances[].taskInstanceId` for task inspection; a definition's `taskCode` is not a task instance ID.
22
+
23
+ ### Status fields are command-specific
24
+
25
+ The paths below are relative to the command payload, as used by CLI `--jq`. Prefix them with `.data` when reading the default JSON envelope with an external jq command.
26
+
27
+ | Command | Execution status path | Trigger path |
28
+ |---|---|---|
29
+ | `+list_flows` | `.flows[].latestProductionInstance.flowInstanceStatus` | `.flows[].latestProductionInstance.historyCmd` |
30
+ | `+search_flow_instances` | `.instances[].status` | `.instances[].triggerType` |
31
+ | `+get_flow_instance_detail` | `.flowInstance.status`; task states in `.taskInstances[].status` | `.flowInstance.triggerType` |
32
+ | `+get_task_instance_detail` | `.status` | Use its flow instance |
33
+
34
+ `flowStatus` is the workflow definition state, such as `RELEASED`, not execution completion. `flowScheduleStatus` is a separate scheduling state; neither it nor the latest-flow summary replaces inspection of a specific run.
35
+
36
+ If the selected instance's execution status is missing, null, or unknown, stop status-based polling and inspect the same read command without `--jq`. A missing field projects to null without a jq error; `ok:true` does not make that status usable. If the raw status remains unavailable, report uncertainty rather than treating it as running or successful. Do not apply this rule to legitimately nullable fields such as `endTime`.
37
+
21
38
  ---
22
39
 
23
40
  ## Workflow A: Find and View Flow Execution Status
@@ -43,6 +60,17 @@ ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
43
60
  --flowCode ${flowCode} --flowInstanceId ${flowInstanceId}
44
61
  ```
45
62
 
63
+ `flowKeyword` must be a discovered flow name, remark, or instance ID; search does not match `flowCode`. Filter returned instances by the exact `flowCode` and the requested execution/base date, following `hasMore` when needed. For a newly submitted run, compare the submission time and prior instance list; if several candidates still match, report ambiguity instead of inspecting an arbitrary latest instance.
64
+
65
+ Once the intended instance is verified, keep its `flowInstanceId` fixed and poll its detail, not all matching instances. A previous instance's `SUCCESS` cannot prove that the requested run completed. If the current CLI cannot correlate `executeId` to one instance confidently, retain the execution handle and report the limitation.
66
+
67
+ ```bash
68
+ # Inspect only the previously verified instance; reuse these IDs while polling.
69
+ ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
70
+ --flowCode "${flowCode}" --flowInstanceId "${flowInstanceId}" \
71
+ --jq '{flowInstanceId: .flowInstance.flowInstanceId, status: .flowInstance.status}'
72
+ ```
73
+
46
74
  ---
47
75
 
48
76
  ## Workflow B: Troubleshoot Task Failures
@@ -56,31 +84,35 @@ ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
56
84
  ae-cli dataops_operations +get_task_instance_detail --spaceCode "${spaceCode}" \
57
85
  --flowCode ${flowCode} --flowInstanceId ${flowInstanceId} \
58
86
  --taskInstanceId ${taskInstanceId} --includeLog true
59
-
60
- # Step 3: Re-execute after fixing
61
- ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
62
- --flowCode ${flowCode}
63
87
  ```
64
88
 
89
+ Report the failed task, historical instance/base date, and relevant log evidence first. These commands cannot rerun only failed nodes of that historical instance. Do not substitute `+execute_flow` (a new whole-flow PROD run) or a backfill job for that request. If the user explicitly requests a new whole-flow run, establish its business date, release any required DEV fixes using [the release workflow](dataops-flow-create.md#step-8-preview-release), and continue below; pass `--baseDate` for a requested historical date rather than assuming today's date is equivalent.
90
+
65
91
  ---
66
92
 
67
93
  ## Workflow C: Manual Execution and Stop
68
94
 
69
95
  ```bash
70
- # Manually trigger PROD execution
96
+ # Run only when the user requested a new whole-flow PROD execution
71
97
  ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
72
98
  --flowCode ${flowCode}
73
- # Returns executeId
99
+ # data.result.operationStatus=SUCCESS means accepted; retain data.result.executeId
74
100
 
75
101
  # Monitor recent executions
76
102
  ae-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
77
103
  --keyword "${flowKeyword}" --startDate "${startDate}" --endDate "${endDate}" --pageSize 20
78
104
 
79
- # Stop execution if necessary (irreversible)
105
+ # Stop only when requested, using the verified execution handle
80
106
  ae-cli dataops_operations +stop_flow_instance --spaceCode "${spaceCode}" \
81
107
  --flowCode ${flowCode} --executeId ${executeId}
82
108
  ```
83
109
 
110
+ An accepted execution is not a completed run. Inspect `data.result.operationStatus`; a non-`SUCCESS` result stops the submission path. Once the instance is identified, read `data.flowInstance.status` from `+get_flow_instance_detail`: `SUCCESS` is successful execution, `FAIL` is failure, and `STOP` or `PAUSE` is not success. `RUNNING` and `READY_PAUSE` are unfinished flow instance states. Task detail uses `data.status`; instance detail uses `data.taskInstances[].status`. A successful read of a failed instance is still a successful query.
111
+
112
+ For `+search_flow_instances --status`, use only `RUNNING,SUCCESS,FAIL,READY_PAUSE,PAUSE,STOP`. `WAITING` filtering is not supported by the current backend; an empty result with that filter does not establish that no work is waiting. Do not infer support from a different command's scheduling or download status.
113
+
114
+ When completion is requested, set a finite polling deadline and use read-only instance queries. Stop on a terminal/paused state, an actionable error, or the deadline; then report the IDs, last observed state, and relevant failure evidence. A temporarily absent instance or a timeout does not authorize another `+execute_flow`; retain the original `executeId` and report uncertain correlation or completion. Read the instance again after a stop request to distinguish stop acceptance from an observed `STOP` state.
115
+
84
116
  ---
85
117
 
86
118
  ## Workflow D: View Flow Structure
@@ -107,19 +139,19 @@ ae-cli dataops_flow +get_flow_overview --spaceCode "${spaceCode}" \
107
139
  | `dataops_operations +get_flow_instance_detail` | Operations instance DAG and task statuses | `--spaceCode` `--flowCode` `--flowInstanceId` |
108
140
  | `dataops_operations +get_task_instance_detail` | Operations task detail and optional logs | `--spaceCode` `--flowCode` `--flowInstanceId` (`--taskInstanceId` or `--taskCode` or `--taskName`) `[--includeLog]` |
109
141
  | `dataops_operations +stop_flow_instance` | Stop a running execution | `--spaceCode` `--flowCode` exactly one of `--executeId` or `--flowInstanceId` |
110
- | `+get_task_params` | DEV task parameter list | `--spaceCode` `--flowCode` `--taskCode` |
111
142
  | `+update_flow` | Update DEV workflow name and/or remark | `--spaceCode` `--flowCode` `[--flowName]` `[--remark]` |
112
143
 
113
144
  ## Parameter Notes
114
145
 
115
146
  - **Execution**: `+execute_flow` requires `--spaceCode` and `--flowCode`; `--baseDate` is optional and maps to runtime parameter `bd`. It always runs PROD and returns `action/result/status`; `result` includes `flowCode`, `executeId`, `operationStatus`, `nextAction`, and optional `flowInstanceId`.
116
147
  - **Stop flow instance**: `dataops_operations +stop_flow_instance` requires `--spaceCode`, `--flowCode`, and exactly one of `--executeId` or `--flowInstanceId`. `--executeId` comes from `dataops_flow +execute_flow`; `--flowInstanceId` comes from `dataops_operations +search_flow_instances`. It returns `success`, `selector`, and selector-specific stop result fields.
117
- - **Flow overview**: `+get_flow_overview` requires `--spaceCode` and either `--flowCode` or exact `--flowName`; `--flowCode` takes precedence. `--env` defaults to `DEV`; use `PROD` to include latest production instance fields when available. It returns `success`, `env`, `resolvedBy`, `flow`, `schedule`, `dag`, and `summary`.
148
+ - **Flow overview**: `+get_flow_overview` requires `--spaceCode` and either `--flowCode` or exact `--flowName`; `--flowCode` takes precedence. `--env` defaults to `DEV`; use `PROD` to include latest production instance fields when available. It requires `dwWorkflowEdit` in both environments and returns `success`, `env`, `resolvedBy`, `flow`, `flowParams`, `schedule`, `dag`, and `summary`. `flowParams` contains all custom flow parameters, including unused ones; `dag.tasks[].taskParams` contains each task's references with `paramKey`, `paramType`, `paramDataType`, `paramFrom`, `paramValue`, `paramDesc`, and built-in flags such as `isBd`. Both lists use the selected environment and are empty arrays when no parameters exist. Definition values preserve expression text: a FLOW reference to `p=${bd}` keeps `paramDataType=QUOTE` and `paramValue=${bd}`; it is not an execution date.
118
149
  - **Flow update**: `+update_flow` requires `--spaceCode`, `--flowCode`, and at least one of `--flowName` or `--remark`. It returns `action/result/status`; `result` is an array with items containing `flowCode`, `operationStatus`, `nameChanged`, and optional `flowName`.
119
150
  - **High-frequency release flows**: `+list_high_frequency_release_flows` requires `--spaceCode`; `--days`, `--topN`, `--minCount`, and `--status` are optional. Defaults are `days=30`, `topN=10`, and `status=SUCCESS`. It returns `period`, `filters`, `flows`, `returnedCount`, and `nextAction`; flow items include `rank`, `flowCode`, `flowName`, `releaseCount`, `lastReleaseTime`, and `avgIntervalHours`.
120
- - **Operations instance search**: `dataops_operations +search_flow_instances` requires `--spaceCode`; `--keyword`, `--startDate`, `--endDate`, `--status`, `--pageNum`, and `--pageSize` are optional. `--keyword` fuzzy-matches `flowName` or `flowCode`. `pageNum` defaults to `1`; `pageSize` defaults to `20` and maxes at `100`. It returns `totalCount`, `returnedCount`, `pageNum`, `pageSize`, `hasMore`, `instances`, `statusCounts`, `triggerTypeCounts`, and `ownerCounts`.
121
- - **Task instance detail**: `dataops_operations +get_task_instance_detail` requires `--spaceCode`, `--flowCode`, `--flowInstanceId`, and one selector: `--taskInstanceId`, `--taskCode`, or exact `--taskName`. Prefer `--taskInstanceId` because retries can create multiple instances with the same task code/name. `--includeLog` is optional and defaults to `false`. It returns `success`, `flowInstanceId`, `taskCode`, `taskName`, `taskInstanceId`, `status`, `task`, `taskInstance`, `definition`, and `log` only when requested.
151
+ - **Operations instance search**: `dataops_operations +search_flow_instances` requires `--spaceCode`; `--keyword`, `--startDate`, `--endDate`, `--status`, `--pageNum`, and `--pageSize` are optional. `--keyword` fuzzy-matches instance ID, workflow name, or workflow remark, not `flowCode`; filter the returned `flowCode` exactly. `pageNum` defaults to `1`; `pageSize` defaults to `20` and maxes at `100`. It returns `totalCount`, `returnedCount`, `pageNum`, `pageSize`, `hasMore`, `instances`, `statusCounts`, `triggerTypeCounts`, and `ownerCounts`.
152
+ - **Flow instance detail**: `dataops_operations +get_flow_instance_detail` requires `dwOMInstanceView` and returns the historical execution snapshot in `flowInstance.instanceParamMap`. A missing snapshot is null or omitted; the server does not substitute current parameter definitions. Later definition changes do not change historical execution values.
153
+ - **Task instance detail**: `dataops_operations +get_task_instance_detail` requires `--spaceCode`, `--flowCode`, `--flowInstanceId`, and one selector: `--taskInstanceId`, `--taskCode`, or exact `--taskName`. Prefer `--taskInstanceId` because retries can create multiple instances with the same task code/name. It requires `dwOMInstanceView`. `--includeLog` is optional and defaults to `false`; when enabled, `log.params` contains the task's historical actual execution values. It returns `success`, `flowInstanceId`, `taskCode`, `taskName`, `taskInstanceId`, `status`, `task`, `taskInstance`, `definition`, and `log` only when requested.
122
154
  - **env**: `DEV` (development) | `PROD` (production, instance list defaults to PROD)
123
155
  - **flow list paging**: `+list_flows` requires `--spaceCode`; `--keyword`, `--pageNum`, and `--pageSize` are optional. It returns `flows`, `totalCount`, `returnedCount`, `pageNum`, `pageSize`, and `hasMore`; flow items include `latestProductionInstance` only when available. `pageSize` defaults to `20` and maxes at `100`.
124
- - **Task parameters**: `+get_task_params` requires `--spaceCode`, `--flowCode`, and `--taskCode`; it has no optional flags. It queries DEV and returns `data` as an array. Items include fields such as `paramKey`, `paramType`, `paramDataType`, `paramFrom`, and built-in flags like `isBd`.
125
- - **Task Status**: `success` / `failure` / `running` / `waiting`
156
+ - **Task parameter sources**: Read `dag.tasks[].taskParams` from `+get_flow_overview`. `paramFrom=FLOW` means a flow definition supplies the value; `SPACE` means a space parameter does. A code-parsed reference remains in the list when neither source exists; its `paramFrom` and `paramValue` fields are null or omitted, both meaning no configured source/value. Deleting a flow parameter falls back to a same-name space parameter, or to code-parsed status. Unpublished DEV changes do not alter PROD values or sources.
157
+ - **Task Status**: `WAITING` / `RUNNING` / `SUCCESS` / `FAIL` / `STOP` / `PAUSE`; only `SUCCESS` means successful completion.
@@ -16,14 +16,17 @@ Use the `dataops_integration` subcommand to manage datasources and sync solution
16
16
  **Core Rules:**
17
17
  - Configuration parameters differ significantly between preset repositories (te_etl) and non-preset repositories, must strictly follow templates
18
18
  - sourceConfig/sinkConfig/channelConfig/fieldsMapping are all JSON strings
19
- - Sync solution updates are not partial patches: call `+get_sync_detail --withParams true` first, then submit complete sourceConfig and sinkConfig JSON. Pass channelConfig and fieldsMapping when keeping or updating them
19
+ - Sync solution updates are not partial patches. The current detail command returns a summary, not complete editable configs; follow [Update an Existing Sync Solution](#workflow-d-update-an-existing-sync-solution).
20
20
  - Test datasource connection before creating sync solution
21
21
  - `sinkConfig.dataSaveMode` codes are fixed: `1 = APPEND_DATA` (insert/append new data; UI label `插入新数据`), `2 = OVERWRITE` (`DROP_DATA`; UI label `覆盖写入`). Never describe `dataSaveMode=1` as overwrite; use `2` for overwrite.
22
+ - Integration `--dry-run` only previews the local request. It does not validate nested JSON, permissions, connections, or execution readiness on the server.
22
23
 
23
24
  ---
24
25
 
25
26
  ## Workflow A: Create Datasource
26
27
 
28
+ Use the selected component's returned template. The values in angle brackets below are illustrative; replace them with the user's real connection settings before execution. For MySQL, `jdbcUrl` and the separate `database` field are both required.
29
+
27
30
  ```bash
28
31
  # Step 1: View supported datasource component types. No spaceCode is required.
29
32
  ae-cli dataops_integration +list_datasource_components
@@ -34,18 +37,22 @@ ae-cli dataops_integration +get_datasource_component_template \
34
37
 
35
38
  # Step 3: Create datasource
36
39
  ae-cli dataops_integration +add_datasource --spaceCode "${spaceCode}" \
37
- --componentName "MySQL" --dataSourceName "Production MySQL" \
40
+ --componentName "MySQL" --dataSourceName "${datasourceName}" \
38
41
  --sharedConfig true \
39
- --envJsonList '[{"host":"localhost","port":3306,"username":"root","password":"xxx"}]'
42
+ --envJsonList '[{"jdbcUrl":"jdbc:mysql://<host>:3306/<database>","database":"<database>","username":"<username>","password":"<password>"}]'
40
43
 
41
44
  # Step 4: Test the saved datasource connection by name.
42
45
  ae-cli dataops_integration +test_datasource_connect --spaceCode "${spaceCode}" \
43
- --datasourceName "Production MySQL"
46
+ --datasourceName "${datasourceName}"
44
47
  ```
45
48
 
49
+ Read back the exact datasource name with `+get_datasource_detail`. Creation does not prove connectivity: require `connectStatus=SUCCESS` from the connection test and inspect `connectFails` otherwise. For `+online_datasource`, inspect both `successDataSourceNames` and `failDataSources`; a successful command envelope can contain individual failures.
50
+
46
51
  ---
47
52
 
48
- ## Workflow B: Create Sync Solution (Complete Process)
53
+ ## Workflow B: Create Sync Solution
54
+
55
+ Resolve the source/sink identities and source selection (table, filtered table, or custom query), target table and field mapping, and intended write mode before saving. Daily frequency does not imply append or overwrite: settle `dataSaveMode` from the user's intent, including the supported upsert keys when upsert is selected. Ask only for choices that remain unresolved after discovery. Use `+get_sync_detail --withParams true` for an existing solution's `usedParams` before deciding whether a manual run needs `baseDate` (`bd`).
49
56
 
50
57
  ### Step 1: View Available Datasources
51
58
 
@@ -63,6 +70,8 @@ ae-cli dataops_integration +list_sync_datasources --spaceCode "${spaceCode}" --e
63
70
 
64
71
  ### Step 2: Browse Source Table Structure
65
72
 
73
+ Current limitation: `+list_datasource_databases` cannot discover databases for a `te_etl@...` datasource because the backend requires `bizClassify`, which this CLI command does not expose. Report that step as unsupported; do not invent a flag or bypass the CLI through an undocumented API. The commands below cover ordinary external datasources. A catalog value must come from trusted configuration or discovery, not a guessed default.
74
+
66
75
  ```bash
67
76
  # List databases under datasource
68
77
  ae-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}" \
@@ -70,38 +79,38 @@ ae-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}"
70
79
 
71
80
  # List tables under database
72
81
  ae-cli dataops_integration +list_datasource_tables --spaceCode "${spaceCode}" \
73
- --datasourceId "${datasourceId}" --database "test"
82
+ --datasourceId "${datasourceId}" --database "${database}"
74
83
  # For catalog-based sources such as Databricks, add --catalog "${catalog}".
75
84
 
76
85
  # Get table structure (columns and partitionColumns)
77
86
  ae-cli dataops_integration +get_table_structure --spaceCode "${spaceCode}" \
78
- --datasourceId "${datasourceId}" --database "test" --tablePath "users"
87
+ --datasourceId "${datasourceId}" --database "${database}" --tablePath "${tablePath}"
79
88
  # For catalog-based sources such as Databricks, add --catalog "${catalog}".
80
89
  ```
81
90
 
82
91
  ### Step 3: Create Sync Solution
83
92
 
84
- **Key: Must strictly follow JSON templates below to generate parameters**
93
+ Generate the four JSON strings from the templates below with real discovered values, then place them in the shell variables used here. `sourceConfig.datasourceId` must equal `--srcDatasourceId`; `sinkConfig.datasourceId` must equal `--sinkDatasourceId`. Use the same selected `spaceCode` throughout. Do not execute template IDs or table names. Gaia fills `gatewayConfig` from the selected space on creation; do not supply a guessed `companyId` or copy another space's gateway settings.
85
94
 
86
95
  ```bash
87
96
  ae-cli dataops_integration +add_sync_solution --spaceCode "${spaceCode}" \
88
- --syncName "MySQL to Preset Repository Sync" \
97
+ --syncName "${syncName}" \
89
98
  --srcComponent "MySQL" --srcDatasourceId "${mysqlDatasourceId}" \
90
99
  --sinkComponent "te_etl" --sinkDatasourceId "te_etl@TASK_ENGINE_TRINO" \
91
- --sourceConfig '{"component":"MySQL","datasourceId":"xxx","database":"test","tablePath":"users","batchSize":1000}' \
92
- --sinkConfig '{"component":"te_etl","datasourceId":"te_etl@TASK_ENGINE_TRINO","database":"","tablePath":"ods_users_mysql","tableType":"PHYSICAL_TABLE","bizClassify":"CURRENT","dbBizType":"TASK_ENV_DB","authedSpace":"","partitionKeys":[],"dataSaveMode":1,"batchSize":20000}' \
93
- --channelConfig '{"limitType":"0","gatewayConfig":{"engineFlag":"TASK_ENGINE_TRINO","companyId":1,"appDefinition":"APP_GAIA","bizFlag":"BIZ_GAIA_TASK_RELEASE","repoCode":"te_etl","spaceCode":"default"}}' \
94
- --fieldsMapping '{"mapping":[{"source":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false},"target":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false}}]}'
100
+ --sourceConfig "${sourceConfigJson}" --sinkConfig "${sinkConfigJson}" \
101
+ --channelConfig "${channelConfigJson}" --fieldsMapping "${fieldsMappingJson}"
95
102
  ```
96
103
 
97
- ### Step 4: Execute Sync
104
+ On creation, take `syncId` from `data.result.syncId` in the default CLI JSON envelope, then read `+get_sync_detail` for that exact ID. This verifies the saved identity and exposed summary fields; it does not prove all original JSON fields or a successful run. Preserve the submitted complete configuration when later edits may be needed.
105
+
106
+ ### Step 4: Execute Sync Only When Requested
98
107
 
99
108
  ```bash
100
109
  # Manually execute sync solution
101
110
  ae-cli dataops_integration +exec_sync_solution --spaceCode "${spaceCode}" \
102
111
  --syncId "${syncId}" --baseDate "${baseDate}"
103
112
 
104
- # List manual sync runs
113
+ # Read back the submitted taskId among manual sync runs
105
114
  ae-cli dataops_integration +list_sync_runs --spaceCode "${spaceCode}" \
106
115
  --syncId "${syncId}" --limit 20
107
116
 
@@ -110,10 +119,32 @@ ae-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
110
119
  --syncId "${syncId}" --taskId "${taskId}"
111
120
  ```
112
121
 
122
+ Execution returns `data.result.taskId` and `data.result.status`. The outer `ok:true` and `data.status=SUCCESS` acknowledge the action, not completed data movement. Match that exact `taskId` in `data.runs[]` from `+list_sync_runs`; do not use the newest row without matching its ID. Run states are `WAIT`, `RUNNING`, or `RETRY` while active; `FINISHED` is successful completion, `FAILED` is failure, and `KILLED` is termination. After a stop request, read back the same task instead of treating the stop response as proof of termination. If the task is missing, has an unknown status, or remains active beyond the available wait, report the original `syncId/taskId` and last observed state; do not submit another run to obtain a clearer result.
123
+
124
+ ---
125
+
126
+ ## Workflow C: Schedule Daily Sync Without a Manual Run
127
+
128
+ For an intent such as "sync this MySQL table daily, publish it, but do not run it now":
129
+
130
+ 1. Complete datasource/solution discovery and configuration above, including target write mode and required date parameters. Reuse the exact existing solution when it already matches the intent.
131
+ 2. Follow [Create or Modify Integration Sync Task](dataops-flow-create.md#step-3-create-or-modify-integration-sync-task): create or select the intended flow and bind its DEV integration node to the real `syncId`.
132
+ 3. Follow [Configure Schedule](dataops-flow-create.md#step-7-configure-schedule), using the requested daily time and the platform's configured timezone. Resolve an unspecified or ambiguous time/timezone before setting the schedule.
133
+ 4. Preview the release, check its business result, and release using [Release to Production and Verify](dataops-flow-create.md#step-9-release-to-production-and-verify). Read `+get_flow_overview --env PROD` for the same `flowCode` and verify the visible integration node, dependencies, enabled schedule, and CRON. The overview does not expose the node's `syncId`; retain the create/update result's `taskCode/syncId` and available release-preview evidence for the binding, and state this readback limit.
134
+ 5. Finish with the saved `syncId`, published `flowCode`, verified visible PROD configuration, and the fact that no manual execution was submitted. Do not call `+exec_sync_solution` or `+execute_flow` for this intent. Publication establishes configuration readiness, not the success of a future scheduled run.
135
+
136
+ ## Workflow D: Update an Existing Sync Solution
137
+
138
+ First inspect the exact `syncId` with `+get_sync_detail --withParams true`. Its `source`, `sink`, and `fieldMapping` are summaries; `withParams` adds `usedParams`, not complete editable `sourceConfig`, `sinkConfig`, `channelConfig`, or `fieldsMapping`. Do not reverse-engineer a replacement payload from those summaries.
139
+
140
+ Use `+save_sync_solution` only when a trusted, complete original configuration for that same solution is available, such as the retained creation payload, and there is no indication it has since changed. Change the requested values in that configuration and preserve the remaining fields, including channel settings and field mappings. If the original is unavailable or stale, report the current CLI/backend capability gap and request the complete current configuration; do not guess it or create a replacement solution. After saving, read back the same `syncId` and verify the exposed changed fields, explicitly identifying any fields the summary cannot verify.
141
+
113
142
  ---
114
143
 
115
144
  ## JSON Configuration Templates
116
145
 
146
+ These examples show structure, not real resource identities. Replace `ds-id`, database/table names, field names/types, and sample query/filter values from the selected resources. Keep component-specific key names and JSON value types. The MySQL query source intentionally omits `database` and `tablePath`.
147
+
117
148
  ### Preset Repository as Source (sourceConfig)
118
149
 
119
150
  ```json
@@ -228,22 +259,18 @@ MySQL Sink requires non-empty `database` and `tablePath` and must not contain a
228
259
  `1000` to `10000`; omit it to use default `1000`. Strings, enum names, decimals,
229
260
  booleans, `null`, and out-of-range values are rejected.
230
261
 
231
- ### channelConfig (must include gatewayConfig when source or target involves preset repository)
262
+ ### channelConfig
263
+
264
+ For a new solution, Gaia generates `gatewayConfig` from its space. The minimal no-limit example is:
232
265
 
233
266
  ```json
234
267
  {
235
- "limitType": "0",
236
- "gatewayConfig": {
237
- "engineFlag": "TASK_ENGINE_TRINO",
238
- "companyId": 1,
239
- "appDefinition": "APP_GAIA",
240
- "bizFlag": "BIZ_GAIA_TASK_RELEASE",
241
- "repoCode": "te_etl",
242
- "spaceCode": "default"
243
- }
268
+ "limitType": "0"
244
269
  }
245
270
  ```
246
271
 
272
+ For updates, preserve the complete trusted original channel settings. If `gatewayConfig` is omitted, Gaia retains the existing gateway settings or derives them from the space; never fabricate tenant or space values.
273
+
247
274
  ### fieldsMapping (bidirectional column mapping)
248
275
 
249
276
  ```json
@@ -310,12 +337,12 @@ booleans, `null`, and out-of-range values are rejected.
310
337
  - **Datasource modification**: `+modify_datasource` creates no preview; it updates only provided optional fields (`risk: write`). Use `--envJsonList` with the same JSON array format as `+add_datasource`.
311
338
  - **Datasource online**: `+online_datasource` requires `--spaceCode` and `--dataSourceNames`. It executes directly (`risk: write`) and returns `failDataSources` and `successDataSourceNames`.
312
339
  - **Sync datasources**: `+list_sync_datasources` requires `--spaceCode`; `--env` is optional and defaults to `DEV`. It returns `sourceComponentSet` and `sinkComponentSet`, grouped by component, with `dataSourceList` and `supportableComponent`.
313
- - **Datasource databases**: `+list_datasource_databases` requires `--spaceCode` and `--datasourceId`; `--catalog` and `--env` are optional, and `--env` defaults to `DEV`. It returns an array of objects with `databaseName`.
340
+ - **Datasource databases**: `+list_datasource_databases` requires `--spaceCode` and `--datasourceId`; `--catalog` and `--env` are optional, and `--env` defaults to `DEV`. It returns an array of objects with `databaseName`. Current `te_etl@...` database discovery is unsupported because the CLI lacks the required `bizClassify` parameter.
314
341
  - **Datasource tables**: `+list_datasource_tables` requires `--spaceCode`, `--datasourceId`, and `--database`; `--catalog` and `--env` are optional, and `--env` defaults to `DEV`. It returns table metadata including `database`, `tableName`, `tableType`, `tableComment`, `engine`, `disabled`, `disabledReasons`, `sameVersion`, and `supportSharding`.
315
342
  - **Datasource table structure**: `+get_table_structure` requires `--spaceCode`, `--datasourceId`, `--database`, and `--tablePath`; `--catalog` and `--env` are optional, and `--env` defaults to `DEV`. It returns `columns` and `partitionColumns`.
316
343
  - **Sync solution list**: `+list_sync_solutions` requires only `--spaceCode`. It returns sync metadata including `syncId`, `syncName`, source/sink datasource and table fields, last execution/schedule status codes, owner, remark, and timestamps.
317
- - **Sync detail**: `+get_sync_detail` requires `--spaceCode` and `--syncId`; `--withParams` is optional and defaults to `false`. It returns source, sink, field mapping, last execution/schedule status, owner, and `nextAction`; `withParams=true` also returns `usedParams`.
318
- - **Sync update**: `+save_sync_solution` requires `--spaceCode`, `--syncId`, `--sourceConfig`, and `--sinkConfig`; `--syncName`, `--channelConfig`, `--fieldsMapping`, and `--remark` are optional. `syncName` is accepted for compatibility but ignored; the current name is preserved. `sourceConfig` and `sinkConfig` must be complete JSON strings from `+get_sync_detail --withParams true`; pass complete `channelConfig` and `fieldsMapping` when keeping or updating them.
344
+ - **Sync detail**: `+get_sync_detail` requires `--spaceCode` and `--syncId`; `--withParams` is optional and defaults to `false`. It returns source, sink, and field mapping summaries, last execution/schedule status, owner, and `nextAction`; `withParams=true` also returns `usedParams`. It does not return complete editable configs.
345
+ - **Sync update**: `+save_sync_solution` requires `--spaceCode`, `--syncId`, `--sourceConfig`, and `--sinkConfig`; `--syncName`, `--channelConfig`, `--fieldsMapping`, and `--remark` are optional. `syncName` is accepted for compatibility but ignored; the current name is preserved. `sourceConfig` and `sinkConfig` must be complete JSON strings from a trusted original configuration, not reconstructed from `+get_sync_detail`; pass complete `channelConfig` and `fieldsMapping` when keeping or updating them. Follow [the update workflow](#workflow-d-update-an-existing-sync-solution) when the original configuration is unavailable.
319
346
  - **Sync runs**: `+list_sync_runs` requires `--spaceCode` and `--syncId`; `--limit` is optional and defaults to `20`. It returns `runs`, `returnedCount`, `limit`, and `nextAction`; each run includes `taskId`, `execType`, `status`, `execTime`, `channelMode`, and `submitter`.
320
347
  - **Stop sync run**: `+stop_sync_solution` requires `--spaceCode`, `--syncId`, and `--taskId`. Use `taskId` from `+list_sync_runs` for an active run. It returns `action`, `result` with `execStatus`, `syncId`, and `taskId`, and top-level `status`.
321
348
 
@@ -354,7 +381,7 @@ Sharded cluster:
354
381
 
355
382
  1. **Table name rule**: When writing to preset repository, if table name not specified, use `ods_${source_table_name}_${component_name_lowercase}`
356
383
  2. **tablePath**: PostgreSQL uses `schema.table_name`, other components use table name directly
357
- 3. **Preset repository database is empty**, non-preset repository database is required
358
- 4. **channelConfig**: Must include gatewayConfig when involving preset repository
384
+ 3. **Database**: Preset repository configurations use an empty database. External table-mode source/sink configurations require a database; MySQL custom query sources must omit database and the other table-only fields.
385
+ 4. **channelConfig**: Gaia builds gatewayConfig from the selected space on creation. Preserve trusted channel settings on update; do not guess companyId or copy gateway settings from another space.
359
386
  5. **Field mapping**: Each field object must include manual/partitionKey/primaryKey/shardingKey/sortingKey/upsertKey properties
360
387
  6. **Conditional required parameters**: Some components (e.g., MongoDB) have additional required fields based on mode, see "Component Conditional Required Parameters" above
@@ -61,24 +61,41 @@ ae-cli dataops_ide +search_tables --spaceCode "${spaceCode}" --searchKey "user"
61
61
 
62
62
  ## Workflow C: Execute SQL Query (Download-Centered Async Flow)
63
63
 
64
- Use this flow for exactly one read-only SQL query. Result rows are not returned through MCP/CLI; submit creates a Gaia download-center task directly. The query semantics are preserved, but the result remains platform-bounded; when present, `downloadRowLimit` reports that cap. This is not an unlimited or full export.
64
+ Use this flow for exactly one read-only SQL query. Result rows are not returned through MCP/CLI; submit creates a Gaia download-center task directly. The result remains platform-bounded; when present, `downloadRowLimit` reports that cap. This is not an unlimited or full export.
65
+
66
+ For a question such as "how many orders yesterday?", first discover the table and date/count columns, establish DEV/PROD and the business date/time zone, and use an aggregate query when appropriate. Ask only about unresolved choices that change the answer. The existing query endpoint performs platform parameter replacement; single-quote any literal `${...}` intended for the platform so the shell does not expand it first.
65
67
 
66
68
  ```bash
67
69
  # Step 1: Submit SQL and create a download task. Defaults: repoCode=te_etl, engineType=TASK_ENGINE_TRINO.
68
70
  ae-cli dataops_ide +submit_sql_query --spaceCode "${spaceCode}" --repoCode "te_etl" \
69
- --sql "SELECT * FROM hive.ws_default_dev.dwd_user LIMIT 10" \
71
+ --sql "${verifiedReadOnlySql}" \
70
72
  --engineType "TASK_ENGINE_TRINO"
71
73
 
72
- # Step 2: Poll the download task status by spaceCode/downloadTaskId. Rows are not returned through MCP/CLI.
74
+ # Step 2: Continue only when submission returned data.downloadTaskId, not data.status=REJECTED.
75
+ # Poll the same download task. Rows are not returned through MCP/CLI.
73
76
  ae-cli dataops_ide +get_sql_query_status --spaceCode "${spaceCode}" --downloadTaskId ${downloadTaskId}
74
77
 
75
78
  # Step 3: CLI-only streaming save after downloadStatus=SUCCESS. The target is replaced only after the stream completes.
76
79
  ae-cli dataops_ide +get_sql_query_status --spaceCode "${spaceCode}" --downloadTaskId ${downloadTaskId} --downloadTo "./result.zip"
77
80
 
78
- # (Optional) Cancel the download task.
81
+ # Optional: cancel only when requested, not automatically when the local wait budget expires.
79
82
  ae-cli dataops_ide +cancel_sql_query --spaceCode "${spaceCode}" --downloadTaskId ${downloadTaskId}
80
83
  ```
81
84
 
85
+ Read the actual business state even when outer `ok:true` is present:
86
+
87
+ | Response | Next action |
88
+ |---|---|
89
+ | Submit `data.status=REJECTED` or no `data.downloadTaskId` | Report the returned reason/requestId; do not invent an ID or start polling |
90
+ | Status `data.downloadStatus=WAITING/RUNNING/FAILED_RETRY` | Continue bounded polling of the same ID; do not resubmit SQL |
91
+ | Status `UNKNOWN` | Recheck the same task within the wait budget; unresolved is not success or proof that it never ran |
92
+ | Status `FAILED/CANCELLED` | Stop and report the message and task ID; do not silently resubmit |
93
+ | Status `SUCCESS` | Download if the user needs the result; status alone is not the result data |
94
+
95
+ After `--downloadTo`, require a returned `data.localFile` and inspect the actual ZIP contents. Use a fresh local target; do not assume the internal filename, encoding, delimiter, or headers. For a data question, read the result and answer using its real values and agreed scope; do not end by giving only a download task ID or asking the user to read the file. For an export request, provide the saved file and disclose any result limit. If downloading or reading fails, report that boundary without inventing rows.
96
+
97
+ When the wait budget expires, return the original `downloadTaskId` and latest observed status with the status command needed to continue. Do not equate a locally timed-out wait with a cancelled or failed server task.
98
+
82
99
  ---
83
100
 
84
101
  ## Command Quick Reference