@thinkingai/ae-cli 6.1.24 → 6.1.25

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (135) hide show
  1. package/README.md +7 -1
  2. package/README.zh.md +7 -1
  3. package/dist/{auth-XZNXFRJS.js → auth-LHM7NQPR.js} +167 -7
  4. package/dist/{capability-FYTU3X6L.js → capability-SJI5KOCK.js} +11 -6
  5. package/dist/{chunk-CCS66K4K.js → chunk-2FJM4HRV.js} +3 -3
  6. package/dist/{chunk-57RRFUEZ.js → chunk-5XRQ3CZR.js} +4 -4
  7. package/dist/{chunk-EQ255HKO.js → chunk-7WZACOAI.js} +2 -2
  8. package/dist/{chunk-DX5CDO34.js → chunk-BW7JUDEI.js} +84 -19
  9. package/dist/{chunk-7XGFSUOM.js → chunk-DU72X2IO.js} +271 -76
  10. package/dist/{chunk-V2NUHPXF.js → chunk-GK7WDY7E.js} +1 -1
  11. package/dist/{chunk-7FTTULED.js → chunk-IG3DYEIR.js} +4 -1
  12. package/dist/{chunk-HBGADCKA.js → chunk-KJDOTPYU.js} +3 -3
  13. package/dist/{chunk-4NRCTUZJ.js → chunk-KS4DE3MX.js} +4 -4
  14. package/dist/{chunk-JOCDD4ON.js → chunk-N4BLPJC7.js} +1 -1
  15. package/dist/{chunk-HFGZGYCG.js → chunk-QYXDJOLA.js} +252 -6
  16. package/dist/{chunk-DVMLWQD4.js → chunk-SO5LKIWP.js} +2 -2
  17. package/dist/{chunk-VYXXOKYS.js → chunk-XKIW3QMF.js} +7 -4
  18. package/dist/{chunk-MAPZ7VB6.js → chunk-YVGXGTD6.js} +13 -0
  19. package/dist/{community-report-client-CU22W53K.js → community-report-client-6VJDDMJJ.js} +1 -1
  20. package/dist/{config-XO3LI3SA.js → config-XQWOPDHN.js} +25 -6
  21. package/dist/{context-IFJI3LIT.js → context-M3HDGEIR.js} +4 -4
  22. package/dist/{data-integration-HNGUKFIC.js → data-integration-XBTCGXQT.js} +6 -6
  23. package/dist/index.js +78 -41
  24. package/dist/{local-data-upload-client-GTRIT64F.js → local-data-upload-client-3P7NIVQ4.js} +1 -1
  25. package/dist/{memory-QGPSNCQS.js → memory-6APEB2JB.js} +4 -4
  26. package/dist/{metadata-DI5BMYFU.js → metadata-JRYKNUS5.js} +7 -7
  27. package/dist/{model-RNPQXIBI.js → model-ZS7HVGKN.js} +4 -4
  28. package/dist/{personal-semantic-preference-5C3WUXFI.js → personal-semantic-preference-IDE2GE5T.js} +8 -6
  29. package/dist/project-semantic-36KOCYN7.js +356 -0
  30. package/dist/{sync-O45NXEQY.js → sync-EQLSTUS7.js} +7 -7
  31. package/dist/{te-agent-K5MSPNHJ.js → te-agent-Z3HCTKR7.js} +348 -55
  32. package/dist/{te-analysis-AISWCOOZ.js → te-analysis-5ZZ6YHVH.js} +940 -144
  33. package/dist/{te-community-4OYU3BDA.js → te-community-Z7JACGF3.js} +3 -3
  34. package/dist/{te-dataops-NFIPB6FX.js → te-dataops-F7GQOIS6.js} +644 -276
  35. package/dist/{te-engage-BAM3GBFX.js → te-engage-RNRMYDO3.js} +6 -6
  36. package/dist/{te-experiment-GUTZXZ5Z.js → te-experiment-BTAC2UNK.js} +6 -6
  37. package/dist/{te-kb-3TS73NO6.js → te-kb-QJLJEX5K.js} +323 -181
  38. package/dist/{te-system-STMHLS5J.js → te-system-IXC43QHZ.js} +6 -6
  39. package/dist/{te-team-3I5AOX3N.js → te-team-BIZIKIHZ.js} +6 -6
  40. package/dist/{update-RZXLPCRH.js → update-W2SGQNKT.js} +4 -4
  41. package/package.json +11 -4
  42. package/skills/ae-agent/SKILL.md +9 -4
  43. package/skills/ae-agent/references/notification.md +40 -0
  44. package/skills/ae-analysis/SKILL.md +43 -20
  45. package/skills/ae-analysis/references/adhoc_export.md +1 -1
  46. package/skills/ae-analysis/references/adhoc_run.md +1 -1
  47. package/skills/ae-analysis/references/agent_review_submit_to_page.md +2 -0
  48. package/skills/ae-analysis/references/ai_models/event.md +6 -1
  49. package/skills/ae-analysis/references/ai_models/funnel.md +7 -1
  50. package/skills/ae-analysis/references/ai_models/heat_map.md +23 -0
  51. package/skills/ae-analysis/references/ai_models/interval.md +24 -0
  52. package/skills/ae-analysis/references/ai_models/path.md +20 -0
  53. package/skills/ae-analysis/references/ai_models/prop_analysis.md +2 -0
  54. package/skills/ae-analysis/references/ai_models/rank_list.md +30 -0
  55. package/skills/ae-analysis/references/ai_models/retention.md +45 -1
  56. package/skills/ae-analysis/references/ai_models/revenue.md +20 -2
  57. package/skills/ae-analysis/references/ai_models/session.md +76 -0
  58. package/skills/ae-analysis/references/ai_models/tag.md +2 -0
  59. package/skills/ae-analysis/references/ai_models.md +73 -3
  60. package/skills/ae-analysis/references/analysis_data_retrieval.md +4 -0
  61. package/skills/ae-analysis/references/analysis_drilldown_contract.md +4 -1
  62. package/skills/ae-analysis/references/asset_batch_info_export.md +10 -0
  63. package/skills/ae-analysis/references/asset_batch_sql_export.md +13 -1
  64. package/skills/ae-analysis/references/asset_export.md +12 -1
  65. package/skills/ae-analysis/references/asset_list.md +1 -0
  66. package/skills/ae-analysis/references/asset_search.md +11 -5
  67. package/skills/ae-analysis/references/bi_panel_list.md +1 -1
  68. package/skills/ae-analysis/references/catalog_list.md +1 -1
  69. package/skills/ae-analysis/references/collaboration.md +48 -0
  70. package/skills/ae-analysis/references/command_index.md +19 -17
  71. package/skills/ae-analysis/references/dashboard_list.md +4 -4
  72. package/skills/ae-analysis/references/dashboard_report_data_export.md +2 -0
  73. package/skills/ae-analysis/references/drilldown_session_details_run.md +41 -0
  74. package/skills/ae-analysis/references/event_export.md +3 -1
  75. package/skills/ae-analysis/references/governance_recommendation_auto_review.md +77 -0
  76. package/skills/ae-analysis/references/governance_recommendation_export.md +11 -0
  77. package/skills/ae-analysis/references/metadata_resolution.md +3 -3
  78. package/skills/ae-analysis/references/metric_export.md +3 -1
  79. package/skills/ae-analysis/references/metric_list.md +1 -1
  80. package/skills/ae-analysis/references/operation_record_export.md +10 -0
  81. package/skills/ae-analysis/references/personal_semantic_preference_add.md +2 -2
  82. package/skills/ae-analysis/references/personal_semantic_preference_get.md +4 -4
  83. package/skills/ae-analysis/references/personal_semantic_preference_list.md +4 -4
  84. package/skills/ae-analysis/references/project_semantic_knowledge_wiki.md +29 -11
  85. package/skills/ae-analysis/references/project_semantic_knowledge_wiki_plan_schema.md +59 -0
  86. package/skills/ae-analysis/references/property_export.md +3 -1
  87. package/skills/ae-analysis/references/report_create.md +5 -1
  88. package/skills/ae-analysis/references/report_data_export.md +6 -0
  89. package/skills/ae-analysis/references/report_get.md +2 -0
  90. package/skills/ae-analysis/references/report_list.md +5 -5
  91. package/skills/ae-analysis/references/report_update.md +7 -1
  92. package/skills/ae-analysis/references/sql_table_columns.md +4 -4
  93. package/skills/ae-analysis/references/sql_table_list.md +5 -5
  94. package/skills/ae-analysis/references/user_cluster_models.md +8 -0
  95. package/skills/ae-analysis/references/user_tag_create.md +7 -1
  96. package/skills/ae-analysis/references/user_tag_get.md +1 -1
  97. package/skills/ae-analysis/references/user_tag_models.md +18 -2
  98. package/skills/ae-analysis/references/user_tag_refresh.md +1 -1
  99. package/skills/ae-analysis/references/user_tag_update.md +2 -2
  100. package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/build-project-semantic-wiki.mjs +337 -12
  101. package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/default-compile-rules.md +7 -2
  102. package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/generate-build-ir.mjs +284 -24
  103. package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/package-wiki-source-zip.mjs +185 -22
  104. package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/precompiled-source.mjs +46 -2
  105. package/skills/ae-capability/SKILL.md +63 -1
  106. package/skills/ae-capability/references/collaboration.md +48 -0
  107. package/skills/ae-community/SKILL.md +5 -1
  108. package/skills/ae-community/references/collaboration.md +48 -0
  109. package/skills/ae-data-integration/SKILL.md +4 -0
  110. package/skills/ae-data-integration/references/collaboration.md +48 -0
  111. package/skills/ae-dataops/SKILL.md +54 -74
  112. package/skills/ae-dataops/references/collaboration.md +48 -0
  113. package/skills/ae-dataops/references/dataops-backfill.md +23 -18
  114. package/skills/ae-dataops/references/dataops-flow-create.md +74 -23
  115. package/skills/ae-dataops/references/dataops-flow-monitor.md +45 -13
  116. package/skills/ae-dataops/references/dataops-integration.md +57 -30
  117. package/skills/ae-dataops/references/dataops-query.md +21 -4
  118. package/skills/ae-dataops/references/dataops-table.md +143 -11
  119. package/skills/ae-engage/SKILL.md +21 -6
  120. package/skills/ae-engage/references/channel-mgmt.md +3 -3
  121. package/skills/ae-engage/references/collaboration.md +48 -0
  122. package/skills/ae-engage/references/scene-config-channel.md +4 -4
  123. package/skills/ae-generate-tracking-plan/SKILL.md +4 -0
  124. package/skills/ae-generate-tracking-plan/references/collaboration.md +48 -0
  125. package/skills/ae-kb/SKILL.md +9 -1
  126. package/skills/ae-kb/references/collaboration.md +48 -0
  127. package/skills/ae-kb/references/schema-import.md +30 -0
  128. package/skills/ae-kb-discovery/SKILL.md +7 -2
  129. package/skills/ae-kb-discovery/references/collaboration.md +48 -0
  130. package/skills/ae-metadata/SKILL.md +20 -12
  131. package/skills/ae-metadata/references/collaboration.md +48 -0
  132. package/dist/project-semantic-3LF6Q6JZ.js +0 -1119
  133. package/skills/ae-project-semantic/SKILL.md +0 -193
  134. package/skills/ae-project-semantic/references/query-routing-v5.md +0 -165
  135. package/skills/ae-project-semantic/references/recommendation-quality.md +0 -68
@@ -18,6 +18,23 @@ Use `dataops_flow` to execute and control flows. Use `dataops_operations` for sp
18
18
  - **flowInstanceId** — Operations workflow instance ID returned by `dataops_operations +search_flow_instances`; use it for operations detail, task logs, and stop
19
19
  - **Instance list queries PROD environment by default** (unlike other tools which default to DEV)
20
20
 
21
+ In the default CLI JSON, take `executeId` from `data.result.executeId` and an already available `flowInstanceId` from `data.result.flowInstanceId`. Otherwise select `data.instances[]` from operations search and keep its `flowCode` and `flowInstanceId` together. Instance detail supplies `data.taskInstances[].taskInstanceId` for task inspection; a definition's `taskCode` is not a task instance ID.
22
+
23
+ ### Status fields are command-specific
24
+
25
+ The paths below are relative to the command payload, as used by CLI `--jq`. Prefix them with `.data` when reading the default JSON envelope with an external jq command.
26
+
27
+ | Command | Execution status path | Trigger path |
28
+ |---|---|---|
29
+ | `+list_flows` | `.flows[].latestProductionInstance.flowInstanceStatus` | `.flows[].latestProductionInstance.historyCmd` |
30
+ | `+search_flow_instances` | `.instances[].status` | `.instances[].triggerType` |
31
+ | `+get_flow_instance_detail` | `.flowInstance.status`; task states in `.taskInstances[].status` | `.flowInstance.triggerType` |
32
+ | `+get_task_instance_detail` | `.status` | Use its flow instance |
33
+
34
+ `flowStatus` is the workflow definition state, such as `RELEASED`, not execution completion. `flowScheduleStatus` is a separate scheduling state; neither it nor the latest-flow summary replaces inspection of a specific run.
35
+
36
+ If the selected instance's execution status is missing, null, or unknown, stop status-based polling and inspect the same read command without `--jq`. A missing field projects to null without a jq error; `ok:true` does not make that status usable. If the raw status remains unavailable, report uncertainty rather than treating it as running or successful. Do not apply this rule to legitimately nullable fields such as `endTime`.
37
+
21
38
  ---
22
39
 
23
40
  ## Workflow A: Find and View Flow Execution Status
@@ -43,6 +60,17 @@ ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
43
60
  --flowCode ${flowCode} --flowInstanceId ${flowInstanceId}
44
61
  ```
45
62
 
63
+ `flowKeyword` must be a discovered flow name, remark, or instance ID; search does not match `flowCode`. Filter returned instances by the exact `flowCode` and the requested execution/base date, following `hasMore` when needed. For a newly submitted run, compare the submission time and prior instance list; if several candidates still match, report ambiguity instead of inspecting an arbitrary latest instance.
64
+
65
+ Once the intended instance is verified, keep its `flowInstanceId` fixed and poll its detail, not all matching instances. A previous instance's `SUCCESS` cannot prove that the requested run completed. If the current CLI cannot correlate `executeId` to one instance confidently, retain the execution handle and report the limitation.
66
+
67
+ ```bash
68
+ # Inspect only the previously verified instance; reuse these IDs while polling.
69
+ ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
70
+ --flowCode "${flowCode}" --flowInstanceId "${flowInstanceId}" \
71
+ --jq '{flowInstanceId: .flowInstance.flowInstanceId, status: .flowInstance.status}'
72
+ ```
73
+
46
74
  ---
47
75
 
48
76
  ## Workflow B: Troubleshoot Task Failures
@@ -56,31 +84,35 @@ ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
56
84
  ae-cli dataops_operations +get_task_instance_detail --spaceCode "${spaceCode}" \
57
85
  --flowCode ${flowCode} --flowInstanceId ${flowInstanceId} \
58
86
  --taskInstanceId ${taskInstanceId} --includeLog true
59
-
60
- # Step 3: Re-execute after fixing
61
- ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
62
- --flowCode ${flowCode}
63
87
  ```
64
88
 
89
+ Report the failed task, historical instance/base date, and relevant log evidence first. These commands cannot rerun only failed nodes of that historical instance. Do not substitute `+execute_flow` (a new whole-flow PROD run) or a backfill job for that request. If the user explicitly requests a new whole-flow run, establish its business date, release any required DEV fixes using [the release workflow](dataops-flow-create.md#step-8-preview-release), and continue below; pass `--baseDate` for a requested historical date rather than assuming today's date is equivalent.
90
+
65
91
  ---
66
92
 
67
93
  ## Workflow C: Manual Execution and Stop
68
94
 
69
95
  ```bash
70
- # Manually trigger PROD execution
96
+ # Run only when the user requested a new whole-flow PROD execution
71
97
  ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
72
98
  --flowCode ${flowCode}
73
- # Returns executeId
99
+ # data.result.operationStatus=SUCCESS means accepted; retain data.result.executeId
74
100
 
75
101
  # Monitor recent executions
76
102
  ae-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
77
103
  --keyword "${flowKeyword}" --startDate "${startDate}" --endDate "${endDate}" --pageSize 20
78
104
 
79
- # Stop execution if necessary (irreversible)
105
+ # Stop only when requested, using the verified execution handle
80
106
  ae-cli dataops_operations +stop_flow_instance --spaceCode "${spaceCode}" \
81
107
  --flowCode ${flowCode} --executeId ${executeId}
82
108
  ```
83
109
 
110
+ An accepted execution is not a completed run. Inspect `data.result.operationStatus`; a non-`SUCCESS` result stops the submission path. Once the instance is identified, read `data.flowInstance.status` from `+get_flow_instance_detail`: `SUCCESS` is successful execution, `FAIL` is failure, and `STOP` or `PAUSE` is not success. `RUNNING` and `READY_PAUSE` are unfinished flow instance states. Task detail uses `data.status`; instance detail uses `data.taskInstances[].status`. A successful read of a failed instance is still a successful query.
111
+
112
+ For `+search_flow_instances --status`, use only `RUNNING,SUCCESS,FAIL,READY_PAUSE,PAUSE,STOP`. `WAITING` filtering is not supported by the current backend; an empty result with that filter does not establish that no work is waiting. Do not infer support from a different command's scheduling or download status.
113
+
114
+ When completion is requested, set a finite polling deadline and use read-only instance queries. Stop on a terminal/paused state, an actionable error, or the deadline; then report the IDs, last observed state, and relevant failure evidence. A temporarily absent instance or a timeout does not authorize another `+execute_flow`; retain the original `executeId` and report uncertain correlation or completion. Read the instance again after a stop request to distinguish stop acceptance from an observed `STOP` state.
115
+
84
116
  ---
85
117
 
86
118
  ## Workflow D: View Flow Structure
@@ -107,19 +139,19 @@ ae-cli dataops_flow +get_flow_overview --spaceCode "${spaceCode}" \
107
139
  | `dataops_operations +get_flow_instance_detail` | Operations instance DAG and task statuses | `--spaceCode` `--flowCode` `--flowInstanceId` |
108
140
  | `dataops_operations +get_task_instance_detail` | Operations task detail and optional logs | `--spaceCode` `--flowCode` `--flowInstanceId` (`--taskInstanceId` or `--taskCode` or `--taskName`) `[--includeLog]` |
109
141
  | `dataops_operations +stop_flow_instance` | Stop a running execution | `--spaceCode` `--flowCode` exactly one of `--executeId` or `--flowInstanceId` |
110
- | `+get_task_params` | DEV task parameter list | `--spaceCode` `--flowCode` `--taskCode` |
111
142
  | `+update_flow` | Update DEV workflow name and/or remark | `--spaceCode` `--flowCode` `[--flowName]` `[--remark]` |
112
143
 
113
144
  ## Parameter Notes
114
145
 
115
146
  - **Execution**: `+execute_flow` requires `--spaceCode` and `--flowCode`; `--baseDate` is optional and maps to runtime parameter `bd`. It always runs PROD and returns `action/result/status`; `result` includes `flowCode`, `executeId`, `operationStatus`, `nextAction`, and optional `flowInstanceId`.
116
147
  - **Stop flow instance**: `dataops_operations +stop_flow_instance` requires `--spaceCode`, `--flowCode`, and exactly one of `--executeId` or `--flowInstanceId`. `--executeId` comes from `dataops_flow +execute_flow`; `--flowInstanceId` comes from `dataops_operations +search_flow_instances`. It returns `success`, `selector`, and selector-specific stop result fields.
117
- - **Flow overview**: `+get_flow_overview` requires `--spaceCode` and either `--flowCode` or exact `--flowName`; `--flowCode` takes precedence. `--env` defaults to `DEV`; use `PROD` to include latest production instance fields when available. It returns `success`, `env`, `resolvedBy`, `flow`, `schedule`, `dag`, and `summary`.
148
+ - **Flow overview**: `+get_flow_overview` requires `--spaceCode` and either `--flowCode` or exact `--flowName`; `--flowCode` takes precedence. `--env` defaults to `DEV`; use `PROD` to include latest production instance fields when available. It requires `dwWorkflowEdit` in both environments and returns `success`, `env`, `resolvedBy`, `flow`, `flowParams`, `schedule`, `dag`, and `summary`. `flowParams` contains all custom flow parameters, including unused ones; `dag.tasks[].taskParams` contains each task's references with `paramKey`, `paramType`, `paramDataType`, `paramFrom`, `paramValue`, `paramDesc`, and built-in flags such as `isBd`. Both lists use the selected environment and are empty arrays when no parameters exist. Definition values preserve expression text: a FLOW reference to `p=${bd}` keeps `paramDataType=QUOTE` and `paramValue=${bd}`; it is not an execution date.
118
149
  - **Flow update**: `+update_flow` requires `--spaceCode`, `--flowCode`, and at least one of `--flowName` or `--remark`. It returns `action/result/status`; `result` is an array with items containing `flowCode`, `operationStatus`, `nameChanged`, and optional `flowName`.
119
150
  - **High-frequency release flows**: `+list_high_frequency_release_flows` requires `--spaceCode`; `--days`, `--topN`, `--minCount`, and `--status` are optional. Defaults are `days=30`, `topN=10`, and `status=SUCCESS`. It returns `period`, `filters`, `flows`, `returnedCount`, and `nextAction`; flow items include `rank`, `flowCode`, `flowName`, `releaseCount`, `lastReleaseTime`, and `avgIntervalHours`.
120
- - **Operations instance search**: `dataops_operations +search_flow_instances` requires `--spaceCode`; `--keyword`, `--startDate`, `--endDate`, `--status`, `--pageNum`, and `--pageSize` are optional. `--keyword` fuzzy-matches `flowName` or `flowCode`. `pageNum` defaults to `1`; `pageSize` defaults to `20` and maxes at `100`. It returns `totalCount`, `returnedCount`, `pageNum`, `pageSize`, `hasMore`, `instances`, `statusCounts`, `triggerTypeCounts`, and `ownerCounts`.
121
- - **Task instance detail**: `dataops_operations +get_task_instance_detail` requires `--spaceCode`, `--flowCode`, `--flowInstanceId`, and one selector: `--taskInstanceId`, `--taskCode`, or exact `--taskName`. Prefer `--taskInstanceId` because retries can create multiple instances with the same task code/name. `--includeLog` is optional and defaults to `false`. It returns `success`, `flowInstanceId`, `taskCode`, `taskName`, `taskInstanceId`, `status`, `task`, `taskInstance`, `definition`, and `log` only when requested.
151
+ - **Operations instance search**: `dataops_operations +search_flow_instances` requires `--spaceCode`; `--keyword`, `--startDate`, `--endDate`, `--status`, `--pageNum`, and `--pageSize` are optional. `--keyword` fuzzy-matches instance ID, workflow name, or workflow remark, not `flowCode`; filter the returned `flowCode` exactly. `pageNum` defaults to `1`; `pageSize` defaults to `20` and maxes at `100`. It returns `totalCount`, `returnedCount`, `pageNum`, `pageSize`, `hasMore`, `instances`, `statusCounts`, `triggerTypeCounts`, and `ownerCounts`.
152
+ - **Flow instance detail**: `dataops_operations +get_flow_instance_detail` requires `dwOMInstanceView` and returns the historical execution snapshot in `flowInstance.instanceParamMap`. A missing snapshot is null or omitted; the server does not substitute current parameter definitions. Later definition changes do not change historical execution values.
153
+ - **Task instance detail**: `dataops_operations +get_task_instance_detail` requires `--spaceCode`, `--flowCode`, `--flowInstanceId`, and one selector: `--taskInstanceId`, `--taskCode`, or exact `--taskName`. Prefer `--taskInstanceId` because retries can create multiple instances with the same task code/name. It requires `dwOMInstanceView`. `--includeLog` is optional and defaults to `false`; when enabled, `log.params` contains the task's historical actual execution values. It returns `success`, `flowInstanceId`, `taskCode`, `taskName`, `taskInstanceId`, `status`, `task`, `taskInstance`, `definition`, and `log` only when requested.
122
154
  - **env**: `DEV` (development) | `PROD` (production, instance list defaults to PROD)
123
155
  - **flow list paging**: `+list_flows` requires `--spaceCode`; `--keyword`, `--pageNum`, and `--pageSize` are optional. It returns `flows`, `totalCount`, `returnedCount`, `pageNum`, `pageSize`, and `hasMore`; flow items include `latestProductionInstance` only when available. `pageSize` defaults to `20` and maxes at `100`.
124
- - **Task parameters**: `+get_task_params` requires `--spaceCode`, `--flowCode`, and `--taskCode`; it has no optional flags. It queries DEV and returns `data` as an array. Items include fields such as `paramKey`, `paramType`, `paramDataType`, `paramFrom`, and built-in flags like `isBd`.
125
- - **Task Status**: `success` / `failure` / `running` / `waiting`
156
+ - **Task parameter sources**: Read `dag.tasks[].taskParams` from `+get_flow_overview`. `paramFrom=FLOW` means a flow definition supplies the value; `SPACE` means a space parameter does. A code-parsed reference remains in the list when neither source exists; its `paramFrom` and `paramValue` fields are null or omitted, both meaning no configured source/value. Deleting a flow parameter falls back to a same-name space parameter, or to code-parsed status. Unpublished DEV changes do not alter PROD values or sources.
157
+ - **Task Status**: `WAITING` / `RUNNING` / `SUCCESS` / `FAIL` / `STOP` / `PAUSE`; only `SUCCESS` means successful completion.
@@ -16,14 +16,17 @@ Use the `dataops_integration` subcommand to manage datasources and sync solution
16
16
  **Core Rules:**
17
17
  - Configuration parameters differ significantly between preset repositories (te_etl) and non-preset repositories, must strictly follow templates
18
18
  - sourceConfig/sinkConfig/channelConfig/fieldsMapping are all JSON strings
19
- - Sync solution updates are not partial patches: call `+get_sync_detail --withParams true` first, then submit complete sourceConfig and sinkConfig JSON. Pass channelConfig and fieldsMapping when keeping or updating them
19
+ - Sync solution updates are not partial patches. The current detail command returns a summary, not complete editable configs; follow [Update an Existing Sync Solution](#workflow-d-update-an-existing-sync-solution).
20
20
  - Test datasource connection before creating sync solution
21
21
  - `sinkConfig.dataSaveMode` codes are fixed: `1 = APPEND_DATA` (insert/append new data; UI label `插入新数据`), `2 = OVERWRITE` (`DROP_DATA`; UI label `覆盖写入`). Never describe `dataSaveMode=1` as overwrite; use `2` for overwrite.
22
+ - Integration `--dry-run` only previews the local request. It does not validate nested JSON, permissions, connections, or execution readiness on the server.
22
23
 
23
24
  ---
24
25
 
25
26
  ## Workflow A: Create Datasource
26
27
 
28
+ Use the selected component's returned template. The values in angle brackets below are illustrative; replace them with the user's real connection settings before execution. For MySQL, `jdbcUrl` and the separate `database` field are both required.
29
+
27
30
  ```bash
28
31
  # Step 1: View supported datasource component types. No spaceCode is required.
29
32
  ae-cli dataops_integration +list_datasource_components
@@ -34,18 +37,22 @@ ae-cli dataops_integration +get_datasource_component_template \
34
37
 
35
38
  # Step 3: Create datasource
36
39
  ae-cli dataops_integration +add_datasource --spaceCode "${spaceCode}" \
37
- --componentName "MySQL" --dataSourceName "Production MySQL" \
40
+ --componentName "MySQL" --dataSourceName "${datasourceName}" \
38
41
  --sharedConfig true \
39
- --envJsonList '[{"host":"localhost","port":3306,"username":"root","password":"xxx"}]'
42
+ --envJsonList '[{"jdbcUrl":"jdbc:mysql://<host>:3306/<database>","database":"<database>","username":"<username>","password":"<password>"}]'
40
43
 
41
44
  # Step 4: Test the saved datasource connection by name.
42
45
  ae-cli dataops_integration +test_datasource_connect --spaceCode "${spaceCode}" \
43
- --datasourceName "Production MySQL"
46
+ --datasourceName "${datasourceName}"
44
47
  ```
45
48
 
49
+ Read back the exact datasource name with `+get_datasource_detail`. Creation does not prove connectivity: require `connectStatus=SUCCESS` from the connection test and inspect `connectFails` otherwise. For `+online_datasource`, inspect both `successDataSourceNames` and `failDataSources`; a successful command envelope can contain individual failures.
50
+
46
51
  ---
47
52
 
48
- ## Workflow B: Create Sync Solution (Complete Process)
53
+ ## Workflow B: Create Sync Solution
54
+
55
+ Resolve the source/sink identities and source selection (table, filtered table, or custom query), target table and field mapping, and intended write mode before saving. Daily frequency does not imply append or overwrite: settle `dataSaveMode` from the user's intent, including the supported upsert keys when upsert is selected. Ask only for choices that remain unresolved after discovery. Use `+get_sync_detail --withParams true` for an existing solution's `usedParams` before deciding whether a manual run needs `baseDate` (`bd`).
49
56
 
50
57
  ### Step 1: View Available Datasources
51
58
 
@@ -63,6 +70,8 @@ ae-cli dataops_integration +list_sync_datasources --spaceCode "${spaceCode}" --e
63
70
 
64
71
  ### Step 2: Browse Source Table Structure
65
72
 
73
+ Current limitation: `+list_datasource_databases` cannot discover databases for a `te_etl@...` datasource because the backend requires `bizClassify`, which this CLI command does not expose. Report that step as unsupported; do not invent a flag or bypass the CLI through an undocumented API. The commands below cover ordinary external datasources. A catalog value must come from trusted configuration or discovery, not a guessed default.
74
+
66
75
  ```bash
67
76
  # List databases under datasource
68
77
  ae-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}" \
@@ -70,38 +79,38 @@ ae-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}"
70
79
 
71
80
  # List tables under database
72
81
  ae-cli dataops_integration +list_datasource_tables --spaceCode "${spaceCode}" \
73
- --datasourceId "${datasourceId}" --database "test"
82
+ --datasourceId "${datasourceId}" --database "${database}"
74
83
  # For catalog-based sources such as Databricks, add --catalog "${catalog}".
75
84
 
76
85
  # Get table structure (columns and partitionColumns)
77
86
  ae-cli dataops_integration +get_table_structure --spaceCode "${spaceCode}" \
78
- --datasourceId "${datasourceId}" --database "test" --tablePath "users"
87
+ --datasourceId "${datasourceId}" --database "${database}" --tablePath "${tablePath}"
79
88
  # For catalog-based sources such as Databricks, add --catalog "${catalog}".
80
89
  ```
81
90
 
82
91
  ### Step 3: Create Sync Solution
83
92
 
84
- **Key: Must strictly follow JSON templates below to generate parameters**
93
+ Generate the four JSON strings from the templates below with real discovered values, then place them in the shell variables used here. `sourceConfig.datasourceId` must equal `--srcDatasourceId`; `sinkConfig.datasourceId` must equal `--sinkDatasourceId`. Use the same selected `spaceCode` throughout. Do not execute template IDs or table names. Gaia fills `gatewayConfig` from the selected space on creation; do not supply a guessed `companyId` or copy another space's gateway settings.
85
94
 
86
95
  ```bash
87
96
  ae-cli dataops_integration +add_sync_solution --spaceCode "${spaceCode}" \
88
- --syncName "MySQL to Preset Repository Sync" \
97
+ --syncName "${syncName}" \
89
98
  --srcComponent "MySQL" --srcDatasourceId "${mysqlDatasourceId}" \
90
99
  --sinkComponent "te_etl" --sinkDatasourceId "te_etl@TASK_ENGINE_TRINO" \
91
- --sourceConfig '{"component":"MySQL","datasourceId":"xxx","database":"test","tablePath":"users","batchSize":1000}' \
92
- --sinkConfig '{"component":"te_etl","datasourceId":"te_etl@TASK_ENGINE_TRINO","database":"","tablePath":"ods_users_mysql","tableType":"PHYSICAL_TABLE","bizClassify":"CURRENT","dbBizType":"TASK_ENV_DB","authedSpace":"","partitionKeys":[],"dataSaveMode":1,"batchSize":20000}' \
93
- --channelConfig '{"limitType":"0","gatewayConfig":{"engineFlag":"TASK_ENGINE_TRINO","companyId":1,"appDefinition":"APP_GAIA","bizFlag":"BIZ_GAIA_TASK_RELEASE","repoCode":"te_etl","spaceCode":"default"}}' \
94
- --fieldsMapping '{"mapping":[{"source":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false},"target":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false}}]}'
100
+ --sourceConfig "${sourceConfigJson}" --sinkConfig "${sinkConfigJson}" \
101
+ --channelConfig "${channelConfigJson}" --fieldsMapping "${fieldsMappingJson}"
95
102
  ```
96
103
 
97
- ### Step 4: Execute Sync
104
+ On creation, take `syncId` from `data.result.syncId` in the default CLI JSON envelope, then read `+get_sync_detail` for that exact ID. This verifies the saved identity and exposed summary fields; it does not prove all original JSON fields or a successful run. Preserve the submitted complete configuration when later edits may be needed.
105
+
106
+ ### Step 4: Execute Sync Only When Requested
98
107
 
99
108
  ```bash
100
109
  # Manually execute sync solution
101
110
  ae-cli dataops_integration +exec_sync_solution --spaceCode "${spaceCode}" \
102
111
  --syncId "${syncId}" --baseDate "${baseDate}"
103
112
 
104
- # List manual sync runs
113
+ # Read back the submitted taskId among manual sync runs
105
114
  ae-cli dataops_integration +list_sync_runs --spaceCode "${spaceCode}" \
106
115
  --syncId "${syncId}" --limit 20
107
116
 
@@ -110,10 +119,32 @@ ae-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
110
119
  --syncId "${syncId}" --taskId "${taskId}"
111
120
  ```
112
121
 
122
+ Execution returns `data.result.taskId` and `data.result.status`. The outer `ok:true` and `data.status=SUCCESS` acknowledge the action, not completed data movement. Match that exact `taskId` in `data.runs[]` from `+list_sync_runs`; do not use the newest row without matching its ID. Run states are `WAIT`, `RUNNING`, or `RETRY` while active; `FINISHED` is successful completion, `FAILED` is failure, and `KILLED` is termination. After a stop request, read back the same task instead of treating the stop response as proof of termination. If the task is missing, has an unknown status, or remains active beyond the available wait, report the original `syncId/taskId` and last observed state; do not submit another run to obtain a clearer result.
123
+
124
+ ---
125
+
126
+ ## Workflow C: Schedule Daily Sync Without a Manual Run
127
+
128
+ For an intent such as "sync this MySQL table daily, publish it, but do not run it now":
129
+
130
+ 1. Complete datasource/solution discovery and configuration above, including target write mode and required date parameters. Reuse the exact existing solution when it already matches the intent.
131
+ 2. Follow [Create or Modify Integration Sync Task](dataops-flow-create.md#step-3-create-or-modify-integration-sync-task): create or select the intended flow and bind its DEV integration node to the real `syncId`.
132
+ 3. Follow [Configure Schedule](dataops-flow-create.md#step-7-configure-schedule), using the requested daily time and the platform's configured timezone. Resolve an unspecified or ambiguous time/timezone before setting the schedule.
133
+ 4. Preview the release, check its business result, and release using [Release to Production and Verify](dataops-flow-create.md#step-9-release-to-production-and-verify). Read `+get_flow_overview --env PROD` for the same `flowCode` and verify the visible integration node, dependencies, enabled schedule, and CRON. The overview does not expose the node's `syncId`; retain the create/update result's `taskCode/syncId` and available release-preview evidence for the binding, and state this readback limit.
134
+ 5. Finish with the saved `syncId`, published `flowCode`, verified visible PROD configuration, and the fact that no manual execution was submitted. Do not call `+exec_sync_solution` or `+execute_flow` for this intent. Publication establishes configuration readiness, not the success of a future scheduled run.
135
+
136
+ ## Workflow D: Update an Existing Sync Solution
137
+
138
+ First inspect the exact `syncId` with `+get_sync_detail --withParams true`. Its `source`, `sink`, and `fieldMapping` are summaries; `withParams` adds `usedParams`, not complete editable `sourceConfig`, `sinkConfig`, `channelConfig`, or `fieldsMapping`. Do not reverse-engineer a replacement payload from those summaries.
139
+
140
+ Use `+save_sync_solution` only when a trusted, complete original configuration for that same solution is available, such as the retained creation payload, and there is no indication it has since changed. Change the requested values in that configuration and preserve the remaining fields, including channel settings and field mappings. If the original is unavailable or stale, report the current CLI/backend capability gap and request the complete current configuration; do not guess it or create a replacement solution. After saving, read back the same `syncId` and verify the exposed changed fields, explicitly identifying any fields the summary cannot verify.
141
+
113
142
  ---
114
143
 
115
144
  ## JSON Configuration Templates
116
145
 
146
+ These examples show structure, not real resource identities. Replace `ds-id`, database/table names, field names/types, and sample query/filter values from the selected resources. Keep component-specific key names and JSON value types. The MySQL query source intentionally omits `database` and `tablePath`.
147
+
117
148
  ### Preset Repository as Source (sourceConfig)
118
149
 
119
150
  ```json
@@ -228,22 +259,18 @@ MySQL Sink requires non-empty `database` and `tablePath` and must not contain a
228
259
  `1000` to `10000`; omit it to use default `1000`. Strings, enum names, decimals,
229
260
  booleans, `null`, and out-of-range values are rejected.
230
261
 
231
- ### channelConfig (must include gatewayConfig when source or target involves preset repository)
262
+ ### channelConfig
263
+
264
+ For a new solution, Gaia generates `gatewayConfig` from its space. The minimal no-limit example is:
232
265
 
233
266
  ```json
234
267
  {
235
- "limitType": "0",
236
- "gatewayConfig": {
237
- "engineFlag": "TASK_ENGINE_TRINO",
238
- "companyId": 1,
239
- "appDefinition": "APP_GAIA",
240
- "bizFlag": "BIZ_GAIA_TASK_RELEASE",
241
- "repoCode": "te_etl",
242
- "spaceCode": "default"
243
- }
268
+ "limitType": "0"
244
269
  }
245
270
  ```
246
271
 
272
+ For updates, preserve the complete trusted original channel settings. If `gatewayConfig` is omitted, Gaia retains the existing gateway settings or derives them from the space; never fabricate tenant or space values.
273
+
247
274
  ### fieldsMapping (bidirectional column mapping)
248
275
 
249
276
  ```json
@@ -310,12 +337,12 @@ booleans, `null`, and out-of-range values are rejected.
310
337
  - **Datasource modification**: `+modify_datasource` creates no preview; it updates only provided optional fields (`risk: write`). Use `--envJsonList` with the same JSON array format as `+add_datasource`.
311
338
  - **Datasource online**: `+online_datasource` requires `--spaceCode` and `--dataSourceNames`. It executes directly (`risk: write`) and returns `failDataSources` and `successDataSourceNames`.
312
339
  - **Sync datasources**: `+list_sync_datasources` requires `--spaceCode`; `--env` is optional and defaults to `DEV`. It returns `sourceComponentSet` and `sinkComponentSet`, grouped by component, with `dataSourceList` and `supportableComponent`.
313
- - **Datasource databases**: `+list_datasource_databases` requires `--spaceCode` and `--datasourceId`; `--catalog` and `--env` are optional, and `--env` defaults to `DEV`. It returns an array of objects with `databaseName`.
340
+ - **Datasource databases**: `+list_datasource_databases` requires `--spaceCode` and `--datasourceId`; `--catalog` and `--env` are optional, and `--env` defaults to `DEV`. It returns an array of objects with `databaseName`. Current `te_etl@...` database discovery is unsupported because the CLI lacks the required `bizClassify` parameter.
314
341
  - **Datasource tables**: `+list_datasource_tables` requires `--spaceCode`, `--datasourceId`, and `--database`; `--catalog` and `--env` are optional, and `--env` defaults to `DEV`. It returns table metadata including `database`, `tableName`, `tableType`, `tableComment`, `engine`, `disabled`, `disabledReasons`, `sameVersion`, and `supportSharding`.
315
342
  - **Datasource table structure**: `+get_table_structure` requires `--spaceCode`, `--datasourceId`, `--database`, and `--tablePath`; `--catalog` and `--env` are optional, and `--env` defaults to `DEV`. It returns `columns` and `partitionColumns`.
316
343
  - **Sync solution list**: `+list_sync_solutions` requires only `--spaceCode`. It returns sync metadata including `syncId`, `syncName`, source/sink datasource and table fields, last execution/schedule status codes, owner, remark, and timestamps.
317
- - **Sync detail**: `+get_sync_detail` requires `--spaceCode` and `--syncId`; `--withParams` is optional and defaults to `false`. It returns source, sink, field mapping, last execution/schedule status, owner, and `nextAction`; `withParams=true` also returns `usedParams`.
318
- - **Sync update**: `+save_sync_solution` requires `--spaceCode`, `--syncId`, `--sourceConfig`, and `--sinkConfig`; `--syncName`, `--channelConfig`, `--fieldsMapping`, and `--remark` are optional. `syncName` is accepted for compatibility but ignored; the current name is preserved. `sourceConfig` and `sinkConfig` must be complete JSON strings from `+get_sync_detail --withParams true`; pass complete `channelConfig` and `fieldsMapping` when keeping or updating them.
344
+ - **Sync detail**: `+get_sync_detail` requires `--spaceCode` and `--syncId`; `--withParams` is optional and defaults to `false`. It returns source, sink, and field mapping summaries, last execution/schedule status, owner, and `nextAction`; `withParams=true` also returns `usedParams`. It does not return complete editable configs.
345
+ - **Sync update**: `+save_sync_solution` requires `--spaceCode`, `--syncId`, `--sourceConfig`, and `--sinkConfig`; `--syncName`, `--channelConfig`, `--fieldsMapping`, and `--remark` are optional. `syncName` is accepted for compatibility but ignored; the current name is preserved. `sourceConfig` and `sinkConfig` must be complete JSON strings from a trusted original configuration, not reconstructed from `+get_sync_detail`; pass complete `channelConfig` and `fieldsMapping` when keeping or updating them. Follow [the update workflow](#workflow-d-update-an-existing-sync-solution) when the original configuration is unavailable.
319
346
  - **Sync runs**: `+list_sync_runs` requires `--spaceCode` and `--syncId`; `--limit` is optional and defaults to `20`. It returns `runs`, `returnedCount`, `limit`, and `nextAction`; each run includes `taskId`, `execType`, `status`, `execTime`, `channelMode`, and `submitter`.
320
347
  - **Stop sync run**: `+stop_sync_solution` requires `--spaceCode`, `--syncId`, and `--taskId`. Use `taskId` from `+list_sync_runs` for an active run. It returns `action`, `result` with `execStatus`, `syncId`, and `taskId`, and top-level `status`.
321
348
 
@@ -354,7 +381,7 @@ Sharded cluster:
354
381
 
355
382
  1. **Table name rule**: When writing to preset repository, if table name not specified, use `ods_${source_table_name}_${component_name_lowercase}`
356
383
  2. **tablePath**: PostgreSQL uses `schema.table_name`, other components use table name directly
357
- 3. **Preset repository database is empty**, non-preset repository database is required
358
- 4. **channelConfig**: Must include gatewayConfig when involving preset repository
384
+ 3. **Database**: Preset repository configurations use an empty database. External table-mode source/sink configurations require a database; MySQL custom query sources must omit database and the other table-only fields.
385
+ 4. **channelConfig**: Gaia builds gatewayConfig from the selected space on creation. Preserve trusted channel settings on update; do not guess companyId or copy gateway settings from another space.
359
386
  5. **Field mapping**: Each field object must include manual/partitionKey/primaryKey/shardingKey/sortingKey/upsertKey properties
360
387
  6. **Conditional required parameters**: Some components (e.g., MongoDB) have additional required fields based on mode, see "Component Conditional Required Parameters" above
@@ -61,24 +61,41 @@ ae-cli dataops_ide +search_tables --spaceCode "${spaceCode}" --searchKey "user"
61
61
 
62
62
  ## Workflow C: Execute SQL Query (Download-Centered Async Flow)
63
63
 
64
- Use this flow for exactly one read-only SQL query. Result rows are not returned through MCP/CLI; submit creates a Gaia download-center task directly. The query semantics are preserved, but the result remains platform-bounded; when present, `downloadRowLimit` reports that cap. This is not an unlimited or full export.
64
+ Use this flow for exactly one read-only SQL query. Result rows are not returned through MCP/CLI; submit creates a Gaia download-center task directly. The result remains platform-bounded; when present, `downloadRowLimit` reports that cap. This is not an unlimited or full export.
65
+
66
+ For a question such as "how many orders yesterday?", first discover the table and date/count columns, establish DEV/PROD and the business date/time zone, and use an aggregate query when appropriate. Ask only about unresolved choices that change the answer. The existing query endpoint performs platform parameter replacement; single-quote any literal `${...}` intended for the platform so the shell does not expand it first.
65
67
 
66
68
  ```bash
67
69
  # Step 1: Submit SQL and create a download task. Defaults: repoCode=te_etl, engineType=TASK_ENGINE_TRINO.
68
70
  ae-cli dataops_ide +submit_sql_query --spaceCode "${spaceCode}" --repoCode "te_etl" \
69
- --sql "SELECT * FROM hive.ws_default_dev.dwd_user LIMIT 10" \
71
+ --sql "${verifiedReadOnlySql}" \
70
72
  --engineType "TASK_ENGINE_TRINO"
71
73
 
72
- # Step 2: Poll the download task status by spaceCode/downloadTaskId. Rows are not returned through MCP/CLI.
74
+ # Step 2: Continue only when submission returned data.downloadTaskId, not data.status=REJECTED.
75
+ # Poll the same download task. Rows are not returned through MCP/CLI.
73
76
  ae-cli dataops_ide +get_sql_query_status --spaceCode "${spaceCode}" --downloadTaskId ${downloadTaskId}
74
77
 
75
78
  # Step 3: CLI-only streaming save after downloadStatus=SUCCESS. The target is replaced only after the stream completes.
76
79
  ae-cli dataops_ide +get_sql_query_status --spaceCode "${spaceCode}" --downloadTaskId ${downloadTaskId} --downloadTo "./result.zip"
77
80
 
78
- # (Optional) Cancel the download task.
81
+ # Optional: cancel only when requested, not automatically when the local wait budget expires.
79
82
  ae-cli dataops_ide +cancel_sql_query --spaceCode "${spaceCode}" --downloadTaskId ${downloadTaskId}
80
83
  ```
81
84
 
85
+ Read the actual business state even when outer `ok:true` is present:
86
+
87
+ | Response | Next action |
88
+ |---|---|
89
+ | Submit `data.status=REJECTED` or no `data.downloadTaskId` | Report the returned reason/requestId; do not invent an ID or start polling |
90
+ | Status `data.downloadStatus=WAITING/RUNNING/FAILED_RETRY` | Continue bounded polling of the same ID; do not resubmit SQL |
91
+ | Status `UNKNOWN` | Recheck the same task within the wait budget; unresolved is not success or proof that it never ran |
92
+ | Status `FAILED/CANCELLED` | Stop and report the message and task ID; do not silently resubmit |
93
+ | Status `SUCCESS` | Download if the user needs the result; status alone is not the result data |
94
+
95
+ After `--downloadTo`, require a returned `data.localFile` and inspect the actual ZIP contents. Use a fresh local target; do not assume the internal filename, encoding, delimiter, or headers. For a data question, read the result and answer using its real values and agreed scope; do not end by giving only a download task ID or asking the user to read the file. For an export request, provide the saved file and disclose any result limit. If downloading or reading fails, report that boundary without inventing rows.
96
+
97
+ When the wait budget expires, return the original `downloadTaskId` and latest observed status with the status command needed to continue. Do not equate a locally timed-out wait with a cancelled or failed server task.
98
+
82
99
  ---
83
100
 
84
101
  ## Command Quick Reference
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: dataops-table
3
- version: 1.0.0
4
- description: "Data table and view management: search tables, view table details, create physical tables/views. Trigger keywords: create table, table creation, view, data dictionary, table details, datatable, DDL."
3
+ version: 1.2.0
4
+ description: "Data table and view management: search and create tables/views, mutate one table field, recycle an entity, list the recycle bin, and permanently delete a recycled entity. Trigger keywords: table, view, column, recycle bin, permanent deletion, data dictionary, DDL."
5
5
  metadata:
6
6
  requires:
7
7
  bins: ["ae-cli"]
@@ -11,16 +11,19 @@ metadata:
11
11
 
12
12
  > **Prerequisites:** Read [`ae-dataops/SKILL.md`](../SKILL.md) for general rules.
13
13
 
14
- Use the `dataops_datatable` subcommand to manage data tables.
14
+ Use `dataops_datatable` for all commands in this reference, including table/view recycling and permanent deletion.
15
15
 
16
16
  **Core Rules:**
17
- - Creating/modifying/deleting workspace task tables **must use dataops_datatable**, prohibited to use dataops_ide
17
+ - Creating/modifying/deleting workspace task tables **must use the commands in this reference**, never `dataops_ide`
18
18
  - Use `+dict_search_tables` as the default discovery command for the visible DataOps catalog
19
19
  - Use `dataops_ide +search_tables` only when raw engine-side metadata is needed
20
20
  - Use `dataops_ide +ide_list_tables` only after catalog/schema are known and you need schema browsing
21
21
  - Confirm no table with the same name exists before creation
22
22
  - When creating views or data tables, generate DDL according to **Trino DDL specifications**
23
23
  - `+create_table` and `+create_view` create objects in the DEV environment only. Publish PROD separately with `+publish_entity`.
24
+ - Field mutation commands resolve one TASK_ENV Hive physical table by `spaceCode + tableName` and change DEV only. Publish PROD separately.
25
+ - Never add, modify, or delete a partition field. Field rename and batch mutation are not supported.
26
+ - Recycling and permanent deletion affect the exact entity's existing DEV and PRODUCT mappings immediately; they are not DEV-only edits awaiting publication.
24
27
 
25
28
  ---
26
29
 
@@ -70,15 +73,131 @@ ae-cli dataops_datatable +publish_entity --spaceCode "${spaceCode}" \
70
73
 
71
74
  ---
72
75
 
76
+ ## Workflow D: Add, Modify, or Delete One Field
77
+
78
+ Read the current DEV table first. Call the command once per field and wait for each actual result before changing the next field.
79
+
80
+ ```bash
81
+ ae-cli dataops_datatable +get_table_detail --spaceCode "${spaceCode}" \
82
+ --tableName "orders" --manageMode TASK_ENV --env DEV --entityType TABLE
83
+
84
+ # Add one nullable ordinary field at the end. --dry-run performs a server semantic preview.
85
+ ae-cli dataops_datatable +add_table_field --spaceCode "${spaceCode}" \
86
+ --tableName "orders" --fieldName "amount" --fieldType "decimal(18,2)" \
87
+ --comment "Order amount" --dry-run
88
+ ae-cli dataops_datatable +add_table_field --spaceCode "${spaceCode}" \
89
+ --tableName "orders" --fieldName "amount" --fieldType "decimal(18,2)" \
90
+ --comment "Order amount"
91
+
92
+ # Change type and comment together. Omit an option to preserve that attribute.
93
+ ae-cli dataops_datatable +modify_table_field --spaceCode "${spaceCode}" \
94
+ --tableName "orders" --fieldName "amount" --fieldType "double" \
95
+ --comment "Order amount in settlement currency" --dry-run
96
+ ae-cli dataops_datatable +modify_table_field --spaceCode "${spaceCode}" \
97
+ --tableName "orders" --fieldName "amount" --fieldType "double" \
98
+ --comment "Order amount in settlement currency"
99
+
100
+ # Clear only the physical field comment.
101
+ ae-cli dataops_datatable +modify_table_field --spaceCode "${spaceCode}" \
102
+ --tableName "orders" --fieldName "amount" --clearComment
103
+
104
+ # Preview deletion without confirmation. Execute with --yes only after explicit user confirmation.
105
+ ae-cli dataops_datatable +delete_table_field --spaceCode "${spaceCode}" \
106
+ --tableName "orders" --fieldName "amount" --dry-run
107
+ ae-cli dataops_datatable +delete_table_field --spaceCode "${spaceCode}" \
108
+ --tableName "orders" --fieldName "amount" --yes
109
+ ```
110
+
111
+ The semantic preview returns `executable`, `wouldChange`, `plannedDiff`, blockers, and warnings without writing. Execution returns the actual post-readback `outcome`: `CHANGED`, `UNCHANGED`, `PARTIAL`, or `FAILED`. `UNCHANGED` includes a reason. `PARTIAL` and `FAILED` are command failures and exit non-zero.
112
+
113
+ Partition field mutations are always blocked. Standard TASK_ENV Hive external tables are supported, but `EXTERNAL_FILE_SCHEMA_NOT_VERIFIED` means the external file schema was not validated or rewritten. A type change also reports `DATA_COMPATIBILITY_NOT_VERIFIED`.
114
+
115
+ ---
116
+
117
+ ## Entity Lifecycle: Scope and Safety
118
+
119
+ These commands accept one TASK_ENV entity in the current space's built-in `te_etl` / `hive` warehouse: an ordinary physical table (including external tables) or an ordinary view. View SQL modification, restore, batch deletion, clearing the whole recycle bin, and cascading dependency deletion are not supported.
120
+
121
+ Both writes require `--spaceCode`, `--entityId`, and `--name`. The server verifies that the ID and name identify the same current-space entity; a missing ID never falls back to a new same-name object. `--dry-run` sends `preview: true` for validation without deletion, and does not require `--yes`.
122
+
123
+ `RECYCLE_NAME_CONFLICT` means an old same-name entity already exists in the recycle bin. Inspect the recycle bin and obtain separate explicit authorization before deleting that old entity. Recycling never deletes it automatically. Do not change IDs or retry by name to bypass a conflict.
124
+
125
+ The recycle-bin list keeps different entity IDs separate even when their names match. Each entry includes `entityId`, `name`, `entityType`, `environments`, and `recycleTime`; totals include `totalCount`, `returnedCount`, and `hasMore`. The server defaults to 100 results, maximum 1000. Refine `--search` if truncated.
126
+
127
+ An already recycled entity returns `UNCHANGED` with a reason. Permanent deletion accepts only recycled entities, not active or mixed-state entities. Writes return `PREVIEW`, `CHANGED`, `UNCHANGED`, `FAILED`, or `PARTIAL`; actual state changes appear in `diff`. `FAILED` and `PARTIAL` exit non-zero. Re-read the same entity ID after a failure; deletion across storage and metadata is not atomic. Existing physical-table locking is reused, but this does not add a shared lock across all view operations.
128
+
129
+ Permissions: recycle requires `dwDataTableEdit`, list requires `dwDataTableView`, and permanent deletion requires `dwDeleteTable`. External file deletion remains governed by existing engine behavior; the CLI adds no external-file cleanup action.
130
+
131
+ ### Workflow E: Recycle a Table or View
132
+
133
+ An ordinary "delete table/view" request follows this workflow only. It does not authorize permanent deletion.
134
+
135
+ Find the active entity with `+dict_search_tables` and inspect `+get_table_detail`. Verify TASK_ENV scope and the existing DEV/PRODUCT mappings. Copy its exact `entityId` and name; never derive an ID from a name. Then preview:
136
+
137
+ ```bash
138
+ ae-cli dataops_datatable +entity_recycle --spaceCode "${spaceCode}" \
139
+ --entityId "${entityId}" --name "${name}" --dry-run
140
+ ```
141
+
142
+ Explain the planned changes and that execution immediately affects existing DEV and PRODUCT mappings. Only after explicit user confirmation, execute:
143
+
144
+ ```bash
145
+ ae-cli dataops_datatable +entity_recycle --spaceCode "${spaceCode}" \
146
+ --entityId "${entityId}" --name "${name}" --yes
147
+ ```
148
+
149
+ Inspect the actual outcome and confirm the exact recycled entity in `+recycle_bin_list`. Report `UNCHANGED` as already recycled, not as a new change. Do not infer completion from a preview, HTTP success, or a name match alone. On `FAILED`, `PARTIAL`, or unverified state, report the observed result, reconcile the same entity ID, and stop without offering permanent deletion.
150
+
151
+ After actual recycling is confirmed, the Agent may ask:
152
+
153
+ > The entity is in the recycle bin. Would you like to permanently delete it? This is irreversible and may delete internal table data.
154
+
155
+ This question belongs in the Agent's reply, not CLI output. Stop here unless the user gives a new explicit confirmation; a successful recycle is not permission to run the next workflow.
156
+
157
+ ### Workflow F: Permanently Delete a Recycled Table or View
158
+
159
+ Use this workflow only for an explicit permanent-deletion request or a new affirmative answer to the post-recycle question. Never append it automatically to ordinary deletion.
160
+
161
+ Recycled objects are absent from the active table catalog. List the recycle bin, then copy and verify the exact `entityId`, name, entity type, and affected environments. If the object is active or mixed-state, stop; do not silently recycle it or choose a same-name recycled object.
162
+
163
+ ```bash
164
+ ae-cli dataops_datatable +recycle_bin_list --spaceCode "${spaceCode}" \
165
+ --search "${name}" --maxResults 100
166
+ ```
167
+
168
+ Preview only the verified recycled object:
169
+
170
+ ```bash
171
+ ae-cli dataops_datatable +recycle_bin_delete --spaceCode "${spaceCode}" \
172
+ --entityId "${recycledEntityId}" --name "${name}" --dry-run
173
+ ```
174
+
175
+ Explain the irreversible effect, including possible internal table data deletion, and obtain explicit confirmation for this exact target and environment scope before execution. Recycling authorization does not authorize permanent deletion.
176
+
177
+ ```bash
178
+ ae-cli dataops_datatable +recycle_bin_delete --spaceCode "${spaceCode}" \
179
+ --entityId "${recycledEntityId}" --name "${name}" --yes
180
+ ```
181
+
182
+ Inspect the actual outcome and `diff`, then read the recycle bin again for the same ID. A truncated list or a missing name alone is not proof of deletion. On failure or partial completion, report the observed state and reconcile the same ID before any retry; do not switch to a newly created same-name entity.
183
+
184
+ ---
185
+
73
186
  ## Command Quick Reference
74
187
 
75
- | Command | Purpose | Key Flags |
76
- |---------|---------|-----------|
77
- | `+dict_search_tables` | DataOps table catalog search, default 50 results | `--spaceCode` `--search` `--maxResults` |
78
- | `+get_table_detail` | DataOps catalog detail | `--spaceCode` `--tableName` `--manageMode` `--env` `--entityType TABLE|VIEW` |
79
- | `+create_table` | Create DataOps physical table in DEV | `--spaceCode` `--ddl` |
80
- | `+create_view` | Create DataOps view in DEV | `--spaceCode` `--ddl` |
81
- | `+publish_entity` | Publish table/view from DEV to PROD | `--spaceCode` `--name` `[--entityId]` `[--entityType]` |
188
+ | Command | Purpose | Risk | Key Flags |
189
+ |---------|---------|------|-----------|
190
+ | `ae-cli dataops_datatable +dict_search_tables` | DataOps table catalog search, default 50 results | read | `--spaceCode` `--search` `--maxResults` |
191
+ | `ae-cli dataops_datatable +get_table_detail` | DataOps catalog detail | read | `--spaceCode` `--tableName` `--manageMode` `--env` `--entityType TABLE\|VIEW` |
192
+ | `ae-cli dataops_datatable +create_table` | Create DataOps physical table in DEV | write | `--spaceCode` `--ddl` |
193
+ | `ae-cli dataops_datatable +create_view` | Create DataOps view in DEV | write | `--spaceCode` `--ddl` |
194
+ | `ae-cli dataops_datatable +publish_entity` | Publish table/view from DEV to PROD | write | `--spaceCode` `--name` `[--entityId]` `[--entityType]` |
195
+ | `ae-cli dataops_datatable +add_table_field` | Append one ordinary field in DEV | write | `--spaceCode` `--tableName` `--fieldName` `--fieldType` `[--comment]` |
196
+ | `ae-cli dataops_datatable +modify_table_field` | Change one ordinary field type and/or comment in DEV | write | `--spaceCode` `--tableName` `--fieldName` `[--fieldType]` `[--comment\|--clearComment]` |
197
+ | `ae-cli dataops_datatable +delete_table_field` | Delete one ordinary field in DEV | high-risk-write | `--spaceCode` `--tableName` `--fieldName`; `--yes` after confirmation |
198
+ | `ae-cli dataops_datatable +entity_recycle` | Move one table/view and its DEV/PRODUCT mappings to the recycle bin | high-risk-write | `--spaceCode` `--entityId` `--name`; `--yes` after confirmation |
199
+ | `ae-cli dataops_datatable +recycle_bin_list` | Find recycled entities by ID, including same-name objects | read | `--spaceCode` `[--search]` `[--maxResults]` |
200
+ | `ae-cli dataops_datatable +recycle_bin_delete` | Permanently delete one recycled entity | high-risk-write | `--spaceCode` `--entityId` `--name`; `--yes` after confirmation |
82
201
 
83
202
  ## Parameter Notes
84
203
 
@@ -87,5 +206,18 @@ ae-cli dataops_datatable +publish_entity --spaceCode "${spaceCode}" \
87
206
  - **create_table**: Requires `--spaceCode` and `--ddl`; no optional flags. Creates a DataOps physical table in DEV only. The backend parses Trino-compatible DDL and saves TASK_ENV metadata in the default workspace warehouse (`repo=te_etl`, `catalog=hive`). Publish by name with `+publish_entity --name <tableName>`.
88
207
  - **create_view**: Requires `--spaceCode` and `--ddl`; no optional command flags. Creates a DataOps view in DEV only. The backend saves TASK_ENV metadata through the DataView save flow in the default workspace warehouse (`repo=te_etl`, `catalog=hive`). Publish by name with `+publish_entity --name <viewName>`. Keep the literal `${env}` placeholder when referencing current-space task tables, for example `ws_${spaceCode}_${env}`.
89
208
  - **publish_entity**: Requires `--spaceCode` and `--name`. Publishes one existing TASK_ENV table/view from DEV to PROD. Optional `--entityId` disambiguates same-name matches; optional `--entityType TABLE|VIEW` validates the resolved type. Returns `action/result/status`; result includes published ids/names and `ONLINE` status, or `errorType`/`candidates`/`errors`.
209
+ - **add_table_field**: Requires an exact table and field name plus one complete Trino-compatible `--fieldType`. Optional `--comment` sets the physical column comment. An identical existing definition returns `UNCHANGED`; a conflicting existing field returns `FIELD_ALREADY_EXISTS`.
210
+ - **modify_table_field**: Requires at least one of `--fieldType`, `--comment`, or `--clearComment`. `--comment` and `--clearComment` are mutually exclusive. A missing field returns `FIELD_NOT_FOUND`; the command never adds or upserts it.
211
+ - **delete_table_field**: High-risk write. Run the semantic preview first, obtain explicit user confirmation, and only then use `--yes`. A missing field returns `FIELD_NOT_FOUND`.
212
+ - **field mutation result**: Only the target field and attributes that actually changed appear in `diff`. None of the three field mutation commands accepts `entityId`, `expectedVersion`, environment, rename, position, default value, `NOT NULL`, or a field array.
90
213
  - **schema naming**: DEV environment uses `ws_${spaceCode}_dev`, PROD environment uses `ws_${spaceCode}_product`. Do not hardcode either schema in current-space view DDL; use `ws_${spaceCode}_${env}` with literal `${env}`.
91
214
  - **Table name rule**: `^[a-z][0-9a-z_]{0,127}$`
215
+
216
+ ## Transitional Status
217
+
218
+ - Transition status: transitional
219
+ - Owning module: gaia-mcp-datatable
220
+ - Current transport: Gaia CLI REST
221
+ - Gateway target: TBD
222
+ - Review after: 2026-12-04
223
+ - Exit condition: Equivalent Gateway capabilities preserve semantic preview, single-field input, exact entity-ID lifecycle operations, recycle-bin discovery, deletion confirmation, and structured actual outcomes.