@thinkingai/ae-cli 6.0.18 → 6.0.22
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -10
- package/README.zh.md +9 -11
- package/dist/{auth-DYC2BGLM.js → auth-I2G7OAU2.js} +2 -4
- package/dist/{auth-GHBMJ5ZR.js → auth-RDN6D5HU.js} +32 -104
- package/dist/{chunk-5OC7CYCG.js → chunk-2W7HULUD.js} +14 -14
- package/dist/chunk-4EUZRJPO.js +163 -0
- package/dist/{chunk-DMMEPCST.js → chunk-5FBLRSMU.js} +25 -115
- package/dist/{chunk-BJX3GXWC.js → chunk-E34XH4GW.js} +2 -2
- package/dist/chunk-FPE5LGVU.js +105 -0
- package/dist/{chunk-I5CRVM2S.js → chunk-H52CSLJW.js} +44 -17
- package/dist/{chunk-RSHMZ3LG.js → chunk-U6QPKCN3.js} +6 -20
- package/dist/{chunk-6W4QOZAL.js → chunk-WPXHCP4A.js} +23 -13
- package/dist/{chunk-PN3OS34E.js → chunk-XE2PI5DS.js} +1 -1
- package/dist/{chunk-2DUEAXV3.js → chunk-YACCPUNQ.js} +1 -1
- package/dist/cli-token-6AGEDJF4.js +14 -0
- package/dist/{client-2UJ63MVQ.js → client-VTJMUZ4O.js} +3 -3
- package/dist/{config-NCUGK6NS.js → config-2SDTLGOC.js} +3 -3
- package/dist/index.js +314 -197
- package/dist/metadata-JZ2OTWBK.js +420 -0
- package/dist/{model-SJSOZTRS.js → model-ZYMRN4VS.js} +2 -2
- package/dist/{raw-7NVWSVJP.js → raw-NGYHAL3F.js} +4 -4
- package/dist/{sync-GRVO4K3U.js → sync-JVLX6TGW.js} +3 -3
- package/dist/{te-agent-KZRIEN4N.js → te-agent-QPZTLHPH.js} +545 -21
- package/dist/{te-analysis-VAR4KUOK.js → te-analysis-WWX3Z6RP.js} +352 -52
- package/dist/te-audience-UXGHUVBB.js +599 -0
- package/dist/{te-common-WNCZB7R7.js → te-common-7XXH73MF.js} +5 -4
- package/dist/{te-community-JEEFXPB3.js → te-community-QKE67HR2.js} +5 -4
- package/dist/te-dataops-6W7NNYST.js +1841 -0
- package/dist/{te-engage-4XHCTS6I.js → te-engage-NCFAQFX3.js} +5 -4
- package/dist/{te-kb-B62KFP2A.js → te-kb-K5BQYUCM.js} +50 -26
- package/dist/{te-meta-EPZGQN56.js → te-meta-LJZBOSMO.js} +48 -24
- package/dist/{te-team-YY24VORI.js → te-team-DHK5R7MV.js} +5 -6
- package/package.json +3 -2
- package/skills/ae-agent/SKILL.md +103 -4
- package/skills/ae-analysis/SKILL.md +75 -21
- package/skills/ae-analysis/references/build_attribution_analysis_qp.md +62 -0
- package/skills/ae-analysis/references/build_cluster_definition.md +62 -0
- package/skills/ae-analysis/references/build_distribution_analysis_qp.md +67 -0
- package/skills/ae-analysis/references/build_event_analysis_qp.md +2 -0
- package/skills/ae-analysis/references/build_funnel_analysis_qp.md +2 -0
- package/skills/ae-analysis/references/build_heat_map_analysis_qp.md +54 -0
- package/skills/ae-analysis/references/build_interval_analysis_qp.md +60 -0
- package/skills/ae-analysis/references/build_path_analysis_qp.md +53 -0
- package/skills/ae-analysis/references/build_prop_analysis_qp.md +2 -0
- package/skills/ae-analysis/references/build_rank_list_analysis_qp.md +63 -0
- package/skills/ae-analysis/references/build_retention_analysis_qp.md +2 -0
- package/skills/ae-analysis/references/build_tag_definition.md +65 -0
- package/skills/ae-analysis/references/cancel_query.md +2 -3
- package/skills/ae-analysis/references/copy_dashboard.md +33 -0
- package/skills/ae-analysis/references/create_id_cluster.md +40 -0
- package/skills/ae-analysis/references/create_id_tag.md +39 -0
- package/skills/ae-analysis/references/delete_alert.md +27 -0
- package/skills/ae-analysis/references/delete_cluster.md +27 -0
- package/skills/ae-analysis/references/delete_dashboard.md +30 -0
- package/skills/ae-analysis/references/delete_metric.md +27 -0
- package/skills/ae-analysis/references/delete_report.md +27 -0
- package/skills/ae-analysis/references/delete_tag.md +27 -0
- package/skills/ae-analysis/references/drilldown_user_events.md +4 -4
- package/skills/ae-analysis/references/drilldown_users.md +3 -3
- package/skills/ae-analysis/references/freeze_dashboards.md +29 -0
- package/skills/ae-analysis/references/get_table_columns.md +11 -12
- package/skills/ae-analysis/references/list_cluster_members.md +4 -4
- package/skills/ae-analysis/references/list_clusters.md +5 -3
- package/skills/ae-analysis/references/list_dashboards.md +3 -3
- package/skills/ae-analysis/references/list_entities.md +1 -1
- package/skills/ae-analysis/references/list_events.md +7 -4
- package/skills/ae-analysis/references/list_metrics.md +5 -3
- package/skills/ae-analysis/references/list_properties.md +6 -4
- package/skills/ae-analysis/references/list_reports.md +3 -3
- package/skills/ae-analysis/references/list_spaces.md +27 -0
- package/skills/ae-analysis/references/list_tag_members.md +4 -4
- package/skills/ae-analysis/references/list_tags.md +4 -2
- package/skills/ae-analysis/references/move_dashboard.md +33 -0
- package/skills/ae-analysis/references/query_adhoc.md +2 -2
- package/skills/ae-analysis/references/query_bi_panel_data.md +2 -2
- package/skills/ae-analysis/references/query_dashboard_report_data.md +4 -4
- package/skills/ae-analysis/references/query_entity_details.md +4 -4
- package/skills/ae-analysis/references/query_event_details.md +3 -3
- package/skills/ae-analysis/references/query_report_data.md +37 -3
- package/skills/ae-analysis/references/update_id_cluster.md +36 -0
- package/skills/ae-analysis/references/update_id_tag.md +37 -0
- package/skills/ae-analysis/references/update_report.md +35 -0
- package/skills/ae-analysis-intent/SKILL.md +1 -1
- package/skills/ae-dataops/SKILL.md +54 -522
- package/skills/ae-dataops/references/dataops-flow-create.md +78 -75
- package/skills/ae-dataops/references/dataops-flow-monitor.md +58 -51
- package/skills/ae-dataops/references/dataops-integration.md +80 -41
- package/skills/ae-dataops/references/dataops-query.md +54 -53
- package/skills/ae-dataops/references/dataops-table.md +28 -59
- package/skills/ae-generate-tracking-code/SKILL.md +68 -31
- package/skills/ae-generate-tracking-code/references/snippet-delivery.md +26 -1
- package/skills/ae-generate-tracking-code/references/te-api.md +47 -164
- package/skills/ae-generate-tracking-plan/SKILL.md +109 -32
- package/skills/ae-generate-tracking-plan/references/te-api.md +57 -27
- package/skills/ae-metadata/SKILL.md +90 -0
- package/skills/ae-metadata/references/metadata_data_table_csv_delete.md +15 -0
- package/skills/ae-metadata/references/metadata_data_table_csv_write.md +29 -0
- package/skills/ae-metadata/references/metadata_data_table_download.md +17 -0
- package/skills/ae-metadata/references/metadata_data_table_get.md +24 -0
- package/skills/ae-metadata/references/metadata_data_table_list.md +27 -0
- package/skills/ae-metadata/references/metadata_data_table_property_bindings_update.md +17 -0
- package/skills/ae-metadata/references/metadata_data_table_sql_delete.md +15 -0
- package/skills/ae-metadata/references/metadata_data_table_sql_write.md +31 -0
- package/skills/ae-metadata/references/metadata_event_get.md +52 -0
- package/skills/ae-metadata/references/metadata_input_file_upload.md +29 -0
- package/skills/ae-metadata/references/metadata_property_dimension_table_bind_existing.md +27 -0
- package/skills/ae-metadata/references/metadata_property_dimension_table_create_and_bind_csv.md +30 -0
- package/skills/ae-metadata/references/metadata_property_get.md +54 -0
- package/dist/te-audience-KMKYI2YJ.js +0 -400
- package/dist/te-dataops-GD3XBYCS.js +0 -1830
|
@@ -13,7 +13,7 @@ metadata:
|
|
|
13
13
|
|
|
14
14
|
Use the `dataops_flow` subcommand to manage flow lifecycle.
|
|
15
15
|
|
|
16
|
-
**Flow Lifecycle: Create → Configure Nodes → Configure Schedule →
|
|
16
|
+
**Flow Lifecycle: Create → Configure Nodes → Configure Schedule → Preview Release → Release to PROD → Manual PROD Execution / Online Schedule**
|
|
17
17
|
|
|
18
18
|
---
|
|
19
19
|
|
|
@@ -25,68 +25,58 @@ Follow these steps in order to create a production-ready flow from scratch.
|
|
|
25
25
|
|
|
26
26
|
```bash
|
|
27
27
|
ae-cli dataops_flow +create_flow --spaceCode "${spaceCode}" \
|
|
28
|
-
--flowName "Daily ETL Process" --remark "Process user data"
|
|
28
|
+
--flowName "Daily ETL Process" --remark "Process user data"
|
|
29
29
|
# Returns flowCode, required for subsequent steps
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
### Step 2: Create Task Nodes (can be called multiple times)
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
|
-
# Create SQL task
|
|
36
|
-
ae-cli dataops_flow +
|
|
37
|
-
--flowCode
|
|
38
|
-
|
|
35
|
+
# Create SQL task and save SQL content
|
|
36
|
+
ae-cli dataops_flow +create_sql_task --spaceCode "${spaceCode}" \
|
|
37
|
+
--flowCode ${flowCode} --taskName "Process User Data" \
|
|
38
|
+
--sql "SELECT * FROM dwd_user"
|
|
39
|
+
# Returns result.taskCode
|
|
39
40
|
|
|
40
41
|
# Create task with upstream dependency
|
|
41
|
-
ae-cli dataops_flow +
|
|
42
|
-
--flowCode
|
|
43
|
-
--preTaskCode
|
|
42
|
+
ae-cli dataops_flow +create_sql_task --spaceCode "${spaceCode}" \
|
|
43
|
+
--flowCode ${flowCode} --taskName "Export Results" \
|
|
44
|
+
--preTaskCode ${upstreamTaskCode} \
|
|
45
|
+
--sql "INSERT INTO ads_user SELECT * FROM dwd_user"
|
|
44
46
|
```
|
|
45
47
|
|
|
46
|
-
|
|
48
|
+
`+create_sql_task` creates DEV Trino SQL task nodes and saves the SQL content.
|
|
47
49
|
|
|
48
|
-
|
|
49
|
-
|------|-------------|-------------|
|
|
50
|
-
| TRINO_SQL | Trino SQL query | Data processing, ETL |
|
|
51
|
-
| SHELL | Shell script | System commands, file operations |
|
|
52
|
-
| FLOW_CHECK | Flow check | Check if upstream flow is complete |
|
|
53
|
-
| TASK_CHECK | Task check | Check if specified task is complete |
|
|
54
|
-
| OFFLINE_SYNC | Offline sync | Integration solution data sync (requires syncId) |
|
|
55
|
-
| APP_SYNC | App sync | App table data sync (requires syncId) |
|
|
56
|
-
| PLACE_HOLDER | Placeholder | Flow control, logical grouping |
|
|
57
|
-
|
|
58
|
-
### Step 3: Save Task Definition (configure SQL/Shell content)
|
|
50
|
+
### Step 3: Create or Modify Integration Sync Task
|
|
59
51
|
|
|
60
52
|
```bash
|
|
61
|
-
#
|
|
62
|
-
ae-cli dataops_flow +
|
|
63
|
-
--flowCode
|
|
64
|
-
--
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
--
|
|
70
|
-
--taskContentJson '{"script":"#!/bin/bash\necho hello"}' \
|
|
71
|
-
--confirmed true
|
|
53
|
+
# Create integration sync task from an existing DataOps sync solution
|
|
54
|
+
ae-cli dataops_flow +create_integration_task --spaceCode "${spaceCode}" \
|
|
55
|
+
--flowCode ${flowCode} --taskName "Sync Employees" \
|
|
56
|
+
--syncId "${syncId}"
|
|
57
|
+
|
|
58
|
+
# Rebind an existing integration sync task to another sync solution
|
|
59
|
+
ae-cli dataops_flow +update_integration_task --spaceCode "${spaceCode}" \
|
|
60
|
+
--flowCode ${flowCode} --taskCode ${taskCode} \
|
|
61
|
+
--syncId "${syncId}"
|
|
72
62
|
```
|
|
73
63
|
|
|
74
|
-
|
|
64
|
+
Use `dataops_integration +list_sync_solutions` or `+get_sync_detail` to find the `syncId` first. These commands expose workflow `OFFLINE_SYNC` tasks only; app sync tasks are not part of this flow.
|
|
65
|
+
|
|
66
|
+
### Step 4: Modify SQL Task Content
|
|
75
67
|
|
|
76
68
|
```bash
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
--
|
|
80
|
-
--
|
|
81
|
-
# Check valid=true, otherwise review message and errorPosition
|
|
69
|
+
# Update SQL task content without changing owner, remark, dependencies, retry, or timeout
|
|
70
|
+
ae-cli dataops_flow +update_sql_task --spaceCode "${spaceCode}" \
|
|
71
|
+
--flowCode ${flowCode} --taskCode ${taskCode} \
|
|
72
|
+
--sql "SELECT * FROM dwd_user"
|
|
82
73
|
```
|
|
83
74
|
|
|
84
75
|
### Step 5: Add Task Dependencies (DAG connections)
|
|
85
76
|
|
|
86
77
|
```bash
|
|
87
78
|
ae-cli dataops_flow +add_task_relation --spaceCode "${spaceCode}" \
|
|
88
|
-
--flowCode
|
|
89
|
-
--confirmed true
|
|
79
|
+
--flowCode ${flowCode} --preTaskCode ${upstreamTaskCode} --taskCode ${downstreamTaskCode}
|
|
90
80
|
```
|
|
91
81
|
|
|
92
82
|
### Step 6: Configure Schedule
|
|
@@ -94,8 +84,8 @@ ae-cli dataops_flow +add_task_relation --spaceCode "${spaceCode}" \
|
|
|
94
84
|
```bash
|
|
95
85
|
# CRON expression for scheduled execution
|
|
96
86
|
ae-cli dataops_flow +save_schedule_config --spaceCode "${spaceCode}" \
|
|
97
|
-
--flowCode
|
|
98
|
-
--cron "0 0 2 * * ?"
|
|
87
|
+
--flowCode ${flowCode} --enabled true \
|
|
88
|
+
--cron "0 0 2 * * ?"
|
|
99
89
|
# Example: 0 0 2 * * ? = Daily at 2 AM
|
|
100
90
|
```
|
|
101
91
|
|
|
@@ -103,28 +93,34 @@ ae-cli dataops_flow +save_schedule_config --spaceCode "${spaceCode}" \
|
|
|
103
93
|
- `0 0 */4 * * ?` — Every 4 hours
|
|
104
94
|
- `0 30 8 * * 1-5` — Weekdays at 8:30
|
|
105
95
|
|
|
106
|
-
### Step 7:
|
|
96
|
+
### Step 7: Preview Release
|
|
107
97
|
|
|
108
98
|
```bash
|
|
109
|
-
#
|
|
110
|
-
ae-cli dataops_flow +
|
|
111
|
-
--flowCode
|
|
112
|
-
|
|
113
|
-
# Check execution status
|
|
114
|
-
ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
115
|
-
|
|
116
|
-
# View task logs (get bsTaskInstanceId and taskCode from DAG)
|
|
117
|
-
ae-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${instanceId}" --taskCode "${taskCode}"
|
|
118
|
-
|
|
119
|
-
# Adjust configuration based on logs, then repeat testing
|
|
99
|
+
# Preview pending DEV-to-PROD release changes before publishing
|
|
100
|
+
ae-cli dataops_flow +preview_release_flow --spaceCode "${spaceCode}" \
|
|
101
|
+
--flowCode ${flowCode}
|
|
120
102
|
```
|
|
121
103
|
|
|
122
|
-
### Step 8: Release to Production
|
|
104
|
+
### Step 8: Release to Production and Verify
|
|
123
105
|
|
|
124
106
|
```bash
|
|
125
107
|
ae-cli dataops_flow +release_flow --spaceCode "${spaceCode}" \
|
|
126
|
-
--flowCode
|
|
127
|
-
|
|
108
|
+
--flowCode ${flowCode}
|
|
109
|
+
|
|
110
|
+
# Manually trigger PROD execution when verification is needed
|
|
111
|
+
ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
112
|
+
--flowCode ${flowCode}
|
|
113
|
+
|
|
114
|
+
# Check execution status
|
|
115
|
+
ae-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
|
|
116
|
+
--keyword "${flowCode}" --pageSize 20
|
|
117
|
+
|
|
118
|
+
# Inspect one instance and task logs when troubleshooting
|
|
119
|
+
ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
|
|
120
|
+
--flowCode ${flowCode} --flowInstanceId ${flowInstanceId}
|
|
121
|
+
ae-cli dataops_operations +get_task_instance_detail --spaceCode "${spaceCode}" \
|
|
122
|
+
--flowCode ${flowCode} --flowInstanceId ${flowInstanceId} \
|
|
123
|
+
--taskName "${taskName}" --includeLog true
|
|
128
124
|
```
|
|
129
125
|
|
|
130
126
|
---
|
|
@@ -133,31 +129,38 @@ ae-cli dataops_flow +release_flow --spaceCode "${spaceCode}" \
|
|
|
133
129
|
|
|
134
130
|
| Command | Purpose | Key Flags |
|
|
135
131
|
|---------|---------|-----------|
|
|
136
|
-
| `+create_flow` | Create flow | `--spaceCode` `--flowName`
|
|
137
|
-
| `+
|
|
138
|
-
| `+
|
|
139
|
-
| `+
|
|
140
|
-
| `+
|
|
141
|
-
| `+
|
|
142
|
-
| `+
|
|
143
|
-
| `+get_task_params` | View task
|
|
144
|
-
| `+execute_flow` | Manual execution | `--spaceCode` `--flowCode`
|
|
145
|
-
|
|
|
146
|
-
|
|
|
147
|
-
| `+
|
|
132
|
+
| `+create_flow` | Create DEV flow | `--spaceCode` `--flowName` `[--remark]` |
|
|
133
|
+
| `+create_sql_task` | Create DEV Trino SQL task and save SQL | `--spaceCode` `--flowCode` `--taskName` `--sql` `[--preSql]` `[--postSql]` `[--preTaskCode]` `[--remark]` |
|
|
134
|
+
| `+update_sql_task` | Update DEV Trino SQL task content | `--spaceCode` `--flowCode` `--taskCode` `--sql` `[--preSql]` `[--postSql]` |
|
|
135
|
+
| `+create_integration_task` | Create DEV integration sync task from an existing sync solution | `--spaceCode` `--flowCode` `--taskName` `--syncId` `[--preTaskCode]` `[--remark]` |
|
|
136
|
+
| `+update_integration_task` | Rebind DEV integration sync task to a sync solution | `--spaceCode` `--flowCode` `--taskCode` `--syncId` |
|
|
137
|
+
| `+add_task_relation` | Add DEV dependency | `--spaceCode` `--flowCode` `--preTaskCode` `--taskCode` |
|
|
138
|
+
| `+save_schedule_config` | Save DEV schedule config | `--spaceCode` `--flowCode` `--enabled` `[--cron]` |
|
|
139
|
+
| `+get_task_params` | View DEV task parameter list | `--spaceCode` `--flowCode` `--taskCode` |
|
|
140
|
+
| `+execute_flow` | Manual PROD execution | `--spaceCode` `--flowCode` `[--baseDate]` |
|
|
141
|
+
| `dataops_operations +get_flow_instance_detail` | View operations instance DAG and task statuses | `--spaceCode` `--flowCode` `--flowInstanceId` |
|
|
142
|
+
| `dataops_operations +get_task_instance_detail` | View operations task detail and optional logs | `--spaceCode` `--flowCode` `--flowInstanceId` (`--taskCode` or `--taskName`) `[--includeLog]` |
|
|
143
|
+
| `+preview_release_flow` | Preview pending DEV-to-PROD release changes without publishing | `--spaceCode` `--flowCode` |
|
|
144
|
+
| `+release_flow` | Submit DEV-to-PROD release | `--spaceCode` `--flowCode` |
|
|
148
145
|
|
|
149
146
|
## Parameter Notes
|
|
150
147
|
|
|
151
148
|
- **Parameter Reference**: Reference workspace parameters in tasks using `${paramKey}` (e.g., `${ws_run_date}`)
|
|
152
|
-
- **
|
|
153
|
-
- **
|
|
154
|
-
|
|
155
|
-
|
|
149
|
+
- **Execution**: `+execute_flow` requires `--spaceCode` and `--flowCode`; `--baseDate` is optional and maps to runtime parameter `bd`. It always runs PROD and returns `action/result/status`; `result` includes `flowCode`, `executeId`, `operationStatus`, `nextAction`, and optional `flowInstanceId`.
|
|
150
|
+
- **Schedule config**: `+save_schedule_config` requires `--spaceCode`, `--flowCode`, and `--enabled`. `--cron` is required only when `--enabled true`; omit it when disabling scheduling. It returns `action/result/status`; `result` includes `enabled`, `flow`, `message`, and `cron` only when enabled.
|
|
151
|
+
- **Release preview**: `+preview_release_flow` requires `--spaceCode` and `--flowCode`; it has no optional flags. It returns `flowCode`, `releaseStatus`, `message`, and `changes`. Each change may include `scheduleConfigChange` and `tasks`; task entries may include `changed`, `contentCompare`, and `targetTable`.
|
|
152
|
+
- **Release**: `+release_flow` requires `--spaceCode` and `--flowCode`; it has no optional flags. It returns `action/result/status`; `result` includes `flowCode`, `releaseStatus`, `message`, optional `packageCode`, and optional `changes`. Each change may include `scheduleConfigChange` and `tasks`; task entries include `changed`.
|
|
153
|
+
- **SQL task creation**: `+create_sql_task` requires `--spaceCode`, `--flowCode`, `--taskName`, and `--sql`; `--preSql`, `--postSql`, `--preTaskCode`, and `--remark` are optional. It returns `action/result/status`; `result` includes `flowCode`, `taskCode`, `taskName`, `taskType=TRINO_SQL`, and `sqlSaved=true`.
|
|
154
|
+
- **SQL task update**: `+update_sql_task` requires `--spaceCode`, `--flowCode`, `--taskCode`, and `--sql`; `--preSql` and `--postSql` are optional and keep existing values when omitted. It returns `action/result/status`; `result` includes `sqlSaved`, `flowCode`, `taskCode`, `taskType=TRINO_SQL`, and `task`.
|
|
155
|
+
- **Task dependency**: `+add_task_relation` requires `--spaceCode`, `--flowCode`, `--preTaskCode`, and `--taskCode`. `preTaskCode` is upstream and `taskCode` is downstream. It returns `action/result/status`; `result` includes `status`, `flowCode`, `preTaskCode`, `taskCode`, and `message`.
|
|
156
|
+
- **Task parameters**: `+get_task_params` requires `--spaceCode`, `--flowCode`, and `--taskCode`; it has no optional flags. It queries DEV and returns `data` as an array. Items include fields such as `paramKey`, `paramType`, `paramDataType`, `paramFrom`, and built-in flags like `isBd`.
|
|
157
|
+
- **Integration task creation**: `+create_integration_task` requires `--spaceCode`, `--flowCode`, `--taskName`, and `--syncId`; `--preTaskCode` and `--remark` are optional. It returns `action/result/status`; `result` includes `syncTaskSaved`, `flowCode`, `taskCode`, `taskName`, `taskType=OFFLINE_SYNC`, `syncId`, and `nextAction`.
|
|
158
|
+
- **SQL task dependencies**: Use optional `--preTaskCode` to create one upstream dependency with the task. Use `+add_task_relation` for additional DAG dependencies.
|
|
156
159
|
|
|
157
160
|
## Important Notes
|
|
158
161
|
|
|
159
162
|
1. **Getting flowCode**: Use `+list_flows` (see `dataops-flow-monitor` Skill)
|
|
160
|
-
2. **Getting taskCode**: Returned by `+
|
|
161
|
-
3. **Release Impact**:
|
|
163
|
+
2. **Getting taskCode**: Returned by `+create_sql_task` / `+create_integration_task` or query via `+get_flow_overview`
|
|
164
|
+
3. **Release Impact**: Submits current DEV changes to PROD; released schedule/config applies to future PROD runs
|
|
162
165
|
4. **SQL Validation**: Recommended to validate before saving TRINO_SQL tasks
|
|
163
166
|
5. **Cannot create circular dependencies**
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dataops-flow-monitor
|
|
3
3
|
version: 1.0.0
|
|
4
|
-
description: "Flow execution and monitoring: view execution instances,
|
|
4
|
+
description: "Flow execution and monitoring: view execution instances, inspect instance DAGs and task logs, manually execute/stop flows. Trigger keywords: execute flow, running instance, monitor, logs, stop, troubleshoot, instance, flow instance."
|
|
5
5
|
metadata:
|
|
6
6
|
requires:
|
|
7
7
|
bins: ["ae-cli"]
|
|
@@ -11,11 +11,11 @@ metadata:
|
|
|
11
11
|
|
|
12
12
|
> **Prerequisites:** Read [`ae-dataops/SKILL.md`](../SKILL.md) for general rules.
|
|
13
13
|
|
|
14
|
-
Use
|
|
14
|
+
Use `dataops_flow` to execute and control flows. Use `dataops_operations` for space-level operations instance search, instance DAG inspection, and task logs.
|
|
15
15
|
|
|
16
16
|
**Core Concepts:**
|
|
17
|
-
- **executeId** — Execution
|
|
18
|
-
- **flowInstanceId** — Operations instance ID
|
|
17
|
+
- **executeId** — Execution ID returned immediately by `dataops_flow +execute_flow`; useful for stopping before the scheduler instance is available
|
|
18
|
+
- **flowInstanceId** — Operations workflow instance ID returned by `dataops_operations +search_flow_instances`; use it for operations detail, task logs, and stop
|
|
19
19
|
- **Instance list queries PROD environment by default** (unlike other tools which default to DEV)
|
|
20
20
|
|
|
21
21
|
---
|
|
@@ -23,18 +23,24 @@ Use the `dataops_flow` subcommand to execute and monitor flows.
|
|
|
23
23
|
## Workflow A: Find and View Flow Execution Status
|
|
24
24
|
|
|
25
25
|
```bash
|
|
26
|
-
# Step 1:
|
|
27
|
-
ae-cli dataops_flow +list_flows --spaceCode "${spaceCode}" --keyword "etl"
|
|
26
|
+
# Step 1: List/search current-space flows and get flowCode
|
|
27
|
+
ae-cli dataops_flow +list_flows --spaceCode "${spaceCode}" --keyword "etl" --pageSize 20
|
|
28
28
|
|
|
29
|
-
#
|
|
30
|
-
ae-cli dataops_flow +
|
|
31
|
-
--
|
|
29
|
+
# Optional: find frequently released flows in the last 30 days
|
|
30
|
+
ae-cli dataops_flow +list_high_frequency_release_flows --spaceCode "${spaceCode}" \
|
|
31
|
+
--days 30 --topN 10 --status SUCCESS
|
|
32
32
|
|
|
33
|
-
#
|
|
34
|
-
ae-cli dataops_flow +
|
|
33
|
+
# Optional: get one-environment overview by exact name or flowCode
|
|
34
|
+
ae-cli dataops_flow +get_flow_overview --spaceCode "${spaceCode}" \
|
|
35
|
+
--flowName "etl" --env "PROD"
|
|
35
36
|
|
|
36
|
-
# Step
|
|
37
|
-
ae-cli
|
|
37
|
+
# Step 2: Search operation instances across the whole space, with statistics
|
|
38
|
+
ae-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
|
|
39
|
+
--keyword "${flowKeyword}" --startDate "${startDate}" --endDate "${endDate}" --pageSize 20
|
|
40
|
+
|
|
41
|
+
# Step 3: Inspect one instance DAG and task statuses
|
|
42
|
+
ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
|
|
43
|
+
--flowCode ${flowCode} --flowInstanceId ${flowInstanceId}
|
|
38
44
|
```
|
|
39
45
|
|
|
40
46
|
---
|
|
@@ -42,21 +48,18 @@ ae-cli dataops_flow +get_execute_record --spaceCode "${spaceCode}" --executeId "
|
|
|
42
48
|
## Workflow B: Troubleshoot Task Failures
|
|
43
49
|
|
|
44
50
|
```bash
|
|
45
|
-
# Step 1:
|
|
46
|
-
ae-cli
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
# Step 2: View failed task logs
|
|
50
|
-
ae-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${bsTaskInstanceId}" \
|
|
51
|
-
--taskCode "${taskCode}" --limit 100
|
|
51
|
+
# Step 1: Inspect one workflow instance and locate the failed task
|
|
52
|
+
ae-cli dataops_operations +get_flow_instance_detail --spaceCode "${spaceCode}" \
|
|
53
|
+
--flowCode ${flowCode} --flowInstanceId ${flowInstanceId}
|
|
52
54
|
|
|
53
|
-
# Step
|
|
54
|
-
ae-cli
|
|
55
|
-
--flowCode
|
|
55
|
+
# Step 2: View task detail and logs
|
|
56
|
+
ae-cli dataops_operations +get_task_instance_detail --spaceCode "${spaceCode}" \
|
|
57
|
+
--flowCode ${flowCode} --flowInstanceId ${flowInstanceId} \
|
|
58
|
+
--taskName "${taskName}" --includeLog true
|
|
56
59
|
|
|
57
|
-
# Step
|
|
60
|
+
# Step 3: Re-execute after fixing
|
|
58
61
|
ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
59
|
-
--flowCode
|
|
62
|
+
--flowCode ${flowCode}
|
|
60
63
|
```
|
|
61
64
|
|
|
62
65
|
---
|
|
@@ -64,17 +67,18 @@ ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
|
64
67
|
## Workflow C: Manual Execution and Stop
|
|
65
68
|
|
|
66
69
|
```bash
|
|
67
|
-
# Manually trigger execution
|
|
70
|
+
# Manually trigger PROD execution
|
|
68
71
|
ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
69
|
-
--flowCode
|
|
72
|
+
--flowCode ${flowCode}
|
|
70
73
|
# Returns executeId
|
|
71
74
|
|
|
72
|
-
# Monitor
|
|
73
|
-
ae-cli
|
|
75
|
+
# Monitor recent executions
|
|
76
|
+
ae-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
|
|
77
|
+
--keyword "${flowKeyword}" --startDate "${startDate}" --endDate "${endDate}" --pageSize 20
|
|
74
78
|
|
|
75
79
|
# Stop execution if necessary (irreversible)
|
|
76
|
-
ae-cli
|
|
77
|
-
--flowCode
|
|
80
|
+
ae-cli dataops_operations +stop_flow_instance --spaceCode "${spaceCode}" \
|
|
81
|
+
--flowCode ${flowCode} --executeId ${executeId}
|
|
78
82
|
```
|
|
79
83
|
|
|
80
84
|
---
|
|
@@ -82,14 +86,11 @@ ae-cli dataops_flow +stop_flow --spaceCode "${spaceCode}" \
|
|
|
82
86
|
## Workflow D: View Flow Structure
|
|
83
87
|
|
|
84
88
|
```bash
|
|
85
|
-
# View flow
|
|
86
|
-
ae-cli dataops_flow +
|
|
87
|
-
|
|
88
|
-
# View flow basic information
|
|
89
|
-
ae-cli dataops_flow +get_flow_detail --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
89
|
+
# View agent-friendly flow overview for one environment
|
|
90
|
+
ae-cli dataops_flow +get_flow_overview --spaceCode "${spaceCode}" \
|
|
91
|
+
--flowName "etl" --env "DEV"
|
|
90
92
|
|
|
91
|
-
#
|
|
92
|
-
ae-cli dataops_flow +get_schedule_config --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
93
|
+
# The current schedule is included in +get_flow_overview output as "schedule".
|
|
93
94
|
```
|
|
94
95
|
|
|
95
96
|
---
|
|
@@ -98,21 +99,27 @@ ae-cli dataops_flow +get_schedule_config --spaceCode "${spaceCode}" --flowCode "
|
|
|
98
99
|
|
|
99
100
|
| Command | Purpose | Key Flags |
|
|
100
101
|
|---------|---------|-----------|
|
|
101
|
-
| `+list_flows` |
|
|
102
|
-
| `+
|
|
103
|
-
| `+
|
|
104
|
-
| `+
|
|
105
|
-
|
|
|
106
|
-
|
|
|
107
|
-
|
|
|
108
|
-
|
|
|
109
|
-
| `+
|
|
110
|
-
| `+
|
|
111
|
-
| `+get_task_params` | Task parameters | `--spaceCode` `--flowCode` `--taskCode` `--env` |
|
|
112
|
-
| `+update_flow` | Update flow | `--spaceCode` `--flowCode` `--flowName` `--remark` `--confirmed` |
|
|
102
|
+
| `+list_flows` | List/search current-space flows | `--spaceCode` `[--keyword]` `[--pageNum]` `[--pageSize]` |
|
|
103
|
+
| `+list_high_frequency_release_flows` | List flows ranked by release count, not execution count | `--spaceCode` `[--days]` `[--topN]` `[--minCount]` `[--status]` |
|
|
104
|
+
| `+get_flow_overview` | DEV/PROD flow overview | `--spaceCode` (`--flowCode` or `--flowName`) `[--env]` |
|
|
105
|
+
| `+execute_flow` | Manual PROD execution | `--spaceCode` `--flowCode` `[--baseDate]` |
|
|
106
|
+
| `dataops_operations +search_flow_instances` | Search operations workflow instances | `--spaceCode` `[--keyword]` `[--startDate]` `[--endDate]` `[--status]` `[--pageNum]` `[--pageSize]` |
|
|
107
|
+
| `dataops_operations +get_flow_instance_detail` | Operations instance DAG and task statuses | `--spaceCode` `--flowCode` `--flowInstanceId` |
|
|
108
|
+
| `dataops_operations +get_task_instance_detail` | Operations task detail and optional logs | `--spaceCode` `--flowCode` `--flowInstanceId` (`--taskCode` or `--taskName`) `[--includeLog]` |
|
|
109
|
+
| `dataops_operations +stop_flow_instance` | Stop a running execution | `--spaceCode` `--flowCode` exactly one of `--executeId` or `--flowInstanceId` |
|
|
110
|
+
| `+get_task_params` | DEV task parameter list | `--spaceCode` `--flowCode` `--taskCode` |
|
|
111
|
+
| `+update_flow` | Update DEV workflow name and/or remark | `--spaceCode` `--flowCode` `[--flowName]` `[--remark]` |
|
|
113
112
|
|
|
114
113
|
## Parameter Notes
|
|
115
114
|
|
|
115
|
+
- **Execution**: `+execute_flow` requires `--spaceCode` and `--flowCode`; `--baseDate` is optional and maps to runtime parameter `bd`. It always runs PROD and returns `action/result/status`; `result` includes `flowCode`, `executeId`, `operationStatus`, `nextAction`, and optional `flowInstanceId`.
|
|
116
|
+
- **Stop flow instance**: `dataops_operations +stop_flow_instance` requires `--spaceCode`, `--flowCode`, and exactly one of `--executeId` or `--flowInstanceId`. `--executeId` comes from `dataops_flow +execute_flow`; `--flowInstanceId` comes from `dataops_operations +search_flow_instances`. It returns `success`, `selector`, and selector-specific stop result fields.
|
|
117
|
+
- **Flow overview**: `+get_flow_overview` requires `--spaceCode` and either `--flowCode` or exact `--flowName`; `--flowCode` takes precedence. `--env` defaults to `DEV`; use `PROD` to include latest production instance fields when available. It returns `success`, `env`, `resolvedBy`, `flow`, `schedule`, `dag`, and `summary`.
|
|
118
|
+
- **Flow update**: `+update_flow` requires `--spaceCode`, `--flowCode`, and at least one of `--flowName` or `--remark`. It returns `action/result/status`; `result` is an array with items containing `flowCode`, `operationStatus`, `nameChanged`, and optional `flowName`.
|
|
119
|
+
- **High-frequency release flows**: `+list_high_frequency_release_flows` requires `--spaceCode`; `--days`, `--topN`, `--minCount`, and `--status` are optional. Defaults are `days=30`, `topN=10`, and `status=SUCCESS`. It returns `period`, `filters`, `flows`, `returnedCount`, and `nextAction`; flow items include `rank`, `flowCode`, `flowName`, `releaseCount`, `lastReleaseTime`, and `avgIntervalHours`.
|
|
120
|
+
- **Operations instance search**: `dataops_operations +search_flow_instances` requires `--spaceCode`; `--keyword`, `--startDate`, `--endDate`, `--status`, `--pageNum`, and `--pageSize` are optional. `--keyword` fuzzy-matches `flowName` or `flowCode`. `pageNum` defaults to `1`; `pageSize` defaults to `20` and maxes at `100`. It returns `totalCount`, `returnedCount`, `pageNum`, `pageSize`, `hasMore`, `instances`, `statusCounts`, `triggerTypeCounts`, and `ownerCounts`.
|
|
121
|
+
- **Task instance detail**: `dataops_operations +get_task_instance_detail` requires `--spaceCode`, `--flowCode`, `--flowInstanceId`, and either `--taskCode` or exact `--taskName`. `--taskCode` takes precedence when both are passed. `--includeLog` is optional and defaults to `false`. It returns `success`, `flowInstanceId`, `taskCode`, `taskName`, `taskInstanceId`, `status`, `task`, `taskInstance`, `definition`, and `log` only when requested.
|
|
116
122
|
- **env**: `DEV` (development) | `PROD` (production, instance list defaults to PROD)
|
|
123
|
+
- **flow list paging**: `+list_flows` requires `--spaceCode`; `--keyword`, `--pageNum`, and `--pageSize` are optional. It returns `flows`, `totalCount`, `returnedCount`, `pageNum`, `pageSize`, and `hasMore`; flow items include `latestProductionInstance` only when available. `pageSize` defaults to `20` and maxes at `100`.
|
|
124
|
+
- **Task parameters**: `+get_task_params` requires `--spaceCode`, `--flowCode`, and `--taskCode`; it has no optional flags. It queries DEV and returns `data` as an array. Items include fields such as `paramKey`, `paramType`, `paramDataType`, `paramFrom`, and built-in flags like `isBd`.
|
|
117
125
|
- **Task Status**: `success` / `failure` / `running` / `waiting`
|
|
118
|
-
- **DAG Difference**: `+get_flow_dag` returns static definition structure, `+get_execute_dag` returns runtime structure with execution status
|