@thinkingai/ae-cli 1.0.1 → 1.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -13
- package/dist/auth-BQC5DBE7.js +19 -0
- package/dist/auth-DXKYLMO7.js +81 -0
- package/dist/chunk-3CDLNXTD.js +218 -0
- package/dist/chunk-53FNNCZW.js +174 -0
- package/dist/chunk-EBFUVWIZ.js +127 -0
- package/dist/chunk-ZYXIGPG7.js +148 -0
- package/dist/client-T4VODLA6.js +16 -0
- package/dist/community-FVDAWGS4.js +684 -0
- package/dist/config-V7AXKDMB.js +242 -0
- package/dist/index.js +16 -15
- package/dist/raw-G3GKHAPZ.js +61 -0
- package/dist/te-analysis-CHFTDN55.js +871 -0
- package/dist/te-audience-M6LQY6M7.js +350 -0
- package/dist/te-common-BLQ7DVGC.js +144 -0
- package/dist/te-dataops-V4OTKY35.js +1853 -0
- package/dist/te-engage-DXE4R5EW.js +1470 -0
- package/dist/te-meta-FMM7KCVI.js +479 -0
- package/package.json +7 -5
- package/skills/te-analysis/SKILL.md +10 -10
- package/skills/te-analysis/references/build-entity-details-sql.md +10 -10
- package/skills/te-analysis/references/build-event-details-sql.md +10 -10
- package/skills/te-analysis/references/create-alert.md +4 -4
- package/skills/te-analysis/references/create-dashboard.md +4 -4
- package/skills/te-analysis/references/create-or-update-dashboard-note.md +4 -4
- package/skills/te-analysis/references/create-public-access-link.md +4 -4
- package/skills/te-analysis/references/create-report.md +10 -10
- package/skills/te-analysis/references/create-result-cluster.md +4 -4
- package/skills/te-analysis/references/drilldown-user-events.md +6 -6
- package/skills/te-analysis/references/drilldown-users.md +4 -4
- package/skills/te-analysis/references/get-alert-definition-schema.md +3 -3
- package/skills/te-analysis/references/get-alert.md +3 -3
- package/skills/te-analysis/references/get-analysis-query-schema.md +4 -4
- package/skills/te-analysis/references/get-filter-schema.md +3 -3
- package/skills/te-analysis/references/get-groupby-schema.md +3 -3
- package/skills/te-analysis/references/get-report-definition.md +3 -3
- package/skills/te-analysis/references/get-table-columns.md +3 -3
- package/skills/te-analysis/references/list-alerts.md +4 -4
- package/skills/te-analysis/references/list-dashboards.md +4 -4
- package/skills/te-analysis/references/list-public-access-links.md +3 -3
- package/skills/te-analysis/references/list-reports.md +4 -4
- package/skills/te-analysis/references/load-filters.md +4 -4
- package/skills/te-analysis/references/query-adhoc.md +10 -10
- package/skills/te-analysis/references/query-dashboard-detail.md +3 -3
- package/skills/te-analysis/references/query-dashboard-report-data.md +9 -9
- package/skills/te-analysis/references/query-entity-details.md +10 -10
- package/skills/te-analysis/references/query-event-details.md +10 -10
- package/skills/te-analysis/references/query-report-data.md +11 -11
- package/skills/te-analysis/references/update-alert.md +4 -4
- package/skills/te-analysis/references/update-dashboard.md +4 -4
- package/skills/te-analysis/references/update-public-access-link.md +4 -4
- package/skills/te-audience/SKILL.md +5 -5
- package/skills/te-audience/references/create-cluster.md +10 -10
- package/skills/te-audience/references/create-tag.md +10 -10
- package/skills/te-audience/references/get-cluster-definition-schema.md +9 -9
- package/skills/te-audience/references/get-clusters-by-name.md +3 -3
- package/skills/te-audience/references/get-tag-definition-schema.md +3 -3
- package/skills/te-audience/references/get-tags-by-name.md +3 -3
- package/skills/te-audience/references/list-cluster-members.md +4 -4
- package/skills/te-audience/references/list-clusters.md +4 -4
- package/skills/te-audience/references/list-tag-members.md +4 -4
- package/skills/te-audience/references/list-tags.md +4 -4
- package/skills/te-audience/references/refresh-cluster.md +3 -3
- package/skills/te-audience/references/refresh-tag.md +3 -3
- package/skills/te-audience/references/update-cluster.md +9 -9
- package/skills/te-audience/references/update-tag.md +10 -10
- package/skills/te-common/SKILL.md +2 -2
- package/skills/te-common/references/get-resource-url.md +5 -5
- package/skills/te-common/references/list-projects.md +3 -3
- package/skills/te-community/SKILL.md +61 -69
- package/skills/te-community/references/community-activity-analysis.md +43 -43
- package/skills/te-community/references/community-analyzing-official-content.md +47 -47
- package/skills/te-community/references/community-analyzing-theme-comment.md +87 -87
- package/skills/te-community/references/community-character-analysis.md +44 -44
- package/skills/te-community/references/community-daily-report.md +56 -53
- package/skills/te-community/references/community-hottopic-insight.md +81 -87
- package/skills/te-community/references/community-weekly-report.md +54 -53
- package/skills/te-community/references/get_channel_info.md +9 -9
- package/skills/te-community/references/get_comment_tag_analysis.md +15 -15
- package/skills/te-community/references/get_comments_summary.md +15 -15
- package/skills/te-community/references/get_corpus_tags.md +9 -9
- package/skills/te-community/references/get_daily_summary.md +10 -10
- package/skills/te-community/references/get_hot_topics.md +12 -12
- package/skills/te-community/references/get_livestream_analysis.md +9 -9
- package/skills/te-community/references/get_livestream_detail.md +19 -19
- package/skills/te-community/references/get_livestream_list.md +13 -13
- package/skills/te-community/references/get_livestream_overview.md +11 -11
- package/skills/te-community/references/get_livestream_room_metrics.md +12 -12
- package/skills/te-community/references/get_livestream_rooms.md +12 -12
- package/skills/te-community/references/get_overview_metrics.md +12 -12
- package/skills/te-community/references/get_post_detail.md +21 -21
- package/skills/te-community/references/get_risk_content.md +23 -23
- package/skills/te-community/references/get_sentiment_overview.md +16 -16
- package/skills/te-community/references/get_tag_trends.md +13 -13
- package/skills/te-community/references/get_topic_detail.md +12 -12
- package/skills/te-community/references/search_posts.md +26 -26
- package/skills/te-dataops/SKILL.md +390 -392
- package/skills/te-dataops/references/dataops-flow-create.md +87 -87
- package/skills/te-dataops/references/dataops-flow-monitor.md +65 -65
- package/skills/te-dataops/references/dataops-integration.md +90 -85
- package/skills/te-dataops/references/dataops-query.md +58 -58
- package/skills/te-dataops/references/dataops-table.md +63 -63
- package/skills/te-engage/SKILL.md +103 -103
- package/skills/te-engage/references/add-approver.md +11 -11
- package/skills/te-engage/references/add-channel.md +34 -34
- package/skills/te-engage/references/approver-list.md +8 -8
- package/skills/te-engage/references/cancel-query-by-request-id.md +10 -10
- package/skills/te-engage/references/channel-detail.md +13 -13
- package/skills/te-engage/references/channel-list.md +15 -15
- package/skills/te-engage/references/config-channel-detail.md +10 -10
- package/skills/te-engage/references/config-channel-list.md +11 -11
- package/skills/te-engage/references/config-item-analysis-report.md +21 -21
- package/skills/te-engage/references/config-item-detail.md +10 -10
- package/skills/te-engage/references/config-item-list.md +8 -8
- package/skills/te-engage/references/config-item-strategy-comparison.md +14 -14
- package/skills/te-engage/references/config-item-trigger-report.md +26 -26
- package/skills/te-engage/references/copy-config-template.md +16 -16
- package/skills/te-engage/references/delete-channel.md +11 -11
- package/skills/te-engage/references/delete-config-channel.md +11 -11
- package/skills/te-engage/references/delete-config-item.md +12 -12
- package/skills/te-engage/references/delete-flow.md +11 -11
- package/skills/te-engage/references/flow-ab-split-node-report.md +29 -29
- package/skills/te-engage/references/flow-detail.md +19 -19
- package/skills/te-engage/references/flow-list.md +8 -8
- package/skills/te-engage/references/flow-node-config-schema.md +11 -11
- package/skills/te-engage/references/flow-node-detail-report.md +30 -30
- package/skills/te-engage/references/flow-node-overview-report.md +27 -27
- package/skills/te-engage/references/flow-process-report.md +32 -32
- package/skills/te-engage/references/manage-flow.md +44 -44
- package/skills/te-engage/references/manage-strategy.md +37 -37
- package/skills/te-engage/references/manage-task.md +20 -20
- package/skills/te-engage/references/modify-flow-base-info.md +14 -14
- package/skills/te-engage/references/save-flow.md +264 -263
- package/skills/te-engage/references/strategy-detail.md +16 -16
- package/skills/te-engage/references/strategy-list.md +10 -10
- package/skills/te-engage/references/task-data-detail.md +47 -47
- package/skills/te-engage/references/task-data-overview.md +23 -23
- package/skills/te-engage/references/task-detail.md +18 -18
- package/skills/te-engage/references/task-experiment-report.md +26 -26
- package/skills/te-engage/references/task-list.md +35 -35
- package/skills/te-engage/references/task-metric-detail.md +31 -31
- package/skills/te-engage/references/task-stats.md +31 -31
- package/skills/te-engage/references/update-channel-status.md +13 -13
- package/skills/te-engage/references/update-config-channel-status.md +14 -14
- package/skills/te-engage/references/validate-flow-node-config.md +15 -15
- package/skills/te-engage/references/whitelist-list.md +8 -8
- package/skills/te-meta/SKILL.md +5 -5
- package/skills/te-meta/references/batch-create-metadata.md +4 -4
- package/skills/te-meta/references/batch-edit-metadata.md +3 -3
- package/skills/te-meta/references/create-metric.md +12 -12
- package/skills/te-meta/references/create-project-mark-time.md +4 -4
- package/skills/te-meta/references/create-virtual-event.md +10 -10
- package/skills/te-meta/references/create-virtual-property.md +8 -8
- package/skills/te-meta/references/delete-project-mark-times.md +3 -3
- package/skills/te-meta/references/delete-track-items.md +3 -3
- package/skills/te-meta/references/get-metric.md +3 -3
- package/skills/te-meta/references/get-project-config.md +3 -3
- package/skills/te-meta/references/get-track-program.md +3 -3
- package/skills/te-meta/references/list-entities.md +4 -4
- package/skills/te-meta/references/list-events.md +4 -4
- package/skills/te-meta/references/list-metrics.md +4 -4
- package/skills/te-meta/references/list-project-mark-times.md +4 -4
- package/skills/te-meta/references/list-project-users.md +3 -3
- package/skills/te-meta/references/list-properties.md +4 -4
- package/skills/te-meta/references/save-track-items.md +3 -3
- package/skills/te-meta/references/update-metric.md +10 -10
- package/skills/te-meta/references/update-project-mark-time.md +4 -4
- package/skills/te-shared/SKILL.md +8 -8
- package/skills/te-community/references/community-release-analysis.md +0 -79
- package/skills/te-dataops/references/dataops-operations.md +0 -107
|
@@ -1,163 +1,163 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dataops-flow-create
|
|
3
3
|
version: 1.0.0
|
|
4
|
-
description: "
|
|
4
|
+
description: "Flow creation and configuration: create flows, add task nodes, configure dependencies, save task definitions, configure scheduling, test, and release to production. Trigger keywords: create flow, new workflow, configure schedule, add task node, release, cron, scheduled execution."
|
|
5
5
|
metadata:
|
|
6
6
|
requires:
|
|
7
|
-
bins: ["
|
|
7
|
+
bins: ["ae-cli"]
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
-
# DataOps
|
|
10
|
+
# DataOps Flow Creation and Configuration
|
|
11
11
|
|
|
12
|
-
>
|
|
12
|
+
> **Prerequisites:** Read [`te-dataops/SKILL.md`](../SKILL.md) for general rules.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
Use the `dataops_flow` subcommand to manage flow lifecycle.
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
**Flow Lifecycle: Create → Configure Nodes → Configure Schedule → Test in DEV → Release to PROD → Online Schedule**
|
|
17
17
|
|
|
18
18
|
---
|
|
19
19
|
|
|
20
|
-
##
|
|
20
|
+
## Complete Flow Creation Process
|
|
21
21
|
|
|
22
|
-
|
|
22
|
+
Follow these steps in order to create a production-ready flow from scratch.
|
|
23
23
|
|
|
24
|
-
### Step 1:
|
|
24
|
+
### Step 1: Create Flow
|
|
25
25
|
|
|
26
26
|
```bash
|
|
27
|
-
|
|
28
|
-
--flowName "
|
|
29
|
-
#
|
|
27
|
+
ae-cli dataops_flow +create_flow --spaceCode "${spaceCode}" \
|
|
28
|
+
--flowName "Daily ETL Process" --remark "Process user data" --confirmed true
|
|
29
|
+
# Returns flowCode, required for subsequent steps
|
|
30
30
|
```
|
|
31
31
|
|
|
32
|
-
### Step 2:
|
|
32
|
+
### Step 2: Create Task Nodes (can be called multiple times)
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
|
-
#
|
|
36
|
-
|
|
37
|
-
--flowCode "${flowCode}" --taskName "
|
|
38
|
-
#
|
|
39
|
-
|
|
40
|
-
#
|
|
41
|
-
|
|
42
|
-
--flowCode "${flowCode}" --taskName "
|
|
35
|
+
# Create SQL task
|
|
36
|
+
ae-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
|
|
37
|
+
--flowCode "${flowCode}" --taskName "Process User Data" --taskType "TRINO_SQL" --confirmed true
|
|
38
|
+
# Returns taskCode
|
|
39
|
+
|
|
40
|
+
# Create task with upstream dependency
|
|
41
|
+
ae-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
|
|
42
|
+
--flowCode "${flowCode}" --taskName "Export Results" --taskType "TRINO_SQL" \
|
|
43
43
|
--preTaskCode "${upstreamTaskCode}" --confirmed true
|
|
44
44
|
```
|
|
45
45
|
|
|
46
|
-
|
|
46
|
+
**Task Node Types:**
|
|
47
47
|
|
|
48
|
-
|
|
|
49
|
-
|
|
50
|
-
| TRINO_SQL | Trino SQL
|
|
51
|
-
| SHELL | Shell
|
|
52
|
-
| FLOW_CHECK |
|
|
53
|
-
| TASK_CHECK |
|
|
54
|
-
| OFFLINE_SYNC |
|
|
55
|
-
| APP_SYNC |
|
|
56
|
-
| PLACE_HOLDER |
|
|
48
|
+
| Type | Description | Typical Use |
|
|
49
|
+
|------|-------------|-------------|
|
|
50
|
+
| TRINO_SQL | Trino SQL query | Data processing, ETL |
|
|
51
|
+
| SHELL | Shell script | System commands, file operations |
|
|
52
|
+
| FLOW_CHECK | Flow check | Check if upstream flow is complete |
|
|
53
|
+
| TASK_CHECK | Task check | Check if specified task is complete |
|
|
54
|
+
| OFFLINE_SYNC | Offline sync | Integration solution data sync (requires syncId) |
|
|
55
|
+
| APP_SYNC | App sync | App table data sync (requires syncId) |
|
|
56
|
+
| PLACE_HOLDER | Placeholder | Flow control, logical grouping |
|
|
57
57
|
|
|
58
|
-
### Step 3:
|
|
58
|
+
### Step 3: Save Task Definition (configure SQL/Shell content)
|
|
59
59
|
|
|
60
60
|
```bash
|
|
61
|
-
# SQL
|
|
62
|
-
|
|
61
|
+
# SQL task
|
|
62
|
+
ae-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
|
|
63
63
|
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
64
64
|
--taskContentJson '{"sql":"SELECT * FROM dwd_user","repoCode":"te_etl","catalog":"hive","schema":"ws_default_dev"}' \
|
|
65
65
|
--failRetryTimes 3 --failRetryInterval 30 --timeout 60 --confirmed true
|
|
66
66
|
|
|
67
|
-
# Shell
|
|
68
|
-
|
|
67
|
+
# Shell task
|
|
68
|
+
ae-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
|
|
69
69
|
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
70
70
|
--taskContentJson '{"script":"#!/bin/bash\necho hello"}' \
|
|
71
71
|
--confirmed true
|
|
72
72
|
```
|
|
73
73
|
|
|
74
|
-
### Step 4:
|
|
74
|
+
### Step 4: Validate SQL (recommended, TRINO_SQL type only)
|
|
75
75
|
|
|
76
76
|
```bash
|
|
77
|
-
|
|
77
|
+
ae-cli dataops_flow +validate_task_sql --spaceCode "${spaceCode}" \
|
|
78
78
|
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
79
79
|
--sql "SELECT * FROM dwd_user" --repoCode "te_etl" \
|
|
80
80
|
--catalog "hive" --schema "ws_default_dev"
|
|
81
|
-
#
|
|
81
|
+
# Check valid=true, otherwise review message and errorPosition
|
|
82
82
|
```
|
|
83
83
|
|
|
84
|
-
### Step 5:
|
|
84
|
+
### Step 5: Add Task Dependencies (DAG connections)
|
|
85
85
|
|
|
86
86
|
```bash
|
|
87
|
-
|
|
87
|
+
ae-cli dataops_flow +add_task_relation --spaceCode "${spaceCode}" \
|
|
88
88
|
--flowCode "${flowCode}" --preTaskCode "${upstreamTaskCode}" --taskCode "${downstreamTaskCode}" \
|
|
89
89
|
--confirmed true
|
|
90
90
|
```
|
|
91
91
|
|
|
92
|
-
### Step 6:
|
|
92
|
+
### Step 6: Configure Schedule
|
|
93
93
|
|
|
94
94
|
```bash
|
|
95
|
-
# CRON
|
|
96
|
-
|
|
95
|
+
# CRON expression for scheduled execution
|
|
96
|
+
ae-cli dataops_flow +save_schedule_config --spaceCode "${spaceCode}" \
|
|
97
97
|
--flowCode "${flowCode}" --scheduled 1 --scheduleType "CRON" \
|
|
98
98
|
--cron "0 0 2 * * ?" --confirmed true
|
|
99
|
-
#
|
|
99
|
+
# Example: 0 0 2 * * ? = Daily at 2 AM
|
|
100
100
|
```
|
|
101
101
|
|
|
102
|
-
**CRON
|
|
103
|
-
- `0 0 */4 * * ?` —
|
|
104
|
-
- `0 30 8 * * 1-5` —
|
|
102
|
+
**CRON Format** (6 fields): `second minute hour day month weekday`
|
|
103
|
+
- `0 0 */4 * * ?` — Every 4 hours
|
|
104
|
+
- `0 30 8 * * 1-5` — Weekdays at 8:30
|
|
105
105
|
|
|
106
|
-
### Step 7: DEV
|
|
106
|
+
### Step 7: Test in DEV Environment
|
|
107
107
|
|
|
108
108
|
```bash
|
|
109
|
-
#
|
|
110
|
-
|
|
109
|
+
# Manually trigger execution
|
|
110
|
+
ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
111
111
|
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
112
112
|
|
|
113
|
-
#
|
|
114
|
-
|
|
113
|
+
# Check execution status
|
|
114
|
+
ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
115
115
|
|
|
116
|
-
#
|
|
117
|
-
|
|
116
|
+
# View task logs (get bsTaskInstanceId and taskCode from DAG)
|
|
117
|
+
ae-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${instanceId}" --taskCode "${taskCode}"
|
|
118
118
|
|
|
119
|
-
#
|
|
119
|
+
# Adjust configuration based on logs, then repeat testing
|
|
120
120
|
```
|
|
121
121
|
|
|
122
|
-
### Step 8:
|
|
122
|
+
### Step 8: Release to Production
|
|
123
123
|
|
|
124
124
|
```bash
|
|
125
|
-
|
|
125
|
+
ae-cli dataops_flow +release_flow --spaceCode "${spaceCode}" \
|
|
126
126
|
--flowCode "${flowCode}" --confirmed true
|
|
127
|
-
#
|
|
127
|
+
# After release, PROD schedule takes effect immediately without interrupting running instances
|
|
128
128
|
```
|
|
129
129
|
|
|
130
130
|
---
|
|
131
131
|
|
|
132
|
-
##
|
|
133
|
-
|
|
134
|
-
|
|
|
135
|
-
|
|
136
|
-
| `+create_flow` |
|
|
137
|
-
| `+create_task` |
|
|
138
|
-
| `+save_task_definition` |
|
|
139
|
-
| `+validate_task_sql` |
|
|
140
|
-
| `+add_task_relation` |
|
|
141
|
-
| `+save_schedule_config` |
|
|
142
|
-
| `+get_flow_dag` |
|
|
143
|
-
| `+get_task_params` |
|
|
144
|
-
| `+execute_flow` |
|
|
145
|
-
| `+get_execute_dag` |
|
|
146
|
-
| `+get_task_instance_log` |
|
|
147
|
-
| `+release_flow` |
|
|
148
|
-
|
|
149
|
-
##
|
|
150
|
-
|
|
151
|
-
-
|
|
152
|
-
-
|
|
153
|
-
- **taskContentJson**: JSON
|
|
132
|
+
## Command Quick Reference
|
|
133
|
+
|
|
134
|
+
| Command | Purpose | Key Flags |
|
|
135
|
+
|---------|---------|-----------|
|
|
136
|
+
| `+create_flow` | Create flow | `--spaceCode` `--flowName` `--remark` `--confirmed` |
|
|
137
|
+
| `+create_task` | Create task node | `--spaceCode` `--flowCode` `--taskName` `--taskType` `--syncId` `--preTaskCode` `--confirmed` |
|
|
138
|
+
| `+save_task_definition` | Save task definition | `--spaceCode` `--flowCode` `--taskCode` `--taskContentJson` `--failRetryTimes` `--failRetryInterval` `--timeout` `--confirmed` |
|
|
139
|
+
| `+validate_task_sql` | Validate SQL | `--spaceCode` `--flowCode` `--taskCode` `--sql` `--repoCode` `--catalog` `--schema` |
|
|
140
|
+
| `+add_task_relation` | Add dependency | `--spaceCode` `--flowCode` `--preTaskCode` `--taskCode` `--confirmed` |
|
|
141
|
+
| `+save_schedule_config` | Configure schedule | `--spaceCode` `--flowCode` `--scheduled` `--scheduleType` `--cron` `--confirmed` |
|
|
142
|
+
| `+get_flow_dag` | View DAG structure | `--spaceCode` `--flowCode` `--env` |
|
|
143
|
+
| `+get_task_params` | View task parameters | `--spaceCode` `--flowCode` `--taskCode` `--env` |
|
|
144
|
+
| `+execute_flow` | Manual execution | `--spaceCode` `--flowCode` `--env` `--confirmed` |
|
|
145
|
+
| `+get_execute_dag` | View runtime DAG | `--spaceCode` `--executeId` |
|
|
146
|
+
| `+get_task_instance_log` | View task logs | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
|
|
147
|
+
| `+release_flow` | Release to production | `--spaceCode` `--flowCode` `--confirmed` |
|
|
148
|
+
|
|
149
|
+
## Parameter Notes
|
|
150
|
+
|
|
151
|
+
- **Parameter Reference**: Reference workspace parameters in tasks using `${paramKey}` (e.g., `${ws_run_date}`)
|
|
152
|
+
- **Environment**: `DEV` (default) | `PROD`
|
|
153
|
+
- **taskContentJson**: JSON string, format varies by taskType:
|
|
154
154
|
- TRINO_SQL: `{"sql":"SELECT ...","repoCode":"xxx","catalog":"hive","schema":"db"}`
|
|
155
155
|
- SHELL: `{"script":"#!/bin/bash\n..."}`
|
|
156
156
|
|
|
157
|
-
##
|
|
157
|
+
## Important Notes
|
|
158
158
|
|
|
159
|
-
1. **flowCode
|
|
160
|
-
2. **taskCode
|
|
161
|
-
3.
|
|
162
|
-
4. **SQL
|
|
163
|
-
5.
|
|
159
|
+
1. **Getting flowCode**: Use `+list_flows` (see `dataops-flow-monitor` Skill)
|
|
160
|
+
2. **Getting taskCode**: Returned by `+create_task` or query via `+get_flow_dag`
|
|
161
|
+
3. **Release Impact**: Syncs all DEV changes to PROD, PROD schedule takes effect immediately
|
|
162
|
+
4. **SQL Validation**: Recommended to validate before saving TRINO_SQL tasks
|
|
163
|
+
5. **Cannot create circular dependencies**
|
|
@@ -1,118 +1,118 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dataops-flow-monitor
|
|
3
3
|
version: 1.0.0
|
|
4
|
-
description: "
|
|
4
|
+
description: "Flow execution and monitoring: view execution instances, runtime DAG, task logs, manually execute/stop flows. Trigger keywords: execute flow, running instance, monitor, logs, stop, DAG, troubleshoot, instance, flow instance."
|
|
5
5
|
metadata:
|
|
6
6
|
requires:
|
|
7
|
-
bins: ["
|
|
7
|
+
bins: ["ae-cli"]
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
-
# DataOps
|
|
10
|
+
# DataOps Flow Execution and Monitoring
|
|
11
11
|
|
|
12
|
-
>
|
|
12
|
+
> **Prerequisites:** Read [`te-dataops/SKILL.md`](../SKILL.md) for general rules.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
Use the `dataops_flow` subcommand to execute and monitor flows.
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
- **executeId** — flow_*
|
|
18
|
-
- **flowInstanceId** — operations_*
|
|
19
|
-
-
|
|
16
|
+
**Core Concepts:**
|
|
17
|
+
- **executeId** — Execution record ID used by flow_* tools (obtained via `+list_flow_instances`)
|
|
18
|
+
- **flowInstanceId** — Operations instance ID used by operations_* tools (these two are not interchangeable)
|
|
19
|
+
- **Instance list queries PROD environment by default** (unlike other tools which default to DEV)
|
|
20
20
|
|
|
21
21
|
---
|
|
22
22
|
|
|
23
|
-
##
|
|
23
|
+
## Workflow A: Find and View Flow Execution Status
|
|
24
24
|
|
|
25
25
|
```bash
|
|
26
|
-
# Step 1:
|
|
27
|
-
|
|
26
|
+
# Step 1: Search flows (get flowCode)
|
|
27
|
+
ae-cli dataops_flow +list_flows --spaceCode "${spaceCode}" --keyword "etl"
|
|
28
28
|
|
|
29
|
-
# Step 2:
|
|
30
|
-
|
|
29
|
+
# Step 2: View execution instance list (default PROD, sorted by time descending)
|
|
30
|
+
ae-cli dataops_flow +list_flow_instances --spaceCode "${spaceCode}" \
|
|
31
31
|
--flowCode "${flowCode}" --env "PROD" --size 30
|
|
32
32
|
|
|
33
|
-
# Step 3:
|
|
34
|
-
|
|
33
|
+
# Step 3: View runtime DAG (with execution status for each node)
|
|
34
|
+
ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
35
35
|
|
|
36
|
-
# Step 4:
|
|
37
|
-
|
|
36
|
+
# Step 4: View execution record details
|
|
37
|
+
ae-cli dataops_flow +get_execute_record --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
38
38
|
```
|
|
39
39
|
|
|
40
40
|
---
|
|
41
41
|
|
|
42
|
-
##
|
|
42
|
+
## Workflow B: Troubleshoot Task Failures
|
|
43
43
|
|
|
44
44
|
```bash
|
|
45
|
-
# Step 1:
|
|
46
|
-
|
|
47
|
-
#
|
|
45
|
+
# Step 1: Get runtime DAG, locate failed node
|
|
46
|
+
ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
47
|
+
# Find node with status=FAILED, get its bsTaskInstanceId and taskCode
|
|
48
48
|
|
|
49
|
-
# Step 2:
|
|
50
|
-
|
|
49
|
+
# Step 2: View failed task logs
|
|
50
|
+
ae-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${bsTaskInstanceId}" \
|
|
51
51
|
--taskCode "${taskCode}" --limit 100
|
|
52
52
|
|
|
53
|
-
# Step 3:
|
|
54
|
-
|
|
53
|
+
# Step 3: View task definition (confirm SQL/configuration)
|
|
54
|
+
ae-cli dataops_flow +get_task_params --spaceCode "${spaceCode}" \
|
|
55
55
|
--flowCode "${flowCode}" --taskCode "${taskCode}"
|
|
56
56
|
|
|
57
|
-
# Step 4:
|
|
58
|
-
|
|
57
|
+
# Step 4: Re-execute after fixing
|
|
58
|
+
ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
59
59
|
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
60
60
|
```
|
|
61
61
|
|
|
62
62
|
---
|
|
63
63
|
|
|
64
|
-
##
|
|
64
|
+
## Workflow C: Manual Execution and Stop
|
|
65
65
|
|
|
66
66
|
```bash
|
|
67
|
-
#
|
|
68
|
-
|
|
67
|
+
# Manually trigger execution (default DEV, can specify PROD)
|
|
68
|
+
ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
69
69
|
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
70
|
-
#
|
|
70
|
+
# Returns executeId
|
|
71
71
|
|
|
72
|
-
#
|
|
73
|
-
|
|
72
|
+
# Monitor execution progress
|
|
73
|
+
ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
74
74
|
|
|
75
|
-
#
|
|
76
|
-
|
|
75
|
+
# Stop execution if necessary (irreversible)
|
|
76
|
+
ae-cli dataops_flow +stop_flow --spaceCode "${spaceCode}" \
|
|
77
77
|
--flowCode "${flowCode}" --executeId "${executeId}" --confirmed true
|
|
78
78
|
```
|
|
79
79
|
|
|
80
80
|
---
|
|
81
81
|
|
|
82
|
-
##
|
|
82
|
+
## Workflow D: View Flow Structure
|
|
83
83
|
|
|
84
84
|
```bash
|
|
85
|
-
#
|
|
86
|
-
|
|
85
|
+
# View flow definition DAG (static structure, dependencies)
|
|
86
|
+
ae-cli dataops_flow +get_flow_dag --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
87
87
|
|
|
88
|
-
#
|
|
89
|
-
|
|
88
|
+
# View flow basic information
|
|
89
|
+
ae-cli dataops_flow +get_flow_detail --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
90
90
|
|
|
91
|
-
#
|
|
92
|
-
|
|
91
|
+
# View schedule configuration
|
|
92
|
+
ae-cli dataops_flow +get_schedule_config --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
93
93
|
```
|
|
94
94
|
|
|
95
95
|
---
|
|
96
96
|
|
|
97
|
-
##
|
|
98
|
-
|
|
99
|
-
|
|
|
100
|
-
|
|
101
|
-
| `+list_flows` |
|
|
102
|
-
| `+get_flow_detail` |
|
|
103
|
-
| `+get_flow_dag` |
|
|
104
|
-
| `+get_schedule_config` |
|
|
105
|
-
| `+execute_flow` |
|
|
106
|
-
| `+stop_flow` |
|
|
107
|
-
| `+list_flow_instances` |
|
|
108
|
-
| `+get_execute_record` |
|
|
109
|
-
| `+get_execute_dag` |
|
|
110
|
-
| `+get_task_instance_log` |
|
|
111
|
-
| `+get_task_params` |
|
|
112
|
-
| `+update_flow` |
|
|
113
|
-
|
|
114
|
-
##
|
|
115
|
-
|
|
116
|
-
- **env**: `DEV
|
|
117
|
-
-
|
|
118
|
-
- **DAG
|
|
97
|
+
## Command Quick Reference
|
|
98
|
+
|
|
99
|
+
| Command | Purpose | Key Flags |
|
|
100
|
+
|---------|---------|-----------|
|
|
101
|
+
| `+list_flows` | Search flows | `--spaceCode` `--keyword` |
|
|
102
|
+
| `+get_flow_detail` | Flow information | `--spaceCode` `--flowCode` `--env` |
|
|
103
|
+
| `+get_flow_dag` | Definition DAG | `--spaceCode` `--flowCode` `--env` |
|
|
104
|
+
| `+get_schedule_config` | Schedule configuration | `--spaceCode` `--flowCode` `--env` |
|
|
105
|
+
| `+execute_flow` | Manual execution | `--spaceCode` `--flowCode` `--env` `--confirmed` |
|
|
106
|
+
| `+stop_flow` | Stop execution | `--spaceCode` `--flowCode` `--executeId` `--confirmed` |
|
|
107
|
+
| `+list_flow_instances` | Instance list | `--spaceCode` `--flowCode` `--env` `--size` |
|
|
108
|
+
| `+get_execute_record` | Execution record | `--spaceCode` `--executeId` |
|
|
109
|
+
| `+get_execute_dag` | Runtime DAG | `--spaceCode` `--executeId` |
|
|
110
|
+
| `+get_task_instance_log` | Task logs | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
|
|
111
|
+
| `+get_task_params` | Task parameters | `--spaceCode` `--flowCode` `--taskCode` `--env` |
|
|
112
|
+
| `+update_flow` | Update flow | `--spaceCode` `--flowCode` `--flowName` `--remark` `--confirmed` |
|
|
113
|
+
|
|
114
|
+
## Parameter Notes
|
|
115
|
+
|
|
116
|
+
- **env**: `DEV` (development) | `PROD` (production, instance list defaults to PROD)
|
|
117
|
+
- **Task Status**: `success` / `failure` / `running` / `waiting`
|
|
118
|
+
- **DAG Difference**: `+get_flow_dag` returns static definition structure, `+get_execute_dag` returns runtime structure with execution status
|